ollama¶

Description¶

ollama website

Get up and running with Llama 3.1, Mistral, Gemma 2, and other large language models.

Environment Modules¶

Run module spider ollama to find out what environment modules are available for this application.

Environment Variables¶

HPC_OLLAMA_DIR - installation directory

Additional Usage Information¶

Notice on LLMs from Countries of Concern

The University strictly prohibits the deployment or utilization of large‑language models that are developed, operated, or otherwise hosted by organizations located in, or controlled by, a foreign country of concern. Accordingly, any use of such models on university computing resources, or the processing of any university data through them, is not permitted.

Section 288.860(1)(a) of the Florida Statutes defines a "foreign country of concern" as the People's Republic of China, the Russian Federation, the Islamic Republic of Iran, the Democratic People’s Republic of Korea, the Republic of Cuba, the Venezuelan regime of Nicolás Maduro and the Syrian Arab Republic, including any agency of or any other entity under significant control of such foreign country of concern.

The UFIT Information Security Office maintains a list of Fast Path Solutions, including a filter for “Not Permitted” items, which includes the Prohibited use of Large Language Models. Do not work with language models developed in countries of concern on HiPerGator or other UF computing resources.

Interactive OLLAMA use¶

Users need to start an interactive HiperGator Desktop session session on a GPU node at Open Ondemand and launch two terminals, one to start the ollama server and the other to chat with LLMs. Screen or tmux may also be used to activate two terminals.

In terminal 1, load the ollama module and start the server with either default or custom environmental settings:

Default settings
```
   $ ml ollama
   $ ollama serve
```

Custom settings

   $ ml ollama
   $ env {options} ollama serve (pass environmental variables to server)

For example: set custom path to LLMs models (recommended! since default location will quickly consume $HOME storage.

   $ env OLLAMA_MODELS=/blue/group/$USER/ollama/models ollama serve

In terminal 2, pull a model and start chatting. For example, llama3.2:

Default settings

  $ ml ollama
  $ ollama pull llama3.2 #  models stored in /home/$USER/.ollama/models
  $ ollama run llama3.2

Custom settings

   $ ml ollama
   $ env OLLAMA_MODELS=/blue/group/$USER/ollama/models ollama pull llama3.2
   $ env OLLAMA_MODELS=/blue/group/$USER/ollama/models ollama run llama3.2

Tip

ollama serve and ollama pull commands may be combined to pull models in a sngle command

For example:

   $ ml ollama
   $ env OLLAMA_MODELS=/blue/group/$USER/ollama/models ollama serve & ollama pull llama3.2

OLLAMA as a Slurm job¶

Example sbatch script:¶

start ollama server with custom model path
download the 'mistral' LLM
use a personal langchain env to run a python script

#!/bin/bash
#SBATCH --job-name=ollama
#SBATCH --output=ollama_%j.log
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=8
#SBATCH --mem=20gb
#SBATCH --partition=hpg-b200
#SBATCH --gpus=1
#SBATCH --time=01:00:00
date;hostname;pwd

module load ollama

#add conda env with langchain to path
env_path=/my/conda/env/bin 
export PATH=$env_path:$PATH

# start ollama server with custom model path and download the mistral model
env OLLAMA_MODELS=/blue/group/$USER/ollama/models ollama serve &
ollama pull mistral

# run python script
python my_ollama_python_script.py >> my_ollama_output.txt

Example python script:¶

from langchain_ollama import OllamaLLM
llm = OllamaLLM(model="mistral")
print(llm.invoke("why is the sky blue"))

Categories¶

AI