401 - Running LLMs with Ollama

Ollama is a tool that simplifies running and managing pretrained large language models. Rather than manually configuring complex model dependencies, Ollama provides a command-line interface for downloading, launching, and interacting with compatible models.

Why Use Ollama?

• Simplifies model execution: Run compatible LLMs using straightforward terminal commands.
• Supports multiple models: Download and switch between different pretrained language models.
• Supports GPU acceleration: Use compatible GPUs to improve inference performance.
• Provides a local API: Integrate LLM inference into Python scripts and research applications.
• Supports experimentation: Compare model outputs without repeatedly configuring model-loading code.

Ollama vs. Cloud-Based LLM APIs

Important: Ollama does not train an LLM from scratch. Its primary purpose is running inference with existing models. If your research involves training or fine-tuning models across multiple GPUs, see Module 403 - Distributed Training.

Understanding LLM Resource Requirements

Before running a language model, it is important to understand the computing resources it requires.

CPU vs. GPUs

A CPU (Central Processing Unit) handles general-purpose computing tasks, while a GPU (Graphics Processing Unit) is designed for highly parallel computations. LLMs perform large numbers of matrix operations, making GPUs particularly useful for accelerating inference. Models can run on CPUs, but a compatible GPU can improve response speed.

What is VRAM?

VRAM (Video Random Access Memory) is the memory available on a GPU. When running an LLM, memory is needed to store model weights, intermediate computations, and the context being processed. Models with more parameters generally require more memory.

Why Model Selection Matters

A model that is too large for the available GPU memory may:

• Fail to load.
• Run partially or entirely on the CPU.
• Produce responses more slowly than expected.
• Encounter out-of-memory errors.

For your first experiments, choose a small, quantized model and check its published memory requirements.

Requesting GPU Resources on ICE

Step 1: Connect to ICE

Log in to ICE using SSH or Open OnDemand using the steps in Module 100.

Step 2: Check Available GPU Resources

Run:

sinfo

This command displays information about the cluster’s Slurm partitions and node states. Available GPU types, partitions, accounts, and limits may vary. Refer to your course or research group’s approved ICE configuration before requesting resources.

Step 3: Request an Interactive GPU Session Run this command to display a basic interactive allocation:

salloc --nodes=1 --ntasks=1 --gpus=1 --time=01:00:00

This requests:

• 1 compute node
• 1 task
• 1 GPU
• 1 hour of computing time

Your environment may require additional options such as an account, partition or GPU type. Depending on Slurm’s configuration, salloc may also require launching a separate step with srun before commands execute on the allocated compute node.

Step 4: Verify GPU Access Once you are executing commands on an allocated NVIDIA GPU compute node, run:

hostname
nvidia-smi

• hostname identifies your current machine
• nvidia-smi displays information about compatible NVIDIA GPUs, including memory usage and GPU processes

Setting Up Ollama

Step 1: Check for Ollama

Run:

command -v ollama

If Ollama is available, check its version:

ollama --version

You can also inspect the environment’s available software modules:

module avail

If Ollama is unavailable, check the official Ollama Linux documentation and PACE software guidance.

Important Do not use sudo or install system-wide services on ICE. If Ollama is not installed, follow an approved user-space installation procedure or contact your research supervisor or PACE support. Confirm GPU compatibility before installation.

Step 2: Configure Model Storage

Ollama stores downloaded model files, which can consume significant disk space. The OLLAMA_MODELS environment variable can be used to specify a custom storage location.

For example, after identifying an approved directory:

export OLLAMA_MODELS="/path/to/approved/model/storage"

Replace the example path with a real directory you have permission to use.

Step 3: Start the Ollama Server

When Ollama is installed and you are working within an allocated compute session, you can start its inference service.

For a manually launched Ollama process:

export OLLAMA_HOST=127.0.0.1:11434
ollama serve

The OLLAMA_HOST setting binds the server to the compute node’s loopback interface rather than exposing it on the network.

Keep this terminal running while you use Ollama. In another terminal or shell on the same allocated compute node, you can execute model commands.

Note: Accessing the same compute node from multiple terminals may require an additional Slurm job step. Follow the access method supported by your ICE environment.

Running Your First LLM

After starting Ollama, you can download and interact with a compatible language model.

Step 1: Choose a Model

When selecting a model, consider:

• Number of parameters
• Download size
• Estimated GPU memory requirements
• Model license and permitted use
• Whether the model is suitable for your research task

Step 2: Run a Model

For example, a small model available through Ollama is:

ollama run llama3.2:1b

If the model is not already installed, Ollama will attempt to download it. This requires network access to the model registry, which may be restricted on a compute node. If downloads are blocked, use a PACE-approved model transfer or caching workflow.

Step 3: Submit a Prompt

Once the model loads, try entering:

Explain high-performance computing to a beginner in three sentences.

The model should generate a response based on your prompt. You have now performed a basic LLM inference task.

Step 4: Exit the Interactive Model

Enter:

/bye

This exits the interactive model conversation without stopping the Ollama server.

Managing Models

Ollama provides commands for inspecting downloaded models and managing local storage.

List Downloaded Models

ollama list

View Running Models

ollama ps

Displays models currently loaded by Ollama, including information about whether they are using CPU or GPU resources.

Inspect Model Information

ollama show llama3.2:1b

Displays information about the selected model.

ollama rm llama3.2:1b

Removes the downloaded model from Ollama’s managed storage.

Only remove models you own or have permission to manage.

Using Ollama with Python

Researchers may want to interact with language models programmatically rather than through the terminal. Ollama provides a local HTTP API that can be called from Python.

For example, while the Ollama server is running on the same compute node, you can use:

import json
import urllib.request

url = "http://127.0.0.1:11434/api/generate"

payload = {
    "model": "llama3.2:1b",
    "prompt": "What is high-performance computing?",
    "stream": False
}

request = urllib.request.Request(
    url,
    data=json.dumps(payload).encode("utf-8"),
    headers={"Content-Type": "application/json"}
)

with urllib.request.urlopen(request) as response:
    result = json.load(response)

print(result["response"])

This example sends a prompt to Ollama and prints the model’s generated response.

The script assumes that the model has already been downloaded and the Ollama server is accessible at the specified local address.

Programmatic access is particularly useful for:

• Processing multiple text inputs
• Comparing model responses
• Building experimental pipelines
• Automating research tasks
• Recording outputs for later analysis

For reproducible experiments, record the exact model version, prompt, generation settings, and relevant software versions.

Troubleshooting Common Issues

If a problem persists, consult PACE documentation or contact the appropriate support team.

Best Practices for Running LLMs on ICE

When running Ollama or other LLM inference tools on a shared HPC cluster, follow these practices:

1. Run inference on allocated compute nodes. Avoid computationally intensive workloads on login nodes.

2. Select appropriate model sizes. Begin with smaller models before attempting larger workloads.

3. Monitor resource usage. Check GPU memory utilization and identify unnecessary processes.

4. Manage model storage. Avoid retaining multiple large models that are no longer needed.

5. Keep services private. Do not expose an Ollama endpoint publicly without authorization.

6. Protect research data. Do not submit restricted or sensitive datasets unless the environment and workflow have been approved for that information.

7. Document your experiments. Record model versions, configurations, and results.

8. Release resources when finished. Stop Ollama and end your Slurm allocation when it is no longer needed.

Further reading

• Ollama documentation
• PACE documentation