# Ollama GPU Setup and Verification

This document explains the process of setting up **Ollama** to use the GPU in a Docker container, and how to verify if the GPU is being used.

## 1. Installing NVIDIA Container Toolkit

To use **NVIDIA GPU** with Docker containers, you need to install the **NVIDIA Container Toolkit**. Follow the steps below to install it.

### A. Install NVIDIA Container Toolkit (using Apt)

1. Configure the repository:

   curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey        | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
   curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list        | sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g'        | sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
   sudo apt-get update

2. Install the NVIDIA Container Toolkit:

   sudo apt-get install -y nvidia-container-toolkit

### B. Configure Docker to Use NVIDIA GPU

Once the toolkit is installed, configure Docker to use the NVIDIA runtime:

sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

## 2. Running the Ollama Docker Container with GPU

Once Docker is configured to use GPU, you can run the **Ollama** container with GPU support.

### A. Start the Ollama container with GPU

Run the following command to start **Ollama** with GPU support:

docker run --gpus all -d -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama

### B. Verify if Ollama is using the GPU

After starting the container, you can verify if **Ollama** is using the GPU by following these steps:

#### Step 1: Enter the running container

docker exec -it ollama /bin/bash

#### Step 2: Check if the GPU is available inside the container

Inside the container, run the following command:

nvidia-smi

If the GPU is being used correctly, you should see output showing the GPU details, such as:

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 525.105.17   Driver Version: 525.105.17   CUDA Version: 12.0     |
|-------------------------------+----------------------+----------------------+|
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA GeForce ...  Off  | 00000000:01:00.0 Off |                  N/A |
|  0%   40C    P8    32W / 370W |      6MiB / 12288MiB |      0%      Default |
+-----------------------------------------------------------------------------+

This confirms that the GPU is available and being used by **Ollama**.

## 3. Additional Steps for Troubleshooting

If the GPU is not recognized, ensure that:

1. **NVIDIA drivers** and **CUDA** are properly installed on the host machine.
2. Docker is using the `nvidia-docker` runtime, which is essential for enabling GPU usage inside Docker containers.

To test Docker's ability to use GPU, you can run a basic NVIDIA CUDA container:

docker run --gpus all nvidia/cuda:11.2-base nvidia-smi

If this command works and shows GPU information, it confirms that Docker is configured correctly to use the GPU.