Install gpu driver (AMD ROCm)

sudo apt update
sudo apt upgrade
# recommended to reboot
sudo apt install "linux-headers-$(uname -r)" "linux-modules-extra-$(uname -r)"
sudo usermod -a -G render,video $LOGNAME # Add the current user to the render and video groups
wget https://repo.radeon.com/amdgpu-install/6.2/ubuntu/noble/amdgpu-install_6.2.60200-1_all.deb
sudo apt install ./amdgpu-install_6.2.60200-1_all.deb
sudo apt update
sudo apt install amdgpu-dkms rocm

Install Ollama locally

Install Ollama running this one-liner:

curl -fsSL https://ollama.com/install.sh | sh

Start Ollama using systemd:

sudo systemctl start ollama

Setting environment variables on Linux

If Ollama is run as a systemd service, environment variables should be set using systemctl:

  1. Edit the systemd service by calling sudo systemctl edit ollama.service. This will open an editor.

  2. For each environment variable, add a line Environment under section [Service]:

    [Service]
Environment="OLLAMA_HOST=0.0.0.0:11434"
Environment="HSA_OVERRIDE_GFX_VERSION=10.3.0"
Environment="ROCM_PATH=/opt/rocm"

  3. Save and exit.

  4. Reload systemd and restart Ollama:

    sudo systemctl daemon-reload
sudo systemctl restart ollama

Viewing logs to check if it worked

To view logs of Ollama running as a startup service, run:

journalctl -e -u ollama

Install Ollama through docker

Set up Docker’s apt repository.

# Add Docker's official GPG key:
sudo apt-get update
sudo apt-get install ca-certificates curl
sudo install -m 0755 -d /etc/apt/keyrings
sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc
sudo chmod a+r /etc/apt/keyrings/docker.asc

# Add the repository to Apt sources:
echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu \
  $(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \
  sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt-get update

To install the latest version of Docker packages

sudo apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

Create ollama volume

docker volume create ollama

Create directory for docker compose files

sudo mkdir /docker_compose_files && cd /docker_compose_files

sudo mkdir ./ollama && cd ./ollama

Create docker compose file

sudo nano ./compose.yaml

copy in code the save and exit

version: '3.8'

services:
  ollama:
    image: ollama/ollama:rocm
    container_name: ollama
    restart: unless-stopped
    ports:
      - "11434:11434"
    volumes:
      - ollama:/root/.ollama
    devices:
      - /dev/kfd:/dev/kfd
      - /dev/dri:/dev/dri
    environment:
      - HSA_OVERRIDE_GFX_VERSION=10.3.0

volumes:
  ollama:
    external: true

create env

mk ./.env && sudo nano ./.env

copy in code the save and exit

ollama=ollama

start container

sudo docker compose up

Check that its working

docker ps