How-to install Ollama in Ubuntu with vGPU on the ITS Private Cloud

By Orlando Ramirez
5 min

Run large language models (LLMs) with ollama on Ubuntu Linux, utilizing vGPU resources powered by U of T ITS Private Cloud Infrastructure. Utilize the VSS-CLI commands to facilitate this process.

We've pinpointed a bug in the NVIDIA driver version 525_525.147.05 when used alongside Ubuntu kernel 5.15.0-105-generic or newer. To ensure smooth operation of this tutorial, we recommend sticking to kernel version 5.15.0-102-generic. For further details, refer to the following link: Bug Report

ollama is an LLM serving platform written in golang. It makes LLMs built on Llama standards easy to run with an API.

Table of Contents


Steps

Virtual Machine Deployment

  1. Download and update the following attributes:

    1. machine.folder: target logical folder. List available folders with vss-cli compute folder ls

    2. metadata.client : your department client.

    3. metadata.inform: email address for automated notifications

  2. Deploy your file as follows:

    vss-cli --wait compute vm mk from-file ubuntu-llm-ollama.yaml

  3. Add a virtual GPU of 16GB, specifically the 16q profile. For more information in the profile used, check the following document How-to Request a Virtual GPU

    vss-cli compute vm set <VM_ID> gpu mk --profile 16q

  4. Once the VM has been deployed, a confirmation email will be sent with the assigned IP address and credentials.

  5. Power on virtual machine

    vss-cli compute vm set ubuntu-llm state on

NVIDIA Driver and Licensing

Current supported driver is nvidia-linux-grid-535_535.183.01_amd64.deb

  1. Login to the server via ssh. Note that the username may change if you further customized the VM with cloud-init

    ssh -p 2222 vss-admin@XXX.XXX.XXX.XXX

  2. Download the NVIDIA drivers from VKSEY-STOR:

    scp {vss-user}@vskey-stor.eis.utoronto.ca:/ut-vss-lib/nvidia-grid-vsphere-7.0-535.183.04-535.183.01-538.67/Guest_Drivers/nvidia-linux-grid-535_535.183.01_amd64.deb /tmp/

  3. Install the drivers as privileged user:

    apt install dkms nvtop apt install /tmp/nvidia-linux-grid-535_535.183.01_amd64.deb

  4. Create directory: ClientConfigToken

    mkdir /etc/nvidia/ClientConfigToken

  5. Create the NVIDIA token file:

    echo -n -e $(vmware-rpctool "info-get guestinfo.ut.vss.nvidia_token") > /etc/nvidia/ClientConfigToken/client_configuration_token_12-05-2023-11-26-05.tok

  6. Set permissions to the NVIDIA token:

    chmod 744 /etc/nvidia/ClientConfigToken/client_configuration_token_12-05-2023-11-26-05.tok

  7. Set the FeatureType to 2 for “NVIDIA RTX Virtual Workstation” in /etc/nvidia/gridd.conf with the following command:

    sed -i 's/FeatureType=0/FeatureType=2/g' /etc/nvidia/gridd.conf

  8. Restart nvidia-gridd service to pick up the new license token:

    systemctl restart nvidia-gridd

  9. Check for any Error or successful activation:

    journalctl -u nvidia-gridd

    output:

    Dec 13 11:23:20 ubu-llm systemd[1]: Stopped NVIDIA Grid Daemon. Dec 13 11:23:20 ubu-llm systemd[1]: Starting NVIDIA Grid Daemon... Dec 13 11:23:20 ubu-llm systemd[1]: Started NVIDIA Grid Daemon. Dec 13 11:23:20 ubu-llm nvidia-gridd[2017]: Started (2017) Dec 13 11:23:20 ubu-llm nvidia-gridd[2017]: vGPU Software package (0) Dec 13 11:23:20 ubu-llm nvidia-gridd[2017]: Ignore service provider and node-locked licensing Dec 13 11:23:20 ubu-llm nvidia-gridd[2017]: NLS initialized Dec 13 11:23:20 ubu-llm nvidia-gridd[2017]: Acquiring license. (Info: vss-nvidia-ls.eis.utoronto.ca; NVIDIA RTX Virtual Workstation) Dec 13 11:23:22 ubu-llm nvidia-gridd[2017]: License acquired successfully. (Info: vss-nvidia-ls.eis.utoronto.ca, NVIDIA RTX Virtual Workstation; Expiry: 2023-12-14 16:23:22 GMT) Dec 13 14:59:24 ubu-llm nvidia-gridd[2017]: License renewed successfully. (Info: vss-nvidia-ls.eis.utoronto.ca, NVIDIA RTX Virtual Workstation; Expiry: 2023-12-14 19:59:23 GMT)

  10. Verify GPU status with nvidia-smi:

    user@test:~$ nvidia-smi Wed Dec 13 14:18:27 2023 +-----------------------------------------------------------------------------+ | NVIDIA-SMI 525.147.05 Driver Version: 525.147.05 CUDA Version: 12.0 | |-------------------------------+----------------------+----------------------+ | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |===============================+======================+======================| | 0 GRID P6-16Q On | 00000000:02:00.0 Off | N/A | | N/A N/A P8 N/A / N/A | 6426MiB / 16384MiB | 0% Default | | | | Disabled | +-------------------------------+----------------------+----------------------+ +-----------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=============================================================================| | 0 N/A N/A 3458 C ...ld/cuda/bin/ollama-runner 6426MiB | +-----------------------------------------------------------------------------+

  11. You can also monitor in console the gpu usage with nvtop :



Install the Ollama service

The following installation will be done as the user vss-admin.

Prerequisites

  1. Download Anaconda package

    curl https://repo.anaconda.com/archive/Anaconda3-2023.07-2-Linux-x86_64.sh --output anaconda.sh

  2. Install Anaconda package.

    bash anaconda.sh



Install and configure Ollama service

  1. Download & install ollama

curl https://ollama.ai/install.sh | sh

  1. For testing purposes, open a terminal:

    1. Second terminal: run the following command:

      ollama pull llama2 ollama run llama2

      1. You can now test by asking questions:

    2. Proceed to cancel the terminal.

  2. You can exit the model by typing: control + d

Install & configure Ollama Web UI

Prerequisites

  1. Download and install nvm

    curl https://raw.githubusercontent.com/creationix/nvm/master/install.sh | bash

  2. Load the environment or execute the command below:

    source ~/.bashrc

  3. Install nodejs

    nvm install v21.4.0 nvm use v21.4.0

  4. Install python 3

    sudo apt install python3


Install and Configure Ollama Web UI

 

  1. Download and install ollama-webui:

    cd /home/vss-admin git clone https://github.com/ollama-webui/ollama-webui.git && cd ollama-webui/

  2. Create ollama-webui environment file: .env

    cp -RPp .env.example .env

  3. Install libraries and build the ollama-webui project

    npm i npm run build

  4. Install python3-pip and python3-venv:

    apt install python3-pip python3-venv

  5. Create virtual environment to isolate dependencies:

    cd ./backend python3 -m venv venv && . venv/bin/activate

  6. Install libraries and run the ollama backend

    cd ./backend pip install -r requirements.txt sh start.sh

  7. Test the UI application by opening a browser with the your servers ip address and port 8080:
    http://XXX.XXX.XXX.XXX:8080




     

  8. Select a model: llama2 and ask a message:

     

     

Troubleshoot

  1. How to verify if the nvidia licenses has been installed properly?
    Solution. Run the following command as root:

    journalctl -fu nvidia-gridd

    output:

    Dec 13 11:23:20 ubu-llm systemd[1]: Stopped NVIDIA Grid Daemon. Dec 13 11:23:20 ubu-llm systemd[1]: Starting NVIDIA Grid Daemon... Dec 13 11:23:20 ubu-llm systemd[1]: Started NVIDIA Grid Daemon. Dec 13 11:23:20 ubu-llm nvidia-gridd[2017]: Started (2017) Dec 13 11:23:20 ubu-llm nvidia-gridd[2017]: vGPU Software package (0) Dec 13 11:23:20 ubu-llm nvidia-gridd[2017]: Ignore service provider and node-locked licensing Dec 13 11:23:20 ubu-llm nvidia-gridd[2017]: NLS initialized Dec 13 11:23:20 ubu-llm nvidia-gridd[2017]: Acquiring license. (Info: vss-nvidia-ls.eis.utoronto.ca; NVIDIA RTX Virtual Workstation) Dec 13 11:23:22 ubu-llm nvidia-gridd[2017]: License acquired successfully. (Info: vss-nvidia-ls.eis.utoronto.ca, NVIDIA RTX Virtual Workstation; Expiry: 2023-12-14 16:23:22 GMT) Dec 13 14:59:24 ubu-llm nvidia-gridd[2017]: License renewed successfully. (Info: vss-nvidia-ls.eis.utoronto.ca, NVIDIA RTX Virtual Workstation; Expiry: 2023-12-14 19:59:23 GMT)

 

Reference links

VSS Cloud documentation

Anaconda

nvidia-smi commands

Ollama

Ollama-webui

nvtop

Linux

 

Related content

University of Toronto - Since 1827