How-to verify NVIDIA vGPU inference on Ubuntu with an ML workload
Overview
This guide covers verifying that NVIDIA vGPU compute is working on a provisioned Ubuntu VM by running a small Python inference workload. You will set up a Python virtual environment with PyTorch and Hugging Face transformers, run a text-generation test against the GPU, and observe GPU utilization in real time with nvtop.
By the end, you will have confirmed end-to-end that the GPU is licensed for compute, accessible from Python, and actively used during inference.
This guide assumes you already have a provisioned Ubuntu VM with a licensed NVIDIA vGPU. See How-to provision an Ubuntu VM with NVIDIA vGPU if you need to set one up first. Full prerequisites are listed in the next section.
Prerequisites
This guide assumes you have:
A provisioned Ubuntu 22.04 VM with a working NVIDIA vGPU. See How to provision an Ubuntu VM with NVIDIA vGPU if you need to set one up.
FeatureType=4(Virtual Compute Server) configured in/etc/nvidia/gridd.conf. Other feature types restrict CUDA compute and will cause inference to hang at 0% GPU utilization.SSH access to the VM as
vss-admin(or another user with sudo).Internet access from the VM (to download the venv setup script and Hugging Face models on first run).
Verify before proceeding
Confirm the GPU is visible (should see the GPU listed with a driver version and no errors)
nvidia-smivss-admin@nvidial4-6-test-01:~$ nvidia-smi
Tue May 5 16:58:53 2026
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 580.105.08 Driver Version: 580.105.08 CUDA Version: 13.0 |
+-----------------------------------------+------------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
| 0 NVIDIA L4-6C On | 00000000:02:00.0 Off | N/A |
| N/A N/A P0 N/A / N/A | 0MiB / 6144MiB | 0% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+
+-----------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=========================================================================================|
| No running processes found |
+-----------------------------------------------------------------------------------------Confirm the GPU is licensed
nvidia-smi -q | grep -i licensed
Step by Step
Step 1 — Set Up Python ML Environment
Download the venv setup script
curl -O https://eis.utoronto.ca/ut-vss-lib/nvidia/venv-setup/setup-venv-ml-py310.shℹ️ About the script used in this section: setup-venv-ml-py310.sh creates a Python 3.10 virtual environment at ~/venv-ml and installs a complete ML/data-science stack:
PyTorch (CUDA 12.8 build) for GPU compute
Hugging Face transformers, datasets, tokenizers, accelerate, peft, trl, sentence-transformers
Core ML/data science: numpy, pandas, scipy, scikit-learn, xgboost, lightgbm
Visualization: matplotlib, seaborn, plotly, bokeh
MLOps: mlflow, wandb, tensorboard, optuna
Jupyter with a registered kernel named after the venv directory
Make executable and run
chmod +x setup-venv-ml-py310.sh
./setup-venv-ml-py310.shActivate the virtual environment:
source /home/vss-admin/venv-ml/bin/activateℹ️ Note: The venv includes PyTorch, transformers, diffusers, and other ML packages pre-configured for the GPU environment.
Step 2 — Test GPU Inference
Run a quick test with distilgpt2 (82M parameters, ~160MB) to confirm GPU inference works
python3 -c "
from transformers import pipeline
import torch
# distilgpt2 - only 82M params, ~160MB
pipe = pipeline('text-generation', model='distilgpt2', device=0)
result = pipe('The future of AI is', max_new_tokens=200)
print(result[0]['generated_text'])
"(venv-ml) vss-admin@nvidial4-6-test-03:~$ python3 -c "
from transformers import pipeline
import torch
# distilgpt2 - only 82M params, ~160MB
pipe = pipeline('text-generation', model='distilgpt2', device=0)
result = pipe('The future of AI is', max_new_tokens=200)
print(result[0]['generated_text'])
"
Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.
config.json: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████| 762/762 [00:00<00:00, 1.33MB/s]
model.safetensors: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████| 353M/353M [00:02<00:00, 160MB/s]
Loading weights: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████| 76/76 [00:00<00:00, 22891.71it/s]
generation_config.json: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████| 124/124 [00:00<00:00, 663kB/s]
tokenizer_config.json: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████| 26.0/26.0 [00:00<00:00, 150kB/s]
vocab.json: 1.04MB [00:00, 18.0MB/s]
merges.txt: 456kB [00:00, 38.3MB/s]
tokenizer.json: 1.36MB [00:00, 29.7MB/s]
[transformers] Passing `generation_config` together with generation-related arguments=({'max_new_tokens'}) is deprecated and will be removed in future versions. Please pass either a `generation_config` object OR all generation parameters explicitly, but not both.
[transformers] Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.
[transformers] Both `max_new_tokens` (=200) and `max_length`(=50) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
[transformers] Ignoring clean_up_tokenization_spaces=True for BPE tokenizer GPT2Tokenizer. The clean_up_tokenization post-processing step is designed for WordPiece tokenizers and is destructive for BPE (it strips spaces before punctuation). Set clean_up_tokenization_spaces=False to suppress this warning, or set clean_up_tokenization_spaces_for_bpe_even_though_it_will_corrupt_output=True to force cleanup anyway.
The future of AI is to be based on the ideas of the AI scientists, not on science-based engineering. But we will also be looking at the ways to make AI smarter.
The AI ecosystem is in the process of becoming completely artificial.
But we will also be looking at the limits of what the next AI will do to our human intelligence.
The first thing we will do is start thinking about AI-based engineering, in the future.
The first thing we will do is start thinking about AI-based engineering, in the future.
The second thing we will do is start thinking about AI-based engineering, in the future.
This will allow us to do a lot more than just look at the potential of artificial intelligence.
It will allow us to do a lot more than just look at the potential of artificial intelligence.
We will also be looking at the limits of what the next AI will do to our human intelligence.
The first thing we will do is start⚠️ Note: If inference hangs with 0% GPU utilization, check license status with
nvidia-smi -q | grep -i license. Usedevice=-1(CPU) as a workaround while resolving licensing.
Step 3 - View real time GPU statistics using nvtop
Open a second terminal session and ssh to the VM
ssh -p 2222 vss-admin@128.100.159.10Install nvtop
sudo apt install nvtopRun nvtop
nvtop
Move back to the original terminal, run the inference test again with distilgpt2 (82M parameters, ~160MB) and once you see the output move back to the nvtop terminal :
python3 -c "
from transformers import pipeline
import torch
# distilgpt2 - only 82M params, ~160MB
pipe = pipeline('text-generation', model='distilgpt2', device=0)
result = pipe('The future of AI is', max_new_tokens=200)
print(result[0]['generated_text'])
"You should see a spike in GPU% and GPU mem%
Next steps
Your VM is now verified for GPU inference workloads.
To release the license and decommission the VM when you're finished
See Step 7 of How-to provision an Ubuntu VM with NVIDIA vGPU