diff --git a/agent/README.md b/agent/README.md index 8084e23..7d51eee 100644 --- a/agent/README.md +++ b/agent/README.md @@ -19,6 +19,20 @@ docker run --rm --gpus all nvidia/cuda:13.0.3-base-ubuntu24.04 nvidia-smi # the header's CUDA version must be 13.0 or later (driver 580+) ``` +### After a driver update: regenerate the CDI spec +If the agent's first log lines say `accel: torch is NOT on the GPU` or report +`cudaGetDeviceCount: unknown error (999)` while `nvidia-smi` still works, the +toolkit's saved device list (`/etc/cdi/nvidia.yaml`) is out of date. The +`nvidia-uvm` device number changes between driver versions, and a spec +generated before the update hands the container a device node that no longer +exists (2026-09-24: host `511,0`, container `235,0`). Compare +`ls -l /dev/nvidia-uvm` on the host with the same inside the container, then: +```sh +sudo nvidia-ctk cdi generate --output=/etc/cdi/nvidia.yaml +# if your toolkit ships it, this keeps it current on every driver update: +sudo systemctl enable --now nvidia-cdi-refresh.path +``` + ## 1. Get a token In FC: **Settings → Tagging → GPU agent → Generate token** (or Rotate). Copy it.