docs(agent): a driver update leaves the CDI spec naming a stale nvidia-uvm device — how to spot and regenerate it (1451)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR
2026-09-24 19:04:03 -04:00
parent 42a40d71a4
commit e39ec1c550
1 changed files with 14 additions and 0 deletions
@@ -19,6 +19,20 @@ docker run --rm --gpus all nvidia/cuda:13.0.3-base-ubuntu24.04 nvidia-smi
 # the header's CUDA version must be 13.0 or later (driver 580+)
 ```

+### After a driver update: regenerate the CDI spec
+If the agent's first log lines say `accel: torch is NOT on the GPU` or report
+`cudaGetDeviceCount: unknown error (999)` while `nvidia-smi` still works, the
+toolkit's saved device list (`/etc/cdi/nvidia.yaml`) is out of date. The
+`nvidia-uvm` device number changes between driver versions, and a spec
+generated before the update hands the container a device node that no longer
+exists (2026-09-24: host `511,0`, container `235,0`). Compare
+`ls -l /dev/nvidia-uvm` on the host with the same inside the container, then:
+```sh
+sudo nvidia-ctk cdi generate --output=/etc/cdi/nvidia.yaml
+# if your toolkit ships it, this keeps it current on every driver update:
+sudo systemctl enable --now nvidia-cdi-refresh.path
+```
+
 ## 1. Get a token
 In FC: **Settings → Tagging → GPU agent → Generate token** (or Rotate). Copy it.