From e39ec1c550a07a2d25efeaf3f2398688838d2efb Mon Sep 17 00:00:00 2001
From: Bryan Van Deusen <bryan.vandeusen@fabledsword.com>
Date: Thu, 24 Sep 2026 19:04:03 -0400
Subject: [PATCH] =?UTF-8?q?docs(agent):=20a=20driver=20update=20leaves=20t?=
 =?UTF-8?q?he=20CDI=20spec=20naming=20a=20stale=20nvidia-uvm=20device=20?=
 =?UTF-8?q?=E2=80=94=20how=20to=20spot=20and=20regenerate=20it=20(1451)?=
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR
---
 agent/README.md | 14 ++++++++++++++
 1 file changed, 14 insertions(+)

diff --git a/agent/README.md b/agent/README.md
index 8084e23..7d51eee 100644
--- a/agent/README.md
+++ b/agent/README.md
@@ -19,6 +19,20 @@ docker run --rm --gpus all nvidia/cuda:13.0.3-base-ubuntu24.04 nvidia-smi
 # the header's CUDA version must be 13.0 or later (driver 580+)
 ```
 
+### After a driver update: regenerate the CDI spec
+If the agent's first log lines say `accel: torch is NOT on the GPU` or report
+`cudaGetDeviceCount: unknown error (999)` while `nvidia-smi` still works, the
+toolkit's saved device list (`/etc/cdi/nvidia.yaml`) is out of date. The
+`nvidia-uvm` device number changes between driver versions, and a spec
+generated before the update hands the container a device node that no longer
+exists (2026-09-24: host `511,0`, container `235,0`). Compare
+`ls -l /dev/nvidia-uvm` on the host with the same inside the container, then:
+```sh
+sudo nvidia-ctk cdi generate --output=/etc/cdi/nvidia.yaml
+# if your toolkit ships it, this keeps it current on every driver update:
+sudo systemctl enable --now nvidia-cdi-refresh.path
+```
+
 ## 1. Get a token
 In FC: **Settings → Tagging → GPU agent → Generate token** (or Rotate). Copy it.