feat(host_agent): NFS delay per mount, read from /proc/self/mountstats (#5996)

Each sample carries each NFS mount's requests since the last one: how many
a second, their round trip and execute time (READ and WRITE on their own),
and how many were sent again or hit a major timeout. mountstats is the
kernel's own count and never touches the share, so it reads while one is
stuck. Steward stores them per mount and the slowest on the host as
nfs_exe_max_ms, for an "NFS slow" rule and the host page's NFS chart beside
mounts_stalled; each mount's bar says what a request took.

Agent 1.22.0 (which also carries the backlog order from 41fc807).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

docs/plugins/host-agent-design.md

@@ -124,6 +124,12 @@ Python 3.8+ stdlib only. `urllib.request` for the POST, plus `json`, `os`, `shut
 
 Each mount is read in a daemon thread under one 5 s deadline. A hard-mounted NFS share whose server stops blocks `statvfs` in the kernel with no timeout; read inline, that froze the whole agent. A mount that misses the deadline is sent as `{"mount", "stalled": true}`, and isn't probed again while its last probe is still blocked. Steward counts them as `mounts_stalled`.
 
+### NFS delay per mount (1.22.0, #5996)
+
+`mounts_stalled` says a mount is stuck; it can't say a share was slow before it stuck, or how slow it was when it came back. Each sample also reads `/proc/self/mountstats`, the kernel's own cumulative count of every NFS request each mount has finished. Reading it never touches the share, so it answers while one is stuck. The agent sends each NFS mount's requests since the last sample as `nfs: [{mount, ops_per_sec, retrans, timeouts, rtt_ms, exe_ms, read_ms, write_ms}]`: `rtt_ms` is the network and server's round trip, `exe_ms` that plus the time queued on the host, which is what a program waits. The delays are averages over the requests that finished, and are left out when none did. A mount new since the last sample, or whose counters went back (remounted), waits for the next one.
+
+Steward stores them per mount (`nfs_*` on `<host>:<mount>`), and the slowest mount's `exe_ms` on the host as `nfs_exe_max_ms`, for a rule ("NFS slow": over 1000 ms for 3 checks) and for the host page's NFS chart beside `mounts_stalled`. Each mount's bar on the host page says what a request took and how many were sent again or timed out. A request is counted when it finishes, so a stuck share shows nothing until it comes back, then its long requests all at once.
+
 ### Every Docker read is bounded (1.15.0, #5764)
 
 One Docker answer must arrive within 15 s and fit in 32 MiB, and one log answer within 5 s and 4 MiB. Past either limit the read is cut off, not waited on. A per-read timeout alone never ends a body that keeps trickling in. Container logs are read since each container's cursor, but only its newest 2,000 lines, all containers within 10 s. A container that writes more than that gets a `_steward` note saying older lines were skipped. A container whose answer is too big skips ahead to now. Before this, a 1.13.0 agent re-read a growing backlog every interval and went silent for a day at 2.1 GB. Each sample also carries the agent's own `agent` footprint (`rss_bytes`, `cpu_pct`), which Steward records as `agent_rss_bytes` and `agent_cpu_pct`.

plugins/host_agent/__init__.py

@@ -91,6 +91,15 @@ def get_alert_metrics() -> dict[str, list[str]]:
     the whole host ("<host>") or one interface ("<host>:net:<iface>") or disk
     ("<host>:diskio:<device>").
 
+    NFS delay (agent ≥ 1.22.0, #5996), per NFS mount on "<host>:<mount>",
+    over the requests it finished since the last sample: nfs_exe_ms is the
+    average time a program waited on one (queueing included), nfs_rtt_ms the
+    server's round trip, nfs_read_ms / nfs_write_ms the same for reads and
+    writes alone, nfs_ops_per_sec how many, and nfs_retrans / nfs_timeouts the
+    retransmissions and major timeouts among them. nfs_exe_max_ms is the
+    host's slowest mount, for a covering rule: "NFS slow" is > 1000 for 3
+    checks. A healthy share answers in a few ms.
+
     Drive health (agent ≥ 1.20.0, #5789), read hourly with smartctl by the
     root helper and judged in drives.py, per drive on "<host>:drive:<device>":
     drive_failing is 1 when the drive fails its own self-assessment, raises
@@ -115,6 +124,8 @@ def get_alert_metrics() -> dict[str, list[str]]:
         "agent_rss_bytes", "agent_cpu_pct",
         "updates_pending", "updates_security", "reboot_required", "maintenance_failed",
         "temp_c_max", "temp_c", "net_rx_bps", "net_tx_bps", "disk_read_bps", "disk_write_bps",
+        "nfs_exe_max_ms", "nfs_exe_ms", "nfs_rtt_ms", "nfs_read_ms", "nfs_write_ms",
+        "nfs_ops_per_sec", "nfs_retrans", "nfs_timeouts",
         *DRIVE_METRICS,
     ]}
 

plugins/host_agent/agent.py

@@ -92,6 +92,7 @@ UPTIME_PATH = "/proc/uptime"
 OS_RELEASE_PATH = "/etc/os-release"
 NETDEV_PATH = "/proc/net/dev"
 DISKSTATS_PATH = "/proc/diskstats"
+MOUNTSTATS_PATH = "/proc/self/mountstats"
 HWMON_DIR = "/sys/class/hwmon"
 # Where the kernel lists virtual network devices (bridges, veth, docker0,
 # vxlan, tun, VLANs, bonds) and the block devices (a disk has a `device`).
@@ -394,6 +395,86 @@ def collect_diskio_raw() -> dict:
     return _physical_only(out, _virtual_disk)
 
 
+# ─── NFS delay per mount (#5996) ─────────────────────────────────────────────
+#
+# The kernel's own count of every NFS request each mount has finished, read
+# from /proc/self/mountstats. Reading it never touches the share, so it answers
+# while one is stuck. Each op's line is cumulative: ops, transmissions, major
+# timeouts, bytes sent, bytes received, then queue, round-trip and execute
+# time in ms. A request is counted when it finishes, so a share that is stuck
+# shows nothing until it comes back, and then its long requests at once
+# (mounts_stalled says it is stuck meanwhile).
+
+
+def read_mountstats(text: str) -> dict:
+    """{mount point: {op: (ops, transmissions, timeouts, queue_ms, rtt_ms,
+    execute_ms)}} for each NFS mount in a mountstats text."""
+    out: dict = {}
+    mount = None
+    in_ops = False
+    for line in text.splitlines():
+        if line.startswith("device "):
+            mount, in_ops = None, False
+            parts = line.split()
+            try:
+                at = parts.index("on")
+                fstype = parts[parts.index("fstype") + 1]
+            except (ValueError, IndexError):
+                continue
+            if fstype.startswith("nfs") and at + 1 < len(parts):
+                mount = parts[at + 1].replace("\\040", " ")
+                out[mount] = {}
+        elif mount is None:
+            continue
+        elif line.strip() == "per-op statistics":
+            in_ops = True
+        elif in_ops:
+            name, sep, rest = line.strip().partition(":")
+            nums = rest.split()
+            if sep and len(nums) >= 8 and all(n.isdigit() for n in nums[:8]):
+                ops, trans, timeouts, _sent, _recv, queue, rtt, exe = (int(n) for n in nums[:8])
+                out[mount][name] = (ops, trans, timeouts, queue, rtt, exe)
+    return out
+
+
+def collect_nfs_raw() -> dict:
+    try:
+        return read_mountstats(_read_file(MOUNTSTATS_PATH))
+    except OSError:
+        return {}
+
+
+def nfs_delays(cur: dict, prev: dict, dt: float) -> list:
+    """Each NFS mount's requests since the last sample: how many a second,
+    their average round trip and execute time (queueing included, what a
+    program waits), READ and WRITE on their own, and the retransmissions and
+    major timeouts among them. A mount new since the last sample, or whose
+    counters went back (remounted), is left for the next one."""
+    out: list = []
+    if dt <= 0:
+        return out
+    for mount, ops in cur.items():
+        before = prev.get(mount)
+        if before is None:
+            continue
+        d = {op: tuple(a - b for a, b in zip(v, before.get(op, (0,) * 6))) for op, v in ops.items()}
+        if any(x < 0 for v in d.values() for x in v):
+            continue
+        n = sum(v[0] for v in d.values())
+        row = {"mount": mount, "ops_per_sec": round(n / dt, 2),
+               "retrans": max(0, sum(v[1] - v[0] for v in d.values())),
+               "timeouts": sum(v[2] for v in d.values())}
+        if n:
+            row["rtt_ms"] = round(sum(v[4] for v in d.values()) / n, 2)
+            row["exe_ms"] = round(sum(v[5] for v in d.values()) / n, 2)
+        for op, key in (("READ", "read_ms"), ("WRITE", "write_ms")):
+            v = d.get(op)
+            if v and v[0]:
+                row[key] = round(v[5] / v[0], 2)
+        out.append(row)
+    return out
+
+
 def collect_temps() -> list:
     """Return [{label, celsius}] from /sys/class/hwmon (best-effort, may be empty)."""
     out: list = []
@@ -1789,7 +1870,13 @@ def build_sample(mounts: list[str], state: dict,
         {"device": n, "read_bps": round(rd, 1), "write_bps": round(wr, 1)}
         for n, (rd, wr) in _rates(disk_raw, state.get("disk", {}), dt).items()
     ]
-    state["net"], state["disk"], state["mono"] = net_raw, disk_raw, now_mono
+    try:
+        nfs_raw = collect_nfs_raw()
+    except Exception:
+        nfs_raw = {}
+    if nfs_raw:
+        sample["nfs"] = nfs_delays(nfs_raw, state.get("nfs", {}), dt)
+    state["net"], state["disk"], state["nfs"], state["mono"] = net_raw, disk_raw, nfs_raw, now_mono
 
     # Docker's status rides every sample, so Steward can tell "no containers"
     # (status ok, docker: []) from "couldn't ask" (unavailable / error).

plugins/host_agent/metrics_query.py

@@ -21,10 +21,20 @@ SOURCE_MODULE = "host_agent"
 HISTORY_METRICS = (
     "cpu_pct", "mem_used_pct", "disk_used_pct_worst", "load_1m",
     "net_rx_bps", "net_tx_bps", "disk_read_bps", "disk_write_bps",
-    "temp_c_max", "psi_mem_some_avg10",
+    "temp_c_max", "psi_mem_some_avg10", "nfs_exe_max_ms", "mounts_stalled",
 )
 
 
+async def has_nfs(session, host_name: str) -> bool:
+    """Whether the host's agent has reported an NFS mount's delay (#5996)."""
+    from steward.models.metrics import PluginMetricLatest
+    return (await session.execute(
+        select(PluginMetricLatest.metric_name).where(
+            PluginMetricLatest.source_module == SOURCE_MODULE,
+            PluginMetricLatest.resource_name.startswith(host_name + ":"),
+            PluginMetricLatest.metric_name == "nfs_ops_per_sec").limit(1))).first() is not None
+
+
 async def _latest_metrics_for_host(session, host_name: str) -> dict[str, dict[str, float]]:
     """{resource_name: {metric: value}} — latest sample per (resource, metric) for
     a host + its sub-resources, from plugin_metric_latest (#5208). A host whose

plugins/host_agent/routes.py

@@ -36,6 +36,7 @@ from .metrics_query import (
     SOURCE_MODULE,
     _history_for_host,
     _latest_metrics_for_host,
+    has_nfs,
 )
 
 host_agent_bp = Blueprint("host_agent", __name__, template_folder="templates")
@@ -290,6 +291,25 @@ def _expand_sample_to_metrics(
             row("disk_used_pct_worst", host_name, worst_pct)
         row("mounts_stalled", host_name, stalled)
 
+    # NFS delay per mount (agent ≥ 1.22.0, #5996): what its requests took
+    # since the last sample. The host's slowest mount is its own metric, for
+    # a rule; recorded only when some mount finished a request, since a mount
+    # with none, idle or stuck, has no delay to give (mounts_stalled covers
+    # stuck).
+    slowest = None
+    for nfs in sample.get("nfs") or []:
+        mount_res = f"{host_name}:{nfs['mount']}"
+        for key, metric in (("ops_per_sec", "nfs_ops_per_sec"), ("rtt_ms", "nfs_rtt_ms"),
+                            ("exe_ms", "nfs_exe_ms"), ("read_ms", "nfs_read_ms"),
+                            ("write_ms", "nfs_write_ms"), ("retrans", "nfs_retrans"),
+                            ("timeouts", "nfs_timeouts")):
+            if isinstance(nfs.get(key), (int, float)):
+                row(metric, mount_res, nfs[key])
+        if isinstance(nfs.get("exe_ms"), (int, float)):
+            slowest = max(slowest or 0.0, nfs["exe_ms"])
+    if slowest is not None:
+        row("nfs_exe_max_ms", host_name, slowest)
+
     # Per-core CPU. Sub-resources carry a ':' so the fleet widget filters them out.
     for i, core_pct in enumerate(sample.get("cpu_cores") or []):
         if core_pct is not None:
@@ -883,9 +903,10 @@ async def host_detail(host_id: str):
             select(Host).where(Host.id == host_id))).scalar_one_or_none()
         if host is None:
             return _error(404, "not_found")
+        nfs = await has_nfs(session, host.name)
     return await render_template(
         "host_detail.html",
-        host=host, current_range=range_key, range_options=RANGE_OPTIONS,
+        host=host, current_range=range_key, range_options=RANGE_OPTIONS, has_nfs=nfs,
         poll_seconds=_DETAIL_POLL_SECONDS, chart_poll_seconds=_CHART_POLL_SECONDS,
     )
 

plugins/host_agent/templates/_host_metrics.html

@@ -114,6 +114,17 @@
       <span style="color:var(--text-muted);">{{ m.get('disk_used_bytes')|bytes }} / {{ m.get('disk_total_bytes')|bytes }}{% if m.get('disk_used_pct') is not none %} · {{ "%.0f"|format(m.get('disk_used_pct')) }}%{% endif %}</span>
     </div>
     {{ bar(m.get('disk_used_pct'), 'disk') }}
+    {% if m.get('nfs_ops_per_sec') is not none %}
+    {# NFS delay (#5996): what a request took since the agent's last sample. #}
+    {% set exe = m.get('nfs_exe_ms') %}
+    <div style="font-size:0.72rem;color:var(--text-dim);margin-top:0.2rem;">
+      NFS {% if m.get('nfs_ops_per_sec') > 0 and exe is not none -%}
+      <span class="{{ 'tone-crit' if exe >= 1000 else ('tone-warn' if exe >= 100 else '') }}">{{ "%.1f"|format(exe) }} ms a request</span> · {{ "%.3g"|format(m.get('nfs_ops_per_sec')) }}/s
+      {%- else %}idle{% endif %}
+      {%- if m.get('nfs_retrans') %} · <span class="tone-warn">{{ m.get('nfs_retrans')|int }} sent again</span>{% endif %}
+      {%- if m.get('nfs_timeouts') %} · <span class="tone-crit">{{ m.get('nfs_timeouts')|int }} timed out</span>{% endif %}
+    </div>
+    {% endif %}
   </div>
   {% endfor %}
 </div>

plugins/host_agent/templates/host_detail.html

@@ -35,6 +35,12 @@
     <div class="section-title" style="margin-bottom:0.5rem;">Load &amp; pressure — last <span class="hm-chart-range">{{ current_range }}</span></div>
     <div style="height:220px;"><canvas id="chart-pressure"></canvas></div>
   </div>
+  {% if has_nfs %}
+  <div class="card" style="margin-bottom:0;">
+    <div class="section-title" style="margin-bottom:0.5rem;">NFS delay (ms) — last <span class="hm-chart-range">{{ current_range }}</span></div>
+    <div style="height:220px;"><canvas id="chart-nfs"></canvas></div>
+  </div>
+  {% endif %}
 </div>
 {# Hidden poller: fetches the history series and applies it to the charts above
    (no visible swap). Lazy on load → never blocks paint; refreshed slowly + on
@@ -79,6 +85,10 @@
       { key: "psi_mem_some_avg10", label: "Mem PSI %", color: stewardColor("--red") },
       { key: "temp_c_max", label: "Temp °C", color: stewardColor("--yellow") },
     ] },
+    "chart-nfs": { ymax: undefined, ds: [
+      { key: "nfs_exe_max_ms", label: "Slowest mount, ms a request", color: stewardColor("--blue") },
+      { key: "mounts_stalled", label: "Mounts not answering", color: stewardColor("--red") },
+    ] },
   };
   const charts = {};
   for (const id of Object.keys(defs)) {

steward/core/formatting.py

@@ -341,6 +341,14 @@ METRICS: dict[str, tuple[str, str]] = {
     "seconds_since_seen": ("Time since the agent reported", "over 3 minutes means the agent is offline"),
     "clock_off_secs": ("Clock off by", "how far the host's clock is from Steward's, either way"),
     "mounts_stalled": ("Mounts not answering", "mounts the agent couldn't read in 5 s, e.g. an NFS share whose server stopped"),
+    "nfs_exe_max_ms": ("NFS delay (slowest mount)", "the host's slowest NFS mount: how long a request took, queueing included"),
+    "nfs_exe_ms": ("NFS delay", "one NFS mount (resource host:mount): how long a request took on average, queueing included"),
+    "nfs_rtt_ms": ("NFS round trip", "one NFS mount: how long the server took to answer a request"),
+    "nfs_read_ms": ("NFS read delay", "one NFS mount: how long a read took on average"),
+    "nfs_write_ms": ("NFS write delay", "one NFS mount: how long a write took on average"),
+    "nfs_ops_per_sec": ("NFS requests", "one NFS mount: requests finished a second"),
+    "nfs_retrans": ("NFS retransmissions", "one NFS mount: requests sent again since the last sample"),
+    "nfs_timeouts": ("NFS timeouts", "one NFS mount: requests that timed out since the last sample"),
     "agent_rss_bytes": ("Agent memory", "the host agent's own resident memory; tens of MB is normal"),
     "agent_cpu_pct": ("Agent CPU", "the host agent's own CPU, % of one core; a few % is normal"),
     "updates_pending": ("Updates waiting", "packages the host's package manager would upgrade, checked hourly"),

tests/core/test_metric_catalog.py

@@ -25,17 +25,19 @@ def _const(node):
 
 
 def _loop_names(func: ast.AST, var: str) -> set[str]:
-    """Literal first elements of `for var, _ in [("a", x), ("b", y)]` loops."""
+    """Literal elements bound to `var` by `for var, _ in [("a", x), ("b", y)]`
+    loops, wherever `var` sits in the target (`for _, var in ...` too)."""
     names: set[str] = set()
     for node in ast.walk(func):
         if not (isinstance(node, ast.For) and isinstance(node.iter, (ast.List, ast.Tuple))):
             continue
-        target = node.target
-        first = target.elts[0] if isinstance(target, ast.Tuple) and target.elts else target
-        if isinstance(first, ast.Name) and first.id == var:
-            for elt in node.iter.elts:
-                if isinstance(elt, ast.Tuple) and elt.elts and _const(elt.elts[0]):
-                    names.add(elt.elts[0].value)
+        targets = node.target.elts if isinstance(node.target, ast.Tuple) else [node.target]
+        at = next((i for i, t in enumerate(targets) if isinstance(t, ast.Name) and t.id == var), None)
+        if at is None:
+            continue
+        for elt in node.iter.elts:
+            if isinstance(elt, ast.Tuple) and len(elt.elts) > at and _const(elt.elts[at]):
+                names.add(elt.elts[at].value)
     return names
 
 
@@ -112,17 +114,30 @@ def host_agent_rows(source: str) -> set[str]:
     for func in ast.walk(ast.parse(source)):
         if isinstance(func, ast.FunctionDef) and func.name == "_expand_sample_to_metrics":
             for call in ast.walk(func):
-                if (isinstance(call, ast.Call) and isinstance(call.func, ast.Name)
-                        and call.func.id == "row" and call.args and _const(call.args[0])):
+                if not (isinstance(call, ast.Call) and isinstance(call.func, ast.Name)
+                        and call.func.id == "row" and call.args):
+                    continue
+                if _const(call.args[0]):
                     names.add(call.args[0].value)
+                elif isinstance(call.args[0], ast.Name):
+                    names |= _loop_names(func, call.args[0].id)
     return names
 
 
 def test_every_metric_host_agent_ingest_writes_is_in_the_catalog():
     """temp_c_max was recorded for weeks and no rule could watch it (#5787)."""
     names = host_agent_rows((_REPO / "plugins/host_agent/routes.py").read_text())
-    assert {"cpu_pct", "temp_c_max", "mounts_stalled"} <= names and len(names) >= 20, names
+    assert {"cpu_pct", "temp_c_max", "mounts_stalled", "nfs_exe_ms"} <= names and len(names) >= 20, names
     from plugins.host_agent import NOT_ALERTABLE
     missing = sorted(names - catalog()["host_agent"] - NOT_ALERTABLE)
     assert not missing, f"host_agent records these but no rule can watch them: {missing}"
     assert NOT_ALERTABLE <= names, "NOT_ALERTABLE names something host_agent no longer records"
+
+
+def test_a_loop_name_is_read_wherever_it_sits_in_the_tuple():
+    code = '''
+def _expand_sample_to_metrics():
+    for key, metric in (("rtt_ms", "nfs_rtt_ms"), ("exe_ms", "nfs_exe_ms")):
+        row(metric, "h:/nfs", 1)
+'''
+    assert host_agent_rows(code) == {"nfs_rtt_ms", "nfs_exe_ms"}

tests/plugins/host_agent/test_agent_collectors.py

@@ -151,3 +151,52 @@ def test_a_mount_that_errors_is_left_out_not_the_whole_list():
             patch.object(a.os, "statvfs", side_effect=OSError):
         out = a.collect_storage(["/", "/nfs/gone"])
     assert out == [{"mount": "/", "total_bytes": 10, "used_bytes": 1}]
+
+
+# ─── NFS delay per mount (#5996) ─────────────────────────────────────────────
+
+MOUNTSTATS = (
+    "device proc mounted on /proc with fstype proc\n"
+    "device 192.168.0.7:/volume1/data mounted on /nfs/data with fstype nfs4 statvers=1.1\n"
+    "\topts:\trw,vers=4.1,hard,proto=tcp\n"
+    "\tper-op statistics\n"
+    "\t        NULL: 0 0 0 0 0 0 0 0 0\n"
+    "\t        READ: 10 10 0 1200 40960 5 30 40 0\n"
+    "\t       WRITE: 4 4 0 8192 512 2 20 24 0\n"
+    "\n"
+    "device 192.168.0.5:/volume1/My Media mounted on /nfs/my\\040media with fstype nfs statvers=1.1\n"
+    "\tper-op statistics\n"
+    "\t     GETATTR: 3 3 0 300 300 0 3 3 0\n"
+    "device /dev/sda1 mounted on / with fstype ext4\n"
+)
+
+
+def test_mountstats_reads_each_nfs_mount_s_ops_and_skips_the_rest():
+    got = a.read_mountstats(MOUNTSTATS)
+    assert set(got) == {"/nfs/data", "/nfs/my media"}
+    assert got["/nfs/data"]["READ"] == (10, 10, 0, 5, 30, 40)
+    assert got["/nfs/data"]["NULL"] == (0, 0, 0, 0, 0, 0)
+    assert got["/nfs/my media"] == {"GETATTR": (3, 3, 0, 0, 3, 3)}
+
+
+def test_mountstats_that_cannot_be_read_is_no_mounts(tmp_path):
+    with patch.object(a, "MOUNTSTATS_PATH", str(tmp_path / "missing")):
+        assert a.collect_nfs_raw() == {}
+
+
+def test_nfs_delay_is_what_the_requests_since_the_last_sample_took():
+    prev = {"/nfs/data": {"READ": (10, 10, 0, 5, 30, 40), "WRITE": (4, 4, 0, 2, 20, 24)},
+            "/nfs/idle": {"GETATTR": (3, 3, 0, 0, 3, 3)},
+            "/nfs/remounted": {"READ": (50, 50, 0, 0, 50, 50)}}
+    cur = {"/nfs/data": {"READ": (20, 21, 1, 7, 130, 240), "WRITE": (4, 4, 0, 2, 20, 24)},
+           "/nfs/idle": {"GETATTR": (3, 3, 0, 0, 3, 3)},
+           "/nfs/remounted": {"READ": (2, 2, 0, 0, 2, 2)},
+           "/nfs/new": {"READ": (5, 5, 0, 0, 5, 5)}}
+    got = {r["mount"]: r for r in a.nfs_delays(cur, prev, 10.0)}
+    # ten READs in 10 s, 100 ms of round trip and 200 ms of waiting between
+    # them, one sent twice and one that hit a major timeout; no WRITE finished
+    assert got["/nfs/data"] == {"mount": "/nfs/data", "ops_per_sec": 1.0, "retrans": 1, "timeouts": 1,
+                                "rtt_ms": 10.0, "exe_ms": 20.0, "read_ms": 20.0}
+    assert got["/nfs/idle"] == {"mount": "/nfs/idle", "ops_per_sec": 0.0, "retrans": 0, "timeouts": 0}
+    assert set(got) == {"/nfs/data", "/nfs/idle"}          # counters gone back, or no baseline yet
+    assert a.nfs_delays(cur, prev, 0) == []

tests/plugins/host_agent/test_expand_metrics.py

@@ -181,3 +181,25 @@ def test_the_host_page_lists_filesystems_fullest_first():
               "h:/data": {"disk_used_pct": 30.0}}
     *_, mounts = _split_host_metrics("h", latest)
     assert list(mounts) == ["/srv", "/var/lib/docker", "/", "/boot", "/data"]
+
+
+def test_nfs_delay_is_recorded_per_mount_and_the_slowest_on_the_host():
+    """#5996: what each NFS mount's requests took, and the host's slowest for a rule."""
+    sample = {"ts": "2026-10-17T12:00:00+00:00",
+              "nfs": [{"mount": "/nfs/data", "ops_per_sec": 4.0, "retrans": 2, "timeouts": 0,
+                       "rtt_ms": 3.5, "exe_ms": 1250.0, "write_ms": 1400.0},
+                      {"mount": "/nfs/media", "ops_per_sec": 1.0, "retrans": 0, "timeouts": 0,
+                       "rtt_ms": 1.0, "exe_ms": 2.0, "read_ms": 2.0}]}
+    idx = _index(_expand_sample_to_metrics(sample, "d3", datetime.now(timezone.utc)))
+    assert idx[("nfs_exe_ms", "d3:/nfs/data")] == 1250.0
+    assert idx[("nfs_write_ms", "d3:/nfs/data")] == 1400.0 and ("nfs_read_ms", "d3:/nfs/data") not in idx
+    assert idx[("nfs_retrans", "d3:/nfs/data")] == 2 and idx[("nfs_ops_per_sec", "d3:/nfs/media")] == 1.0
+    assert idx[("nfs_exe_max_ms", "d3")] == 1250.0
+
+
+def test_a_host_whose_nfs_mounts_finished_nothing_has_no_nfs_delay():
+    sample = {"ts": "2026-10-17T12:00:00+00:00",
+              "nfs": [{"mount": "/nfs/data", "ops_per_sec": 0.0, "retrans": 0, "timeouts": 0}]}
+    idx = _index(_expand_sample_to_metrics(sample, "d3", datetime.now(timezone.utc)))
+    assert idx[("nfs_ops_per_sec", "d3:/nfs/data")] == 0.0
+    assert ("nfs_exe_max_ms", "d3") not in idx and ("nfs_exe_ms", "d3:/nfs/data") not in idx

tests/plugins/host_agent/test_host_health.py

@@ -95,3 +95,22 @@ async def test_sensors_not_counted_are_listed_once_per_reason(app):
     page = " ".join(page.split())
     assert (f"Not counted, likely nothing wired to them: temp3 62°C ({falls}); "
             f"SYSTIN 115°C, AUXTIN0 110°C ({held}).") in page
+
+
+async def test_an_nfs_mount_says_what_a_request_took_and_a_local_one_says_nothing(app):
+    """#5996: the delay under each NFS mount's bar, graded at 100 and 1000 ms."""
+    async with app.test_request_context("/plugins/host_agent/h1/metrics"):
+        page = await render_plugin_template(
+            app, "host_agent", "_host_metrics.html", host=SimpleNamespace(address="10.0.0.2"), stale=False,
+            reg=SimpleNamespace(distro=None, kernel=None, arch=None, agent_version=None,
+                                last_seen_at=datetime.now(timezone.utc)),
+            hostlvl={}, cores=[], nets={}, disks_io={}, temps={},
+            mounts={"/nfs/data": {"disk_used_pct": 40.0, "nfs_ops_per_sec": 4.0, "nfs_exe_ms": 250.0,
+                                  "nfs_retrans": 3.0},
+                    "/nfs/media": {"disk_used_pct": 30.0, "nfs_ops_per_sec": 0.0},
+                    "/": {"disk_used_pct": 20.0}})
+    page = re.sub(r">\s+<", "><", " ".join(page.split()))
+    assert '<span class="tone-warn">250.0 ms a request</span> · 4/s · <span class="tone-warn">3 sent again</span>' in page
+    assert "NFS idle" in page and page.count("NFS ") == 2
+    assert "timed out" not in page
+    assert not hex_colours(page), hex_colours(page)