fix(host_agent): a stalled NFS mount no longer freezes the agent (#5760)
CI / lint (push) Successful in 3s
CI / integration (push) Successful in 1m21s
CI / unit (push) Successful in 1m33s
CI / publish (push) Successful in 1m3s

#5760 "One stalled NFS mount freezes the whole agent: statvfs has no
deadline, so the host goes silent while it still answers ping". The agent
read every mount inline in its only loop; a hard-mounted share whose server
stops blocks statvfs in the kernel with no timeout. Each mount is now read
in a daemon thread under one 5 s deadline. One that misses it is sent as
stalled, and isn't probed again while its last probe is still blocked. A
mount that errors (a stale handle) is left out instead of emptying the list.

Steward counts them as mounts_stalled (alertable: "an NFS share stopped"),
and skips the fullest-disk figure when every mount stalled.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-10-14 17:12:27 -04:00
co-authored by Claude Opus 5.5
parent 7feb2b1ce2
commit b6c4ce510d
7 changed files with 128 additions and 19 deletions
+4 -2
View File
@@ -63,10 +63,12 @@ def get_alert_metrics() -> dict[str, list[str]]:
clock_off_secs is how far the host's clock is from Steward's, either way
(agent ≥ 1.12.0): "clock drifting" is > 30. load_pct is load_1m as a %
of the host's cores. inodes_used_pct is per mount (agent ≥ 1.13.0), and
disk_used_pct_worst is the fullest mount by space or inodes."""
disk_used_pct_worst is the fullest mount by space or inodes.
mounts_stalled counts the mounts that didn't answer the agent in time
(agent ≥ 1.14.0): "an NFS share stopped" is > 0."""
return {"host_agent": [
"cpu_pct", "mem_used_pct", "mem_available_bytes", "swap_used_bytes",
"disk_used_pct_worst", "disk_used_pct", "inodes_used_pct", "load_1m", "load_5m", "load_15m",
"load_pct",
"uptime_secs", "seconds_since_seen", "clock_off_secs",
"uptime_secs", "seconds_since_seen", "clock_off_secs", "mounts_stalled",
]}