feat(host_agent): NFS delay per mount, read from /proc/self/mountstats (#5996)
CI / lint (push) Successful in 2s
CI / unit (push) Failing after 1m32s
CI / integration (push) Successful in 1m31s
CI / publish (push) Skipped

Each sample carries each NFS mount's requests since the last one: how many
a second, their round trip and execute time (READ and WRITE on their own),
and how many were sent again or hit a major timeout. mountstats is the
kernel's own count and never touches the share, so it reads while one is
stuck. Steward stores them per mount and the slowest on the host as
nfs_exe_max_ms, for an "NFS slow" rule and the host page's NFS chart beside
mounts_stalled; each mount's bar says what a request took.

Agent 1.22.0 (which also carries the backlog order from 41fc807).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-10-17 09:46:14 -04:00
co-authored by Claude Opus 5.5
parent 2df8431de9
commit 3f9aae7f46
12 changed files with 282 additions and 13 deletions
+6
View File
@@ -124,6 +124,12 @@ Python 3.8+ stdlib only. `urllib.request` for the POST, plus `json`, `os`, `shut
Each mount is read in a daemon thread under one 5 s deadline. A hard-mounted NFS share whose server stops blocks `statvfs` in the kernel with no timeout; read inline, that froze the whole agent. A mount that misses the deadline is sent as `{"mount", "stalled": true}`, and isn't probed again while its last probe is still blocked. Steward counts them as `mounts_stalled`.
### NFS delay per mount (1.22.0, #5996)
`mounts_stalled` says a mount is stuck; it can't say a share was slow before it stuck, or how slow it was when it came back. Each sample also reads `/proc/self/mountstats`, the kernel's own cumulative count of every NFS request each mount has finished. Reading it never touches the share, so it answers while one is stuck. The agent sends each NFS mount's requests since the last sample as `nfs: [{mount, ops_per_sec, retrans, timeouts, rtt_ms, exe_ms, read_ms, write_ms}]`: `rtt_ms` is the network and server's round trip, `exe_ms` that plus the time queued on the host, which is what a program waits. The delays are averages over the requests that finished, and are left out when none did. A mount new since the last sample, or whose counters went back (remounted), waits for the next one.
Steward stores them per mount (`nfs_*` on `<host>:<mount>`), and the slowest mount's `exe_ms` on the host as `nfs_exe_max_ms`, for a rule ("NFS slow": over 1000 ms for 3 checks) and for the host page's NFS chart beside `mounts_stalled`. Each mount's bar on the host page says what a request took and how many were sent again or timed out. A request is counted when it finishes, so a stuck share shows nothing until it comes back, then its long requests all at once.
### Every Docker read is bounded (1.15.0, #5764)
One Docker answer must arrive within 15 s and fit in 32 MiB, and one log answer within 5 s and 4 MiB. Past either limit the read is cut off, not waited on. A per-read timeout alone never ends a body that keeps trickling in. Container logs are read since each container's cursor, but only its newest 2,000 lines, all containers within 10 s. A container that writes more than that gets a `_steward` note saying older lines were skipped. A container whose answer is too big skips ahead to now. Before this, a 1.13.0 agent re-read a growing backlog every interval and went silent for a day at 2.1 GB. Each sample also carries the agent's own `agent` footprint (`rss_bytes`, `cpu_pct`), which Steward records as `agent_rss_bytes` and `agent_cpu_pct`.