fix(host_agent): every Docker read is bounded, and the agent reports its own footprint (#5764)
CI / lint (push) Successful in 2s
CI / integration (push) Successful in 1m20s
CI / unit (push) Failing after 1m32s
CI / publish (push) Skipped

#5764 "The agent reads a container's whole log backlog into memory, on its
main loop, with no overall deadline". Docker03's 1.13.0 agent went silent
for a day at 2.1 GB and 46 h of CPU, then ignored SIGTERM.

- One shared reader, _docker_get: each answer has an overall deadline and a
  byte cap, not only a per-recv timeout. A trickling or endless body is cut
  off (15 s / 32 MiB for JSON, 5 s / 4 MiB for a log read).
- Container logs: since the cursor but only the newest 2,000 lines, all
  within 10 s. A note says when older lines were skipped, and a container
  whose answer is too big skips ahead to now instead of re-reading its
  backlog every interval.
- The agent reports its own RSS and CPU; Steward records agent_rss_bytes
  and agent_cpu_pct, so a runaway agent shows before it goes silent.

Agent 1.15.0.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-10-14 18:01:45 -04:00
co-authored by Claude Opus 5.5
parent 82a2a3d41f
commit a805e1d539
7 changed files with 292 additions and 53 deletions
+2
View File
@@ -340,6 +340,8 @@ METRICS: dict[str, tuple[str, str]] = {
"seconds_since_seen": ("Time since the agent reported", "over 3 minutes means the agent is offline"),
"clock_off_secs": ("Clock off by", "how far the host's clock is from Steward's, either way"),
"mounts_stalled": ("Mounts not answering", "mounts the agent couldn't read in 5 s, e.g. an NFS share whose server stopped"),
"agent_rss_bytes": ("Agent memory", "the host agent's own resident memory; tens of MB is normal"),
"agent_cpu_pct": ("Agent CPU", "the host agent's own CPU, % of one core; a few % is normal"),
}
# Where a rule's metric comes from, as the picker and the rules list say it.