fix(host_agent): every Docker read is bounded, and the agent reports its own footprint (#5764)
CI / lint (push) Successful in 2s
CI / integration (push) Successful in 1m20s
CI / unit (push) Failing after 1m32s
CI / publish (push) Skipped

#5764 "The agent reads a container's whole log backlog into memory, on its
main loop, with no overall deadline". Docker03's 1.13.0 agent went silent
for a day at 2.1 GB and 46 h of CPU, then ignored SIGTERM.

- One shared reader, _docker_get: each answer has an overall deadline and a
  byte cap, not only a per-recv timeout. A trickling or endless body is cut
  off (15 s / 32 MiB for JSON, 5 s / 4 MiB for a log read).
- Container logs: since the cursor but only the newest 2,000 lines, all
  within 10 s. A note says when older lines were skipped, and a container
  whose answer is too big skips ahead to now instead of re-reading its
  backlog every interval.
- The agent reports its own RSS and CPU; Steward records agent_rss_bytes
  and agent_cpu_pct, so a runaway agent shows before it goes silent.

Agent 1.15.0.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-10-14 18:01:45 -04:00
co-authored by Claude Opus 5.5
parent 82a2a3d41f
commit a805e1d539
7 changed files with 292 additions and 53 deletions
+4 -1
View File
@@ -65,10 +65,13 @@ def get_alert_metrics() -> dict[str, list[str]]:
of the host's cores. inodes_used_pct is per mount (agent ≥ 1.13.0), and
disk_used_pct_worst is the fullest mount by space or inodes.
mounts_stalled counts the mounts that didn't answer the agent in time
(agent ≥ 1.14.0): "an NFS share stopped" is > 0."""
(agent ≥ 1.14.0): "an NFS share stopped" is > 0. agent_rss_bytes and
agent_cpu_pct are the agent's own memory and CPU (agent ≥ 1.15.0): a
healthy agent sits in tens of MB and a few %."""
return {"host_agent": [
"cpu_pct", "mem_used_pct", "mem_available_bytes", "swap_used_bytes",
"disk_used_pct_worst", "disk_used_pct", "inodes_used_pct", "load_1m", "load_5m", "load_15m",
"load_pct",
"uptime_secs", "seconds_since_seen", "clock_off_secs", "mounts_stalled",
"agent_rss_bytes", "agent_cpu_pct",
]}