fix(host_agent): every Docker read is bounded, and the agent reports its own footprint (#5764)
#5764 "The agent reads a container's whole log backlog into memory, on its main loop, with no overall deadline". Docker03's 1.13.0 agent went silent for a day at 2.1 GB and 46 h of CPU, then ignored SIGTERM. - One shared reader, _docker_get: each answer has an overall deadline and a byte cap, not only a per-recv timeout. A trickling or endless body is cut off (15 s / 32 MiB for JSON, 5 s / 4 MiB for a log read). - Container logs: since the cursor but only the newest 2,000 lines, all within 10 s. A note says when older lines were skipped, and a container whose answer is too big skips ahead to now instead of re-reading its backlog every interval. - The agent reports its own RSS and CPU; Steward records agent_rss_bytes and agent_cpu_pct, so a runaway agent shows before it goes silent. Agent 1.15.0. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -340,6 +340,8 @@ METRICS: dict[str, tuple[str, str]] = {
|
||||
"seconds_since_seen": ("Time since the agent reported", "over 3 minutes means the agent is offline"),
|
||||
"clock_off_secs": ("Clock off by", "how far the host's clock is from Steward's, either way"),
|
||||
"mounts_stalled": ("Mounts not answering", "mounts the agent couldn't read in 5 s, e.g. an NFS share whose server stopped"),
|
||||
"agent_rss_bytes": ("Agent memory", "the host agent's own resident memory; tens of MB is normal"),
|
||||
"agent_cpu_pct": ("Agent CPU", "the host agent's own CPU, % of one core; a few % is normal"),
|
||||
}
|
||||
|
||||
# Where a rule's metric comes from, as the picker and the rules list say it.
|
||||
|
||||
Reference in New Issue
Block a user