fix(host_agent): every Docker read is bounded, and the agent reports its own footprint (#5764)
#5764 "The agent reads a container's whole log backlog into memory, on its main loop, with no overall deadline". Docker03's 1.13.0 agent went silent for a day at 2.1 GB and 46 h of CPU, then ignored SIGTERM. - One shared reader, _docker_get: each answer has an overall deadline and a byte cap, not only a per-recv timeout. A trickling or endless body is cut off (15 s / 32 MiB for JSON, 5 s / 4 MiB for a log read). - Container logs: since the cursor but only the newest 2,000 lines, all within 10 s. A note says when older lines were skipped, and a container whose answer is too big skips ahead to now instead of re-reading its backlog every interval. - The agent reports its own RSS and CPU; Steward records agent_rss_bytes and agent_cpu_pct, so a runaway agent shows before it goes silent. Agent 1.15.0. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -65,10 +65,13 @@ def get_alert_metrics() -> dict[str, list[str]]:
|
||||
of the host's cores. inodes_used_pct is per mount (agent ≥ 1.13.0), and
|
||||
disk_used_pct_worst is the fullest mount by space or inodes.
|
||||
mounts_stalled counts the mounts that didn't answer the agent in time
|
||||
(agent ≥ 1.14.0): "an NFS share stopped" is > 0."""
|
||||
(agent ≥ 1.14.0): "an NFS share stopped" is > 0. agent_rss_bytes and
|
||||
agent_cpu_pct are the agent's own memory and CPU (agent ≥ 1.15.0): a
|
||||
healthy agent sits in tens of MB and a few %."""
|
||||
return {"host_agent": [
|
||||
"cpu_pct", "mem_used_pct", "mem_available_bytes", "swap_used_bytes",
|
||||
"disk_used_pct_worst", "disk_used_pct", "inodes_used_pct", "load_1m", "load_5m", "load_15m",
|
||||
"load_pct",
|
||||
"uptime_secs", "seconds_since_seen", "clock_off_secs", "mounts_stalled",
|
||||
"agent_rss_bytes", "agent_cpu_pct",
|
||||
]}
|
||||
|
||||
Reference in New Issue
Block a user