fix(docker): reap vanished containers; clarify the resources widget
CI / lint (push) Successful in 2s
CI / unit (push) Successful in 44s
CI / integration (push) Successful in 2m20s
CI / publish (push) Successful in 1m0s

The container persister upserted current state keyed (host_id, name) but — unlike
the image/swarm/disk persisters — never deleted containers that vanished from the
host's latest listing. Every removed one-shot container (CI job runners, buildkit
builders, codex jobs) left a permanent "stopped" row, so the dashboard counts
ballooned (e.g. 856 stopped) and read as "not dedup'd". It wasn't dedup — it was
a missing reaper.

- ingest.py: after the upsert loop, delete this host's containers whose name is
  notin the newest snapshot (the listing is authoritative — event derivation
  already treats absence as removal). Mirrors images/swarm/disk. The existing
  `if not snapshots: return` keeps swarm/disk-only samples from touching
  containers; an empty container snapshot legitimately means none exist.
- widget_resources.html: this widget shows the busiest running containers by CPU.
  Replaced the two unlabeled hairline bars with labeled CPU/MEM bars (label · bar ·
  % value, warn/crit colors) so it's self-explanatory.
- Regression test: a container present then absent across snapshots is reaped.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC
This commit is contained in:
2026-06-20 22:06:43 -04:00
parent 51682f130a
commit 0c055ed6fa
3 changed files with 69 additions and 26 deletions
+13
View File
@@ -321,3 +321,16 @@ async def persist_host_docker(session, host, snapshots, swarm=None, disk=None) -
resource_name=resource, metric_name="is_healthy",
value=1.0 if health == "healthy" else 0.0,
)
# Reap containers that vanished from the host's latest listing. Without this,
# removed one-shot containers (CI job runners, buildkit builders, codex jobs)
# accumulate forever as permanent "stopped" rows and inflate every count.
# The listing is authoritative (event derivation already treats absence as
# removal), so anything not in the newest snapshot no longer exists on the
# host. Mirrors the image/swarm/disk persisters. An empty snapshot legitimately
# means the host has no containers, so the unfiltered delete is correct.
seen_names = {c["name"] for c in latest_containers if c.get("name")}
reap = delete(DockerContainer).where(DockerContainer.host_id == host.id)
if seen_names:
reap = reap.where(DockerContainer.name.notin_(seen_names))
await session.execute(reap)