fix(docker): reap vanished containers; clarify the resources widget
CI / lint (push) Successful in 2s
CI / unit (push) Successful in 44s
CI / integration (push) Successful in 2m20s
CI / publish (push) Successful in 1m0s

The container persister upserted current state keyed (host_id, name) but — unlike
the image/swarm/disk persisters — never deleted containers that vanished from the
host's latest listing. Every removed one-shot container (CI job runners, buildkit
builders, codex jobs) left a permanent "stopped" row, so the dashboard counts
ballooned (e.g. 856 stopped) and read as "not dedup'd". It wasn't dedup — it was
a missing reaper.

- ingest.py: after the upsert loop, delete this host's containers whose name is
  notin the newest snapshot (the listing is authoritative — event derivation
  already treats absence as removal). Mirrors images/swarm/disk. The existing
  `if not snapshots: return` keeps swarm/disk-only samples from touching
  containers; an empty container snapshot legitimately means none exist.
- widget_resources.html: this widget shows the busiest running containers by CPU.
  Replaced the two unlabeled hairline bars with labeled CPU/MEM bars (label · bar ·
  % value, warn/crit colors) so it's self-explanatory.
- Regression test: a container present then absent across snapshots is reaped.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC
This commit is contained in:
2026-06-20 22:06:43 -04:00
parent 51682f130a
commit 0c055ed6fa
3 changed files with 69 additions and 26 deletions
+38
View File
@@ -231,6 +231,44 @@ def test_lifecycle_events_derived_across_snapshots(app):
assert not any(e[0] == "start" for e in events)
@_NEEDS_DB
def test_vanished_container_is_reaped(app):
"""A container absent from the latest snapshot is deleted, not left behind as
a permanent stopped row. Regression: removed one-shot containers (CI jobs,
buildkit builders) accumulated forever and inflated the dashboard counts."""
from sqlalchemy import text
from steward.models.hosts import Host
persist = _persist_fn(app)
t1 = datetime(2026, 6, 19, 12, 0, 0, tzinfo=timezone.utc)
t2 = datetime(2026, 6, 19, 12, 0, 30, tzinfo=timezone.utc)
base = {"status": "running", "cpu_pct": 1.0, "mem_pct": 1.0,
"restart_count": 0, "exit_code": None, "oom_killed": False, "health": None}
keep = {**base, "name": "keep"}
gone = {**base, "name": "ci-job-123"}
async def _go():
async with app.db_sessionmaker() as s:
async with s.begin():
await s.execute(text("DELETE FROM docker_containers"))
h = Host(id=str(uuid.uuid4()), name="reap", address="10.9.9.8")
s.add(h)
await s.flush()
await persist(s, h, [(t1, [keep, gone])]) # both present
before = [r[0] for r in (await s.execute(text(
"SELECT name FROM docker_containers WHERE host_id = :h ORDER BY name"),
{"h": h.id})).all()]
await persist(s, h, [(t2, [keep])]) # ci-job-123 vanished
after = [r[0] for r in (await s.execute(text(
"SELECT name FROM docker_containers WHERE host_id = :h ORDER BY name"),
{"h": h.id})).all()]
return before, after
before, after = asyncio.run(_go())
assert before == ["ci-job-123", "keep"]
assert after == ["keep"] # the vanished container was reaped
@_NEEDS_DB
def test_swarm_topology_persisted(app):
from sqlalchemy import text