fix(docker): reap vanished containers; clarify the resources widget
The container persister upserted current state keyed (host_id, name) but — unlike the image/swarm/disk persisters — never deleted containers that vanished from the host's latest listing. Every removed one-shot container (CI job runners, buildkit builders, codex jobs) left a permanent "stopped" row, so the dashboard counts ballooned (e.g. 856 stopped) and read as "not dedup'd". It wasn't dedup — it was a missing reaper. - ingest.py: after the upsert loop, delete this host's containers whose name is notin the newest snapshot (the listing is authoritative — event derivation already treats absence as removal). Mirrors images/swarm/disk. The existing `if not snapshots: return` keeps swarm/disk-only samples from touching containers; an empty container snapshot legitimately means none exist. - widget_resources.html: this widget shows the busiest running containers by CPU. Replaced the two unlabeled hairline bars with labeled CPU/MEM bars (label · bar · % value, warn/crit colors) so it's self-explanatory. - Regression test: a container present then absent across snapshots is reaped. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC
This commit is contained in:
@@ -231,6 +231,44 @@ def test_lifecycle_events_derived_across_snapshots(app):
|
||||
assert not any(e[0] == "start" for e in events)
|
||||
|
||||
|
||||
@_NEEDS_DB
|
||||
def test_vanished_container_is_reaped(app):
|
||||
"""A container absent from the latest snapshot is deleted, not left behind as
|
||||
a permanent stopped row. Regression: removed one-shot containers (CI jobs,
|
||||
buildkit builders) accumulated forever and inflated the dashboard counts."""
|
||||
from sqlalchemy import text
|
||||
from steward.models.hosts import Host
|
||||
|
||||
persist = _persist_fn(app)
|
||||
t1 = datetime(2026, 6, 19, 12, 0, 0, tzinfo=timezone.utc)
|
||||
t2 = datetime(2026, 6, 19, 12, 0, 30, tzinfo=timezone.utc)
|
||||
base = {"status": "running", "cpu_pct": 1.0, "mem_pct": 1.0,
|
||||
"restart_count": 0, "exit_code": None, "oom_killed": False, "health": None}
|
||||
keep = {**base, "name": "keep"}
|
||||
gone = {**base, "name": "ci-job-123"}
|
||||
|
||||
async def _go():
|
||||
async with app.db_sessionmaker() as s:
|
||||
async with s.begin():
|
||||
await s.execute(text("DELETE FROM docker_containers"))
|
||||
h = Host(id=str(uuid.uuid4()), name="reap", address="10.9.9.8")
|
||||
s.add(h)
|
||||
await s.flush()
|
||||
await persist(s, h, [(t1, [keep, gone])]) # both present
|
||||
before = [r[0] for r in (await s.execute(text(
|
||||
"SELECT name FROM docker_containers WHERE host_id = :h ORDER BY name"),
|
||||
{"h": h.id})).all()]
|
||||
await persist(s, h, [(t2, [keep])]) # ci-job-123 vanished
|
||||
after = [r[0] for r in (await s.execute(text(
|
||||
"SELECT name FROM docker_containers WHERE host_id = :h ORDER BY name"),
|
||||
{"h": h.id})).all()]
|
||||
return before, after
|
||||
|
||||
before, after = asyncio.run(_go())
|
||||
assert before == ["ci-job-123", "keep"]
|
||||
assert after == ["keep"] # the vanished container was reaped
|
||||
|
||||
|
||||
@_NEEDS_DB
|
||||
def test_swarm_topology_persisted(app):
|
||||
from sqlalchemy import text
|
||||
|
||||
Reference in New Issue
Block a user