The agent sent Docker's working set (usage less inactive_file), which still
counts recently used file cache. A file-heavy container with no limit read
191 GB while its processes held 2.2 GB, and the host's own figure, which
already leaves cache out, disagreed with it.
Agent 1.23.0 sends mem_used_bytes / mem_used_pct (anon + shmem, so
Postgres's shared buffers count; v1: rss + shmem) and mem_cache_bytes
apart. The new names keep the two meanings apart in history and in learned
normals: docker_013 renames the columns, clears the old values and adds
mem_cache_bytes. The Docker page, swarm by-node sums, Busiest Containers,
container detail, normals and the MCP tools all read the new figure. File
cache is shown only on the container's page and in get_container.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An agent keeps its log cursors in memory, so after a restart it seeds each
container from its last 50 lines, which Steward already holds; docker_logs
has no unique key, so every update or reboot doubled them. The ingest now
reads what it already has for each container over the batch's own span,
in one query, and skips those lines. A push of new lines overlaps nothing.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
On 2026-10-14 nine swarm services sat at 0/1 and nothing fired. From each
manager's swarm report the docker plugin now records three figures on
resource "swarm": swarm_services_short, swarm_nodes_down and
swarm_managers_unreachable. Drained nodes are left out of the last two,
since a drain is deliberate. The figures are recorded at most once every
30 s whichever manager reported, and are offered to alert rules.
Agent 1.19.0 sends each manager's ManagerStatus.Reachability. Older agents
fall back to the node's status.
A maintenance job on a swarm node also keeps a window over docker/swarm,
so its drain and reboot hold those alerts until the job ends.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5752 "Container CPU and memory history lives in the one metric store;
docker_metrics and docker_metrics_hourly are retired". Container cpu_pct and
mem_pct already went to plugin_metrics for alerting; now mem_usage_bytes does
too, and every snapshot in an ingest (backlog included) is recorded at its
own time. The container history chart, Busiest Containers, a service's
history and the MCP container tools read plugin_metrics under resource
"<host>/<container>", rolled up and pruned by the shared metric retention.
docker_012_drop_metrics drops both tables. Their history is not copied
(lesson #5221 "A data migration's cost is set by the live table"); the
store already holds 7 days of container CPU and memory %. Docker retention
now prunes only lifecycle events, and Settings -> Thresholds has one
"Metric retention" card for every source.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5449 "The hourly Docker log prune ranks every log line in the table to find
the few over their cap": the size ring put a running-total window over every
row of docker_logs each hour, sorting the whole table on NFS-backed storage
to find the handful of containers past docker.logs.max_bytes_per_container.
- prune_docker_logs (capability docker.prune_logs) does the age cutoff, then
a per-container byte total picks the containers at or over the cap, and
only their rows are ranked and trimmed. Same keep rule as before: the
newest line always survives.
- run_docker_retention keeps the metrics rollup and the event prune.
- Cleanup runs the log ring in its own transaction after them.
Arose from #5431 "Ingest dropped for ~3 minutes after each deploy while
startup retention runs".
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- #5304 "Docker disk totals recorded over time per host, with a growth chart and alertable metrics"
- #5305 "Disk page: images grouped by repository, versions folded under it"
- #5306 "Docker overview: one disk line per host linking to the Disk page"
Each `docker system df` report is now also recorded as metrics, with the host
as the resource: disk_total_bytes, its parts, and disk_reclaimable_bytes. An
alert rule can watch them. The Disk page starts with the hosts side by side,
showing how much each grew over the selected range with a line of it, then
gives each host its own section. An image's versions fold under its
repository, which is charged only for the layers its versions don't share.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
persist_host_docker marks docker.status, .swarm, .disk, .logs, .metrics,
.events, .containers, .alerts and .reap on the ingest's phase timer, each
flushed before its mark so a part's writes are charged to it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- The status test imported the plugin's models after the app had mapped
them ("already defined"); it reads docker_host_status with SQL.
- The swarm-expiry count awaited inside a generator expression.
- The page test's since came from the module's NOW, so |ago said "5m 4s";
it is built at test time and the assertion checks the minutes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The agent (1.9.0) said nothing when Docker could not be asked and nothing
when Docker had no containers, so Steward could tell neither apart from the
other, and kept a host's last containers on screen as running either way.
- Agent: collect_docker returns (status, detail, containers). "ok" sends
docker: [] when there are none; "unavailable" (no socket, nothing
listening) and "error" (permission, timeout, bad answer) say why.
collect_swarm says whether /info answered, and a node that is not a
manager sends swarm_manager: false. A failed /info says nothing.
- Ingest: an empty docker list is a snapshot; the newest sample's status
goes to the Docker plugin with the rest.
- docker_host_status (docker_011): per host, status, detail, since and
last report. A host that never had Docker gets no row.
- An "ok" empty snapshot reaps the host's containers; an error keeps them.
swarm_manager: false drops the host's swarm services and nodes.
- Docker page: a "Docker not answering" card names each such host, why,
and since when; its containers read as stale. Duplicate header comment
in rows.html removed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Milestone 465 "Swarm task awareness". Fixes three defects from the live MCP
review #4973:
- #5008 "Swarm task history read as live state"
- #5009 "Crash-loop detection is blind to Swarm services"
- #5011 "Swarm task errors are not captured"
Agent and ingest (agent 1.8.0):
- A manager's agent now reports each service's newest tasks: slot, node,
desired and current state, Swarm's Err/Message, exit code and time.
- They are stored in docker_swarm_services.tasks_json (migration
docker_010_swarm_tasks).
Swarm view (pure):
- A task container is "replaced" when Swarm set its task's desired state off
`running`. Without the task on record, a newer or running task in the same
slot decides.
- Replaced containers are no longer replicas or crashes.
- Each service gets, from the task records:
- failures_1h and failures_24h (failed or rejected tasks);
- crash_looping (Settings → Thresholds count);
- last_error, in Swarm's own words.
MCP:
- New findings: swarm_service_crash_looping (critical) and swarm_task_failed
(warning, on a healthy service with a failure in 24h).
- Down and degraded findings name the last task error.
- list_swarm_services carries the failure fields.
- Container rows mark swarm_replaced and give a replaced task no problems.
Web:
- The containers page swarm cards and the Swarm topology page show crash
loops, recent failures and the last task error.
- The dashboard widget flags a crash loop, and its "failed (24h)" count no
longer counts a rolling update's killed task.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
test_persist_logs_respects_toggle_and_exclude read via s.execute(SELECT) between
two `async with s.begin()` blocks; that SELECT autobegins a transaction, so the
next begin() raised "A transaction is already begun". Move both reads after the
final begin block (the toggle-off persist stores nothing, so end-state still
reflects the exclude phase), and pass a SimpleNamespace stand-in host to sidestep
ORM attribute-expiry across the commit boundaries.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CAGR73DUowdVFVvYzLXC5C
The user-facing half (rule 27). Per-container log viewer + admin controls, so
container logs are something the operator can actually touch.
Viewer (plugins/docker):
- /container/<host>/<name>/logs full page + /logs/lines HTMX fragment polled
every 5s (near-live follow); stream filter (all/stdout/stderr) + case-insensitive
text search; newest-first so the latest lines survive the poll's scroll reset;
empty/loading states; a "View logs" link from the container detail page.
Jinja auto-escapes log content (no markup injection).
Settings (Thresholds & Retention tab):
- global on/off toggle, comma-separated exclude list, retention days + max MB
per container — DB-backed, no restart. The toggle + exclude are enforced
authoritatively at ingest (_persist_logs drops disabled/excluded lines), since
the push model has no channel to tell an agent to stop.
Tests: route-defined smoke; template-parse covers the new templates; integration
test for the ingest-time toggle + exclude enforcement.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CAGR73DUowdVFVvYzLXC5C
Bound docker_logs growth in the periodic cleanup task (same architecture as the
metrics/events retention). run_docker_retention gains logs_retention_days +
logs_max_bytes_per_container: it prunes lines past the age window, then keeps
only the newest ~cap bytes per (host, container) via a window-function ring
(exclusive-prefix sum, so the newest line always survives even if it alone
exceeds the cap). Containers rotate independently.
- settings DEFAULTS: docker.logs.enabled/exclude/retention_days(3)/
max_bytes_per_container(5MB) — operator preference: ~3 days / ~5 MB
- cleanup.py reads the two windows fresh each run (rule 25, no restart)
- integration rotate test: age prune + per-container byte cap + isolation
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CAGR73DUowdVFVvYzLXC5C
Ingest side of container logs. The host_agent ingest route extracts
sample["docker_logs"] as time-series batches (append every line, each carrying
its own Docker ts with recorded_at as fallback) and hands them to the docker
capability alongside the existing container/swarm/disk data — still under the
begin_nested SAVEPOINT so a logs failure can't sink host metrics.
- ingest.py: _persist_logs + pure _log_rows(batches, host_id) that shapes/filters
records (drops the _steward truncation marker, malformed + lineless records,
normalises unknown stream → stdout); persist_host_docker gains a logs= param
- routes.py: accumulate docker_log_batches, add to the guard + capability call
- unit test for _log_rows shaping/filtering; integration push→store→query test
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CAGR73DUowdVFVvYzLXC5C
Dedicated host-scoped table for pushed container log lines (one row per line),
chained after docker_008. host_id FK CASCADE; twin indexes — (host_id,
container_name, ts) for the viewer, ts alone for the age-cutoff prune — mirroring
docker_events. Integration schema-shape test asserts columns + both indexes.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CAGR73DUowdVFVvYzLXC5C
The widget-dedup test imported plugins.docker.routes, which re-imports
plugins.docker.models after the app already loaded the docker plugin —
"Table 'docker_containers' is already defined". Same plugin-loader gotcha the
host_agent query helpers avoid.
Extract dedup_by_container_id into plugins/docker/dedup.py (imports no ORM
models, so it's safe to import in either environment); routes.py imports it
from there and the test targets the model-free module.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC
The running count was inflated: swarm-aware agents on every manager list
cluster-wide tasks, so the same container (identical container_id) was reported
once per manager and the dashboard widgets summed them. The earlier swarm dedup
only covered the Swarm topology page — the widgets read raw rows.
- _dedup_by_container_id(): collapse rows sharing a non-empty container_id
(globally unique, so only true duplicates merge); first occurrence wins under
the running-first ordering. Applied in both the containers and resources
widgets before counting/limiting. Rows without a container_id (older agents)
are kept as-is.
- Replaced the static "stopped" tally — not actionable — with "failed (24h)":
distinct containers with a die/oom event in the last 24h (DISTINCT so the
managers' duplicate events don't double it), rendered red when non-zero.
- widget.html empty-state now keys off total_count.
- Regression test: same container_id from two managers counts once; id-less rows
all kept.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC
The container persister upserted current state keyed (host_id, name) but — unlike
the image/swarm/disk persisters — never deleted containers that vanished from the
host's latest listing. Every removed one-shot container (CI job runners, buildkit
builders, codex jobs) left a permanent "stopped" row, so the dashboard counts
ballooned (e.g. 856 stopped) and read as "not dedup'd". It wasn't dedup — it was
a missing reaper.
- ingest.py: after the upsert loop, delete this host's containers whose name is
notin the newest snapshot (the listing is authoritative — event derivation
already treats absence as removal). Mirrors images/swarm/disk. The existing
`if not snapshots: return` keeps swarm/disk-only samples from touching
containers; an empty container snapshot legitimately means none exist.
- widget_resources.html: this widget shows the busiest running containers by CPU.
Replaced the two unlabeled hairline bars with labeled CPU/MEM bars (label · bar ·
% value, warn/crit colors) so it's self-explanatory.
- Regression test: a container present then absent across snapshots is reaped.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC
docker_metrics.mem_usage_bytes and docker_containers.mem_usage_bytes /
mem_limit_bytes were int4 (max 2,147,483,647). A container using >2.1 GB of RAM
(e.g. 8.18 GB) overflowed the column, so asyncpg raised "value out of int32
range" and the entire docker ingest batch failed — no metrics stored for any
host with a large container.
The I/O counters and the milestone-77 rollup/disk tables already used
BigInteger; this trio (docker_001-era) was missed.
- models.py: DockerMetric.mem_usage_bytes, DockerContainer.mem_usage_bytes,
DockerContainer.mem_limit_bytes → BigInteger.
- migration docker_008_bigint_mem: ALTER COLUMN ... TYPE BIGINT (safe in-place
int4→int8 promotion).
- integration regression test: persist an ~8 GB container, assert the
current-state and time-series rows round-trip.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC
test_disk_usage_persisted opened a second session.begin() after interleaved
SELECTs, which autobegin a transaction → "A transaction is already begun".
Roll back the read transaction before the re-report write. Restores the
integration lane green for the /system/df slice (a840d6f).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC
Backend for the image/disk panel (milestone 77 #942). Agent gains
collect_disk_usage() — one /system/df call (gated on containers existing, so
Docker-less hosts pay nothing), surfacing reclaimable bytes (image size held by
unreferenced images), layers/containers/volumes/build-cache sizes, and the top
50 images by size. Emitted as sample["docker_disk"]; host_agent ingest tracks
the newest sample's copy and hands it to the docker capability as a 5th arg.
New current-state tables docker_disk_usage (one row/host) + docker_images
(per-host image rows), docker_007 migration; ingest upserts the summary and
replaces the image set per host (stale images pruned). Unit tests for the df
parsing/reclaimable math + build_sample gating; integration test for
persistence + image-set replacement.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC
Bounds Docker time-series growth (the main scaling concern). New
docker_metrics_hourly table + docker_006 migration; a plugin retention module
(docker.run_retention capability) rolls raw docker_metrics older than the raw
window into hourly averages (idempotent upsert), deletes the rolled raw rows,
then prunes stale rollups + lifecycle events. Core cleanup.py drives it each
hourly run via the capability (no plugin-model import), reading the three
retention windows fresh from settings so changes apply without restart (rule 25).
Settings → "Thresholds & Retention" gains a Docker retention card (raw /
rolled-up / events windows, working defaults 7/90/30 days). Unit tests cover the
hour-aligned cutoff/bucketing helpers; integration test exercises the real
rollup-average + prune across both windows.
Milestone 77 task #941.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC
Wires the agent's enriched + swarm payloads through the docker.persist_host_
samples capability:
* Swarm topology — persist sample["swarm"] into docker_swarm_services /
docker_swarm_nodes (upsert + prune stale, host-scoped so two managers don't
clobber). Migration docker_005 adds services.placement_json for the
task→node placement the agent now reports.
* Lifecycle events — _derive_events (pure, unit-tested) diffs the newest
snapshot against stored per-container state: start / stop / die (non-zero
exit) / oom / health_change → docker_events rows. Skipped on a host's first
snapshot so the baseline doesn't emit a start per existing container.
* Alerts — record restart_count (always) and is_healthy (1.0/0.0, only when a
HEALTHCHECK exists) alongside cpu/mem, under host-scoped resource names;
METRIC_CATALOG[docker] gains restart_count + is_healthy so they're alertable.
host_agent ingest captures the newest sample's swarm object and threads it to
the capability (now persist_host_docker(session, host, snapshots, swarm=None));
invoked when containers OR swarm are present, under the same SAVEPOINT. Unit
tests cover the event-diff matrix; integration tests cover event derivation
across two snapshots and swarm topology round-trip (incl. placement).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC
Adds the milestone-77 storage that doesn't fit on the per-container row:
* docker_events — lifecycle (start/stop/die/oom/health_change), to be
derived by diffing consecutive host snapshots; host-scoped, indexed for
timeline lookups (host_id, container_name, at) and retention pruning (at).
* docker_swarm_services / docker_swarm_nodes — manager-reported Swarm
topology (desired-vs-running replicas, node role/availability/status).
Migration docker_004 extends the docker branch (down_revision docker_003);
purely additive, no DROP+recreate. event/mode/role are plain strings (no
CHECK whitelist), matching how docker_containers models status. Integration
guard asserts the three new host-scoped tables exist.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC
First slice of milestone 77 (Docker monitoring depth). Surfaces real per-container
stats beyond basic state, all read-only on the existing push model.
- agent (→1.4.0): collect_docker now inspects each container (health, restart
count, exit code, OOM) and reads net + block I/O from the stats payload; pulls
compose project + swarm service/task/node from container labels. Per-container
inspect+stats calls run over a small bounded ThreadPool so the ~1s-per-stats
blocking doesn't stretch the sample on a busy host.
- schema (docker_003): additive columns on docker_containers — health, exit_code,
oom_killed, compose_project, service_name, task_id, node_id, and BigInteger
net/blk byte counters.
- ingest: persists the enrichment + restart_count (.get keeps older agents working).
- ui: Docker page rows now show health badge, uptime ("up 3d 4h"), restart count,
exit code (+OOM) for stopped containers, and compose/service grouping label.
- tests: agent helpers (grouping, inspect fields, net/IO sum) + collect_docker
assembly incl. inspect; integration asserts enrichment round-trips.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC
Docker collection moves off the central single-socket scrape onto the host
agent, giving Docker a real per-host dimension. The Steward host now reports
its own containers like any other host, and same-named containers on different
hosts no longer collide.
- agent: stdlib UDS Docker client (AF_UNIX HTTP/1.1, Connection: close,
chunked-aware), collect_docker() ports the cpu%/mem math; sample["docker"]
added best-effort (silent-skip on absent/unreadable socket). AGENT_VERSION
1.2.0 → 1.3.0; optional docker_socket config key.
- ingest: host_agent ingest hands per-host container snapshots to the docker
plugin via a new "docker.persist_host_samples" capability (no hard import,
no-op when docker disabled), inside a SAVEPOINT so a docker failure never
sinks the host metrics. Resource names are host-scoped ("<host>/<name>").
- schema: docker_containers re-keyed (host_id, name); docker_metrics gains
host_id; docker_002 migration DROP+recreates (dev-only, rule 122).
- ui: Docker page + widgets grouped by host with host links; new per-host
Docker panel embedded on the Hosts hub (gated on docker enabled via a new
enabled_plugins template context). Replaces the SQLite-only strftime
bucketing with DB-agnostic Python bucketing.
- provisioning: install/provision playbooks add steward-agent to the docker
group (best-effort) so the agent can read the socket.
- removed central scrape: docker scheduler.py + scraper.py deleted; plugin.yaml
socket_path/scrape_interval_seconds/include_stopped dropped (plugin 2.0.0).
- tests: agent docker collector units (math, chunked decode, silent-skip,
sample shape, config) + integration (host-scoped schema + persistence).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC
