ade078cabf18e0fd49d84d082a5a6b1c2ea9b66d
11
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  bvandeusenandClaude Opus 5.5
|
bcd0563f39 |
feat(docker): uptime from StartedAt, a restart is an event, host totals count physical devices only (#5153)
- Agent: started_at comes from inspect's State.StartedAt (nanoseconds
trimmed; Docker's zero time is "never started"), not the list's Created,
so a restarted container's uptime counts from its restart.
- A rising restart_count between two snapshots emits a "restart" event
("restart #3", or "3 restarts, now #5"), said instead of a start; a
restart then a death gives both. A count that falls (recreated under the
same name) says nothing. The timeline draws it as ↻.
- Network and disk I/O count what the kernel calls physical: interfaces
not under /sys/devices/virtual/net (bridges, docker0, veth, vxlan, tun),
block devices with a device behind them (not dm, md, zram), so a byte is
not counted twice. Where /sys calls everything virtual (an agent in LXC)
the name-filtered set is kept rather than reporting no traffic.
- Agent tests updated for #5152's (status, detail, containers) and
(is_manager, topology) shapes; the no-Docker disk test now states the
case it was passing for by accident.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
bvandeusenandClaude Opus 5.5
|
8ceb983183 |
feat(docker): Docker's own status reaches Steward — an empty answer clears a host, a lost one greys it, a demoted manager's swarm rows go (#5152)
The agent (1.9.0) said nothing when Docker could not be asked and nothing when Docker had no containers, so Steward could tell neither apart from the other, and kept a host's last containers on screen as running either way. - Agent: collect_docker returns (status, detail, containers). "ok" sends docker: [] when there are none; "unavailable" (no socket, nothing listening) and "error" (permission, timeout, bad answer) say why. collect_swarm says whether /info answered, and a node that is not a manager sends swarm_manager: false. A failed /info says nothing. - Ingest: an empty docker list is a snapshot; the newest sample's status goes to the Docker plugin with the rest. - docker_host_status (docker_011): per host, status, detail, since and last report. A host that never had Docker gets no row. - An "ok" empty snapshot reaps the host's containers; an error keeps them. swarm_manager: false drops the host's swarm services and nodes. - Docker page: a "Docker not answering" card names each such host, why, and since when; its containers read as stale. Duplicate header comment in rows.html removed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
bvandeusenandClaude Opus 5.5
|
58bfc7c361 |
feat(alerts): one evaluation path; host-agent ingest fires rules (#5060)
#5060 "One evaluation path: record_metrics batch API; host-agent ingest evaluates rules". - record_metrics(session, rows) writes metric rows and evaluates the rules that watch them, reading rules, states and maintenance windows once per batch. Each rule sees its key's samples oldest first, so an agent catching up after an outage counts consecutive checks as they happened. - Host-agent ingest now goes through it. Before, it wrote plugin_metrics directly and no host_agent rule could ever fire. The catalog gains the per-mount disk_used_pct ("<host>:<mount>"). - MetricBatch gathers a transaction's samples; the docker ingest, traefik, unifi, snmp, monitor and self-monitor writers record one batch each instead of a rule query per sample. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
bvandeusenandClaude Opus 5.5
|
bac2859455 |
feat(docker): Swarm task awareness — replaced tasks, service crash loops, last task error
Milestone 465 "Swarm task awareness". Fixes three defects from the live MCP review #4973: - #5008 "Swarm task history read as live state" - #5009 "Crash-loop detection is blind to Swarm services" - #5011 "Swarm task errors are not captured" Agent and ingest (agent 1.8.0): - A manager's agent now reports each service's newest tasks: slot, node, desired and current state, Swarm's Err/Message, exit code and time. - They are stored in docker_swarm_services.tasks_json (migration docker_010_swarm_tasks). Swarm view (pure): - A task container is "replaced" when Swarm set its task's desired state off `running`. Without the task on record, a newer or running task in the same slot decides. - Replaced containers are no longer replicas or crashes. - Each service gets, from the task records: - failures_1h and failures_24h (failed or rejected tasks); - crash_looping (Settings → Thresholds count); - last_error, in Swarm's own words. MCP: - New findings: swarm_service_crash_looping (critical) and swarm_task_failed (warning, on a healthy service with a failure in 24h). - Down and degraded findings name the last task error. - list_swarm_services carries the failure fields. - Container rows mark swarm_replaced and give a replaced task no problems. Web: - The containers page swarm cards and the Swarm topology page show crash loops, recent failures and the last task error. - The dashboard widget flags a crash loop, and its "failed (24h)" count no longer counts a rolling update's killed task. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
bvandeusenandClaude Opus 4.8
|
8f1c8c5cf7 |
feat(docker): container log viewer + Settings controls [M79 step 5]
The user-facing half (rule 27). Per-container log viewer + admin controls, so container logs are something the operator can actually touch. Viewer (plugins/docker): - /container/<host>/<name>/logs full page + /logs/lines HTMX fragment polled every 5s (near-live follow); stream filter (all/stdout/stderr) + case-insensitive text search; newest-first so the latest lines survive the poll's scroll reset; empty/loading states; a "View logs" link from the container detail page. Jinja auto-escapes log content (no markup injection). Settings (Thresholds & Retention tab): - global on/off toggle, comma-separated exclude list, retention days + max MB per container — DB-backed, no restart. The toggle + exclude are enforced authoritatively at ingest (_persist_logs drops disabled/excluded lines), since the push model has no channel to tell an agent to stop. Tests: route-defined smoke; template-parse covers the new templates; integration test for the ingest-time toggle + exclude enforcement. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CAGR73DUowdVFVvYzLXC5C |
||
|
bvandeusenandClaude Opus 4.8
|
a8de3570fe |
feat(docker): persist pushed container logs into docker_logs [M79 step 3]
Ingest side of container logs. The host_agent ingest route extracts sample["docker_logs"] as time-series batches (append every line, each carrying its own Docker ts with recorded_at as fallback) and hands them to the docker capability alongside the existing container/swarm/disk data — still under the begin_nested SAVEPOINT so a logs failure can't sink host metrics. - ingest.py: _persist_logs + pure _log_rows(batches, host_id) that shapes/filters records (drops the _steward truncation marker, malformed + lineless records, normalises unknown stream → stdout); persist_host_docker gains a logs= param - routes.py: accumulate docker_log_batches, add to the guard + capability call - unit test for _log_rows shaping/filtering; integration push→store→query test Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CAGR73DUowdVFVvYzLXC5C |
||
|
bvandeusenandClaude Opus 4.8
|
0c055ed6fa |
fix(docker): reap vanished containers; clarify the resources widget
The container persister upserted current state keyed (host_id, name) but — unlike the image/swarm/disk persisters — never deleted containers that vanished from the host's latest listing. Every removed one-shot container (CI job runners, buildkit builders, codex jobs) left a permanent "stopped" row, so the dashboard counts ballooned (e.g. 856 stopped) and read as "not dedup'd". It wasn't dedup — it was a missing reaper. - ingest.py: after the upsert loop, delete this host's containers whose name is notin the newest snapshot (the listing is authoritative — event derivation already treats absence as removal). Mirrors images/swarm/disk. The existing `if not snapshots: return` keeps swarm/disk-only samples from touching containers; an empty container snapshot legitimately means none exist. - widget_resources.html: this widget shows the busiest running containers by CPU. Replaced the two unlabeled hairline bars with labeled CPU/MEM bars (label · bar · % value, warn/crit colors) so it's self-explanatory. - Regression test: a container present then absent across snapshots is reaped. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC |
||
|
bvandeusenandClaude Opus 4.8
|
a840d6f823 |
feat(docker): collect + persist /system/df image/disk usage (agent 1.6.0)
Backend for the image/disk panel (milestone 77 #942). Agent gains collect_disk_usage() — one /system/df call (gated on containers existing, so Docker-less hosts pay nothing), surfacing reclaimable bytes (image size held by unreferenced images), layers/containers/volumes/build-cache sizes, and the top 50 images by size. Emitted as sample["docker_disk"]; host_agent ingest tracks the newest sample's copy and hands it to the docker capability as a 5th arg. New current-state tables docker_disk_usage (one row/host) + docker_images (per-host image rows), docker_007 migration; ingest upserts the summary and replaces the image set per host (stale images pruned). Unit tests for the df parsing/reclaimable math + build_sample gating; integration test for persistence + image-set replacement. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC |
||
|
bvandeusenandClaude Opus 4.8
|
578cc33cc0 |
feat(docker): ingest swarm topology + lifecycle events + health/restart alerts
Wires the agent's enriched + swarm payloads through the docker.persist_host_
samples capability:
* Swarm topology — persist sample["swarm"] into docker_swarm_services /
docker_swarm_nodes (upsert + prune stale, host-scoped so two managers don't
clobber). Migration docker_005 adds services.placement_json for the
task→node placement the agent now reports.
* Lifecycle events — _derive_events (pure, unit-tested) diffs the newest
snapshot against stored per-container state: start / stop / die (non-zero
exit) / oom / health_change → docker_events rows. Skipped on a host's first
snapshot so the baseline doesn't emit a start per existing container.
* Alerts — record restart_count (always) and is_healthy (1.0/0.0, only when a
HEALTHCHECK exists) alongside cpu/mem, under host-scoped resource names;
METRIC_CATALOG[docker] gains restart_count + is_healthy so they're alertable.
host_agent ingest captures the newest sample's swarm object and threads it to
the capability (now persist_host_docker(session, host, snapshots, swarm=None));
invoked when containers OR swarm are present, under the same SAVEPOINT. Unit
tests cover the event-diff matrix; integration tests cover event derivation
across two snapshots and swarm topology round-trip (incl. placement).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC
|
||
|
bvandeusenandClaude Opus 4.8
|
82c3d2cf36 |
feat(docker): per-container enrichment — health, restarts, exit code, I/O, grouping
First slice of milestone 77 (Docker monitoring depth). Surfaces real per-container
stats beyond basic state, all read-only on the existing push model.
- agent (→1.4.0): collect_docker now inspects each container (health, restart
count, exit code, OOM) and reads net + block I/O from the stats payload; pulls
compose project + swarm service/task/node from container labels. Per-container
inspect+stats calls run over a small bounded ThreadPool so the ~1s-per-stats
blocking doesn't stretch the sample on a busy host.
- schema (docker_003): additive columns on docker_containers — health, exit_code,
oom_killed, compose_project, service_name, task_id, node_id, and BigInteger
net/blk byte counters.
- ingest: persists the enrichment + restart_count (.get keeps older agents working).
- ui: Docker page rows now show health badge, uptime ("up 3d 4h"), restart count,
exit code (+OOM) for stopped containers, and compose/service grouping label.
- tests: agent helpers (grouping, inspect fields, net/IO sum) + collect_docker
assembly incl. inspect; integration asserts enrichment round-trips.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC
|
||
|
bvandeusenandClaude Opus 4.8
|
7b80552a7d |
feat(docker): per-host collection via the host agent; drop central scrape
Docker collection moves off the central single-socket scrape onto the host
agent, giving Docker a real per-host dimension. The Steward host now reports
its own containers like any other host, and same-named containers on different
hosts no longer collide.
- agent: stdlib UDS Docker client (AF_UNIX HTTP/1.1, Connection: close,
chunked-aware), collect_docker() ports the cpu%/mem math; sample["docker"]
added best-effort (silent-skip on absent/unreadable socket). AGENT_VERSION
1.2.0 → 1.3.0; optional docker_socket config key.
- ingest: host_agent ingest hands per-host container snapshots to the docker
plugin via a new "docker.persist_host_samples" capability (no hard import,
no-op when docker disabled), inside a SAVEPOINT so a docker failure never
sinks the host metrics. Resource names are host-scoped ("<host>/<name>").
- schema: docker_containers re-keyed (host_id, name); docker_metrics gains
host_id; docker_002 migration DROP+recreates (dev-only, rule 122).
- ui: Docker page + widgets grouped by host with host links; new per-host
Docker panel embedded on the Hosts hub (gated on docker enabled via a new
enabled_plugins template context). Replaces the SQLite-only strftime
bucketing with DB-agnostic Python bucketing.
- provisioning: install/provision playbooks add steward-agent to the docker
group (best-effort) so the agent can read the socket.
- removed central scrape: docker scheduler.py + scraper.py deleted; plugin.yaml
socket_path/scrape_interval_seconds/include_stopped dropped (plugin 2.0.0).
- tests: agent docker collector units (math, chunked decode, silent-skip,
sample shape, config) + integration (host-scoped schema + persistence).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC
|