The agent (1.9.0) said nothing when Docker could not be asked and nothing
when Docker had no containers, so Steward could tell neither apart from the
other, and kept a host's last containers on screen as running either way.
- Agent: collect_docker returns (status, detail, containers). "ok" sends
docker: [] when there are none; "unavailable" (no socket, nothing
listening) and "error" (permission, timeout, bad answer) say why.
collect_swarm says whether /info answered, and a node that is not a
manager sends swarm_manager: false. A failed /info says nothing.
- Ingest: an empty docker list is a snapshot; the newest sample's status
goes to the Docker plugin with the rest.
- docker_host_status (docker_011): per host, status, detail, since and
last report. A host that never had Docker gets no row.
- An "ok" empty snapshot reaps the host's containers; an error keeps them.
swarm_manager: false drops the host's swarm services and nodes.
- Docker page: a "Docker not answering" card names each such host, why,
and since when; its containers read as stale. Duplicate header comment
in rows.html removed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The Docker page opens with a "Needs attention" card: every container
with something wrong, worst problem first — restarting, crash-looping,
unhealthy, OOM-killed, exited non-zero — each with what it does ("is
restarting and keeps dying · exit 1 · since 8m ago"), linked to its page.
It is ranked by the same code as the MCP tools: container_problems,
problem_since and the per-container ranking move from mcp_tools into
service.py (ranked_containers, needs_attention), and mcp_tools' _rows and
findings now read them from there.
Counts follow as stat tiles: needs attention, unhealthy, restarting (or
crash-looping), running, stopped, stale, services, hosts.
A container whose last report is older than the host agent's staleness
window (stale_after_seconds, 180 s by default) carries a Stale badge with
when it last reported, instead of a live-looking green dot.
The per-container CPU and memory history columns are gone from the
overview, and with them one history query per container per refresh; the
container's own page keeps its charts. With nothing left to range over,
the time-range control goes too, and the page now refreshes on the poll
interval. CPU and memory are graded by Settings → Thresholds rather than
50/80 and 70/90 written into the template.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5093 "Shared format filters and status_class driven by Settings → Thresholds".
- steward/core/formatting.py: human_bytes (binary units), human_rate,
human_duration (two largest units), ago (a <time> titled with the absolute
UTC time) and status_class(value, kind, thresholds) -> ok/warn/crit/unknown,
which raises on a kind the thresholds don't name.
- app.py registers them as the bytes, rate, duration and ago filters and the
status_class global; threshold_style is now graded by status_class (and caps
its weight at 500).
- Replaces the copies: docker's _human_bytes/_human_uptime, core alerts'
_duration, the host-agent fmt_bps/fmt_bytes macros, the Traefik and UniFi
byte-rate arithmetic, the swarm-failures ago macro and two uptime spans.
- Tests: a table per helper, and every template compiled against the app's
filters, so an unknown filter fails CI rather than a page.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Milestone 465 "Swarm task awareness". Fixes three defects from the live MCP
review #4973:
- #5008 "Swarm task history read as live state"
- #5009 "Crash-loop detection is blind to Swarm services"
- #5011 "Swarm task errors are not captured"
Agent and ingest (agent 1.8.0):
- A manager's agent now reports each service's newest tasks: slot, node,
desired and current state, Swarm's Err/Message, exit code and time.
- They are stored in docker_swarm_services.tasks_json (migration
docker_010_swarm_tasks).
Swarm view (pure):
- A task container is "replaced" when Swarm set its task's desired state off
`running`. Without the task on record, a newer or running task in the same
slot decides.
- Replaced containers are no longer replicas or crashes.
- Each service gets, from the task records:
- failures_1h and failures_24h (failed or rejected tasks);
- crash_looping (Settings → Thresholds count);
- last_error, in Swarm's own words.
MCP:
- New findings: swarm_service_crash_looping (critical) and swarm_task_failed
(warning, on a healthy service with a failure in 24h).
- Down and degraded findings name the last task error.
- list_swarm_services carries the failure fields.
- Container rows mark swarm_replaced and give a replaced task no problems.
Web:
- The containers page swarm cards and the Swarm topology page show crash
loops, recent failures and the last task error.
- The dashboard widget flags a crash loop, and its "failed (24h)" count no
longer counts a rolling update's killed task.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
MCP-6 #4970 "Automation tools: playbooks, runs, schedules, alert actions".
Viewer: list_playbooks, get_playbook (secret defaults masked), list_inventory
(variable names only), list_ansible_runs, get_ansible_run (log tail + grep),
list_schedules, wait_for_run (polls, 120s cap).
Operator: run_playbook (check mode unless check=false; a steward:confirm
playbook applies only with confirmed=true; secret variables reach the run
but are never stored, audited or returned), cancel_ansible_run, and
create/update/delete_schedule (secrets refused, check by default).
Admin: set_alert_action, and the docker plugin's prune_docker, which
previews from the disk report unless check=false (Ansible's check mode
skips the prune commands, so a check run would show nothing).
Every write is audited as "<user> via MCP" with the token id; the audit
tool's user filter includes those rows. prune_extra_vars moves to the
docker service so the page and the tool share it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
MCP-3 of milestone 461 "MCP server: Steward as an LLM-inspectable,
LLM-operable hub". The /mcp tools run outside Quart's request pipeline, so
the queries they share with the pages now live in plain functions that take
an open session:
- steward/hosts/service.py: uptime, status, monitors, agent metrics,
agent overview (stale threshold passed in, not read from current_app),
CPU sparks.
- plugins/docker/service.py: bucket, container_history, host_map,
list_containers, container_events, query_logs.
- steward/alerts/service.py: active_alerts (also used by core/reports),
list_rules, list_maintenance_windows.
- steward/monitors/service.py: monitor_rows (was routes._rows_data).
Routes are repointed and render the same data. One behavior fix rides
along: the container-log search now escapes % and _ so they match
literally instead of acting as LIKE wildcards.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The user-facing half (rule 27). Per-container log viewer + admin controls, so
container logs are something the operator can actually touch.
Viewer (plugins/docker):
- /container/<host>/<name>/logs full page + /logs/lines HTMX fragment polled
every 5s (near-live follow); stream filter (all/stdout/stderr) + case-insensitive
text search; newest-first so the latest lines survive the poll's scroll reset;
empty/loading states; a "View logs" link from the container detail page.
Jinja auto-escapes log content (no markup injection).
Settings (Thresholds & Retention tab):
- global on/off toggle, comma-separated exclude list, retention days + max MB
per container — DB-backed, no restart. The toggle + exclude are enforced
authoritatively at ingest (_persist_logs drops disabled/excluded lines), since
the push model has no channel to tell an agent to stop.
Tests: route-defined smoke; template-parse covers the new templates; integration
test for the ingest-time toggle + exclude enforcement.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CAGR73DUowdVFVvYzLXC5C
M78 step 2. The /disk page surfaced reclaimable space + stopped counts but
said "prune deferred to a later milestone" — this wires the cleanup.
Adds POST /plugins/docker/disk/<host_id>/prune (admin only) that resolves
the host's linked AnsibleTarget → steward:target:<id> scope and fires the
bundled maintenance/docker_prune.yml through the ansible.run_playbook
capability (audited runner; the collection agent stays read-only). Mirrors
the host_agent update route. `target` ∈ {containers, images, system}:
- containers → docker container prune -f
- images → docker image prune -af (prune_all_images)
- system → docker system prune -f (conservative)
The disk() route now resolves each host group's linked target + capability
availability; disk.html renders three confirm-gated buttons per host, shown
only to admins when Ansible is available and the host has a linked target,
otherwise a clear "link a target" / "runner unavailable" hint (rules 24/26/27).
Tests: the prune_target→extra_vars mapping (the -af-only-for-images rule) is
extracted to a pure helper and unit-tested; disk_prune added to the module
smoke; disk.html covered by the existing template-parse test.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CAGR73DUowdVFVvYzLXC5C
The container list and dashboard widget listed each swarm task as its own cryptic
row (svc.1.<taskid>) and could only show tasks on nodes that run a Steward agent.
Make both views service-centric and cluster-complete.
- swarm_view.build_swarm_services(): model-free builder that merges real container
rows (collected local-per-node, so host_id = the node a task runs on) with the
managers' placement (DockerSwarmService.placement_json). Where placement counts
exceed the local rows on a node — i.e. nodes with no agent — it synthesizes
"ghost" replicas so the replica list matches the cluster. Non-swarm containers
pass through grouped by host.
- /rows + rows.html: a Swarm-services panel collapses each service to one block,
every replica a host chip (status + cpu); ghosts render dashed "N · no agent".
Non-swarm/compose containers keep the per-host table below. New Services stat.
- /widget + widget.html: same service collapse with host chips; standalone
containers stay grouped by host.
- Unit tests for the builder (real+ghost merge, full coverage, partial agent-less,
synthesized service, down state).
No migration — node_id, placement_json and swarm-node hostnames are already stored.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC
The widget-dedup test imported plugins.docker.routes, which re-imports
plugins.docker.models after the app already loaded the docker plugin —
"Table 'docker_containers' is already defined". Same plugin-loader gotcha the
host_agent query helpers avoid.
Extract dedup_by_container_id into plugins/docker/dedup.py (imports no ORM
models, so it's safe to import in either environment); routes.py imports it
from there and the test targets the model-free module.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC
The running count was inflated: swarm-aware agents on every manager list
cluster-wide tasks, so the same container (identical container_id) was reported
once per manager and the dashboard widgets summed them. The earlier swarm dedup
only covered the Swarm topology page — the widgets read raw rows.
- _dedup_by_container_id(): collapse rows sharing a non-empty container_id
(globally unique, so only true duplicates merge); first occurrence wins under
the running-first ordering. Applied in both the containers and resources
widgets before counting/limiting. Rows without a container_id (older agents)
are kept as-is.
- Replaced the static "stopped" tally — not actionable — with "failed (24h)":
distinct containers with a die/oom event in the last 24h (DISTINCT so the
managers' duplicate events don't double it), rendered red when non-zero.
- widget.html empty-state now keys off total_count.
- Regression test: same container_id from two managers counts once; id-less rows
all kept.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC
Swarm services/nodes are cluster-global (every manager's API returns the same
list), but each manager reports them independently (rows keyed by host_id) and
the Swarm page grouped by reporting manager — so two managers in one cluster
listed every service and node twice.
Group the reporting managers into swarms in the swarm() view (managers sharing
any node_id are the same cluster, via union-find over node-set intersection),
then dedup within each: one service per name and one node per node_id, keeping
the freshest. Render one section per swarm ("reported by N managers · names").
node_id is globally unique so node dedup is always safe; grouping by node
overlap also keeps it correct if two separate swarms are ever monitored.
View-layer only — no schema/agent change. The main per-host container page is
unaffected (those are genuinely distinct per-node tasks).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC
Container list now sub-groups each host's containers by compose project (or
swarm service) with a small subheading, preserving the running-first order;
hosts with no such labels render flat as before. The dashboard status widget
links each container to its detail page and surfaces enriched state inline —
health dot (healthy/unhealthy), restart count, and last non-zero exit code for
stopped containers. Completes the #942 UI surface.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC
Two new read-only sub-pages, linked from the Docker index header only when the
data exists (non-swarm / Docker-less installs aren't offered empty pages):
- /plugins/docker/swarm — services with replica health (running/desired,
colour-coded green/amber/red), Swarm nodes (role/availability/status, leader
badge), and task→node placement with node ids resolved to hostnames. Grouped
by reporting manager host. Empty state explains manager-only collection.
- /plugins/docker/disk — per-host reclaimable space, image/layer/container/
volume/build-cache sizes, stopped-container count, and a per-image table
(size, shared, ref count, reclaimable badge). Notes prune actions are deferred.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC
New /plugins/docker/container/<host_id>/<name> detail page (v1 quality):
status/health badge, uptime, CPU/mem, restart count, last exit code (+OOM),
net + block I/O (humanised), image/compose/swarm-service/node/ports, a
range-toggled CPU/mem history graph (HTMX fragment reusing the time-range
selector), and a lifecycle timeline rendered from docker_events (glyph+colour
per event kind). Not-found and empty-history/empty-timeline states included.
Container names in the main list and the per-host hub panel now link to it.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC
First slice of milestone 77 (Docker monitoring depth). Surfaces real per-container
stats beyond basic state, all read-only on the existing push model.
- agent (→1.4.0): collect_docker now inspects each container (health, restart
count, exit code, OOM) and reads net + block I/O from the stats payload; pulls
compose project + swarm service/task/node from container labels. Per-container
inspect+stats calls run over a small bounded ThreadPool so the ~1s-per-stats
blocking doesn't stretch the sample on a busy host.
- schema (docker_003): additive columns on docker_containers — health, exit_code,
oom_killed, compose_project, service_name, task_id, node_id, and BigInteger
net/blk byte counters.
- ingest: persists the enrichment + restart_count (.get keeps older agents working).
- ui: Docker page rows now show health badge, uptime ("up 3d 4h"), restart count,
exit code (+OOM) for stopped containers, and compose/service grouping label.
- tests: agent helpers (grouping, inspect fields, net/IO sum) + collect_docker
assembly incl. inspect; integration asserts enrichment round-trips.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC
Docker collection moves off the central single-socket scrape onto the host
agent, giving Docker a real per-host dimension. The Steward host now reports
its own containers like any other host, and same-named containers on different
hosts no longer collide.
- agent: stdlib UDS Docker client (AF_UNIX HTTP/1.1, Connection: close,
chunked-aware), collect_docker() ports the cpu%/mem math; sample["docker"]
added best-effort (silent-skip on absent/unreadable socket). AGENT_VERSION
1.2.0 → 1.3.0; optional docker_socket config key.
- ingest: host_agent ingest hands per-host container snapshots to the docker
plugin via a new "docker.persist_host_samples" capability (no hard import,
no-op when docker disabled), inside a SAVEPOINT so a docker failure never
sinks the host metrics. Resource names are host-scoped ("<host>/<name>").
- schema: docker_containers re-keyed (host_id, name); docker_metrics gains
host_id; docker_002 migration DROP+recreates (dev-only, rule 122).
- ui: Docker page + widgets grouped by host with host links; new per-host
Docker panel embedded on the Hosts hub (gated on docker enabled via a new
enabled_plugins template context). Replaces the SQLite-only strftime
bucketing with DB-agnostic Python bucketing.
- provisioning: install/provision playbooks add steward-agent to the docker
group (best-effort) so the agent can read the socket.
- removed central scrape: docker scheduler.py + scraper.py deleted; plugin.yaml
socket_path/scrape_interval_seconds/include_stopped dropped (plugin 2.0.0).
- tests: agent docker collector units (math, chunked decode, silent-skip,
sample shape, config) + integration (host-scoped schema + persistence).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC
The fabledscryer->steward rename had only ever reached host_agent. The other
five bundled plugins (http, snmp, traefik, unifi, docker) still imported
`from fabledscryer.*` (package no longer exists) and read FABLEDSCRYER_* env
vars — so every one of them was broken at import since the original rebrand.
CI stayed green only because none are enabled by default and migrations don't
import plugin modules. Now that they version in-tree, complete the rename:
- fabledscryer.* -> steward.* imports across all five plugins
- FABLEDSCRYER_* -> STEWARD_* in plugin migration env.py files
- author/repository/homepage + user-facing 'Fabled Scryer' strings -> Steward
- snmp/scheduler.py: also drop dead `now`/datetime; record_metric from steward
Adds tests/test_no_legacy_names.py — fails if 'scryer'/'roundtable' ever
reappear in shipped code (the drift bit twice; this stops a third time).
Also clears pre-existing ruff lint debt (unused imports, semicolon statements,
mid-file import) surfaced by the new lint lane.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
First-party plugins (host_agent, http, snmp, traefik, unifi, docker) are now
tracked under plugins/ and baked into the image, so they version atomically
with core — ending the cross-repo import drift the roundtable->steward rename
exposed. History for these files is preserved in the archived Roundtable-plugins
repo.
Plugin discovery becomes multi-root: PLUGIN_DIR (single) -> PLUGIN_DIRS
(bundled first, then external) + PLUGIN_INSTALL_DIR. Bundled ships in the image;
third-party plugins still mount at runtime into the external root
(STEWARD_PLUGIN_DIR, default /data/plugins) and downloads/installs land there.
Bundled shadows external on a name collision.
- config.py: load_bootstrap returns plugin_dirs + plugin_install_dir
- app.py: iterate PLUGIN_DIRS at the migration + load sites
- migration_runner.py: discover_all_in() unions every plugin root
- plugin_manager.py: resolve_plugin_path() (pure, first-root-wins); load /
install / hot-reload span all roots; installs target the external root
- settings/routes.py: _discover_plugins scans all roots, dedup bundled-first
- Dockerfile: COPY plugins/ ; docker-compose: drop host bind, document external
- tests/test_plugin_dirs.py: resolution, multi-root discovery, bootstrap split
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
