Host CPU, memory and load per core, container CPU and memory, and every
monitor's response time learn a normal per hour (milestone 513, System 182).
A reading past its fixed cutoff but within its normal reads healthy and says
what is usual at this hour; one well out of it keeps its grade and is a
metric anomaly. Disk, a monitor down and an agent offline keep their cutoffs.
- Each normal's floor, the smallest departure that counts, is a setting:
Settings -> Thresholds -> Normals (norms.floor.*). Defaults: host CPU 10
points, host memory 5, load per core 10, container CPU 10 points of one
core, container memory 5, monitor response 100 ms; Traefik's 2-point 4xx
and 50 ms p95 floors move there from constants. learn_normal takes a
floor function (floor_setting) read at each pass.
- learn_normal(parts=False): a host learns, its cores don't.
- Ingest records load_pct (load_1m / cores x 100), the figure pages grade.
- Shared grading: anomalies.against / graded, worst_reading(normals=),
hosts' host_grades, docker usage(thresholds=, norms=), monitors'
latency_grade. Applied to host vitals, the Hosts widget, MCP overview,
Docker usage, and the monitor list, widget and page.
- The monitor and container pages list their anomalies.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5752 "Container CPU and memory history lives in the one metric store;
docker_metrics and docker_metrics_hourly are retired". Container cpu_pct and
mem_pct already went to plugin_metrics for alerting; now mem_usage_bytes does
too, and every snapshot in an ingest (backlog included) is recorded at its
own time. The container history chart, Busiest Containers, a service's
history and the MCP container tools read plugin_metrics under resource
"<host>/<container>", rolled up and pruned by the shared metric retention.
docker_012_drop_metrics drops both tables. Their history is not copied
(lesson #5221 "A data migration's cost is set by the live table"); the
store already holds 7 days of container CPU and memory %. Docker retention
now prunes only lifecycle events, and Settings -> Thresholds has one
"Metric retention" card for every source.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5468 "Log search by service name: searching "steward_app" refuses because every
deploy makes a new container name".
search_container_logs(container="steward_app") was refused, because no container
carries the service's own name and five task containers partly matched. Now a
service's name reaches every task container Steward holds anything for, live and
removed (docker_svc.service_containers, as the service page's logs already do),
so one search spans deploys. Each match carries the slot and task_id that wrote
it, and the reply names the service and how many tasks were searched. A container
named exactly what was asked still takes precedence. A miss that partly matches
task containers now names their services.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Service page:
- The header carries the verdict, mode, stack, image and when it was reported.
'In trouble since' shows only while the service is in trouble; once recovered,
'Last failed X ago · its logs'.
- The failure card shows only while the service is down, degraded or crash-looping.
- The tile strip and the kv card are gone.
- One Tasks table: running tasks first with CPU, memory and since when, then
Swarm's record. The node is named as its Steward host throughout, and
'N failed in 24h' sits in the caption.
Container page:
- Same shape: header verdict, a 'Last exit' card only while it is stopped
after a failing exit, set-up and I/O in one kv card, and lifecycle as a table.
- A swarm task's breadcrumb goes through its service.
Both pages:
- Resource history is the one fragment (_usage_history.html): now and peak in
cores and bytes, drawn with a floor of 1 core / 1 GiB (sparkline_svg floor=),
so idle noise draws flat.
- The container page's old % sparklines, its private _sparkline and the unused
bucket()/container_history() helpers are removed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- #5304 "Docker disk totals recorded over time per host, with a growth chart and alertable metrics"
- #5305 "Disk page: images grouped by repository, versions folded under it"
- #5306 "Docker overview: one disk line per host linking to the Disk page"
Each `docker system df` report is now also recorded as metrics, with the host
as the resource: disk_total_bytes, its parts, and disk_reclaimable_bytes. An
alert rule can watch them. The Disk page starts with the hosts side by side,
showing how much each grew over the selected range with a line of it, then
gives each host its own section. An image's versions fold under its
repository, which is charged only for the layers its versions don't share.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- #5298 "Docker overview: verdict line; only the problem tiles that aren't zero"
- #5299 "Docker overview: swarm services as a filterable, sortable table with a Stack column"
- #5300 "Docker overview: failures listed once, each with a link to the failed task's logs"
- #5301 "Docker overview: standalone containers in one table with a Host column"
- #5302 "Service page: replicas, image, task history, resource history and logs across all its tasks"
overview.py holds the rows, filters and sort as plain functions. The filter
bar sits outside the polled fragment and the URL keeps its state. A service's
page reads every task container Steward still holds anything for, the live
ones and removed ones known by their events, so a failed task's logs are
reachable after Docker has removed it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Docker panel (#5276): problems first, then the 8 busiest running
containers by CPU; the other running and the stopped fold behind one
"+N more running · M stopped" line. Containers that are not swarm tasks
read the way the Docker page reads them: CI jobs as "CI: workflow · job",
buildx builders as "BuildKit builder · name".
Monitors card (#5277): the edit/delete icon buttons and the inline add
form are gone. Each row links to its monitor's page; "Add monitor" opens
the add form filed under this host, starting from its address, with the
host in its breadcrumbs and Cancel back to it.
Agent card (#5278): Full metrics stays in view; Update, Rotate token,
Unregister agent (was "Remove") and Re-provision sit under "Manage agent".
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The fill bars were a share of the host's capacity, a sliver on hosts with
80 cores. Each figure now sits beside 24 bars of the last 6 hours (1h/24h
per widget), each row scaled to its own peak but never below 1 core or
1 GiB (configurable per widget), so a 0.05-core container reads flat and
a bar's height means real load. One grouped date_bin query feeds them.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The widget names the busiest container in a verdict ("Busiest: tdarr on
VDNT-Docker03, 1.6 of 8 cores."), then a fixed-layout table (container,
host, CPU in cores, memory in bytes) with each figure's share drawn as
shading behind it: of the host's cores, and of the memory limit or the
host's memory.
The numbers were already rendered, but each row grew as wide as its longest
name. A grid item won't shrink below its content, so Gitea CI container
names pushed the figures out of view and the cell scrolled sideways. Names
now come from display_name(): a swarm task by its service (no slot number),
an Actions job container as "CI: <workflow> · <job>". They cut off with …
and show whole on hover.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5209 "Docker views: readable names, CPU/memory in cores and bytes, failures
listed with since-time" (milestone 477 "UI problems first").
- A swarm task reads as its service and slot ("codex_db · 1", or the node
for a global service), with the raw name in the title: Docker page, host
panel, busiest widget, container detail.
- CPU reads in cores (Docker's 100% = one core), toned by its share of the
host's CPUs from the agent's per-core series. Memory reads in bytes, against
a limit only when one is really set (Docker reports host RAM otherwise).
- Exit 137/143 without an OOM kill is a stop, not a failure: no
exited_nonzero, no crash-loop death, no "failed (24h)", dim exit code.
- "Failed (24h)" is a list (Docker page #failures, linked from the widget
and the tile): plain containers by their deaths, swarm services by Swarm's
failed tasks; a second manager's copy of a task death counts once.
- Failing swarm services lead Needs attention; services sort problems first
with since-times on the page, the widget and the Swarm page; nodes that
are down or drained sort first.
- Busiest widget orders by a 5-minute CPU average so the list holds still.
- Host panel: a table, problems first with what is wrong and since when;
replaced swarm tasks hidden and counted.
- Container detail: node hostname, I/O labelled "since start", Restarts
hidden for swarm tasks with a note on where task failures are.
- Widget names: "Docker — Services & failures", "Docker — Busiest containers".
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The Docker page opens with a "Needs attention" card: every container
with something wrong, worst problem first — restarting, crash-looping,
unhealthy, OOM-killed, exited non-zero — each with what it does ("is
restarting and keeps dying · exit 1 · since 8m ago"), linked to its page.
It is ranked by the same code as the MCP tools: container_problems,
problem_since and the per-container ranking move from mcp_tools into
service.py (ranked_containers, needs_attention), and mcp_tools' _rows and
findings now read them from there.
Counts follow as stat tiles: needs attention, unhealthy, restarting (or
crash-looping), running, stopped, stale, services, hosts.
A container whose last report is older than the host agent's staleness
window (stale_after_seconds, 180 s by default) carries a Stale badge with
when it last reported, instead of a live-looking green dot.
The per-container CPU and memory history columns are gone from the
overview, and with them one history query per container per refresh; the
container's own page keeps its charts. With nothing left to range over,
the time-range control goes too, and the page now refreshes on the poll
interval. CPU and memory are graded by Settings → Thresholds rather than
50/80 and 70/90 written into the template.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5010 "Logs of a removed container cannot be reached by name"
- get_container and search_container_logs fall back to the names Steward
still holds logs or events for when no live container matches: the exact
name from logs and events, or a part of it that matches one removed
container's events.
- get_container answers with removed: true, last_seen_at, its history and
events; search_container_logs marks container_removed and notes when it
was last seen, and says when that is older than the range.
- A miss now says the container may have been removed, that Swarm replaces a
dead task under a new name, and where to look instead.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5013 "Overview findings for containers and Swarm services carry since=null"
- Container problems are dated from their own evidence: a crash or restart
loop from its first death in the hour, an OOM kill or error exit from the
death that ended it, a failing healthcheck from its last turn to unhealthy.
list_containers and get_container carry problem_since / problem_age_s.
- Swarm services carry trouble_since: a crash loop's first failure in the
hour, a short service's oldest troubled task in each failing slot, a
recovered service's first failure in 24h. list_swarm_services shows it.
- get_overview findings from the docker plugin now have since and age_s,
so they rank by how long they have stood.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Milestone 465 "Swarm task awareness". Fixes three defects from the live MCP
review #4973:
- #5008 "Swarm task history read as live state"
- #5009 "Crash-loop detection is blind to Swarm services"
- #5011 "Swarm task errors are not captured"
Agent and ingest (agent 1.8.0):
- A manager's agent now reports each service's newest tasks: slot, node,
desired and current state, Swarm's Err/Message, exit code and time.
- They are stored in docker_swarm_services.tasks_json (migration
docker_010_swarm_tasks).
Swarm view (pure):
- A task container is "replaced" when Swarm set its task's desired state off
`running`. Without the task on record, a newer or running task in the same
slot decides.
- Replaced containers are no longer replicas or crashes.
- Each service gets, from the task records:
- failures_1h and failures_24h (failed or rejected tasks);
- crash_looping (Settings → Thresholds count);
- last_error, in Swarm's own words.
MCP:
- New findings: swarm_service_crash_looping (critical) and swarm_task_failed
(warning, on a healthy service with a failure in 24h).
- Down and degraded findings name the last task error.
- list_swarm_services carries the failure fields.
- Container rows mark swarm_replaced and give a replaced task no problems.
Web:
- The containers page swarm cards and the Swarm topology page show crash
loops, recent failures and the last task error.
- The dashboard widget flags a crash loop, and its "failed (24h)" count no
longer counts a rolling update's killed task.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
MCP-6 #4970 "Automation tools: playbooks, runs, schedules, alert actions".
Viewer: list_playbooks, get_playbook (secret defaults masked), list_inventory
(variable names only), list_ansible_runs, get_ansible_run (log tail + grep),
list_schedules, wait_for_run (polls, 120s cap).
Operator: run_playbook (check mode unless check=false; a steward:confirm
playbook applies only with confirmed=true; secret variables reach the run
but are never stored, audited or returned), cancel_ansible_run, and
create/update/delete_schedule (secrets refused, check by default).
Admin: set_alert_action, and the docker plugin's prune_docker, which
previews from the disk report unless check=false (Ansible's check mode
skips the prune commands, so a check run would show nothing).
Every write is audited as "<user> via MCP" with the token id; the audit
tool's user filter includes those rows. prune_extra_vars moves to the
docker service so the page and the tool share it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
MCP-5 of milestone 461 "MCP server: Steward as an LLM-inspectable,
LLM-operable hub". Contributed through each plugin's register_mcp_tools
hook, so they exist only while the plugin is enabled.
docker (viewer, read-only):
- list_containers: filter by host, status, compose project, or problem
(restarting, crash_looping, unhealthy, oom_killed, exited_nonzero);
problems sort first and each row names what is wrong.
- get_container: state, CPU/memory history across the raw/hourly seam,
lifecycle events. A unique part of a swarm task name is enough.
- list_swarm_services: replicas per node (agent-less placements as ghosts)
and the swarm's nodes.
- search_container_logs: text (% and _ literal), stream, range, context
lines, paging by a full-precision `before` cursor; over-long lines keep
both ends and say how much of the middle was cut.
- get_docker_disk_usage: bytes per host, reclaimable first.
- mcp_overview: failing containers and under-replicated swarm services
join get_overview.
host_agent: list_agents (reporting / stale / pending, version, OS), and an
overview finding for agents registered over 30 minutes ago that never
checked in.
Settings > Thresholds gains the TLS-expiry warn/crit days (were 14/3 in
code) and the crash-loop deaths-per-hour count (3), which get_overview
and the docker tools now read.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
MCP-3 of milestone 461 "MCP server: Steward as an LLM-inspectable,
LLM-operable hub". The /mcp tools run outside Quart's request pipeline, so
the queries they share with the pages now live in plain functions that take
an open session:
- steward/hosts/service.py: uptime, status, monitors, agent metrics,
agent overview (stale threshold passed in, not read from current_app),
CPU sparks.
- plugins/docker/service.py: bucket, container_history, host_map,
list_containers, container_events, query_logs.
- steward/alerts/service.py: active_alerts (also used by core/reports),
list_rules, list_maintenance_windows.
- steward/monitors/service.py: monitor_rows (was routes._rows_data).
Routes are repointed and render the same data. One behavior fix rides
along: the container-log search now escapes % and _ so they match
literally instead of acting as LIKE wildcards.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
