The host page gains a "Who's sending" panel, and the MCP a tool, that
put the host's outbound (or inbound) traffic down to its containers and
NFS mounts: read in shared time buckets, so the host's busiest bucket
says who was sending in it. A swarm service's tasks add up under its
name; what none of them accounts for is the host itself.
Also fixes CI 9488: per-container net_rx_bps / net_tx_bps join the
docker alert catalog, so a rule can catch a container flooding the
uplink.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The agent sent Docker's working set (usage less inactive_file), which still
counts recently used file cache. A file-heavy container with no limit read
191 GB while its processes held 2.2 GB, and the host's own figure, which
already leaves cache out, disagreed with it.
Agent 1.23.0 sends mem_used_bytes / mem_used_pct (anon + shmem, so
Postgres's shared buffers count; v1: rss + shmem) and mem_cache_bytes
apart. The new names keep the two meanings apart in history and in learned
normals: docker_013 renames the columns, clears the old values and adds
mem_cache_bytes. The Docker page, swarm by-node sums, Busiest Containers,
container detail, normals and the MCP tools all read the new figure. File
cache is shown only on the container's page and in get_container.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
get_anomaly gives each occurrence against what is usual at that hour, the
metric around the newest one with its normal, what else happened on the
resource's host around each occurrence (alert transitions, Ansible runs,
other anomalies there and a count elsewhere, plus plugins' own events
through a new mcp_events hook: the host agent's maintenance jobs and
downtime, Docker's container starts, deaths and OOM kills), and how earlier
anomalies on the same metric were judged.
get_metric_normal reads the learned slots and the level that would be
flagged. review_anomaly takes one id or several, all or nothing, with a
note. get_anomaly_settings and (admin) set_anomaly_settings read and change
the review window and each normal's floor, as Settings → Thresholds does.
metric_series.points takes an optional `until`; core.alerts gains
host_of_resource; floor_setting names its floor.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
On 2026-10-14 nine swarm services sat at 0/1 and nothing fired. From each
manager's swarm report the docker plugin now records three figures on
resource "swarm": swarm_services_short, swarm_nodes_down and
swarm_managers_unreachable. Drained nodes are left out of the last two,
since a drain is deliberate. The figures are recorded at most once every
30 s whichever manager reported, and are offered to alert rules.
Agent 1.19.0 sends each manager's ManagerStatus.Reachability. Older agents
fall back to the node's status.
A maintenance job on a swarm node also keeps a window over docker/swarm,
so its drain and reboot hold those alerts until the job ends.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
VDNT-Docker03's silent agent read as three overview findings (agent offline,
agent stale, the disk rule's no-data) and every drop emailed the disk rule's
silence and return on top of agent offline.
- The host agent registers its heartbeat (seconds_since_seen) and the
sources it carries (host_agent, docker). While a host's agent is past its
offline window, its other rules' no_data is recorded "while its agent is
offline" and not emailed, nor is their quiet return; a return past a
threshold still fires.
- The overview folds those rules under the host's agent-offline finding
("also silent with it: ..."), and a stale agent whose agent-offline alert
is active is that alert's finding, not a second one.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Host CPU, memory and load per core, container CPU and memory, and every
monitor's response time learn a normal per hour (milestone 513, System 182).
A reading past its fixed cutoff but within its normal reads healthy and says
what is usual at this hour; one well out of it keeps its grade and is a
metric anomaly. Disk, a monitor down and an agent offline keep their cutoffs.
- Each normal's floor, the smallest departure that counts, is a setting:
Settings -> Thresholds -> Normals (norms.floor.*). Defaults: host CPU 10
points, host memory 5, load per core 10, container CPU 10 points of one
core, container memory 5, monitor response 100 ms; Traefik's 2-point 4xx
and 50 ms p95 floors move there from constants. learn_normal takes a
floor function (floor_setting) read at each pass.
- learn_normal(parts=False): a host learns, its cores don't.
- Ingest records load_pct (load_1m / cores x 100), the figure pages grade.
- Shared grading: anomalies.against / graded, worst_reading(normals=),
hosts' host_grades, docker usage(thresholds=, norms=), monitors'
latency_grade. Applied to host vitals, the Hosts widget, MCP overview,
Docker usage, and the monitor list, widget and page.
- The monitor and container pages list their anomalies.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5752 "Container CPU and memory history lives in the one metric store;
docker_metrics and docker_metrics_hourly are retired". Container cpu_pct and
mem_pct already went to plugin_metrics for alerting; now mem_usage_bytes does
too, and every snapshot in an ingest (backlog included) is recorded at its
own time. The container history chart, Busiest Containers, a service's
history and the MCP container tools read plugin_metrics under resource
"<host>/<container>", rolled up and pruned by the shared metric retention.
docker_012_drop_metrics drops both tables. Their history is not copied
(lesson #5221 "A data migration's cost is set by the live table"); the
store already holds 7 days of container CPU and memory %. Docker retention
now prunes only lifecycle events, and Settings -> Thresholds has one
"Metric retention" card for every source.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5449 "The hourly Docker log prune ranks every log line in the table to find
the few over their cap": the size ring put a running-total window over every
row of docker_logs each hour, sorting the whole table on NFS-backed storage
to find the handful of containers past docker.logs.max_bytes_per_container.
- prune_docker_logs (capability docker.prune_logs) does the age cutoff, then
a per-container byte total picks the containers at or over the cap, and
only their rows are ranked and trimmed. Same keep rule as before: the
newest line always survives.
- run_docker_retention keeps the metrics rollup and the event prune.
- Cleanup runs the log ring in its own transaction after them.
Arose from #5431 "Ingest dropped for ~3 minutes after each deploy while
startup retention runs".
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- #5304 "Docker disk totals recorded over time per host, with a growth chart and alertable metrics"
- #5305 "Disk page: images grouped by repository, versions folded under it"
- #5306 "Docker overview: one disk line per host linking to the Disk page"
Each `docker system df` report is now also recorded as metrics, with the host
as the resource: disk_total_bytes, its parts, and disk_reclaimable_bytes. An
alert rule can watch them. The Disk page starts with the hosts side by side,
showing how much each grew over the selected range with a line of it, then
gives each host its own section. An image's versions fold under its
repository, which is charged only for the layers its versions don't share.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5061 "Metric catalog belongs to what emits: plugins register their
metrics; names match what is recorded".
- The static METRIC_CATALOG is gone. metric_catalog() is core's own sources
(the four monitor types and "steward", the app's open_fds/open_fds_pct,
which were recorded but never offered) plus each loaded plugin's
get_alert_metrics() export. A disabled plugin's sources leave the picker.
- traefik offers what it records: error_rate_4xx_pct and error_rate_5xx_pct.
The old "error_rate" was never recorded, so a rule on it could not fire.
- tests/core/test_metric_catalog.py walks every MetricBatch.add /
record_metric call (traefik's tuple loop included) and fails on a recorded
metric the catalog lacks; it shows on a synthetic writer that it can fail.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
MCP-5 of milestone 461 "MCP server: Steward as an LLM-inspectable,
LLM-operable hub". Contributed through each plugin's register_mcp_tools
hook, so they exist only while the plugin is enabled.
docker (viewer, read-only):
- list_containers: filter by host, status, compose project, or problem
(restarting, crash_looping, unhealthy, oom_killed, exited_nonzero);
problems sort first and each row names what is wrong.
- get_container: state, CPU/memory history across the raw/hourly seam,
lifecycle events. A unique part of a swarm task name is enough.
- list_swarm_services: replicas per node (agent-less placements as ghosts)
and the swarm's nodes.
- search_container_logs: text (% and _ literal), stream, range, context
lines, paging by a full-precision `before` cursor; over-long lines keep
both ends and say how much of the middle was cut.
- get_docker_disk_usage: bytes per host, reclaimable first.
- mcp_overview: failing containers and under-replicated swarm services
join get_overview.
host_agent: list_agents (reporting / stale / pending, version, OS), and an
overview finding for agents registered over 30 minutes ago that never
checked in.
Settings > Thresholds gains the TLS-expiry warn/crit days (were 14/3 in
code) and the crash-loop deaths-per-hour count (3), which get_overview
and the docker tools now read.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Plugin UIs had no nav home (only dashboard widgets / typed URLs). Add an
optional get_nav() plugin export and surface it in the sidebar.
- plugin_manager: _PLUGIN_NAV registry + get_plugin_nav() getter + a tolerant
_collect_plugin_nav() (missing hook = fine; raising/malformed = logged &
skipped; idempotent per plugin so hot-reload re-runs cleanly). Collected in
both the startup load path and the hot-reload path.
- app.py: inject plugin_nav into the template context, filtered to enabled
plugins so a hot-disabled plugin's link can't linger before restart.
- base.html: render plugin links under the Infrastructure group, with the same
request.path active-state treatment as core links.
- docker/snmp/unifi/traefik: each exports get_nav() → its /plugins/<name>/ view.
- Tests: collector behavior (collect, missing hook, reload-replace, malformed
item, raising hook, sorted output).
With docker enabled, "Docker" now appears under Infrastructure → its fleet view.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC
Bounds Docker time-series growth (the main scaling concern). New
docker_metrics_hourly table + docker_006 migration; a plugin retention module
(docker.run_retention capability) rolls raw docker_metrics older than the raw
window into hourly averages (idempotent upsert), deletes the rolled raw rows,
then prunes stale rollups + lifecycle events. Core cleanup.py drives it each
hourly run via the capability (no plugin-model import), reading the three
retention windows fresh from settings so changes apply without restart (rule 25).
Settings → "Thresholds & Retention" gains a Docker retention card (raw /
rolled-up / events windows, working defaults 7/90/30 days). Unit tests cover the
hour-aligned cutoff/bucketing helpers; integration test exercises the real
rollup-average + prune across both windows.
Milestone 77 task #941.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC
Docker collection moves off the central single-socket scrape onto the host
agent, giving Docker a real per-host dimension. The Steward host now reports
its own containers like any other host, and same-named containers on different
hosts no longer collide.
- agent: stdlib UDS Docker client (AF_UNIX HTTP/1.1, Connection: close,
chunked-aware), collect_docker() ports the cpu%/mem math; sample["docker"]
added best-effort (silent-skip on absent/unreadable socket). AGENT_VERSION
1.2.0 → 1.3.0; optional docker_socket config key.
- ingest: host_agent ingest hands per-host container snapshots to the docker
plugin via a new "docker.persist_host_samples" capability (no hard import,
no-op when docker disabled), inside a SAVEPOINT so a docker failure never
sinks the host metrics. Resource names are host-scoped ("<host>/<name>").
- schema: docker_containers re-keyed (host_id, name); docker_metrics gains
host_id; docker_002 migration DROP+recreates (dev-only, rule 122).
- ui: Docker page + widgets grouped by host with host links; new per-host
Docker panel embedded on the Hosts hub (gated on docker enabled via a new
enabled_plugins template context). Replaces the SQLite-only strftime
bucketing with DB-agnostic Python bucketing.
- provisioning: install/provision playbooks add steward-agent to the docker
group (best-effort) so the agent can read the socket.
- removed central scrape: docker scheduler.py + scraper.py deleted; plugin.yaml
socket_path/scrape_interval_seconds/include_stopped dropped (plugin 2.0.0).
- tests: agent docker collector units (math, chunked decode, silent-skip,
sample shape, config) + integration (host-scoped schema + persistence).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC
First-party plugins (host_agent, http, snmp, traefik, unifi, docker) are now
tracked under plugins/ and baked into the image, so they version atomically
with core — ending the cross-repo import drift the roundtable->steward rename
exposed. History for these files is preserved in the archived Roundtable-plugins
repo.
Plugin discovery becomes multi-root: PLUGIN_DIR (single) -> PLUGIN_DIRS
(bundled first, then external) + PLUGIN_INSTALL_DIR. Bundled ships in the image;
third-party plugins still mount at runtime into the external root
(STEWARD_PLUGIN_DIR, default /data/plugins) and downloads/installs land there.
Bundled shadows external on a name collision.
- config.py: load_bootstrap returns plugin_dirs + plugin_install_dir
- app.py: iterate PLUGIN_DIRS at the migration + load sites
- migration_runner.py: discover_all_in() unions every plugin root
- plugin_manager.py: resolve_plugin_path() (pure, first-root-wins); load /
install / hot-reload span all roots; installs target the external root
- settings/routes.py: _discover_plugins scans all roots, dedup bundled-first
- Dockerfile: COPY plugins/ ; docker-compose: drop host bind, document external
- tests/test_plugin_dirs.py: resolution, multi-root discovery, bootstrap split
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
