The dashboard's host widgets took 18-60 s. Every "latest reading" query
(hosts overview, host-agent fleet, host page, vitals, SNMP, MCP get_host)
searched plugin_metrics for the newest row per metric. Postgres has no skip
scan, so each one read every raw sample in the 7-day window.
- plugin_metric_latest: one row per (source, resource, metric).
record_metrics upserts it in the same transaction, moving forward only and
in sorted key order. Migration 0030 seeds it by stepping the existing index.
- Retention drops a latest row once its sample leaves the raw window, so a
resource that stopped reporting disappears when its history does, as before.
Renames carry it along.
- A quiet host keeps its last values, marked stale. A time-bounded query
would have shown it as missing.
- Vitals sparklines span the 6 h they claim, averaged in SQL to ~120 points,
instead of fetching 6 h raw and drawing the last 120 samples.
- Dashboard and share-view cells: "Loading…" until the first answer; a 20 s
deadline per fetch; "Couldn't load (why)" with Retry now and an automatic
retry for cells that don't poll; "as of HH:MM" in the header, and
"couldn't refresh · as of" when a refresh fails over shown data.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- One run per host: start_run claims every host it touches (addresses read
from the inventory Steward wrote; a source's own inventory file is claimed
as that file) before it takes a run slot. The claim is all-or-nothing. A run
waiting on a host shows queued and names the run it waits for, honours a
cancel, and fails after HOST_WAIT_SECONDS (1 h) with which host and run.
- A due schedule whose previous run is still queued or running skips the
firing: last_run_at moves on, last_run_id still names the run going, and
last_error says "Skipped: the previous run was still going". The Schedules
page shows that beside the run (muted), and a launch error in red.
- An alert rule that fires while the action run from an earlier firing is
still going starts no new run; the event says so and names the run. An
in-process set closes the gap between the check and the run being linked.
- docs/core/ansible.md: queued/cancelled statuses and the overlap rules.
Tests: tests/core/test_ansible_overlap.py (claim keys, waiting, all-or-nothing,
cancel, deadline) and tests/integration/test_ansible_overlap.py (two real
runs on one host take turns; schedule skip then fire; alert action skip then
start).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Output is read in 64 KiB chunks and split into lines here; a line past
64 KiB is cut and marked instead of raising out of readline's limit.
- ansible-playbook runs in its own session; whatever ends the read (finish,
an error, the task being cancelled) SIGTERMs then SIGKILLs its whole group,
so no ssh or worker it forked outlives the run.
- A run whose task is cancelled (shutdown) records 'cancelled' with
finished_at before re-raising, instead of staying 'running'.
- A cancel that lands after the process exited 0 no longer turns a
successful run into 'cancelled'.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- prune_by_age(column) registers a time-series column with the hourly cleanup;
Traefik (metrics, access summaries, global stats) and UniFi (WAN, client,
WLAN, DPI, events, alarms, speedtests) register theirs in setup().
- Alert events join monitor results and Ansible runs on data.retention_days,
read from Settings each run; Ansible run log files older than it are deleted.
- audit.retention_days (365) keeps the audit log longer, set on Settings > General.
- metric_series.points reads metrics.retention.raw_days, so the chart's
raw/rollup seam is where the rollup put it (RAW_DAYS removed).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
persist_host_docker marks docker.status, .swarm, .disk, .logs, .metrics,
.events, .containers, .alerts and .reap on the ingest's phase timer, each
flushed before its mark so a part's writes are charged to it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- The status test imported the plugin's models after the app had mapped
them ("already defined"); it reads docker_host_status with SQL.
- The swarm-expiry count awaited inside a generator expression.
- The page test's since came from the module's NOW, so |ago said "5m 4s";
it is built at test time and the assertion checks the minutes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The agent (1.9.0) said nothing when Docker could not be asked and nothing
when Docker had no containers, so Steward could tell neither apart from the
other, and kept a host's last containers on screen as running either way.
- Agent: collect_docker returns (status, detail, containers). "ok" sends
docker: [] when there are none; "unavailable" (no socket, nothing
listening) and "error" (permission, timeout, bad answer) say why.
collect_swarm says whether /info answered, and a node that is not a
manager sends swarm_manager: false. A failed /info says nothing.
- Ingest: an empty docker list is a snapshot; the newest sample's status
goes to the Docker plugin with the rest.
- docker_host_status (docker_011): per host, status, detail, since and
last report. A host that never had Docker gets no row.
- An "ok" empty snapshot reaps the host's containers; an error keeps them.
swarm_manager: false drops the host's swarm services and nodes.
- Docker page: a "Docker not answering" card names each such host, why,
and since when; its containers read as stale. Duplicate header comment
in rows.html removed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The first instrumented drop on dev read "dropped after 10.0s … (completed:
parse=0.1s)": the whole stall is inside record_metrics. It now marks its
three steps (flush of the rows, the rule and state reads, the per-sample
evaluation) on the caller's phases, and the drop line carries the request's
samples and metric row count.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5093 "Shared format filters and status_class driven by Settings → Thresholds".
- steward/core/formatting.py: human_bytes (binary units), human_rate,
human_duration (two largest units), ago (a <time> titled with the absolute
UTC time) and status_class(value, kind, thresholds) -> ok/warn/crit/unknown,
which raises on a kind the thresholds don't name.
- app.py registers them as the bytes, rate, duration and ago filters and the
status_class global; threshold_style is now graded by status_class (and caps
its weight at 500).
- Replaces the copies: docker's _human_bytes/_human_uptime, core alerts'
_duration, the host-agent fmt_bps/fmt_bytes macros, the Traefik and UniFi
byte-rate arithmetic, the swarm-failures ago macro and two uptime spans.
- Tests: a table per helper, and every template compiled against the app's
filters, so an unknown filter fails CI rather than a page.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5066 "Alert history page". /alerts/history lists every alert transition
in a range (1h–30d, the shared range picker), newest first, all rules or
one (?rule=). Each row: from → to, the value against the rule's condition
("no data" for a silent one), email and webhook sent or failed with the
error, "muted · <window>" (and "notified when the window ended") taken out
of the event note, other notes, and a link to the Ansible run an action
started. Range and rule changes swap only the rows (HTMX). Viewer role.
The Alerts page links each rule's history and the whole history. The query
moves to alert_svc.history, which MCP get_alert_history now uses too.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5065 "Renames carry their alerts". Alert rules, maintenance-window scopes
and metric history key a thing by (source, name), so a rename used to leave
the rules watching a name nothing reported under, and split the history.
- core/renames.carry_rename moves AlertRule.resource_name,
MaintenanceWindow.scope_resource, plugin_metrics and plugin_metrics_hourly
to the new name in the caller's transaction. A host rename also carries
its parts: host_agent "<host>:…" and docker "<host>/…" (HOST_SOURCES).
Matching is exact (a prefix compare, not LIKE or a collation range).
History left under the new name by something deleted is dropped first
rather than merged in.
- Web monitor edit, MCP update_monitor and web host edit call it; MCP
answers with `carried` (it used to return a warning naming stranded rules).
- A name another monitor of the same type, or another host, already has is
refused on create and rename: two would share one history.
- The rename is audited (monitor.updated / host.renamed) with what moved.
Tests: an integration test renames a firing monitor through the web route
and finds its rule still firing and cleared by a sample on the new name,
its window and raw+hourly history moved and an orphan's history dropped; a
host rename moves agent and container parts but not a longer-named host's
or another source's; a guard that every host_agent resource is "<host>" or
"<host>:…".
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5064 "Notifications that arrive":
- render_webhook: alert.* is escaped to sit inside a JSON string (a quote
or newline in a rule name no longer breaks the body); raw.* is the plain
value for |tojson; a missing value reads "no data"; the template runs
sandboxed and the body is checked to be JSON before it is sent.
- SMTP: 15s timeout; port 465 connects with implicit TLS, otherwise the
box means STARTTLS; a From address setting (falls back to the username).
- A webhook error no longer carries the URL, which is a secret.
- Settings → Notifications: "Send a test email" / "Send a test webhook"
send a sample alert with the form's values (unsaved) and show the result
and the body that went out; template help names alert.* and raw.*.
- Notification audits go through audit_request; the test send is audited
as settings.notification.tested, with the channel and outcome only.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
CI 8343: the wait-for-all-tasks loop also waited on the executor's
5-minute run-output cleanup task (_schedule_cleanup) and hit its 120s
deadline. Wait on alerts._background, then poll the run to a final status.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Notifications and alert actions now start only once the evaluating
transaction commits, and never for one that rolls back. record_metrics
queues them on the session (_after_commit), and after_commit/after_rollback
listeners on the sync session spawn or drop them. Spawned tasks are held in a
module set so they cannot be garbage-collected before they run.
A maintenance window no longer skips evaluation. Alerts in its scope still
change state, but their notifications and actions are held, and each event
says so ("notification muted: maintenance window <name>"; also on no-data
and resume events). notify_after_maintenance, run every minute with the
silence check as the new core task alert_reconcile, sends one notification
for an alert still firing after its window ends and runs its action. The
event's note records the release.
Alert actions go through runner.trigger_run like every other run path, so
the playbook and the inventory must be ones the source lists (#5034 guards).
The firing event links to the run, or its note says why the action did not
start. The maintenance page and form wording now say what a window does.
Tests:
- integration: a rolled-back evaluation notifies nobody, and dispatch waits
for the commit; maintenance mutes while the state moves and notifies once
when the window ends; an action whose playbook is outside its source is
refused, with the note set;
- the alert-action test uses a listed inventory name and waits, with a
deadline, for the run the action starts;
- unit: _muted_by.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent offline is an ordinary rule now. The host_agent scheduler records
host_agent/<host>/seconds_since_seen every minute (from last_seen_at, or
registration time for an agent that never reported), replacing the task
that only logged stale agents. The catalog offers the metric, and picking
it on a new rule fills "> 180".
No data: each rule's state tracks its last sample and its usual gap,
smoothed so that an outage counts as at most three gaps. A core task,
alert_silence_check, runs every 60s and marks a rule no_data_since once its
series has had no sample for three usual gaps (at least 5 minutes; an hour
while the cadence is unknown). A never-sampled rule counts from when it
started watching. The mark sits beside the state, so a firing alert that
goes silent stays firing. Each silence records one event and sends one
notification (rule option notify_no_data, default on; muted in maintenance
windows). The next sample clears it with a "data resumed" event and
notification. Creating, changing, disabling or re-enabling a rule restarts
its watch.
Migration 0029 adds the columns and seeds each existing rule's last sample
and gap from its last day of data, so an upgrade does not flag every rule.
Shown on the Alerts page (a NO DATA badge with its start time), in MCP
list_alerts, list_alert_rules and get_host (state "no_data", no_data_since),
in get_overview findings (alert_no_data, or a firing alert marked silent)
and in the weekly report. The rule form has the notify option, and its
metric reference now matches what traefik records and lists host_agent.
Emails say "no data" and carry the event's note.
The metric-catalog guard also reads positional MetricBatch.add arguments.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Importing plugins.host_agent.models after the app loaded the plugin defines
host_agent_registrations twice (run 8310).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5060 "One evaluation path: record_metrics batch API; host-agent ingest
evaluates rules".
- record_metrics(session, rows) writes metric rows and evaluates the rules
that watch them, reading rules, states and maintenance windows once per
batch. Each rule sees its key's samples oldest first, so an agent catching
up after an outage counts consecutive checks as they happened.
- Host-agent ingest now goes through it. Before, it wrote plugin_metrics
directly and no host_agent rule could ever fire. The catalog gains the
per-mount disk_used_pct ("<host>:<mount>").
- MetricBatch gathers a transaction's samples; the docker ingest, traefik,
unifi, snmp, monitor and self-monitor writers record one batch each instead
of a rule query per sample.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5059 "Alert state machine: acknowledgement holds until resolve; N
consecutive OK to resolve; editing, disabling or deleting a rule resets it".
- The state machine is now next_step(), a pure function of one sample. An
acknowledged alert stays acknowledged while breached; before, the next
breaching sample re-fired it. Firing and acknowledged alerts resolve after
consecutive_ok_required OK checks in a row (new rule column, default 1;
rule form, rules list and MCP create/update_alert_rule clear_after_checks).
- Changing what a rule watches (source, resource, metric, operator,
threshold) or disabling it resets its alert to inactive; a firing or
acknowledged one records a resolved event whose new note says why, and
nobody is notified. Web and MCP edits share alerts.service.apply_rule_change.
- Migration 0028: consecutive_ok_required, consecutive_ok_count,
alert_events.note, and metric_value nullable (a reset has no sample).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5036 "SSO and LDAP: users keep their auth source, no merging into local
accounts, is_active checked, LDAP TLS real and filter escaped".
- users gain auth_source (local/ldap/oidc) and external_id (migration 0027).
A provider account is found by (source, subject) only: the LDAP entry DN
or the OIDC sub. A first sign-in whose username or email another account
holds is refused with a message naming why; a disabled user stays out.
- Local passwords sign in local accounts only.
- LDAP: port 636 is LDAPS, otherwise StartTLS before any bind; the server
certificate is checked against the system CAs or a CA pasted in
Settings -> Auth. The login name is escaped into the filter (RFC 4515),
and an empty password is refused before the directory sees it.
- OIDC: the ID token must carry this sign-in's nonce and the userinfo sub.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The local import shadowed UserRole for the whole function, so its earlier
use read an unassigned name (ruff F823, CI run 8294).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5035 "Sessions: lifetime applied, role and is_active re-read per request,
logout is a POST, failed logins throttled".
- steward/core/proxy.py: trusted_value / is_https / client_ip read
X-Forwarded-Proto and -For only as far as the trusted proxy hops
(Settings → General, default 1, 0 = trust none).
- The session cookie is Secure, and HSTS is sent, only on requests that
arrived over HTTPS by that rule; plain-HTTP and LAN use are unchanged
and nothing redirects.
- A login lasts Settings → General's session lifetime from when it began
(session.permanent + a login_at stamp checked every request — the
cookie's own age slides on every response). A deleted or deactivated
user is logged out on their next request; a role change applies at once.
- Logout is a POST (a form in the sidebar); GET /logout is gone.
- Failed logins pause an address after login.max_failures within
login.lockout_minutes (Settings → General; defaults 5 in 15 min),
keyed by address so nobody can lock a user out by name. Failures are
audited as auth.login_failed with the address.
- to_access_cfg projects the new settings into app.config at startup and
on save, beside the existing projections.
- Integration tests seed real users and a login_at, since every request
now re-reads the session's user.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5031 "Forged-request protection: SameSite cookie, per-session CSRF token
on every non-GET, HTMX header". A page on another site could make a
logged-in browser POST to Steward with the session cookie attached.
- steward/auth/csrf.py: a per-session token, checked in one before_request
for every POST/PUT/PATCH/DELETE (header X-CSRF-Token or form field
csrf_token, constant-time compare). The bearer-authenticated host-agent
ingest is exempt; /mcp never reaches Quart.
- Every POST form renders {{ csrf_field() }}; base.html carries the token
in a meta tag and adds it to every HTMX request; the dashboard's fetch()
calls send it.
- The session cookie is SameSite=Lax; login issues a fresh token.
- PUBLIC_BASE_URL is now loaded from Settings at startup (it was only set
when Settings → General was saved).
- tests/test_csrf.py: refusal, acceptance, rotation, the ingest exemption,
and a guard that every POST form in the templates carries the field.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5014 "Ansible run scope is shown as a raw id"
- inventory_gen.describe_scope / scope_labels turn "steward:target:<uuid>"
into "target web-1", "steward:group:<uuid>" into "group Docker Swarm",
and a deleted one into "target abcdef01 (deleted)".
- MCP run and schedule rows carry scope_label beside the stored scope; the
ansible.run.started audit detail records it too.
- The web run page and schedules table show the name, with the stored
scope as the tooltip.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5012 "get_host cuts `parts` without saying so"
- The agent's sub-resources come back as mounts (fullest first),
interfaces and disk_io (busiest first) and temperatures (hottest first),
each bounded on its own with total, truncated and a `more` that names
parts_limit and query_metrics.
- Per-core CPU is a summary (count, min/avg/max, three busiest), so 38
cores no longer push every interface out of the reply.
- New argument parts_limit (default 20, max 200).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5010 "Logs of a removed container cannot be reached by name"
- get_container and search_container_logs fall back to the names Steward
still holds logs or events for when no live container matches: the exact
name from logs and events, or a part of it that matches one removed
container's events.
- get_container answers with removed: true, last_seen_at, its history and
events; search_container_logs marks container_removed and notes when it
was last seen, and says when that is older than the range.
- A miss now says the container may have been removed, that Swarm replaces a
dead task under a new name, and where to look instead.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5013 "Overview findings for containers and Swarm services carry since=null"
- Container problems are dated from their own evidence: a crash or restart
loop from its first death in the hour, an OOM kill or error exit from the
death that ended it, a failing healthcheck from its last turn to unhealthy.
list_containers and get_container carry problem_since / problem_age_s.
- Swarm services carry trouble_since: a crash loop's first failure in the
hour, a short service's oldest troubled task in each failing slot, a
recovered service's first failure in 24h. list_swarm_services shows it.
- get_overview findings from the docker plugin now have since and age_s,
so they rank by how long they have stood.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Milestone 465 "Swarm task awareness". Fixes three defects from the live MCP
review #4973:
- #5008 "Swarm task history read as live state"
- #5009 "Crash-loop detection is blind to Swarm services"
- #5011 "Swarm task errors are not captured"
Agent and ingest (agent 1.8.0):
- A manager's agent now reports each service's newest tasks: slot, node,
desired and current state, Swarm's Err/Message, exit code and time.
- They are stored in docker_swarm_services.tasks_json (migration
docker_010_swarm_tasks).
Swarm view (pure):
- A task container is "replaced" when Swarm set its task's desired state off
`running`. Without the task on record, a newer or running task in the same
slot decides.
- Replaced containers are no longer replicas or crashes.
- Each service gets, from the task records:
- failures_1h and failures_24h (failed or rejected tasks);
- crash_looping (Settings → Thresholds count);
- last_error, in Swarm's own words.
MCP:
- New findings: swarm_service_crash_looping (critical) and swarm_task_failed
(warning, on a healthy service with a failure in 24h).
- Down and degraded findings name the last task error.
- list_swarm_services carries the failure fields.
- Container rows mark swarm_replaced and give a replaced task no problems.
Web:
- The containers page swarm cards and the Swarm topology page show crash
loops, recent failures and the last task error.
- The dashboard widget flags a crash loop, and its "failed (24h)" count no
longer counts a rolling update's killed task.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
MCP-8 #4972 "Connect card, docs, and end-to-end integration test".
- AI access page: a "Connect a client" card with the /mcp URL (from
Settings → General → Public base URL, else the request host) and recipes
for Claude Code, Claude Desktop (mcp-remote) and other Streamable HTTP
clients, plus the reverse-proxy note. The token-created page shows the
same recipes with the new token filled in, ready to paste.
- docs/core/mcp.md: connecting, the reverse-proxy requirements, the
security model (token ceiling, default deny, check-mode default, secrets,
audit), tools by role, response conventions, adding tools.
docs/architecture.md describes the /mcp dispatch outside the blueprint
pipeline; docs/reference/code-map.md gains an MCP section.
- tests/integration/test_mcp.py: over HTTP, a viewer token lists no write
tools and is refused one by name; an operator token reads get_overview
and its maintenance-window create/end land in the audit log as
"<user> via MCP" with the token id. (Revoked → 401 was already covered.)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
MCP-7 #4971 "Monitoring-config write tools: monitors, alert rules, acks,
maintenance".
The validation the forms did inline (or not at all) moves to the services,
and both the routes and the tools call it, so bad input is refused the same
way with the same reason:
- monitors/service: monitor_fields, build_config, normalise_target. Port
1-65535, status 100-599, timeout 1-60, method GET/HEAD/POST, headers an
object of strings, a key the type does not take is named. The forms now
answer 400 with the reason instead of redirecting silently or a 500.
- alerts/service: METRIC_CATALOG and SOURCE_MODULES, resources_for,
metrics_for, rule_fields (source, resource and metric from the same
choices the rule form offers), add_rule, acknowledge, window_fields.
Tools: create_monitor, update_monitor (config merged over the stored one;
a rename warns about rules still watching the old name),
set_monitor_enabled, create_alert_rule, update_alert_rule,
acknowledge_alert, create_maintenance_window (duration or end),
end_maintenance_window (ends one in progress, cancels an upcoming one) —
operator. delete_monitor, delete_alert_rule — admin. get_alert_rule_options
— viewer. Every write is audited as "<user> via MCP".
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
MCP-6 #4970 "Automation tools: playbooks, runs, schedules, alert actions".
Viewer: list_playbooks, get_playbook (secret defaults masked), list_inventory
(variable names only), list_ansible_runs, get_ansible_run (log tail + grep),
list_schedules, wait_for_run (polls, 120s cap).
Operator: run_playbook (check mode unless check=false; a steward:confirm
playbook applies only with confirmed=true; secret variables reach the run
but are never stored, audited or returned), cancel_ansible_run, and
create/update/delete_schedule (secrets refused, check by default).
Admin: set_alert_action, and the docker plugin's prune_docker, which
previews from the disk report unless check=false (Ansible's check mode
skips the prune commands, so a check run would show nothing).
Every write is audited as "<user> via MCP" with the token id; the audit
tool's user filter includes those rows. prune_extra_vars moves to the
docker service so the page and the tool share it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
MCP-5 of milestone 461 "MCP server: Steward as an LLM-inspectable,
LLM-operable hub". Contributed through each plugin's register_mcp_tools
hook, so they exist only while the plugin is enabled.
docker (viewer, read-only):
- list_containers: filter by host, status, compose project, or problem
(restarting, crash_looping, unhealthy, oom_killed, exited_nonzero);
problems sort first and each row names what is wrong.
- get_container: state, CPU/memory history across the raw/hourly seam,
lifecycle events. A unique part of a swarm task name is enough.
- list_swarm_services: replicas per node (agent-less placements as ghosts)
and the swarm's nodes.
- search_container_logs: text (% and _ literal), stream, range, context
lines, paging by a full-precision `before` cursor; over-long lines keep
both ends and say how much of the middle was cut.
- get_docker_disk_usage: bytes per host, reclaimable first.
- mcp_overview: failing containers and under-replicated swarm services
join get_overview.
host_agent: list_agents (reporting / stale / pending, version, OS), and an
overview finding for agents registered over 30 minutes ago that never
checked in.
Settings > Thresholds gains the TLS-expiry warn/crit days (were 14/3 in
code) and the crash-loop deaths-per-hour count (3), which get_overview
and the docker tools now read.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
MCP-4 of milestone 461 "MCP server: Steward as an LLM-inspectable,
LLM-operable hub". Twelve read-only viewer tools:
- get_overview: what is wrong now, ranked critical-then-oldest. Failing
monitors with their error and when the run of failures began, firing,
pending and still-breaching acknowledged alerts, TLS certificates expiring
within 14 days, agents that stopped reporting, hosts over the configured
CPU/memory/disk thresholds (Settings > Thresholds), failed Ansible runs in
the last 24h, and findings from enabled plugins through a new optional
`mcp_overview(db)` plugin hook. Each finding names the call that shows
its evidence.
- list_hosts, get_host, get_host_metrics
- list_monitors, get_monitor (HTTP header values and URL passwords are
redacted)
- list_alerts, get_alert_history, list_alert_rules, list_maintenance_windows
- query_metrics: the whole metric store, narrowed one argument at a time
(sources > resources > latest values > a series summary)
- list_audit_events
Shared shaping in steward/mcp/tools/_shape.py: UTC ISO times with age_s,
bounded lists that say when and how they were cut, series as
min/avg/max/p95/last plus a ≤60-point series, lookup by id or
case-insensitive name with a did-you-mean, and a range argument that
refuses an unknown key instead of reading it as 24h.
New core reads in steward/core/metric_series.py (sources and resources
via loose index scans, latest values via DISTINCT ON, series across the
raw/hourly-rollup seam). monitors/service.py gains monitor_states,
down_since and recent_failures; agent_overview_by_host now carries the
fullest disk and the last sample time.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Second step of milestone 461. An MCP client (Claude Code, Claude Desktop
via mcp-remote, Open WebUI) connects to /mcp with a personal API token.
- mcp>=2.2 (SDK v2 MCPServer). Mounted by wrapping app.asgi_app, so /mcp
never enters Quart's pipeline; stateless streamable HTTP so a redeploy
never strands a client on a session id the new process didn't issue.
- Auth: Authorization: Bearer stw_… resolves to a Caller (user +
effective role) in a ContextVar; 401 with WWW-Authenticate otherwise.
The web pipeline's database gate runs first, so a DB restart answers
503 instead of failing inside a tool.
- Registry, default-deny: register_tool(fn, role=, read_only=) is the only
door; role and read_only are required. Tools above the caller's role,
or from a disabled plugin, are neither listed nor callable. Undeclared
arguments are refused (not silently dropped); a tool's ValueError
reaches the agent as its reason; a DB failure mid-tool says so.
- Plugins contribute tools via an optional register_mcp_tools(register)
export, all-or-nothing, replaced on hot-reload. Documented in
docs/plugins/writing-a-plugin.md.
- First tool: whoami.
- Tests drive app.asgi_app directly (Quart's test client never reaches
/mcp); the integration test mints a real token against Postgres and
runs initialize → tools/list → whoami → revoke → 401.
Task #4966 "MCP-2 — /mcp transport".
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
First step of milestone 461 (MCP server). A token is the credential an
MCP client presents as `Authorization: Bearer stw_…` to act as its user.
- api_tokens table (0026): sha256 hash only, display prefix, role
ceiling, expiry, revocation, throttled last_used_at.
- steward/core/api_tokens.py: mint, list, revoke, resolve. Resolution
re-reads the user's current role and acts with the lower of that and
the token's ceiling, so demoting or disabling a user weakens or kills
their tokens with nothing to revoke by hand.
- /ai-access/ (sidebar → Automation → AI access): every user manages
their own tokens, can't grant above their own role, sees the token
once. Session-only: a share-link viewer is refused. Create and revoke
are audited.
- Integration tests drive the routes against real Postgres so their
transactions run for real (#884 escaped CI behind a mocked session).
Task #4965 "MCP-1 — Personal API tokens".
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
test_persist_logs_respects_toggle_and_exclude read via s.execute(SELECT) between
two `async with s.begin()` blocks; that SELECT autobegins a transaction, so the
next begin() raised "A transaction is already begun". Move both reads after the
final begin block (the toggle-off persist stores nothing, so end-state still
reflects the exclude phase), and pass a SimpleNamespace stand-in host to sidestep
ORM attribute-expiry across the commit boundaries.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CAGR73DUowdVFVvYzLXC5C
The user-facing half (rule 27). Per-container log viewer + admin controls, so
container logs are something the operator can actually touch.
Viewer (plugins/docker):
- /container/<host>/<name>/logs full page + /logs/lines HTMX fragment polled
every 5s (near-live follow); stream filter (all/stdout/stderr) + case-insensitive
text search; newest-first so the latest lines survive the poll's scroll reset;
empty/loading states; a "View logs" link from the container detail page.
Jinja auto-escapes log content (no markup injection).
Settings (Thresholds & Retention tab):
- global on/off toggle, comma-separated exclude list, retention days + max MB
per container — DB-backed, no restart. The toggle + exclude are enforced
authoritatively at ingest (_persist_logs drops disabled/excluded lines), since
the push model has no channel to tell an agent to stop.
Tests: route-defined smoke; template-parse covers the new templates; integration
test for the ingest-time toggle + exclude enforcement.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CAGR73DUowdVFVvYzLXC5C
Bound docker_logs growth in the periodic cleanup task (same architecture as the
metrics/events retention). run_docker_retention gains logs_retention_days +
logs_max_bytes_per_container: it prunes lines past the age window, then keeps
only the newest ~cap bytes per (host, container) via a window-function ring
(exclusive-prefix sum, so the newest line always survives even if it alone
exceeds the cap). Containers rotate independently.
- settings DEFAULTS: docker.logs.enabled/exclude/retention_days(3)/
max_bytes_per_container(5MB) — operator preference: ~3 days / ~5 MB
- cleanup.py reads the two windows fresh each run (rule 25, no restart)
- integration rotate test: age prune + per-container byte cap + isolation
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CAGR73DUowdVFVvYzLXC5C
Ingest side of container logs. The host_agent ingest route extracts
sample["docker_logs"] as time-series batches (append every line, each carrying
its own Docker ts with recorded_at as fallback) and hands them to the docker
capability alongside the existing container/swarm/disk data — still under the
begin_nested SAVEPOINT so a logs failure can't sink host metrics.
- ingest.py: _persist_logs + pure _log_rows(batches, host_id) that shapes/filters
records (drops the _steward truncation marker, malformed + lineless records,
normalises unknown stream → stdout); persist_host_docker gains a logs= param
- routes.py: accumulate docker_log_batches, add to the guard + capability call
- unit test for _log_rows shaping/filtering; integration push→store→query test
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CAGR73DUowdVFVvYzLXC5C
Dedicated host-scoped table for pushed container log lines (one row per line),
chained after docker_008. host_id FK CASCADE; twin indexes — (host_id,
container_name, ts) for the viewer, ts alone for the age-cutoff prune — mirroring
docker_events. Integration schema-shape test asserts columns + both indexes.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CAGR73DUowdVFVvYzLXC5C
The widget-dedup test imported plugins.docker.routes, which re-imports
plugins.docker.models after the app already loaded the docker plugin —
"Table 'docker_containers' is already defined". Same plugin-loader gotcha the
host_agent query helpers avoid.
Extract dedup_by_container_id into plugins/docker/dedup.py (imports no ORM
models, so it's safe to import in either environment); routes.py imports it
from there and the test targets the model-free module.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC
The running count was inflated: swarm-aware agents on every manager list
cluster-wide tasks, so the same container (identical container_id) was reported
once per manager and the dashboard widgets summed them. The earlier swarm dedup
only covered the Swarm topology page — the widgets read raw rows.
- _dedup_by_container_id(): collapse rows sharing a non-empty container_id
(globally unique, so only true duplicates merge); first occurrence wins under
the running-first ordering. Applied in both the containers and resources
widgets before counting/limiting. Rows without a container_id (older agents)
are kept as-is.
- Replaced the static "stopped" tally — not actionable — with "failed (24h)":
distinct containers with a die/oom event in the last 24h (DISTINCT so the
managers' duplicate events don't double it), rendered red when non-zero.
- widget.html empty-state now keys off total_count.
- Regression test: same container_id from two managers counts once; id-less rows
all kept.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC
The container persister upserted current state keyed (host_id, name) but — unlike
the image/swarm/disk persisters — never deleted containers that vanished from the
host's latest listing. Every removed one-shot container (CI job runners, buildkit
builders, codex jobs) left a permanent "stopped" row, so the dashboard counts
ballooned (e.g. 856 stopped) and read as "not dedup'd". It wasn't dedup — it was
a missing reaper.
- ingest.py: after the upsert loop, delete this host's containers whose name is
notin the newest snapshot (the listing is authoritative — event derivation
already treats absence as removal). Mirrors images/swarm/disk. The existing
`if not snapshots: return` keeps swarm/disk-only samples from touching
containers; an empty container snapshot legitimately means none exist.
- widget_resources.html: this widget shows the busiest running containers by CPU.
Replaced the two unlabeled hairline bars with labeled CPU/MEM bars (label · bar ·
% value, warn/crit colors) so it's self-explanatory.
- Regression test: a container present then absent across snapshots is reaped.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC
plugin_metrics grows by (sources × resources × ~30s cadence); keeping 90d of raw
is a large table. Add a raw→hourly rollup (mirroring the Docker plugin) so only a
short raw window is kept at full resolution, with hourly averages archived longer.
- PluginMetricHourly model + core migration 0024 (plugin_metrics_hourly: avg/max/
count per source/resource/metric/hour, unique bucket constraint + lookup index).
- steward/core/metrics_retention.rollup_plugin_metrics: date_trunc('hour') agg of
raw older than the hour-aligned raw window, idempotent pg upsert into hourly,
delete the rolled raw, prune hourly beyond the rollup window.
- cleanup.py: plugin_metrics is no longer blanket-deleted at data.retention_days;
_run_metrics_retention drives the rollup with windows read live from settings.
- Settings: metrics.retention.raw_days (7) + rollup_days (90), tunable on the
Thresholds & Retention page (new "Host metrics retention" card).
- Chart read: _history_for_host merges the hourly rollup (older part of the range)
with raw date_bin (recent part, capped ≤1h), so 30d charts keep working —
recent at full resolution, older at hourly. Route passes raw_days from settings.
- Tests: unit (cutoff helpers) + integration (rollup aggregates/prunes; history
merges hourly + raw) against Postgres.
Speed was already handled by the indexes + SQL aggregation; this is the storage
lever (raw window ~10x smaller).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC
The previous commit's integration test failed at collection-time import:
importing plugins.host_agent.routes (which imports the host_agent ORM models at
top level) double-registers host_agent_registrations against the app-loaded
plugin's metadata ("Table already defined").
Move the two pure read helpers (_latest_metrics_for_host, _history_for_host)
plus SOURCE_MODULE / HISTORY_METRICS into plugins/host_agent/metrics_query.py,
which imports only the core PluginMetric model — no plugin models. routes.py
imports them back. The integration test now imports from metrics_query and no
longer trips the loader's registration guard. No behavior change.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC
Follow-on to the plugin_metrics indexes. plugin_metrics is already retention-
bounded (core cleanup prunes > data.retention_days, default 90d) and charts top
out at 30d, so the cost wasn't growth — it was the read path shipping raw rows
to Python.
- _history_for_host: bucket + average in SQL via date_bin (epoch-aligned, ~120
buckets) instead of fetching every raw sample (a 30d range was hundreds of
thousands of rows) and downsampling in Python. Uses the new
(source_module, resource_name, recorded_at) index.
- _latest_metrics_for_host: DISTINCT ON (resource_name, metric_name) ORDER BY
recorded_at DESC — newest row per group in one index-ordered pass, replacing
the GROUP-BY-max subquery self-joined back to the whole history.
- Integration test validates both against Postgres.
Deliberately not a materialized raw→hourly rollup: these query-side changes
deliver the speed; a rollup would additionally cut storage and remains a future
option if scale demands it.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC
docker_metrics.mem_usage_bytes and docker_containers.mem_usage_bytes /
mem_limit_bytes were int4 (max 2,147,483,647). A container using >2.1 GB of RAM
(e.g. 8.18 GB) overflowed the column, so asyncpg raised "value out of int32
range" and the entire docker ingest batch failed — no metrics stored for any
host with a large container.
The I/O counters and the milestone-77 rollup/disk tables already used
BigInteger; this trio (docker_001-era) was missed.
- models.py: DockerMetric.mem_usage_bytes, DockerContainer.mem_usage_bytes,
DockerContainer.mem_limit_bytes → BigInteger.
- migration docker_008_bigint_mem: ALTER COLUMN ... TYPE BIGINT (safe in-place
int4→int8 promotion).
- integration regression test: persist an ~8 GB container, assert the
current-state and time-series rows round-trip.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC
test_disk_usage_persisted opened a second session.begin() after interleaved
SELECTs, which autobegin a transaction → "A transaction is already begun".
Roll back the read transaction before the re-report write. Restores the
integration lane green for the /system/df slice (a840d6f).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC
Backend for the image/disk panel (milestone 77 #942). Agent gains
collect_disk_usage() — one /system/df call (gated on containers existing, so
Docker-less hosts pay nothing), surfacing reclaimable bytes (image size held by
unreferenced images), layers/containers/volumes/build-cache sizes, and the top
50 images by size. Emitted as sample["docker_disk"]; host_agent ingest tracks
the newest sample's copy and hands it to the docker capability as a 5th arg.
New current-state tables docker_disk_usage (one row/host) + docker_images
(per-host image rows), docker_007 migration; ingest upserts the summary and
replaces the image set per host (stale images pruned). Unit tests for the df
parsing/reclaimable math + build_sample gating; integration test for
persistence + image-set replacement.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC
Bounds Docker time-series growth (the main scaling concern). New
docker_metrics_hourly table + docker_006 migration; a plugin retention module
(docker.run_retention capability) rolls raw docker_metrics older than the raw
window into hourly averages (idempotent upsert), deletes the rolled raw rows,
then prunes stale rollups + lifecycle events. Core cleanup.py drives it each
hourly run via the capability (no plugin-model import), reading the three
retention windows fresh from settings so changes apply without restart (rule 25).
Settings → "Thresholds & Retention" gains a Docker retention card (raw /
rolled-up / events windows, working defaults 7/90/30 days). Unit tests cover the
hour-aligned cutoff/bucketing helpers; integration test exercises the real
rollup-average + prune across both windows.
Milestone 77 task #941.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC
