The Hosts — Overview widget opens with a verdict ("VDNT-Util01: CPU high."
or "All 3 hosts healthy."), then one row per host under one header: CPU,
memory, disk / and load. Rows sort by host_glance: a down monitor, a
reading past critical, a quiet agent, a warning, then the healthy. Load
reads "16.8 / 8 cores", graded per core as Settings → Thresholds intends
(cores counted from the per-core CPU sub-resources, as on the host page).
A quiet agent's last figures are shown uncoloured, with when it last reported.
Ping and 24h uptime are gone from this widget: they belong to Monitors. So
are the unlabelled CPU sparkline and the 30-day uptime query it no longer
needs.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The Monitors widget opens with a verdict ("1 of 5 monitors down."), then one
row per monitor under a single header: latency and uptime over 24h, 7d and
30d, down monitors first with why and since when, the name linking to the
monitor. Counts follow the type filter. Problems always show; the limit
trims only the healthy rest ("+N more up").
Uptime / SLA (uptime_summary) repeated those numbers per host; migration
0033 moves dashboards onto Monitors. Shared widget styles (.w-verdict,
.w-table, .w-sub, .w-more) carry the widget-shape ruling for the widgets
that follow.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each widget is as tall as its content, never taller than its saved height
(where it scrolls). The dashboard is capped at 2400px instead of 1400px.
Past 12 columns' worth of width it adds columns (one per 110px, up to 24)
and places widgets again in reading order at their saved widths, so more sit
side by side instead of stretching. Edit mode shows and saves the
12-column layout, so a layout doesn't depend on the screen it was edited on.
The share view uses the same layout code (static/dash-layout.js).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- New installs start from widgets.RECOMMENDED, packed row by row for the
enabled plugins; the edit drawer can reset a dashboard to it.
- Host Agent — Resources folds into Hosts — Overview (which gains load 1m),
UniFi — Devices into UniFi — Overview; migration 0032 moves existing
dashboards onto the survivors.
- The Overview labels what it counts (hosts, monitors up/down/pending,
alerts firing) and lists firing alerts and down monitors with since-times.
- Uptime polls every 5 minutes (poll_seconds) instead of every cycle.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5212 "Plain labels: no raw ids, scopes, metric keys or unexplained jargon",
batch 3:
- Disk page: each tile says what it holds. "Writable layers" becomes
"Container files" and Build cache says it is safe to clear. Shared is
"Shared with other images" with a tooltip, and "dangling" is gone.
Untagged images show by id instead of "<none>", a capped list says
"The 50 largest of N images", and each host shows "as of" its report.
- Traefik: an expired certificate reads "expired 12 days ago", not "-12
days" (cert_left). The Traffic empty state knows whether the access log
is on: off points to Settings → Plugins → Traefik; on says to check the
mounted log path.
- Moments use |ago with an exact UTC tooltip: UniFi last seen, alarms,
events, speed test and DPI; AI token last used.
- Settings: the Auth tab is "Authentication" like its breadcrumb, plugin
fields read "Metrics URL" not metrics_url (field_label), a source's pull
interval is "every 1h", and the last report time drops the ISO "T".
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5212 "Plain labels: no raw ids, scopes, metric keys or unexplained jargon",
batch 2:
- A run records where it came from: AnsibleRun.origin ("schedule", "alert"
or "mcp") and origin_id, set by the scheduler, Run now, alert actions and
the MCP tools. Migration 0031 backfills past runs from alert events,
schedules' last runs, and MCP audit entries.
- The runs list has a Started by column: "schedule nightly", "alert rule
nas down", "alice via MCP" or "alice", linked where there is a page.
The raw scope tooltip is gone.
- Run detail: no run id in the subtitle or body, Started by in words, and
the host summary lists failed and unreachable hosts first. The live
output only follows new lines while the reader is at the bottom, and
when the run finishes the page reloads once so the badge and summary
are final.
- list_ansible_runs and get_ansible_run report origin and origin_id.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Run 8504 failed unit: tests/core/test_alert_history_row.py built a rule
with no metric_name, which _history_row now reads to say the reading in
words. The rule now names cpu_pct, and the value and condition asserts
expect "CPU 98%", "No CPU reading" and "CPU > 90%", plus an up/down case.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5212 "Plain labels: no raw ids, scopes, metric keys or unexplained jargon",
batch 1:
- Audit log: each action reads as a sentence ("paused monitor"), with the
code as its tooltip. Detail shows flags as yes/no, lists joined, a run's
scope by name, spans as spans, and no ids the entity column already gives.
The filter is now a picker of areas.
- Alert rules, history and Status say the condition in words: "is down",
"Fullest disk >= 85%". Readings use metric_value, and history times use
|ago.
- Rule form: sources and metrics by name, the threshold's unit, "fires
when it is down/up" for an up/down metric instead of 0.5, and a metric
reference built from the catalog.
- Schedules and Maintenance: viewers no longer see Run now, Pause, Edit,
Delete or New. Intervals and next runs read as "every 2h" and "in 3h";
a scope reads "Every alert on Host agent".
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5211 "Lists load as you scroll: audit, alert history, Ansible runs"
(milestone 477), per preference 172.
- core/paging.py: keyset paging over (time, id), newest first, with an
opaque "<iso>|<id>" cursor; a page reads one row past itself to know
whether another follows. No OFFSET, so rows added while someone scrolls
never shift the pages after them.
- _macros.html scroll_more / scroll_end and static/scroll-more.js, written
once: an IntersectionObserver with a 600px margin fetches the next page
before the reader reaches the end; "Loading more…" while in flight,
"End of the list." at the end, and a failed page says so with a Retry
button instead of re-firing on every scroll.
- Audit log: no "Load more" link and no 500-row cap.
- Alert history: no 200-row stop with "pick a shorter range";
alert_svc.history takes the cursor, so the MCP tool and the page share
one query.
- Ansible runs: no silent cap of 50.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5210 "Problems first, with since-time, across every list" (milestone 477),
second batch.
- Host page: an "Alerts on this host" strip first (rules on its agent
series, its containers or its monitors: alert_svc.watches_host); its
monitors problems first with "Down 12m · reason"; Docker above Ansible.
- Uptime page and widget: worst 24h uptime first, then 7d and 30d; hosts
with no uptime yet last.
- Agent panel and fleet widget: last seen as "3m ago", not a clock time.
- SNMP: a device is "Reachable" only while it answered within the last 3
polls; older readings say "No answer" with their age, and every device
says when it was last read (was: reachable whenever any reading existed).
- Traefik: a router missing from 3 scrapes in a row is listed apart as no
longer reported, by when it was last seen, instead of keeping its last
state (a removed router stayed Failing).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5210 "Problems first, with since-time, across every list" (milestone 477),
first batch.
- Alerts: the Active list reads firing, no data, acknowledged, pending, the
longest-running first, and re-polls itself. An acknowledged alert says how
long it has been breaching (from the alert's events; acknowledging moves
last_transition_at) and, apart from that, who acknowledged it and when.
Rules show what each is doing now; disabled rules sort last. The current
value reads the latest-value table (#5208) instead of plugin_metrics.
- Monitors: a down monitor says "Down 12m · connection refused" in the list.
- Hosts: sorted down, agent gone quiet, pending, healthy. The agent column
shows "—" for a missing reading (was 0%) and a Stale badge with its age.
agent_metrics_by_host is gone; the list reads agent_overview_by_host.
- Schedules: failing first (failed or interrupted run, or a firing that could
not start one), then running, then the rest, paused last; status badge
from the shared states.
- Maintenance windows: active (ending soonest), then upcoming (starting
soonest), with expired folded away; each says when it ends/starts/ended.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Dockerfile: python:3.14-slim, the Python CI tests on (was 3.13). tini from
apt is PID 1 (ENTRYPOINT tini -- entrypoint.sh): it reaps the zombies ssh
and ansible leave and forwards SIGTERM.
- cli: Hypercorn's serve() with one worker (run, stream and scheduler state is
in-process), a 10 s graceful timeout, and SIGTERM/SIGINT as the shutdown
trigger. Access and error lines go through Steward's logging. --debug keeps
Quart's reloading dev server.
- Logging: steward/core/logging_setup.py installs one root handler before
anything logs (until now a library's basicConfig on import was the only
configuration). Its level is the `log.level` setting (DEBUG/INFO/WARNING/
ERROR, default INFO), applied at boot and on save in Settings → General.
- CI publish: build, then "The image boots with no internet" (rule 164,
snippet #3552), then push, so a broken image never reaches a tag (rule 177).
On an --internal network with its own Postgres, the image must migrate and
answer /health, run tini as PID 1, serve first-run setup with its assets
from /static and no third-party host, and exit 0 on SIGTERM. Logs and
clean-up run always.
Tests: tests/core/test_logging_setup.py; the smoke step itself.
Not in this commit: the dependency lockfile (it conflicts with rule 154 and
is waiting on the operator).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The "[queued — waiting for run X on the same host]" and run-slot lines were
broadcast to live viewers and never stored, so a run that sat queued had no
record of why once the live view was gone (CI 8439: the overlap test found
the order right and the explanation missing). _claim_hosts now returns the
line it waited on and start_run seeds the run's output with every wait.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- One run per host: start_run claims every host it touches (addresses read
from the inventory Steward wrote; a source's own inventory file is claimed
as that file) before it takes a run slot. The claim is all-or-nothing. A run
waiting on a host shows queued and names the run it waits for, honours a
cancel, and fails after HOST_WAIT_SECONDS (1 h) with which host and run.
- A due schedule whose previous run is still queued or running skips the
firing: last_run_at moves on, last_run_id still names the run going, and
last_error says "Skipped: the previous run was still going". The Schedules
page shows that beside the run (muted), and a launch error in red.
- An alert rule that fires while the action run from an earlier firing is
still going starts no new run; the event says so and names the run. An
in-process set closes the gap between the check and the run being linked.
- docs/core/ansible.md: queued/cancelled statuses and the overlap rules.
Tests: tests/core/test_ansible_overlap.py (claim keys, waiting, all-or-nothing,
cancel, deadline) and tests/integration/test_ansible_overlap.py (two real
runs on one host take turns; schedule skip then fire; alert action skip then
start).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- git ships in the image; checkouts live under /data/ansible/sources
(STEWARD_ANSIBLE_SOURCE_DIR), on the persistent volume.
- One task, ansible_git_pulls, reads the sources every 30 s and pulls each
as its interval passes: a source added in Settings is pulled without a
restart, a removed one is forgotten.
- git_pull: a 120 s timeout ending the process group, GIT_TERMINAL_PROMPT=0,
pull --ff-only, a failed clone removed; it raises GitPullError with git's
message, so the Sync button stops saying Synced after a failure (and the
message is now escaped).
- An alert action's vars and the firing's steward_alert_* vars reach
ansible-playbook as one JSON extra-vars file, not -e key=value, which
Ansible split on whitespace (a rule named 'High CPU on web').
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Output is read in 64 KiB chunks and split into lines here; a line past
64 KiB is cut and marked instead of raising out of readline's limit.
- ansible-playbook runs in its own session; whatever ends the read (finish,
an error, the task being cancelled) SIGTERMs then SIGKILLs its whole group,
so no ssh or worker it forked outlives the run.
- A run whose task is cancelled (shutdown) records 'cancelled' with
finished_at before re-raising, instead of staying 'running'.
- A cancel that lands after the process exited 0 no longer turns a
successful run into 'cancelled'.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- prune_by_age(column) registers a time-series column with the hourly cleanup;
Traefik (metrics, access summaries, global stats) and UniFi (WAN, client,
WLAN, DPI, events, alarms, speedtests) register theirs in setup().
- Alert events join monitor results and Ansible runs on data.retention_days,
read from Settings each run; Ansible run log files older than it are deleted.
- audit.retention_days (365) keeps the audit log longer, set on Settings > General.
- metric_series.points reads metrics.retention.raw_days, so the chart's
raw/rollup seam is where the rollup put it (RAW_DAYS removed).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Measured on dev (385a579), agent posts cut to five samples still dropped:
flush=8.8s for 990 rows, 6.6s for 680, 8.4s for 1000 — about 110 rows/s,
~9 ms a row, with rules and evaluation near zero. One random read per
row fits a random uuid4 primary key whose index no longer fits in memory:
every new row lands on a random page of it. (Inferred from the rate; the
table and index sizes have not been read.)
- steward.core.ids.time_ordered_id: a UUIDv7-layout string (48-bit ms
timestamp, then random), same 36-char shape as the stored uuid4 ids, so
no migration. Python's uuid7() arrives in 3.14; the image runs 3.13.
- Used for plugin_metrics and its hourly rollup, monitor_results,
audit_events, the docker metrics/hourly/events/logs tables, and the
traefik and unifi time series.
- MAX_SAMPLES_PER_INGEST 5 → 2 (~400 rows) for now: it fits the agent's
10 s even at the old rate, so hosts land data whichever way the new rate
comes out. Raise it once the new rate is measured.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The dispatcher ticked once per MONITORS_POLL_INTERVAL and stamped
last_checked_at when the whole batch finished. With a 60 s tick and a 60 s
monitor the next tick saw 59.x s elapsed and skipped, so the monitor ran
every 120 s; a 30 s monitor could never run faster than the tick.
- The dispatcher ticks every 5 s (DISPATCH_TICK_SECONDS) and reads the
monitors and intervals fresh each tick, so a changed interval applies at
the next one.
- Due is measured from the scheduled start, stamped before the check runs,
with half a tick of slack so tick jitter never costs a whole tick. A
long-overdue monitor runs once.
- Each check runs in its own task and records its own result, so a slow
probe holds up nobody; a monitor still being checked is not started
again, and it is freed in a finally.
- A check past CHECK_DEADLINE_SECONDS (120) is recorded down with "Check did
not finish within 120 s" rather than holding its monitor forever.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Traefik: routers are graded from their latest scrape (router_state: crit on
any real 5xx share, warn on p95 past the latency critical cutoff or a high
4xx share). Failing and slow routers lead as cards that say why; healthy
ones fold into one table that stays open or shut across polls. Proxy
counts are stat tiles; certificates are graded by Settings -> Thresholds
(tls_days).
UniFi: WAN health is a stat strip (latency graded by the thresholds),
devices carry status dots, event categories and firewall actions are
badges, the alarm count reuses .count-badge (renamed from .nav-count).
SNMP: each device wears a status badge and its readings are stat tiles.
Agent fleet: the table comes first, with a Status column (Reporting /
Silent / Never reported, by the host page's stale_after_seconds), silent
agents first, hosts linked, last seen as |ago.
Audit: entries name their entity (one query per type, steward/audit/
service.py) and link it; something deleted since keeps the name its entry
recorded and says "(deleted)".
Tokens: charts read colours through chart-colors.js (stewardColor resolves
a token to the rgb the browser computed; Chart.js defaults take the page's
text and border colours), so host, history, Traefik and SNMP charts drop
their hex. Template hex colours reach 0 (HEX_CEILING = 0), the last rgba
pills go, and heavy weights (600/700/bold, 60 of them) become 500, the
house pair being 400/500; a guard keeps templates there.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Runs read "update.yml on web-1" (run_title over the scope label, beside
describe_scope) on the runs list, the run page's title and breadcrumb, and
the run-started panel. The list shows started |ago (full time on hover),
|duration (counting "so far" while a run is going) and a dry-run tag.
Run statuses join STATES, so the list and the run page use status_badge:
queued dim, running info (blue, so work in progress doesn't read as a
problem), success ok "Succeeded", failed crit, interrupted warn, cancelled
dim. run_started.html leaves its hard-coded colours for tokens; the run
page's facts move to the .kv list and its failures to var(--red). Template
hex colours: 54 -> 47.
Also fixes#5124's sidebar count, which CI run 8389 failed on: the stylesheet
guard allows no weight above 500, and .nav-count asked for 600.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The Alerts page's active list now says each alert in one line around the
resource it watches, e.g. "cpu_pct on web-1 is 94% (> 90%), firing for 12m",
with web-1 linking to its host, monitor or container page (or the plugin's
page for Traefik, UniFi and SNMP). Values take their unit from the metric
name through formatting.metric_value. A silent series reads "No cpu_pct from
web-1 for 20m"; an up/down metric says the word.
Actions follow the routes' roles: acknowledge, edit, toggle and New rule for
operators and admins, Delete for admins only, nothing for viewers.
The sidebar shows how many alerts need attention beside Alerts, from an
async context processor that skips htmx fragments, logged-out pages and the
database-down page, and renders without the count if the query fails.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
NOW is fixed when the module is imported; by the time the detail page
rendered in CI seven seconds had passed, and |ago rightly said "12m 7s
ago". The page was right; the test's clock was stale.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The Monitors list now opens with a count strip (down, up with how many
degraded, pending, paused) and sorts down, degraded, pending, up, paused,
by name within each. Degraded is up but worth a look: uptime at or below
its warning floor, or the latest response past its critical cutoff, both
from Settings → Thresholds; a slow-but-tolerable reply is coloured, not
sorted.
A monitor had no page of its own: every link went to the list. Now
/monitors/<id> shows its state and since when, latency, uptime for 24h/7d/
30d, TLS days, the last checks, recent failures with their errors, and its
settings (HTTP header names, never their values). Status links to it, and
adding or editing returns to it.
Adding is its own page (/monitors/new, form.html shared with edit) behind
an "Add monitor" button; a refused add re-renders the form with the error
and what was typed instead of a bare 400 text. Row actions are labelled
icon buttons, type names read Ping/TCP/DNS/HTTP (|monitor_type), and the
heartbeat uses pill-up/pill-down, so HEX_CEILING drops to 56.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5120 "Status page: what is wrong first".
- A "Needs attention" card heads the Status board: each down monitor with
how long it has been down, and each active alert (firing, acknowledged,
pending, no data) with its condition and how long, linking to its history.
- StatusEntry.since / since_at_least: when the current up/down run began,
from the recent checks (streak_start); "for at least" when every check we
have agrees.
- Counts as stat tiles (Down, Active alerts, Pending, Up); names wrap
instead of being cut off; dots carry their label.
- Uptime and TLS are coloured by status_class against Settings → Thresholds
instead of literal cutoffs; heartbeat pills and DOWN/UP use tokens and
status_badge rather than hex and bold.
- tests/core/test_status_page.py: streak_start cases, and the rows render
with the problem card before the list, each since-time, threshold tones
and no hex colour.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5094 "Status and layout macros, and form classes".
- formatting.STATES: one state -> (tone, label) map for up, down, stale,
firing, pending, acknowledged, no_data, ok, warn, crit, resolved and
inactive; status_info() reads it (an unmapped state is drawn dim under its
own name). required() lets a macro refuse a missing argument.
- _macros.html: status_badge and status_dot (a dot always carries its label,
sr-only unless shown), stat_tile, empty_state, page_header (actions via a
call block) and icon_button, which raises without a label.
- steward.css: tone classes (badge-/dot-/stat- ok, warn, crit, info, dim),
.sr-only, .page-header, .stat-tile, .empty-state, .btn-icon and the form
classes .field, .field-label, .field-hint and .input. Info text is a 40%
mix (--blue): the house --fs-info-fg falls to 4.18:1 on its tint over a
hovered row, 40% holds 5.09:1.
- First users: the Alerts list, history and maintenance pages (headers,
state badges, empty states) and the alert forms' hints; the inline inp/lbl
style sets on the monitor and host forms become .input / .field-label.
- tests/core/test_macros.py renders each macro through the app environment.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5099 "A name containing a quote runs script when an admin clicks Delete".
Inline handlers like onclick="return confirm('Delete {{ rule.name }}?')" are
HTML-decoded before the JavaScript is parsed, so autoescaping did not stop a
name such as x');fetch(…);// from running.
- Every prompt (22 handlers, plus the docker prune and token revoke ones)
is now data-confirm="…", read through dataset by one submit listener in
base.html; nothing rendered is parsed as JavaScript.
- The remaining inline handlers take integers only (|int), and the time
range buttons pass this.dataset.range.
- tests/test_inline_handlers.py fails on any {{ }} in an inline handler that
is not |int, with a self-test; tests/_templates.py is the one template
walker the template guards now share.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5093 "Shared format filters and status_class driven by Settings → Thresholds".
- steward/core/formatting.py: human_bytes (binary units), human_rate,
human_duration (two largest units), ago (a <time> titled with the absolute
UTC time) and status_class(value, kind, thresholds) -> ok/warn/crit/unknown,
which raises on a kind the thresholds don't name.
- app.py registers them as the bytes, rate, duration and ago filters and the
status_class global; threshold_style is now graded by status_class (and caps
its weight at 500).
- Replaces the copies: docker's _human_bytes/_human_uptime, core alerts'
_duration, the host-agent fmt_bps/fmt_bytes macros, the Traefik and UniFi
byte-rate arithmetic, the swarm-failures ago macro and two uptime spans.
- Tests: a table per helper, and every template compiled against the app's
filters, so an unknown filter fails CI rather than a page.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5066 "Alert history page". /alerts/history lists every alert transition
in a range (1h–30d, the shared range picker), newest first, all rules or
one (?rule=). Each row: from → to, the value against the rule's condition
("no data" for a silent one), email and webhook sent or failed with the
error, "muted · <window>" (and "notified when the window ended") taken out
of the event note, other notes, and a link to the Ansible run an action
started. Range and rule changes swap only the rows (HTMX). Viewer role.
The Alerts page links each rule's history and the whole history. The query
moves to alert_svc.history, which MCP get_alert_history now uses too.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The rule name goes into the Subject header, and EmailMessage refuses a
header holding a line break, so the whole email failed. The web form can't
enter one but MCP takes any string. The subject now folds whitespace.
Found by the new From-fallback test in CI run 8349.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5064 "Notifications that arrive":
- render_webhook: alert.* is escaped to sit inside a JSON string (a quote
or newline in a rule name no longer breaks the body); raw.* is the plain
value for |tojson; a missing value reads "no data"; the template runs
sandboxed and the body is checked to be JSON before it is sent.
- SMTP: 15s timeout; port 465 connects with implicit TLS, otherwise the
box means STARTTLS; a From address setting (falls back to the username).
- A webhook error no longer carries the URL, which is a secret.
- Settings → Notifications: "Send a test email" / "Send a test webhook"
send a sample alert with the form's values (unsaved) and show the result
and the body that went out; template help names alert.* and raw.*.
- Notification audits go through audit_request; the test send is audited
as settings.notification.tested, with the channel and outcome only.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Notifications and alert actions now start only once the evaluating
transaction commits, and never for one that rolls back. record_metrics
queues them on the session (_after_commit), and after_commit/after_rollback
listeners on the sync session spawn or drop them. Spawned tasks are held in a
module set so they cannot be garbage-collected before they run.
A maintenance window no longer skips evaluation. Alerts in its scope still
change state, but their notifications and actions are held, and each event
says so ("notification muted: maintenance window <name>"; also on no-data
and resume events). notify_after_maintenance, run every minute with the
silence check as the new core task alert_reconcile, sends one notification
for an alert still firing after its window ends and runs its action. The
event's note records the release.
Alert actions go through runner.trigger_run like every other run path, so
the playbook and the inventory must be ones the source lists (#5034 guards).
The firing event links to the run, or its note says why the action did not
start. The maintenance page and form wording now say what a window does.
Tests:
- integration: a rolled-back evaluation notifies nobody, and dispatch waits
for the commit; maintenance mutes while the state moves and notifies once
when the window ends; an action whose playbook is outside its source is
refused, with the note set;
- the alert-action test uses a listed inventory name and waits, with a
deadline, for the run the action starts;
- unit: _muted_by.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent offline is an ordinary rule now. The host_agent scheduler records
host_agent/<host>/seconds_since_seen every minute (from last_seen_at, or
registration time for an agent that never reported), replacing the task
that only logged stale agents. The catalog offers the metric, and picking
it on a new rule fills "> 180".
No data: each rule's state tracks its last sample and its usual gap,
smoothed so that an outage counts as at most three gaps. A core task,
alert_silence_check, runs every 60s and marks a rule no_data_since once its
series has had no sample for three usual gaps (at least 5 minutes; an hour
while the cadence is unknown). A never-sampled rule counts from when it
started watching. The mark sits beside the state, so a firing alert that
goes silent stays firing. Each silence records one event and sends one
notification (rule option notify_no_data, default on; muted in maintenance
windows). The next sample clears it with a "data resumed" event and
notification. Creating, changing, disabling or re-enabling a rule restarts
its watch.
Migration 0029 adds the columns and seeds each existing rule's last sample
and gap from its last day of data, so an upgrade does not flag every rule.
Shown on the Alerts page (a NO DATA badge with its start time), in MCP
list_alerts, list_alert_rules and get_host (state "no_data", no_data_since),
in get_overview findings (alert_no_data, or a firing alert marked silent)
and in the weekly report. The rule form has the notify option, and its
metric reference now matches what traefik records and lists host_agent.
Emails say "no data" and carry the event's note.
The metric-catalog guard also reads positional MetricBatch.add arguments.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5061 "Metric catalog belongs to what emits: plugins register their
metrics; names match what is recorded".
- The static METRIC_CATALOG is gone. metric_catalog() is core's own sources
(the four monitor types and "steward", the app's open_fds/open_fds_pct,
which were recorded but never offered) plus each loaded plugin's
get_alert_metrics() export. A disabled plugin's sources leave the picker.
- traefik offers what it records: error_rate_4xx_pct and error_rate_5xx_pct.
The old "error_rate" was never recorded, so a rule on it could not fire.
- tests/core/test_metric_catalog.py walks every MetricBatch.add /
record_metric call (traefik's tuple loop included) and fails on a recorded
metric the catalog lacks; it shows on a synthetic writer that it can fail.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5060 "One evaluation path: record_metrics batch API; host-agent ingest
evaluates rules".
- record_metrics(session, rows) writes metric rows and evaluates the rules
that watch them, reading rules, states and maintenance windows once per
batch. Each rule sees its key's samples oldest first, so an agent catching
up after an outage counts consecutive checks as they happened.
- Host-agent ingest now goes through it. Before, it wrote plugin_metrics
directly and no host_agent rule could ever fire. The catalog gains the
per-mount disk_used_pct ("<host>:<mount>").
- MetricBatch gathers a transaction's samples; the docker ingest, traefik,
unifi, snmp, monitor and self-monitor writers record one batch each instead
of a rule query per sample.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5059 "Alert state machine: acknowledgement holds until resolve; N
consecutive OK to resolve; editing, disabling or deleting a rule resets it".
- The state machine is now next_step(), a pure function of one sample. An
acknowledged alert stays acknowledged while breached; before, the next
breaching sample re-fired it. Firing and acknowledged alerts resolve after
consecutive_ok_required OK checks in a row (new rule column, default 1;
rule form, rules list and MCP create/update_alert_rule clear_after_checks).
- Changing what a rule watches (source, resource, metric, operator,
threshold) or disabling it resets its alert to inactive; a firing or
acknowledged one records a resolved event whose new note says why, and
nobody is notified. Web and MCP edits share alerts.service.apply_rule_change.
- Migration 0028: consecutive_ok_required, consecutive_ok_count,
alert_events.note, and metric_value nullable (a reset has no sample).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5037 "Secrets at rest and in run output: webhook URL and git token
encrypted, askpass via env, secret values scrubbed from logs".
- webhook.url joins SECRET_KEYS; existing values are sealed at startup by
migrate_plaintext_secrets. Settings -> Notifications no longer echoes the
URL: it names the saved host, keeps it when left blank, and has a Remove
box.
- Git source http_token is sealed inside ansible.sources on save and at
startup, opened in get_sources, and counted by the undecryptable-secrets
scan as ansible.sources[<name>].http_token.
- The askpass script holds no token: git runs it with STEWARD_GIT_TOKEN in
its environment and printf prints it as data.
- The executor masks every secret the run is handed (become/vault
passwords, bootstrap passwords, sensitive run vars, and the per-host agent
tokens the host_agent plugin now passes as redact) in each line before it
is broadcast, logged or stored.
- Bundled "Write agent config" tasks are no_log; a test fails if any bundled
task templating steward_token lacks it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5034 "Playbook runs: path and inventory checks in trigger_run, connection
vars admin-only, confirm enforced server-side, web actions audited".
Ansible runs inside the Steward container, so an operator who could name
any playbook path, set ansible_connection=local or ProxyCommand ssh args,
or put {{ lookup('pipe', …) }} in a value could run commands on Steward.
- steward/ansible/guards.py: check_playbook (the source must list it),
check_inventory_scope (steward:* or an inventory file the source has),
check_vars_for_role (ansible_* beyond host/port/user/password/become,
and any Jinja in a value, are admin-only), changed_vars (an edit is
held to the role only for what it changes).
- trigger_run applies the path checks, so every run path — web, MCP,
alerts, schedules, plugins — gets them.
- Variables are checked by role on the web run form, host-page runs,
schedule create/update, inventory group/target create/update, and MCP
run_playbook / create_schedule / update_schedule.
- A steward:confirm playbook needs the confirmation ticked on the server
side for a real (non-check) web run, as MCP already required.
- Audited: runs, cancel, schedule create/update/enable/disable/delete/
run-now, playbook create/update/delete, inventory group and target
create/update/delete (variable names only, never values). One
audit_request helper serves the web routes.
- The CSRF form guard and the route guard now each test that they fail
on a synthetic broken input (rule: a guard must be able to fail).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5032 "Plugin install takes its URL and checksum from the catalog, requires
the checksum, and extracts only inside the plugin folder". The install form
carried the download URL and checksum as hidden fields, so any admin POST
could make Steward fetch and import code from anywhere, unchecked, and a zip
member like ../x.py was written wherever it pointed.
- The route takes only the plugin name and reads download_url and
checksum_sha256 from the configured catalogs; a name no repository lists
is refused. Installs are audited (plugin.install / plugin.install_failed).
- The checksum is required; the name must be a plain identifier.
- extract_plugin refuses any member that resolves outside the plugin
folder, caps the download at 50 MB and the unpacked size at 200 MB.
- The zip unpacks into a staging folder and is swapped in whole: a refused
or failed install leaves the installed copy untouched, and an update no
longer leaves the old version's deleted files behind.
- The catalog shows "Can't install: the catalog lists no checksum" in place
of the button for an entry without one.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5014 "Ansible run scope is shown as a raw id"
- inventory_gen.describe_scope / scope_labels turn "steward:target:<uuid>"
into "target web-1", "steward:group:<uuid>" into "group Docker Swarm",
and a deleted one into "target abcdef01 (deleted)".
- MCP run and schedule rows carry scope_label beside the stored scope; the
ansible.run.started audit detail records it too.
- The web run page and schedules table show the name, with the stored
scope as the tooltip.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
MCP-7 #4971 "Monitoring-config write tools: monitors, alert rules, acks,
maintenance".
The validation the forms did inline (or not at all) moves to the services,
and both the routes and the tools call it, so bad input is refused the same
way with the same reason:
- monitors/service: monitor_fields, build_config, normalise_target. Port
1-65535, status 100-599, timeout 1-60, method GET/HEAD/POST, headers an
object of strings, a key the type does not take is named. The forms now
answer 400 with the reason instead of redirecting silently or a 500.
- alerts/service: METRIC_CATALOG and SOURCE_MODULES, resources_for,
metrics_for, rule_fields (source, resource and metric from the same
choices the rule form offers), add_rule, acknowledge, window_fields.
Tools: create_monitor, update_monitor (config merged over the stored one;
a rename warns about rules still watching the old name),
set_monitor_enabled, create_alert_rule, update_alert_rule,
acknowledge_alert, create_maintenance_window (duration or end),
end_maintenance_window (ends one in progress, cancels an upcoming one) —
operator. delete_monitor, delete_alert_rule — admin. get_alert_rule_options
— viewer. Every write is audited as "<user> via MCP".
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The admin banner counts every plugin load failure, but settings/plugins.html
only rendered failure reasons inside the plugin card macro -- which iterates
discovered plugins. A plugin that failed BECAUSE it has no directory was
therefore counted in the banner and shown nowhere on the page the banner links
to. That dead end is what the operator hit: "1 plugin failed to load" on the
dashboard, nothing wrong on the details page, and no way to clean it up.
Folds the reason into the "Configured but not installed" rows rather than
adding a competing section, and drives that list from the UNION of stored
plugin.* settings and recorded load failures instead of stored settings alone.
A failure with no stored row -- a plugin enabled by a DEFAULTS entry, which is
exactly what plugin.http was -- would otherwise still be counted and still be
invisible.
Rows carry `removable`: only a plugin with a real stored row can be cleaned up,
so a default-declared one shows the reason and an explanation instead of a
Remove button that would delete nothing while the default reasserts it. Rows
with a reason read as an error (red, "Failed to load"); rows that merely have
leftover config stay a warning. A discovered plugin that failed is excluded --
its own card already shows the reason.
Replaces find_orphaned_plugin_names outright rather than adding a second
overlapping helper; it had one caller.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The operator's 500 was NOT the startup gap I attributed it to. They found it by
logging in and getting Steward's own error page -- which means create_app had
completed and the app was serving, so the gaierror came from a request handler
acquiring a connection, not from boot. Neither prior fix covers that: the
startup retry never runs, and pool_pre_ping only helps once the database is
back, since its replacement connect fails too while the container is gone.
Adds a before_request gate that acquires a pooled connection with a short
bounded retry (~1.75s over 4 tries) so a database restart is ridden out
invisibly, and renders a distinct 503 "database unavailable" page when the
budget is exhausted. The request budget is deliberately far shorter than the
startup one: nobody watches a container boot, but somebody is watching this
page load, and a page that hangs is worse than one that says what is wrong.
/health and static are exempt -- a liveness probe must stay answerable while
the database is down, or a dependency outage triggers a restart loop.
Credential and missing-database errors still propagate rather than being
reported as "unavailable", which would send the operator chasing the wrong
problem entirely.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Operator saw the `http` plugin -- deleted when ping/dns/http were unified into
the Monitor entity -- still reported as enabled, with no way to clear it.
It was never an orphaned row. `plugin.http` was hardcoded in core DEFAULTS, so
deleting the app_settings row did nothing: the default reasserted it on the
next settings load. That is precisely why no cleanup UI could have fixed it.
Rule 22 says the removed subsystem should have taken its setting with it.
Purged the surviving references -- the DEFAULTS key, "http" in
CAPABILITY_PLUGINS, the stale plugin_manager docstring example, and the
capabilities blurb still advertising HTTP/uptime as a bundled capability --
plus a migration dropping any stored plugin.http row. Untouched: `http` as a
MONITOR TYPE (icmp/tcp/dns/http) everywhere it appears, and http_001_initial,
which is kept deliberately so existing DBs resolve the revision graph.
For the general case, cleanup splits by provenance rather than being uniformly
automatic or uniformly manual:
Bundled plugins ship in the image and version atomically with core, so they
cannot be transiently missing -- a plugin.* default with no bundled directory
is unambiguously a bug. Guarded by a unit test that fails CI, which is the
only part safe to automate.
External plugins live in operator-mounted /data/plugins, where absence is
ambiguous: unmounted volume, failed install, mid-upgrade. Auto-deleting their
config would silently destroy unrecoverable credentials on a transient
condition, so Settings now lists them under "Configured but not installed"
with an explicit, confirmed, audit-logged Remove. The section hides entirely
when empty, and the remove route refuses if the plugin is actually installed.
Orphan detection reads STORED rows, never the DEFAULTS-merged view -- offering
to remove a default-only key would be a lie -- and returns names only, since
plugin config can hold credentials and listing orphans never needs their values.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two connection-recovery gaps, both surfaced by a Postgres restart that left
the app throwing tracebacks while the DB itself was healthy.
pool_pre_ping + pool_recycle on the app engine [#2626]: when the database
restarts, every connection already in the pool is dead at the socket level.
SQLAlchemy only discovered that by failing a real query, so the first
operation after a restart errored out on whatever triggered it. pre_ping
checks liveness on checkout and swaps the dead connection transparently;
pool_recycle caps connection age so a socket stranded by a NAT/conntrack
timeout or a Docker network rebuild is retired on a timer instead.
wait_for_database() gate at startup [#2627]: create_app touches the DB
synchronously (migrations, secret re-encryption, settings load) and assumed
it was both resolvable and accepting connections on the first try. Neither
holds after a host reboot (Docker DNS not yet serving `db` -> gaierror -2)
or an unclean shutdown (Postgres still replaying WAL -> "not yet accepting
connections"). Both are transient, so retry with capped backoff behind one
gate ahead of the first DB touch. Credential and missing-database errors
are classified by SQLSTATE and still fail immediately -- waiting cannot fix
those. Budget is bootstrap-configurable (STEWARD_DB_CONNECT_TIMEOUT /
database.connect_timeout, default 60s) since it governs reaching the DB and
so cannot live in the DB-backed settings.
Tests drive the retry loop off a fake clock, so backoff and timeout
behaviour are deterministic rather than wall-clock dependent.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
M78 step 1. The bundled maintenance/docker_prune.yml was a single
`docker system prune -f`; the upcoming per-host prune buttons need a
granular split. Add a `prune_target` extra-var:
- containers → docker container prune -f
- images → docker image prune -f (+ -a when prune_all_images)
- system → docker system prune -f (+ -a / --volumes) — the DEFAULT,
so existing manual/scheduled callers (which set no var)
keep today's behavior.
Reuses the existing bundled playbook rather than adding parallel files;
keeps the description/category/confirm meta so it still self-describes as
a confirm-gated maintenance run. Adds unit coverage: the playbook stays
valid YAML, its meta still discovers, and all three targets are guarded.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CAGR73DUowdVFVvYzLXC5C
Guardrails so the fd-leak class of bug (Errno 24 lockup; recent SNMP + UniFi
fixes) surfaces early or can't compound, instead of silently killing the app.
- Self-fd watchdog (steward/core/self_monitor.py): records open_fds and
open_fds_pct (% of soft RLIMIT_NOFILE) as "steward"/"process" metrics each
minute through the normal alert pipeline, so the operator can alert on them
via the existing alert-rules UI. Built-in WARNING floor at 80% gives a
zero-config early signal. Stdlib-only (/proc + resource); degrades to a no-op
off Linux. Registered as a core ScheduledTask in app.py.
- Poll-overlap guard (steward/core/scheduler.py): extract a pure _DueTracker
that skips a tick while a task's prior run is still in flight, so a hung poll
can't stack overlapping runs (which amplify per-poll resource/fd use). A
skipped task isn't penalised — it retries the next tick after it completes.
- fd-stability tests (tests/core/): _DueTracker overlap policy, the watchdog
metric/warning/degradation paths, and a real-fd canary that hammers tcp_check
and asserts /proc/self/fd doesn't grow.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
plugin_metrics grows by (sources × resources × ~30s cadence); keeping 90d of raw
is a large table. Add a raw→hourly rollup (mirroring the Docker plugin) so only a
short raw window is kept at full resolution, with hourly averages archived longer.
- PluginMetricHourly model + core migration 0024 (plugin_metrics_hourly: avg/max/
count per source/resource/metric/hour, unique bucket constraint + lookup index).
- steward/core/metrics_retention.rollup_plugin_metrics: date_trunc('hour') agg of
raw older than the hour-aligned raw window, idempotent pg upsert into hourly,
delete the rolled raw, prune hourly beyond the rollup window.
- cleanup.py: plugin_metrics is no longer blanket-deleted at data.retention_days;
_run_metrics_retention drives the rollup with windows read live from settings.
- Settings: metrics.retention.raw_days (7) + rollup_days (90), tunable on the
Thresholds & Retention page (new "Host metrics retention" card).
- Chart read: _history_for_host merges the hourly rollup (older part of the range)
with raw date_bin (recent part, capped ≤1h), so 30d charts keep working —
recent at full resolution, older at hourly. Route passes raw_days from settings.
- Tests: unit (cutoff helpers) + integration (rollup aggregates/prunes; history
merges hourly + raw) against Postgres.
Speed was already handled by the indexes + SQL aggregation; this is the storage
lever (raw window ~10x smaller).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC
Plugin UIs had no nav home (only dashboard widgets / typed URLs). Add an
optional get_nav() plugin export and surface it in the sidebar.
- plugin_manager: _PLUGIN_NAV registry + get_plugin_nav() getter + a tolerant
_collect_plugin_nav() (missing hook = fine; raising/malformed = logged &
skipped; idempotent per plugin so hot-reload re-runs cleanly). Collected in
both the startup load path and the hot-reload path.
- app.py: inject plugin_nav into the template context, filtered to enabled
plugins so a hot-disabled plugin's link can't linger before restart.
- base.html: render plugin links under the Infrastructure group, with the same
request.path active-state treatment as core links.
- docker/snmp/unifi/traefik: each exports get_nav() → its /plugins/<name>/ view.
- Tests: collector behavior (collect, missing hook, reload-replace, malformed
item, raising hook, sorted output).
With docker enabled, "Docker" now appears under Infrastructure → its fleet view.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC
