#5467 "Steward's own health: metrics, rules can watch them, a page that shows them".
The self-monitor tick now records, under source "steward", resource "process":
ingest_posts / ingest_failed / ingest_slowest_ms (the host-agent posts since the
last tick, tallied in steward/core/health.py), db_roundtrip_ms (a timed SELECT 1),
and unwatched_secs (the gap before this tick past watch.GAP_S, from Steward's own
last sample). The hourly cleanup records cleanup_secs. The open-file pair is still
recorded, and is now only skipped on its own where /proc is absent. All of these
are in CORE_METRICS, so an alert rule can watch them under "Steward itself".
Settings → System (/settings/system/, admin) shows a verdict first (failed posts,
a gap in watching, open files near the limit), then each measure now and over the
last day, with a sparkline scaled to its own peak.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5475 "Rule form: pick the host, then its part…". CI run 8909 caught it:
moving from the whole host to a mount kept "CPU" in the mount's Metric list,
because a chosen metric always stayed on offer. Only a rule's own page keeps
its resource and metric after they go quiet (keep=True from edit_rule); the
pickers' swaps drop a metric the new part doesn't report.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5475 "Rule form: pick the host, then its part; metrics that part reports;
a link to groups; a preview of the metric", from the operator's review of
the New rule page: hosts and their parts shared one long picker.
- Host agent rules pick a Host, then a Part: "The whole host", then its
mounts, CPU cores, disks, interfaces and sensors under a heading each,
cores in number order. A part that reports nothing a rule can watch
(an interface's traffic) isn't offered.
- The Metric list holds what the picked resource reports, in catalog order.
- Under it, a preview: the reading now, and the last 24 h low to high with
an hourly sparkline drawn from zero (UI ruling 8), read from one series.
- The group picker links to Ansible > Inventory to make a group, in a new
tab, and re-reads the groups when the tab comes back into view.
- Picking a host or part swaps the pickers alone (/alerts/api/rule_target);
picking a metric swaps the preview (/alerts/api/metric_preview).
- A rule's own resource and metric stay on offer after they go quiet.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5473 "Opening an alert rule takes up to 15 s: the Resource picker scans
every raw host-agent sample". The access log showed the edit page at 6-15 s
(one load gave up) while /alerts/ took 0.2 s. resources_for ran SELECT
DISTINCT over plugin_metrics, and Postgres has no skip scan, so it read every
host-agent sample; every rule page, source swap, rule save and MCP rule call
paid it. resources_for and metrics_for now read plugin_metric_latest, one
row per series, which lists the same names. A test fails if either reads
plugin_metrics again.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5466 "Send test: check a rule's delivery from the rule". A rule's page gets
a Send test button (operator role), and MCP gets send_test_alert. Each sends
a TEST alert for that rule now, by email to everyone who gets alerts and by
webhook. The alert carries the rule's name and condition, at the latest
reading of what it watches; a rule covering hosts uses its first host. The
result for each channel shows under the button, or "Nothing is set up to
send to". Nothing fires, no event is written, and the test is audited as
alert_rule.tested. An alert with no reading now says "has no reading yet"
instead of "is None".
Docs: alerting.md also gains the burst email from #5465 "One email per burst
of alerts".
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5464 "Group rules: one rule covers every host, or every host in a group",
step 2 of milestone 499 "Alerts you can trust". Each rule named one host, so
the live instance carried six rules for three hosts, and a new host had no
agent-offline or disk alert until someone copied them.
- alert_rules.covers / cover_group / parent_id (migration 0041). A rule on
host_agent may cover "all_hosts" or a "group" (an inventory group: the
Hosts ruling, #5294). A covering rule has no state; Steward keeps one
ordinary child rule per covered host, named "<rule>: <host>", so each
host's alert keeps its own state, history and acknowledgement.
- alerts.service.reconcile_children: a covered host that has reported in
the last 7 days gets a child; a host no longer covered loses its child;
children take the parent's condition, checks, notify setting, action and
on/off. Run on save and each minute (alert_reconcile).
- A child is changed through its parent: edit, toggle and delete of a child
are refused (409 / PartOfCoveringRule); its edit page opens the parent.
- Rule form: a Covers choice for host sources, with the group picker; fixed
once the rule exists. Rules list: a covering rule reads "every host in
swarm · 3 hosts" with its hosts' rules nested under it, each opening its
history. The history picker and the rule count leave covering rules out.
- MCP: create_alert_rule takes covers/group; update_alert_rule takes group
and passes changes down; list_alert_rules shows covers, hosts and part_of.
- docs/core/alerting.md: "Rules that cover hosts".
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5463 "Acknowledgements on record: an event and who did it", step 1 of
milestone 499 "Alerts you can trust". The live walk in #5461 found that
acknowledging changed only the alert's current state: once it resolved,
nothing said who had acknowledged it or when.
- alert_events.actor_id (migration 0040): the person behind a transition.
- Acknowledging writes a firing → acknowledged event by the acknowledger.
- A rule edit, disable or enable that resets an alert carries its actor
(web and MCP alike).
- The history page notes "acknowledged by X" and "rule disabled by X"; MCP
get_alert_history returns `by`.
- One usernames() helper in alerts.service, used by the active list, the
history page and the MCP tools.
- docs/core/alerting.md: the state diagram no longer shows an acknowledged
alert re-firing, and says what an event records.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Milestone 495 "Email goes to each account, in its own time zone", on the
operator's ruling (System 156): notifications go to each user, times in
their zone, defaults on, toggled per user or per role.
- users.notify_alerts / notify_report (migration 0036): None follows the
role's default, notify.<kind>.<role> in settings, all on to start.
- mail.pick_recipients: active accounts that want the email and have an
address mail can reach (an SSO placeholder like name@oidc is skipped and
flagged), each in its own zone with no UTC beside it, then the Also send
to addresses in their chosen zone with UTC beside. One copy per address,
all over one SMTP connection.
- Alerts, the weekly report and the test email (to you, in your zone, plus
the typed addresses) send that way. The save-time smtp.tz capture is gone.
- Notifications: a role × email table of defaults, who each email reaches
and who can't be reached, and Also send to with its own zone, which starts
as the first admin's to open the page.
- Account page: your address and your two switches. Users: an Emailed
column, and an admin sets an account's switches on its page.
- The Alerts page counts the accounts it emails. zone_select macro and
formatting.zones_by_region shared by Account and Notifications.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5335 "Emails Steward sends read as designed messages: an HTML alert email
and weekly report, with a plain-text twin".
- core/mail.py renders each email as multipart/alternative from
templates/email/: a table-and-inline-style HTML part and a plain-text
twin carrying the same facts. Colours are the design system's light
theme, resolved from theme.css; nothing is fetched when a mail opens.
- Alert email: state badge, a headline sentence ("Disk used on web01 is
94%."), rule, resource, reading, the condition in words, and the time in
the zone Notifications was saved in with UTC beside it; "View in
Steward" when Steward's address is set. Subject "[Steward] Firing: <rule>
— <resource> 94%". Alerts now carry their operator.
- Weekly report: monitors down, active alerts, the week's alert events,
uptime and busiest Traefik routers as small tables with links. Sent from
the configured From address; the "Generated … UTC" line and the stale
repo footer are gone.
- condition_text() states a condition from a rule's parts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
History (#5316):
- Transitions fold into episodes (steward/alerts/episodes.py): each stretch
out of quiet, the worst state it reached, how long it lasted, its worst
reading against the rule ('3m 21s (fires above 3m)'), who was told across
its changes, and its Ansible runs. The changes themselves fold under it.
- An episode cut by paging says so ('cleared later (above)', 'earlier').
- For one rule: a verdict ('Fired 4 times in the last 7d, 1h 2m in all') and
no Rule column. Notified and Action show only when the range has any.
Tests:
- The confirm guard looks for a confirm( call, not the word in the comment
that says it is gone (lesson #4365).
- The Hosts viewer test looks for the add-host dialog by id, since every page
now carries the confirm dialog.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Confirm dialog (#5314):
- Every data-confirm form now asks in one native <dialog> (base.html + static/confirm.js)
instead of window.confirm.
- The question becomes the heading and what follows it the detail; the act's
button is named after the form's own button; Cancel has the focus and Esc
backs out. Optional data-confirm-title/-detail/-action/-dismiss attributes.
- All 27 existing uses work as they are. Cancelling a run says 'Keep running'.
Alerts list (#5315):
- A verdict line (what is firing, how many rules, where notifications go,
or 'notifies nobody yet' with a link) replaces the empty Active card.
- Each rule's name opens it; rows show 'last fired' and fold fire/clear into
the condition; History and Disable stay; Delete leaves the list.
- .docker-verdict becomes the shared .page-verdict.
Rule page (#5317):
- Named for the rule (title and crumb), with when it last fired and how many
changes of state it holds.
- Three sections: what it watches, when it fires (checks side by side,
seconds echoed as a duration), what happens (where notifications go, no-data,
the playbook, whose fields stay folded until ticked).
- Delete at the foot, behind the dialog, saying its history goes with it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5212 "Plain labels: no raw ids, scopes, metric keys or unexplained jargon",
batch 1:
- Audit log: each action reads as a sentence ("paused monitor"), with the
code as its tooltip. Detail shows flags as yes/no, lists joined, a run's
scope by name, spans as spans, and no ids the entity column already gives.
The filter is now a picker of areas.
- Alert rules, history and Status say the condition in words: "is down",
"Fullest disk >= 85%". Readings use metric_value, and history times use
|ago.
- Rule form: sources and metrics by name, the threshold's unit, "fires
when it is down/up" for an up/down metric instead of 0.5, and a metric
reference built from the catalog.
- Schedules and Maintenance: viewers no longer see Run now, Pause, Edit,
Delete or New. Intervals and next runs read as "every 2h" and "in 3h";
a scope reads "Every alert on Host agent".
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5211 "Lists load as you scroll: audit, alert history, Ansible runs"
(milestone 477), per preference 172.
- core/paging.py: keyset paging over (time, id), newest first, with an
opaque "<iso>|<id>" cursor; a page reads one row past itself to know
whether another follows. No OFFSET, so rows added while someone scrolls
never shift the pages after them.
- _macros.html scroll_more / scroll_end and static/scroll-more.js, written
once: an IntersectionObserver with a 600px margin fetches the next page
before the reader reaches the end; "Loading more…" while in flight,
"End of the list." at the end, and a failed page says so with a Retry
button instead of re-firing on every scroll.
- Audit log: no "Load more" link and no 500-row cap.
- Alert history: no 200-row stop with "pick a shorter range";
alert_svc.history takes the cursor, so the MCP tool and the page share
one query.
- Ansible runs: no silent cap of 50.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5210 "Problems first, with since-time, across every list" (milestone 477),
second batch.
- Host page: an "Alerts on this host" strip first (rules on its agent
series, its containers or its monitors: alert_svc.watches_host); its
monitors problems first with "Down 12m · reason"; Docker above Ansible.
- Uptime page and widget: worst 24h uptime first, then 7d and 30d; hosts
with no uptime yet last.
- Agent panel and fleet widget: last seen as "3m ago", not a clock time.
- SNMP: a device is "Reachable" only while it answered within the last 3
polls; older readings say "No answer" with their age, and every device
says when it was last read (was: reachable whenever any reading existed).
- Traefik: a router missing from 3 scrapes in a row is listed apart as no
longer reported, by when it was last seen, instead of keeping its last
state (a removed router stayed Failing).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5210 "Problems first, with since-time, across every list" (milestone 477),
first batch.
- Alerts: the Active list reads firing, no data, acknowledged, pending, the
longest-running first, and re-polls itself. An acknowledged alert says how
long it has been breaching (from the alert's events; acknowledging moves
last_transition_at) and, apart from that, who acknowledged it and when.
Rules show what each is doing now; disabled rules sort last. The current
value reads the latest-value table (#5208) instead of plugin_metrics.
- Monitors: a down monitor says "Down 12m · connection refused" in the list.
- Hosts: sorted down, agent gone quiet, pending, healthy. The agent column
shows "—" for a missing reading (was 0%) and a Stale badge with its age.
agent_metrics_by_host is gone; the list reads agent_overview_by_host.
- Schedules: failing first (failed or interrupted run, or a firing that could
not start one), then running, then the rest, paused last; status badge
from the shared states.
- Maintenance windows: active (ending soonest), then upcoming (starting
soonest), with expired folded away; each says when it ends/starts/ended.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The Alerts page's active list now says each alert in one line around the
resource it watches, e.g. "cpu_pct on web-1 is 94% (> 90%), firing for 12m",
with web-1 linking to its host, monitor or container page (or the plugin's
page for Traefik, UniFi and SNMP). Values take their unit from the metric
name through formatting.metric_value. A silent series reads "No cpu_pct from
web-1 for 20m"; an up/down metric says the word.
Actions follow the routes' roles: acknowledge, edit, toggle and New rule for
operators and admins, Delete for admins only, nothing for viewers.
The sidebar shows how many alerts need attention beside Alerts, from an
async context processor that skips htmx fragments, logged-out pages and the
database-down page, and renders without the count if the query fails.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5066 "Alert history page". /alerts/history lists every alert transition
in a range (1h–30d, the shared range picker), newest first, all rules or
one (?rule=). Each row: from → to, the value against the rule's condition
("no data" for a silent one), email and webhook sent or failed with the
error, "muted · <window>" (and "notified when the window ended") taken out
of the event note, other notes, and a link to the Ansible run an action
started. Range and rule changes swap only the rows (HTMX). Viewer role.
The Alerts page links each rule's history and the whole history. The query
moves to alert_svc.history, which MCP get_alert_history now uses too.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent offline is an ordinary rule now. The host_agent scheduler records
host_agent/<host>/seconds_since_seen every minute (from last_seen_at, or
registration time for an agent that never reported), replacing the task
that only logged stale agents. The catalog offers the metric, and picking
it on a new rule fills "> 180".
No data: each rule's state tracks its last sample and its usual gap,
smoothed so that an outage counts as at most three gaps. A core task,
alert_silence_check, runs every 60s and marks a rule no_data_since once its
series has had no sample for three usual gaps (at least 5 minutes; an hour
while the cadence is unknown). A never-sampled rule counts from when it
started watching. The mark sits beside the state, so a firing alert that
goes silent stays firing. Each silence records one event and sends one
notification (rule option notify_no_data, default on; muted in maintenance
windows). The next sample clears it with a "data resumed" event and
notification. Creating, changing, disabling or re-enabling a rule restarts
its watch.
Migration 0029 adds the columns and seeds each existing rule's last sample
and gap from its last day of data, so an upgrade does not flag every rule.
Shown on the Alerts page (a NO DATA badge with its start time), in MCP
list_alerts, list_alert_rules and get_host (state "no_data", no_data_since),
in get_overview findings (alert_no_data, or a firing alert marked silent)
and in the weekly report. The rule form has the notify option, and its
metric reference now matches what traefik records and lists host_agent.
Emails say "no data" and carry the event's note.
The metric-catalog guard also reads positional MetricBatch.add arguments.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5061 "Metric catalog belongs to what emits: plugins register their
metrics; names match what is recorded".
- The static METRIC_CATALOG is gone. metric_catalog() is core's own sources
(the four monitor types and "steward", the app's open_fds/open_fds_pct,
which were recorded but never offered) plus each loaded plugin's
get_alert_metrics() export. A disabled plugin's sources leave the picker.
- traefik offers what it records: error_rate_4xx_pct and error_rate_5xx_pct.
The old "error_rate" was never recorded, so a rule on it could not fire.
- tests/core/test_metric_catalog.py walks every MetricBatch.add /
record_metric call (traefik's tuple loop included) and fails on a recorded
metric the catalog lacks; it shows on a synthetic writer that it can fail.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5060 "One evaluation path: record_metrics batch API; host-agent ingest
evaluates rules".
- record_metrics(session, rows) writes metric rows and evaluates the rules
that watch them, reading rules, states and maintenance windows once per
batch. Each rule sees its key's samples oldest first, so an agent catching
up after an outage counts consecutive checks as they happened.
- Host-agent ingest now goes through it. Before, it wrote plugin_metrics
directly and no host_agent rule could ever fire. The catalog gains the
per-mount disk_used_pct ("<host>:<mount>").
- MetricBatch gathers a transaction's samples; the docker ingest, traefik,
unifi, snmp, monitor and self-monitor writers record one batch each instead
of a rule query per sample.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5059 "Alert state machine: acknowledgement holds until resolve; N
consecutive OK to resolve; editing, disabling or deleting a rule resets it".
- The state machine is now next_step(), a pure function of one sample. An
acknowledged alert stays acknowledged while breached; before, the next
breaching sample re-fired it. Firing and acknowledged alerts resolve after
consecutive_ok_required OK checks in a row (new rule column, default 1;
rule form, rules list and MCP create/update_alert_rule clear_after_checks).
- Changing what a rule watches (source, resource, metric, operator,
threshold) or disabling it resets its alert to inactive; a firing or
acknowledged one records a resolved event whose new note says why, and
nobody is notified. Web and MCP edits share alerts.service.apply_rule_change.
- Migration 0028: consecutive_ok_required, consecutive_ok_count,
alert_events.note, and metric_value nullable (a reset has no sample).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
MCP-7 #4971 "Monitoring-config write tools: monitors, alert rules, acks,
maintenance".
The validation the forms did inline (or not at all) moves to the services,
and both the routes and the tools call it, so bad input is refused the same
way with the same reason:
- monitors/service: monitor_fields, build_config, normalise_target. Port
1-65535, status 100-599, timeout 1-60, method GET/HEAD/POST, headers an
object of strings, a key the type does not take is named. The forms now
answer 400 with the reason instead of redirecting silently or a 500.
- alerts/service: METRIC_CATALOG and SOURCE_MODULES, resources_for,
metrics_for, rule_fields (source, resource and metric from the same
choices the rule form offers), add_rule, acknowledge, window_fields.
Tools: create_monitor, update_monitor (config merged over the stored one;
a rename warns about rules still watching the old name),
set_monitor_enabled, create_alert_rule, update_alert_rule,
acknowledge_alert, create_maintenance_window (duration or end),
end_maintenance_window (ends one in progress, cancels an upcoming one) —
operator. delete_monitor, delete_alert_rule — admin. get_alert_rule_options
— viewer. Every write is audited as "<user> via MCP".
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
MCP-3 of milestone 461 "MCP server: Steward as an LLM-inspectable,
LLM-operable hub". The /mcp tools run outside Quart's request pipeline, so
the queries they share with the pages now live in plain functions that take
an open session:
- steward/hosts/service.py: uptime, status, monitors, agent metrics,
agent overview (stale threshold passed in, not read from current_app),
CPU sparks.
- plugins/docker/service.py: bucket, container_history, host_map,
list_containers, container_events, query_logs.
- steward/alerts/service.py: active_alerts (also used by core/reports),
list_rules, list_maintenance_windows.
- steward/monitors/service.py: monitor_rows (was routes._rows_data).
Routes are repointed and render the same data. One behavior fix rides
along: the container-log search now escapes % and _ so they match
literally instead of acting as LIKE wildcards.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
