Run 9093: test_every_logged_action_is_worded wanted the three
prometheus.template_* actions in ACTIONS, and the list's verdict wrote
"doesn't" as a Jinja string literal, which autoescape turns into '.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Prometheus gains a "Widget templates" tab beside Targets. It lists the
built-in templates and an admin's own, any that no longer pass the checker
first. A template opens as its YAML beside a live preview drawn over a
chosen target exactly as the dashboard widget draws it (routes.draw, now
shared by both). An admin writes or pastes a new one, reads a .yaml upload
into the editor to look over, duplicates any template under a free name,
edits and deletes their own; anyone can export the YAML. Built-in
templates are read-only.
Custom templates are rows in prometheus_widget_templates (migration
prometheus_002_templates), kept in backups, keyed "custom:<id>" so a
rename keeps its widgets. Saving needs a template that passes the checker
and a name no other template has. Create, update and delete are audited
and link to the template.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5691 "Generic Prometheus dashboard widget: pick a target and a template",
step 3 of milestone 508 "Prometheus widget templates".
- /plugins/prometheus/widget draws one target with one template in widget
shape: the verdict (led by "isn't answering" when the target is down),
then each block with problems first, figures beside a share bar or the
last 24h of bars, and the healthy rest folding to "+N more healthy" past
six. A missing target or template, or one that no longer checks, says so.
- library.py: bundled templates from plugins/prometheus/widgets/*.yaml,
the picker's target and template lists, and the default title.
- widgets/fabled_archives.yaml: the first bundled template, against the
series Archives reports (refined in #5693).
- Dashboard: a widget setting may take `options_from` (read when the picker
opens) with an `empty` sentence; `default_title`; a Details link may name
settings ("/plugins/prometheus/targets/{target}"). Prometheus is grouped
with the integrations.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5690 "Widget template format: YAML schema, validator and renderer", step 2
of milestone 508 "Prometheus widget templates".
- template_format.check(text) -> Template, or TemplateError with every
problem, each with its line and what to change. Blocks: states, values
(of -> share and bar, thresholds as numbers or shares, sparkline),
times (warn_after / crit_after), table (two labels pivoted), info.
Anchors/aliases refused, 64 KB cap, YAML's on/off/yes/no flagged.
- template_view.render(template, target, latest, now, history) -> View:
a verdict naming the worst problem, then each block with its problems
first (crit, warn, ok, then quiet states). Missing series show as "—"
and say so; a template whose `expects` prefix the target lacks says so.
- shown_value moves to service.py so the renderer can use it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5689 "Prometheus target page: fold state sets, show epoch times, size
bytes", step 1 of milestone 508 "Prometheus widget templates".
- A state set (name ending in _state, a state label, every series 0 or 1;
or the OpenMetrics form, the label named after the metric) reads as one
row per thing it describes, naming the current state, the others on
hover. fabledarchives_jobs{state} is a count and stays as it is.
- label_pairs / state_label / fold_states in service.py, for the widget
templates to reuse.
- After a good scrape, series it no longer has lose their newest value
(forget_gone): Archives' old build_info row read as current beside the
new one. History stays; a failed scrape forgets nothing; alert rules keep
their own last-sample time.
- Epoch times and _bytes sizes were already read by shown_value /
metric_value.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- The audit-wording guard now reads both branches of a conditional action
("backup.made" if ok else "backup.failed"), and its self-test covers that.
- The Backups save test no longer asserts the session's tz, which sign-in
state sets, not the test.
- The restore integration test uses the app's own metadata. all_metadata
without plugin dirs imports the `plugins` package as the tests do; loading
by path under the loader's names is only for `steward restore`. Loading by
path in the shared test process had mapped traefik and unifi tables a
second time for later tests.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5673 "S3 client and the backup job: settings, schedule, history, metrics":
- steward/backup/s3.py: SigV4 over httpx, path-style; put and list only
(an append-only key is enough; retention is the bucket's). Signing is held
to AWS's published examples.
- steward/backup/job.py: a 15-minute tick. The daily backup goes at the hour
chosen on the reader's clock; a missed day is made up, and a failure is
retried an hour later. History of the last 30 runs, audit
(backup.made / backup.failed), and backup_age_secs / backup_bytes under
"Steward itself", resource backup, for alert rules.
- Settings → Backups: verdict first, then the bucket (secret sealed; a
blank box keeps it), the schedule, Test the bucket, Back up now, and the
recent runs.
#5675 "Docs: backing up to Archives, and restoring": docs/core/backups.md.
Also: the audit page now words the Prometheus target actions from #5668,
which showed as codes, and a guard fails on any logged action without
wording.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Run 9075: the restore got through schema and data, then all_metadata's
`import plugins` failed in the image, where plugins/ is a directory the app
loads by path (PLUGIN_DIRS), not an installed package. It now loads each
plugin under the loader's own module name from the bootstrap's plugin
dirs, which `steward restore` passes. A plugin whose models are already
mapped under either name is skipped rather than mapped twice.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Run 9074's smoke restore stopped at "SET transaction_timeout = 0": pg_restore
17 sends it, and the Postgres 16 server doesn't know the setting. Each
section is now written as a script (pg_restore --file=-), that one line is
dropped, and psql -v ON_ERROR_STOP=1 runs the rest, so any other error
still stops the restore. If psql stops, pg_restore is killed rather than
left blocked on its pipe.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The backup file (#5672 "Backup file: pg_dump of config + audit + alert
history, the secret key and a manifest"): steward-<UTC>.tar holding a
pg_dump --format=custom with metrics', check results' and Docker/Traefik/
UniFi rows left out, the secret key, and a manifest. Every table is named
KEPT or EXCLUDED in steward/backup/contents.py, and a guard test fails on a
new one. The image gains postgresql-client-17 (approved, System 180).
The restore (#5674 "Restore: `steward restore` and a CI test that a dump
brings Steward back"): pg_restore in its three sections. Between data and
constraints, a kept row pointing at a left-out row gets what its foreign
key's ON DELETE says (alert_events.ansible_run_id is cleared). It refuses
a non-empty database unless --wipe, and puts the backup's key in
/data/secret.key. Migrations run on Steward's next start.
`steward` stays the server command: it is now a click group that serves
when given no subcommand.
The publish smoke step backs up the booted image's database, restores it
into a second one, checks a marker setting came back and plugin_metrics
is empty, and starts Steward on it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The UniFi password sat in plugin.unifi in plain text: SECRET_KEYS covers
whole settings, and a plugin's settings are one dict. Saving the plugin's
settings page also wiped it, since the password box is never filled back
in and its blank value was stored.
- A plugin field whose name holds password, secret or token is a
credential (is_plugin_secret). set_setting seals it inside the dict,
every read opens it, startup seals any plaintext left from before, and
the undecryptable-secrets banner names it as plugin.<name>.<field>.
- The settings page uses the same test for its password boxes, and a blank
box keeps the saved value.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- tests/test_breadcrumbs.py: the Prometheus target list is its section's
own top page, like SNMP's, so it carries no trail.
- The integration test's exposition has label braces, which str.format read
as fields: the fake scrape raised, the target was recorded down and no
series landed. It substitutes the counter with replace() now.
- docs/plugins/prometheus.md: adding a target, filtering, how series are
named and stored, alerting, and the Fabled Archives recipe.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A bundled prometheus plugin, on by default like the other protocol plugins.
Targets are rows edited on its page: address, bearer token (encrypted),
interval, keep/leave-out name patterns and a series cap. Steward does the
filtering; the endpoint serves everything (Archives' ruling).
- A 5 s tick scrapes due targets at once, each within a 30 s deadline, and
writes the outcome to its row and its series through record_metrics, so
alert rules can watch any of them.
- Gauges are stored as they are; counters (and a histogram's or summary's
_sum/_count) as <name>_per_sec rates from the second scrape. Buckets and
quantiles are skipped. A labelled series is stored under
"<target>:k=v,…". Each target also records is_up, scrape_ms, series_kept
and series_capped.
- Pages: targets not answering first, with why; a target's series grouped
by metric with their newest values; a chart per series; the editor with
Test, which scrapes once and says what would be stored.
- The counter-to-rate keeper moves to steward/core/counter_rates.py; SNMP's
RateKeeper now scales its rate on top of it.
- metric_value reads Prometheus's _seconds as a span and _per_sec as a rate.
- The catalog guard treats an empty list as a dynamic source (snmp,
prometheus), as the rule form does.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
steward/core/prometheus.py parses the 0.0.4 exposition into families keyed
by name, each with its declared TYPE, so a scraper can tell a counter from a
gauge. A histogram or summary owns its _bucket/_sum/_count samples. Label
values are unescaped, and commas or braces inside quotes stay part of the
value. NaN and +/-Inf are read, timestamps are ignored, and a malformed
line is skipped. Traefik's parse_prometheus is now a thin reshaping of it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
At 20:25-20:39Z the NFS storage under steward_db slowed badly (checkpoint
sync 4.6 s, then 70.7 s with one file at 21.9 s, against the usual 0.002 s).
Ingest commits stalled past the agent's 10 s timeout and were dropped, and
last_seen_at, written in that same transaction, went with them. Agents
posting every few seconds read as silent for 3 minutes, and "Agent offline"
fired and emailed for all three hosts.
Ingest now notes each agent's arrival in memory right after the (read-only)
registration lookup, before any write; seconds_since_seen counts from the
later of last_seen_at and that arrival.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Under an hour |ago says "12m 3s ago", so the test looks for the minutes
alone; the reading cell drops an empty class.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The Alerts widget (/alerts/widget) is shaped like every widget: a verdict
("2 firing, 1 silent." / "Nothing firing: all 12 rules are quiet."), then
what is firing or silent with its reading now, what the rule fires on and
since when, then what someone acknowledged and who; a line when alert
email failed in the last 7 days or is retrying (outbox.alert_trouble),
linking admins to Settings -> System; the quiet rules counted, a covering
rule once. It sits beside the Overview in the recommended layout (6x4
each).
The Overview's "+N more" line pointed at the removed /status/; alerts now
link to /alerts/ and monitors to /monitors/.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5509 "Delivery you can see: each alert's email state in history, and
outgoing email on Settings -> System" (milestone 501).
- Alerts -> History: an alert's email reads from the outbox (#5506):
"gathering" with when it sends, "sending", "retrying" with the attempt,
next try and the server's last answer, "sent", or "failed" naming each
reader it didn't reach and why (outbox.delivery / email_states). Events
from before the outbox keep what they recorded. The Notified column
shows once an alert has a queued email.
- Settings -> System gets "Outgoing email": every email of the last 7
days not yet sent, failures first, then the longest retrying, with
recipient, kind, state and last error, and a count of what was sent.
Failures and anything still retrying after 5 minutes lead the page's
verdict (outbox.outgoing).
- get_alert_history (MCP) adds `email`: the same delivery state.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Run 8942 failed this test alone (118 passed): it read the row back in a
second asyncio.run, and the app's pooled connection belonged to the
first, closed loop. The behaviour held ("The alerts this email carried
were deleted"); the settled row is now read from attempt's own result.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5508 "Every other email through the outbox: weekly report, invites,
password resets, and the test buttons" (milestone 501).
- outbox.send_each queues one email per reader (claimed, so the sender
doesn't take it too) and tries them at once, for a page that answers
with what happened; outbox.said words the result. attempt takes a cfg
for a test of the form's unsaved settings and returns the settled rows.
- Weekly report (scheduled and Send now) and invites: queued, tried at
once; one the server couldn't take yet is retried for an hour. An invite
that didn't go shows its link with the reason, as before.
- Password reset: queued in the reset link's own transaction; the sender
delivers it on its next tick.
- Settings -> Notifications test email and a rule's Send test: queued and
recorded with retry=False, tried once (System 156 ruling).
- Removed mail.send, notifications._send_email and _smtp_send_sync;
dispatch_notifications is webhook only. A guard asserts only the outbox
(and smtp_open itself) opens an SMTP connection, with a self-test.
Also fixes run 8941 (#5507's integration test): its Also-send-to
addresses were "@test", which is_real_address rightly drops; they are
example.com now.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5507 "Alert emails through the outbox, gathered for 2 minutes"
(milestone 501). Replaces the in-memory core/alert_burst.py.
- outbox.hold_alert: an alert joins each recipient's held alert email, or
opens one due 2 minutes later (System 156 ruling). A Postgres advisory
lock keeps two alerts at once from opening two windows for one reader.
- email_outbox_events.alert_json (migration 0043) keeps the alert as it
fired, so the email written when the window closes says what it said.
- write_held writes it then, in the reader's zone: the usual alert email
for one alert, the list for several (outbox.alert_words).
- after_settle records on each AlertEvent, once every email carrying it
has settled: email_sent, or email_error naming each reader it didn't
reach and the server's reason.
- A held email survives a restart and goes out when its window closes.
- A refusal of a message's one recipient no longer repeats the address.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5506 "Email outbox: a table of outgoing email and a sender that retries
until accepted, for up to an hour" (milestone 501).
- email_outbox (migration 0042) holds each message per recipient, its
words, its state (held / pending / sending / sent / failed), attempts and
the server's last answer. email_outbox_events links alert events for
#5507.
- core/outbox.py: enqueue; a sender task every 15 s claims what is due
(SKIP LOCKED, stale claims released), sends the batch over one
connection with no session held, and settles each row:
- accepted -> sent;
- a 5xx refusal -> failed at once with the server's words;
- no connection, a timeout, a 4xx -> retried after 30 s, 1, 2, 5, 10,
15 min, the last try at the hour, then failed;
- a test (retry=False) settles on its first answer.
It runs on startup, so a row left by a restart goes out at boot.
- notifications.smtp_open is the one place a connection is opened.
- mail.message takes the html part as optional.
- cleanup ages email_outbox rows on the data window.
Senders move onto it in #5507 and #5508.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5470 "pysnmp-lextudio is deprecated: every startup logs a RuntimeWarning".
Dependency change: the [snmp] extra now installs `pysnmp>=7.1` instead of
`pysnmp-lextudio>=6.2`. pysnmp is the same lextudio lineage under its maintained
name; the 7.1 floor is because below 7 the name is the abandoned 4.x line with
no asyncio API. The poller already finds either API shape (_api(), the engine
close probes), so no code changes; its docstring now names what the image ships.
Approved by the operator 2026-10-09 ("yes please do the swap").
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5469 "Settings → System: table sizes and what the last cleanup removed".
- Storage: each table's size on disk (pg_total_relation_size, indexes and TOAST
included) and Postgres's row estimate (reltuples), largest first, the 15
largest by name and the rest summed, under the database's total size. Read
from the catalog only, never by counting rows.
- Last cleanup: when it ran, how long it took, and what each pass removed (each
aged table, run log files, the metric and Docker rollups, the log ring), or
"Nothing was past its window". run_cleanup now returns that summary and keeps
it as the cleanup.last_run setting, as reports.last_sent_at does; cleanup_secs
(#5467) is recorded from the same figure.
- A hint links to where each retention window is set.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5468 "Log search by service name: searching "steward_app" refuses because every
deploy makes a new container name".
search_container_logs(container="steward_app") was refused, because no container
carries the service's own name and five task containers partly matched. Now a
service's name reaches every task container Steward holds anything for, live and
removed (docker_svc.service_containers, as the service page's logs already do),
so one search spans deploys. Each match carries the slot and task_id that wrote
it, and the reply names the service and how many tasks were searched. A container
named exactly what was asked still takes precedence. A miss that partly matches
task containers now names their services.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The `when` filter returns a <time> element, so title="{{ at|when }}" wrote raw
markup into the attribute. That broke the hover text, and in the rule form's
metric preview (#5475) it leaked a stray `">` into the page. `ago` already
carries the exact time as its own tooltip, so both places drop the title.
Also fixes#5467's page test, which needed the escaped apostrophe.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5467 "Steward's own health: metrics, rules can watch them, a page that shows them".
The self-monitor tick now records, under source "steward", resource "process":
ingest_posts / ingest_failed / ingest_slowest_ms (the host-agent posts since the
last tick, tallied in steward/core/health.py), db_roundtrip_ms (a timed SELECT 1),
and unwatched_secs (the gap before this tick past watch.GAP_S, from Steward's own
last sample). The hourly cleanup records cleanup_secs. The open-file pair is still
recorded, and is now only skipped on its own where /proc is absent. All of these
are in CORE_METRICS, so an alert rule can watch them under "Steward itself".
Settings → System (/settings/system/, admin) shows a verdict first (failed posts,
a gap in watching, open files near the limit), then each measure now and over the
last day, with a sparkline scaled to its own peak.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5490 "A day of one metric reads fast even cold". A cold 24 h series read
measured 18.3 s against 63 ms warm (#5485): every sample's row sat on its own
heap page. ix_plugin_metrics_series_value carries `value` (INCLUDE), so the
read is an index-only scan; it replaces the plain series index from 0023.
It is built once, with CREATE INDEX CONCURRENTLY from a task started after the
app is serving, never in a migration, so the boot isn't held while it builds
over the live table (#5220) and ingest and pages carry on. A build cut short
by a restart leaves an INVALID index; the next boot drops and rebuilds it.
Later boots find it present and only check. Postgres keeps it current on each
insert, as it did the index it replaces.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5487 "MCP: edit inventory group membership". Group membership decides which
hosts a group alert rule covers, and the MCP could only read it. The new
operator tool takes a group and lists to add and remove, each a target's
name or id or the Steward host it is linked to; an unknown name refuses the
whole change. It answers with the group's targets after and the alert rules
covering the group (whose per-host rules follow within a minute), and audits
as ansible.group.updated by "<user> via MCP", as the Inventory page does.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5485 "A rule's page waits up to 25 s for its metric preview: a cold 24 h
series read from plugin_metrics". Opening "Agent offline" took 25.6 s; a
cold day of one series measured 18.3 s over MCP, a warm one 63 ms. The
pickers still come with the page from plugin_metric_latest; the preview is
its own request (hx-trigger=load), showing "Reading the last day…" until it
lands. Picker swaps no longer read it either.
#5486 "Send test where it can be found": the button moves from inside "What
happens" to the page header beside History, its result under the verdict.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5483 "Alerts list: a covering rule's hosts start collapsed behind a toggle;
no dashed rows", from the operator's review of the Alerts list. Each rule is
its own tbody; its hosts sit in another, hidden until "N hosts" is pressed.
A rule with a host that isn't quiet opens on its own and names the host where
"last fired" was, so a problem never hides behind the fold. Host rows lose the
dashed rules for a sunken band under the indent.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5475 "Rule form: pick the host, then its part…". CI run 8909 caught it:
moving from the whole host to a mount kept "CPU" in the mount's Metric list,
because a chosen metric always stayed on offer. Only a rule's own page keeps
its resource and metric after they go quiet (keep=True from edit_rule); the
pickers' swaps drop a metric the new part doesn't report.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5475 "Rule form: pick the host, then its part; metrics that part reports;
a link to groups; a preview of the metric", from the operator's review of
the New rule page: hosts and their parts shared one long picker.
- Host agent rules pick a Host, then a Part: "The whole host", then its
mounts, CPU cores, disks, interfaces and sensors under a heading each,
cores in number order. A part that reports nothing a rule can watch
(an interface's traffic) isn't offered.
- The Metric list holds what the picked resource reports, in catalog order.
- Under it, a preview: the reading now, and the last 24 h low to high with
an hourly sparkline drawn from zero (UI ruling 8), read from one series.
- The group picker links to Ansible > Inventory to make a group, in a new
tab, and re-reads the groups when the tab comes back into view.
- Picking a host or part swaps the pickers alone (/alerts/api/rule_target);
picking a metric swaps the preview (/alerts/api/metric_preview).
- A rule's own resource and metric stay on offer after they go quiet.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5473 "Opening an alert rule takes up to 15 s". With the rule pickers now
reading plugin_metric_latest, nine integration tests failed on rule_fields:
"host_agent has no resource 'box-…'". They seeded a resource by adding a raw
PluginMetric row, which no production path does: host-agent ingest and
record_metric both go through record_metrics, which keeps the latest-value
table. The seeds now go through record_metrics too.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5473 "Opening an alert rule takes up to 15 s: the Resource picker scans
every raw host-agent sample". The access log showed the edit page at 6-15 s
(one load gave up) while /alerts/ took 0.2 s. resources_for ran SELECT
DISTINCT over plugin_metrics, and Postgres has no skip scan, so it read every
host-agent sample; every rule page, source swap, rule save and MCP rule call
paid it. resources_for and metrics_for now read plugin_metric_latest, one
row per series, which lists the same names. A test fails if either reads
plugin_metrics again.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5466 "Send test: check a rule's delivery from the rule". A rule's page gets
a Send test button (operator role), and MCP gets send_test_alert. Each sends
a TEST alert for that rule now, by email to everyone who gets alerts and by
webhook. The alert carries the rule's name and condition, at the latest
reading of what it watches; a rule covering hosts uses its first host. The
result for each channel shows under the button, or "Nothing is set up to
send to". Nothing fires, no event is written, and the test is audited as
alert_rule.tested. An alert with no reading now says "has no reading yet"
instead of "is None".
Docs: alerting.md also gains the burst email from #5465 "One email per burst
of alerts".
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5465 "One email per burst of alerts". When a switch reboots, every host's
agent-offline rule fires at once and each account got one email per rule.
Alert email now waits 30 s from the first alert: each recipient then gets one
message listing every alert in the window, oldest first, in their own zone
("[Steward] 3 alerts: 2 firing, 1 resolved"). One alert alone keeps its
usual email. Webhooks still post per alert, at once. Every alert's event
records the email's outcome. A burst held when Steward stops is not sent;
the alerts stay on the record and on the Alerts page.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5464 "Group rules: one rule covers every host, or every host in a group",
step 2 of milestone 499 "Alerts you can trust". Each rule named one host, so
the live instance carried six rules for three hosts, and a new host had no
agent-offline or disk alert until someone copied them.
- alert_rules.covers / cover_group / parent_id (migration 0041). A rule on
host_agent may cover "all_hosts" or a "group" (an inventory group: the
Hosts ruling, #5294). A covering rule has no state; Steward keeps one
ordinary child rule per covered host, named "<rule>: <host>", so each
host's alert keeps its own state, history and acknowledgement.
- alerts.service.reconcile_children: a covered host that has reported in
the last 7 days gets a child; a host no longer covered loses its child;
children take the parent's condition, checks, notify setting, action and
on/off. Run on save and each minute (alert_reconcile).
- A child is changed through its parent: edit, toggle and delete of a child
are refused (409 / PartOfCoveringRule); its edit page opens the parent.
- Rule form: a Covers choice for host sources, with the group picker; fixed
once the rule exists. Rules list: a covering rule reads "every host in
swarm · 3 hosts" with its hosts' rules nested under it, each opening its
history. The history picker and the rule count leave covering rules out.
- MCP: create_alert_rule takes covers/group; update_alert_rule takes group
and passes changes down; list_alert_rules shows covers, hosts and part_of.
- docs/core/alerting.md: "Rules that cover hosts".
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5463 "Acknowledgements on record: an event and who did it", step 1 of
milestone 499 "Alerts you can trust". The live walk in #5461 found that
acknowledging changed only the alert's current state: once it resolved,
nothing said who had acknowledged it or when.
- alert_events.actor_id (migration 0040): the person behind a transition.
- Acknowledging writes a firing → acknowledged event by the acknowledger.
- A rule edit, disable or enable that resets an alert carries its actor
(web and MCP alike).
- The history page notes "acknowledged by X" and "rule disabled by X"; MCP
get_alert_history returns `by`.
- One usernames() helper in alerts.service, used by the active list, the
history page and the MCP tools.
- docs/core/alerting.md: the state diagram no longer shows an acknowledged
alert re-firing, and says what an event records.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5449 "The hourly Docker log prune ranks every log line in the table to find
the few over their cap": the size ring put a running-total window over every
row of docker_logs each hour, sorting the whole table on NFS-backed storage
to find the handful of containers past docker.logs.max_bytes_per_container.
- prune_docker_logs (capability docker.prune_logs) does the age cutoff, then
a per-container byte total picks the containers at or over the cap, and
only their rows are ranked and trimmed. Same keep rule as before: the
newest line always survives.
- run_docker_retention keeps the metrics rollup and the event prune.
- Cleanup runs the log ring in its own transaction after them.
Arose from #5431 "Ingest dropped for ~3 minutes after each deploy while
startup retention runs".
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Run now set last_run_at to the click, and a schedule's next turn counts
from last_run_at, so a manual run of the nightly prune at 14:00 moved
every later run to 14:00. Operator ruling: "run now shouldn't change the
schedule it should just run the playbook and leave the schedule as it
was."
Run now no longer touches last_run_at. It still records the run as
last_run_id, so the list shows it and a firing during it is skipped.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
_DueTracker counted a task that had never run from 0 on the loop clock,
which is host uptime, so every run_on_startup=False task (hourly data
cleanup with the Docker log prune, the weekly report check, alert
reconcile, agent age, Ansible schedules) fired on the first tick after
each deploy, while agents were flushing their backlog. Agent ingest was
slow or dropped for a few minutes after every deploy.
start_scheduler now calls tracker.begin(): such a task is first due one
interval after the loop starts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each firing stored last_run_at = the moment the ~60 s loop noticed it,
and the next was due a whole interval after that, so every firing's
lateness carried into the next: Nightly Docker Prune moved from 05:44:45
to 05:47:05 over six days.
A firing now stores its due slot, the latest last_run_at + k * interval
not after now (due_slot), for both a launch and a skipped firing. Missed
slots after an outage are skipped, not run in a burst.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
check_interval_s was `m.check_interval_seconds or None`, so a monitor
following monitors.poll_interval_seconds read as null and a reader
couldn't tell how often it ran. Monitor results now carry the effective
interval and `interval_from: "monitor" | "global"`.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
tcp_check swallowed every exception and icmp_check returned on a
non-zero exit without a reason, so every down ICMP/TCP monitor stored
error_msg NULL and the Monitors page and MCP showed it down with no why.
Each failure now names its cause: "No reply within 2 s", "Connection
refused on port N", "No answer on port N within 5 s", "Can't resolve
<name>: ...", the OS error, or ping's own first stderr line. An ICMP
check that fell back to TCP says so, and a ping that outlives its
timeout is killed rather than left running.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
When a silent series' sample also moved the rule (fired, resolved), the
resume and the transition were two events: the resume from the
pre-evaluation state at ingest time, the transition at the sample's
time. Ties sort by a random id, and the resume's "→ inactive" landed
after "→ firing", so the history read "silent, firing, back to normal"
at once and the episode closed while the alert was firing.
_track_samples now only clears the mark and returns its note; the first
sample's evaluation records one event from no_data to wherever that
sample took the rule ("firing" or "resolved" notification), or back to
the state it was in ("data_resumed", still only when notify_no_data).
Also fixes the #5441 integration test: its rule needs a recorded
resource to target (CI 8878).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
After Steward or its database was down, every agent's age and every
series' last sample were old, so all agent-offline rules fired and all
rules went no-data at once, clearing a minute later. With per-account
alert email that would have mailed everyone a dozen notices per restart.
A periodic job now ticks a Watch (steward/core/watch.py) once it has
reached the database; the Watch says since when runs have been unbroken
(process start, or the end of a gap over 120 s). seconds_since_seen and
the no-data sweep count silence from no earlier than that, so a source
must stay quiet while Steward is watching before it counts as silent.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5275 "steward_db killed by its healthcheck twice": on NFS, pg_isready
timed out during slow writes and the orchestrator killed a healthy
database. The operator removed it from the live stack and ruled that
Steward doesn't ship one. Steward waits for the database itself (60 s at
startup, retries per request), so depends_on needs no health condition.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
