#5816 "Restoring mode": while a backup goes back in, the scheduler holds,
ingest and HTMX answer 503 with Retry-After (agents keep their samples),
and every page says what is happening. A safety backup of Steward as it
stands is made first, to /data/backups and the bucket. The backup's key
is put in place, the restore audited, and Steward exits with code 3 so
Swarm, compose or systemd starts it fresh. A failed in-place restore
changes nothing and says why on Backups; a full replace that fails after
the wipe restarts into first-run setup.
#5817 "Settings → Backups: Restore": a Restore button on each sent backup
and on the file card. The chosen backup is checked (the same comparison
as Check, now a shared partial) and held server-side for an hour under a
token; it starts only once the admin types "restore", and an older
backup also needs "Replace everything" ticked. A newer backup, or one
whose key can't open its secrets, gets no button.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- restore_in_place checks the backup's migration heads before pg_restore
writes anything, so a backup from another schema is refused at once.
- The integration tests read their own setting rows directly:
get_all_settings knows only Steward's own keys.
- check.py: drop an import left unused by the move to pgdata.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5815 "Restore engine: in place for a same-version backup".
restore_in_place (steward/backup/restore.py) runs in one transaction:
1. It keeps aside what Steward records as it runs: ACTIVITY columns of rows
that come back, and STATE_SETTINGS.
2. It drops every foreign key touching a replaced table.
3. It truncates the replaced tables and COPYs the backup's rows in
unparsed, through asyncpg.
4. It puts the kept state back and moves sequences past their rows.
5. It puts right the rows whose parent didn't come back, by each key's own
ON DELETE. A RESTRICT key that may be empty is cleared, so a run whose
user is gone keeps its record.
6. It re-adds the keys, which checks every row.
A failure anywhere rolls the whole restore back. A backup made at another
schema is refused with NotSameSchema ("older": a full replace is the way;
"newer": update Steward).
The COPY-text reading moves into steward/backup/pgdata.py, shared by the
check and the restore. The integration tests stand in for pg_restore with
this database's own COPY output:
- settings come back while the audit log and the history of what remains
stay;
- history of what didn't come back goes;
- a failure part-way changes nothing;
- a newer schema is refused.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5823 "Check a backup: compare what it would restore with what is in
place, without restoring".
- steward/backup/check.py: pg_restore writes the backup's rows as COPY
text with no database involved. The rows in place go out through COPY in
the same form. Tables are compared on their primary keys, over the columns
both sides have, and grouped by area.
- Sealed values are opened with each side's own key and compared as
plaintext digests, so a secret sealed twice still matches. They are never
shown. A backup whose key can't open its own secrets fails the check.
- What Steward records as it runs (last-seen times, a schedule's last run,
backup history, scraped UniFi fields) is left out of the comparison:
contents.ACTIVITY, STATE_SETTINGS and HAND_SET_ONLY.
- Settings → Backups:
- a Check button on each sent backup, which fetches it from the bucket
(new s3.get_file);
- "Check a file" to upload one, with the size limit raised before the
CSRF check reads the form;
- a results page.
- Each backup Steward sends checks itself once written, and its history
row says how that went.
- New CLI command `steward check-backup FILE`. CI's image smoke step checks a
real backup against its source and expects a match.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5822 "Backup contents: settings and keys only". The operator's ruling
(System 180, 2026-10-15) overturns 2026-10-12's "Config + audit + alerts":
a backup is for system recovery and carries what would otherwise have to
be set up again, with no logs or history.
- contents.py: audit_events, alert_events, alert_states, metric_anomalies,
metric_anomaly_spans and user_invites move to EXCLUDED.
- Alert state is no longer restored, and evaluation skips a rule with no
state row. ensure_rule_states (alerts/service.py) gives every
single-resource rule lacking one an inactive row; alert_reconcile runs
it each minute.
- Docs, the Backups page and the CLI help say what a backup holds now.
CI's image check also finds the audit log empty after a restore.
- tests/integration/test_backup_restore_fixes.py is removed: its case (an
alert event naming a left-out Ansible run) no longer arises. The SET NULL
fix-up is covered by a unit test on synthetic tables.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An agent keeps its log cursors in memory, so after a restart it seeds each
container from its last 50 lines, which Steward already holds; docker_logs
has no unique key, so every update or reboot doubled them. The ingest now
reads what it already has for each container over the batch's own span,
in one query, and skips those lines. A push of new lines overlaps nothing.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
SQLAlchemy fires after_commit and after_rollback for a savepoint too. The
host-agent ingest releases one after recording its metrics, so an alert
raised by a sample started its email before the outer transaction
committed; twice on 2026-10-12 the outbox row hit the event's foreign key
first and the email was lost. A savepoint rolling back (a docker persist
failing) cleared every pending notification, the outer one's included.
_after_commit now runs its jobs only on the outermost commit, and a
rolled-back savepoint drops only the jobs added inside it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
On 2026-10-14 nine swarm services sat at 0/1 and nothing fired. From each
manager's swarm report the docker plugin now records three figures on
resource "swarm": swarm_services_short, swarm_nodes_down and
swarm_managers_unreachable. Drained nodes are left out of the last two,
since a drain is deliberate. The figures are recorded at most once every
30 s whichever manager reported, and are offered to alert rules.
Agent 1.19.0 sends each manager's ManagerStatus.Reachability. Older agents
fall back to the node's status.
A maintenance job on a swarm node also keeps a window over docker/swarm,
so its drain and reboot hold those alerts until the job ends.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
On the live instance the hourly refresh took 3 min 8 s, averaging a week of
raw temp_c samples each time. The hourly averages are now kept in memory.
The first refresh after startup reads the whole window. Each later one reads
raw samples from the start of the hour the last read ended in, so the hour
that was still filling is read again whole. merge_hours folds the new rows
in and drops hours older than the window.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Rollups exist only for hours older than the raw window (7 days by default),
so the first deploy read an empty week and judged nothing. refresh now
averages the raw samples by the hour and adds any rolled-up hours in the
window. Sensors are named from plugin_metric_latest, so each series is read
through its index instead of a pattern over every host_agent row. An
integration test covers both halves against Postgres.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- #5780 "The error-pages service is graded on other routers' errors". The
Traefik scrape reads which services an errors middleware sends to, and
grades them healthy: their 4xx and 5xx are the codes of the errors they
cover for, already graded on the routers that failed.
- #5781 "The Maintenance failed rules sit at no data until a host has had
a maintenance job". Every host with an agent reads 0 until a job fails.
- #5782 "VDNT-Util01's host page lists SYSTIN at 115 °C in red". The host
page lists sensors that counts_as_hottest leaves out (unwired board
inputs, 0 °C or below) on one muted line, not as heat.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5768 "A maintenance cadence: a group's updates and reboots, one host at a
time, on a schedule". A plan names an inventory group, a weekday and time
in a time zone, a window and a reboot policy (only when required by
default). At its slot a run takes the group's hosts in order and gives each
a #5767 maintenance job. The next host starts only when the last is back in
service: its agent is posting and, for a swarm node, the node is active and
every replicated service is at its replicas. A failure stops the run, and
no host is started once the window has passed. A host's latest failure
shows as host_agent/<host>/maintenance_failed, for an ordinary alert rule.
The page is Hosts → Maintenance.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5767 "Steward hands a host a maintenance job and follows it to the end,
through its own restarts", milestone 514.
A host_maintenance_jobs row per job: queued → updating → [draining →]
rebooting → [undraining →] done, or failed / timed_out. Steward's answer to
an ingest post carries the step that host's agent should carry out now; the
agent (1.18.0) leaves each step id for the root helper once, and the helper
refuses an id it already ran, so repeats are harmless. Results come back as
the #5766 maintenance events, matched by step id; the reboot completes when
the host's `started` event has a boot_at after the reboot was handed out.
The job reboots only when the update says one is needed (or never, if asked).
A swarm node is drained first by itself if it's a manager or by a manager
with the helper, and made active after; with no manager to drain it the job
is refused. A minute tick times out a job past its state's deadline. A
maintenance window over host_agent/<host> lasts the rest of the job.
Everything is on the row, so a Steward restart, or the reboot of the host
Steward runs on, resumes from the next post. Ingest runs it in its own
savepoint so a job going wrong never costs the samples.
The Agent panel leads with a running job and says what a finished one did;
Manage agent has Update now (with "never reboot"), or says why a job can't
start. Integration test: hand off, Steward restarts, the post-boot events
close the job.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5756 "The agent says when its host rebooted and when it lost contact, so
downtime is recorded as what it was". Agent 1.14.0 posts events with the
next post that gets through: started (with the host's boot time), stopping
(on SIGTERM, with whether systemd says the host is going down) and
lost_contact (from its first unanswered post to the one carrying it).
Steward keeps them as the host's downtime (host_downtime): reboot,
agent_stop, unclean (a boot after the agent was last heard that nobody
announced) or lost_contact. Retried posts apply once.
An announced stop explains the silence for reboot_grace_seconds (Settings
-> Plugins -> host_agent, 10 minutes): seconds_since_seen reads 0 until
then, and the host's other rules go quiet with it (agents_away beside
#5721's registry). Past the grace, agent offline fires.
The host's Agent card lists the last 7 days of downtime. Rows are pruned
at data.retention_days.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
VDNT-Docker03's silent agent read as three overview findings (agent offline,
agent stale, the disk rule's no-data) and every drop emailed the disk rule's
silence and return on top of agent offline.
- The host agent registers its heartbeat (seconds_since_seen) and the
sources it carries (host_agent, docker). While a host's agent is past its
offline window, its other rules' no_data is recorded "while its agent is
offline" and not emailed, nor is their quiet return; a return past a
threshold still fires.
- The overview folds those rules under the host's agent-offline finding
("also silent with it: ..."), and a stale agent whose agent-offline alert
is active is that alert's finding, not a second one.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
CI 9128: the host vitals and monitor templates required the new grades, so
callers (and tests) that render them without one failed; they now fall back
to status_class. The new integration test read outside begin(), which
autobegan the session.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Host CPU, memory and load per core, container CPU and memory, and every
monitor's response time learn a normal per hour (milestone 513, System 182).
A reading past its fixed cutoff but within its normal reads healthy and says
what is usual at this hour; one well out of it keeps its grade and is a
metric anomaly. Disk, a monitor down and an agent offline keep their cutoffs.
- Each normal's floor, the smallest departure that counts, is a setting:
Settings -> Thresholds -> Normals (norms.floor.*). Defaults: host CPU 10
points, host memory 5, load per core 10, container CPU 10 points of one
core, container memory 5, monitor response 100 ms; Traefik's 2-point 4xx
and 50 ms p95 floors move there from constants. learn_normal takes a
floor function (floor_setting) read at each pass.
- learn_normal(parts=False): a host learns, its cores don't.
- Ingest records load_pct (load_1m / cores x 100), the figure pages grade.
- Shared grading: anomalies.against / graded, worst_reading(normals=),
hosts' host_grades, docker usage(thresholds=, norms=), monitors'
latency_grade. Applied to host vitals, the Hosts widget, MCP overview,
Docker usage, and the monitor list, widget and page.
- The monitor and container pages list their anomalies.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5741 "Traefik routers are graded against their own normal". CI run 9126:
the new route test got 404, because CI doesn't enable the Traefik plugin
(enabling it would start its scraper against the shared database). The test
now mounts traefik_bp on its own app, as plugin_manager.load_plugins does,
with the plugin.yaml scrape interval.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5741 "Traefik routers are graded against their own normal". The rows
fragment and the Routers widget had no test that ran them against Postgres,
so router_norms and the anomaly list were covered only by template
renders. This one seeds a router answering 60% 4xx with a learned normal of
58% and an open p95 anomaly, and checks both routes answer 200, the page
says the share is usual, and lists the anomaly.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5741 "Traefik routers are graded against their own normal", milestone 513.
- A router's 4xx share and p95 learn their normal (learn_normal in the
plugin's setup). Once a router has 5 days of history, a reading over the
fixed cutoff but inside its normal is healthy, and the healthy table says
it is usual for that router. One out of its normal is a warning that says
what is usual, and a metric anomaly.
- The 5xx share is a failure: fixed cutoff always, whatever the history or
traffic (System 182's ruling).
- A 4xx share over fewer requests than Settings -> Thresholds' new "Fewest
requests to judge 4xx" (30 per 15 minutes) isn't graded on the page, and
isn't judged for anomalies.
- Core: learn_normal(gate=(metric, least)) judges a key only while its
resource's gate metric averages at least `least`; `least` may be a
function, so it follows a setting. learn_normal(since=...) learns only
from history after a moment: Traefik latency learns from #5755's
per-scrape figures only.
- The Traefik page lists the routers' metric anomalies of the last 30 days
(_anomaly_card.html). The dashboard widget keeps its why to problems.
- Thresholds and docs say when the cutoffs give way to a normal.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5752 "Container CPU and memory history lives in the one metric store".
Docker persist now records each snapshot at its own time, so test_docker's
June and early-October snapshots leave raw plugin_metrics rows older than
the 7-day window, and the rollup's table-wide counts include them (CI run
9119: 11 buckets, not 1). The test's own host is still checked exactly:
one hourly row averaging 15 over 2 samples, and one raw sample left.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5752 "Container CPU and memory history lives in the one metric store;
docker_metrics and docker_metrics_hourly are retired". Container cpu_pct and
mem_pct already went to plugin_metrics for alerting; now mem_usage_bytes does
too, and every snapshot in an ingest (backlog included) is recorded at its
own time. The container history chart, Busiest Containers, a service's
history and the MCP container tools read plugin_metrics under resource
"<host>/<container>", rolled up and pruned by the shared metric retention.
docker_012_drop_metrics drops both tables. Their history is not copied
(lesson #5221 "A data migration's cost is set by the live table"); the
store already holds 7 days of container CPU and memory %. Docker retention
now prunes only lifecycle events, and Settings -> Thresholds has one
"Metric retention" card for every source.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The app's pooled connections belong to the loop that opened them, so a
second asyncio.run in one test failed with "attached to a different loop"
(run 9116). Tests that do several passes now await them in one coroutine.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5740 "A metric that departs from its normal is recorded as an anomaly to
review, and is findable after it ends", milestone 513.
- Every minute each learned key's last 15 minutes is judged against its
hour's normal: above the usual hold x1.25 and by the metric's floor opens
an occurrence, which ends when the level comes back.
- metric_anomalies holds one row per item to review; a repeat while it
still waits joins it as an occurrence (metric_anomaly_spans).
- An ended row nobody reviews within anomalies.review_days (14, Settings ->
Thresholds) lapses; an ongoing one never does.
- Learning skips every hour an occurrence covers unless its anomaly was
marked normal: no normal is learned without review.
- Alerts -> Anomalies lists those to review first, then the reviewed and
lapsed under day dividers; marking is operator/admin and audited.
- The Overview widget and get_overview warn of open ones and count those
to review; the host page shows its own; the weekly report lists the
week's and the lapsed; MCP list_anomalies and review_anomaly.
- Reviewed and lapsed rows are pruned at data.retention_days, and kept
in backups like alert history.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A metric registered with learn_normal() gets, for every resource reporting
it, a usual hold (the 90th percentile across past days of each hour's p95)
and a usual peak for each slot:
- an hour of the day once 5 days of history exist;
- an hour of the week once history reaches back 3 weeks, so weekly jobs are
learned too.
Each slot takes in the hours either side, so a job that runs a little late
stays normal. One SQL statement relearns everything hourly into
metric_norms, from the rollup plus the raw window not rolled up yet;
norms_at() reads the slot a moment falls in. Nothing is registered yet:
Traefik (#5741) and hosts (#5742) opt in.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
plugin_metrics_hourly gains value_p95 beside value_avg and value_max. A
two-minute backup spike raises the hour's peak and leaves its p95 low; an
hour spent at that level raises both. Learned normals (milestone 513) need
that difference to call a recurring spike usual without calling the same
level held at another hour usual too. Hours rolled up before this keep a
NULL p95 and read as their average.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The scraper folded traefik_service_requests_total{code} into two shares and
dropped the code, so a problem couldn't say whether it was scanners (404), a
login wall (401/403) or the app down (502/503/504). Each scrape now stores the
responses per router and code that occurred (traefik_status_codes, migration
traefik_006), pruned with the router metrics and left out of backups (System
180 ruling: Traefik history is not backed up). A problem card lists its codes
over the last 15 minutes, most first, or says none occurred, or that codes
aren't recorded yet.
A counter series new since the previous scrape now counts from 0 instead of
being skipped: Traefik creates a router's 502 series at its first 502, so a
first burst of 5xx read as 0% until it happened again.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Run 9095: with the Archives template at version 2, the library test's
"bad" copy no longer had a bad version, saved, and took the name the
real save then wanted.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Prometheus gains a "Widget templates" tab beside Targets. It lists the
built-in templates and an admin's own, any that no longer pass the checker
first. A template opens as its YAML beside a live preview drawn over a
chosen target exactly as the dashboard widget draws it (routes.draw, now
shared by both). An admin writes or pastes a new one, reads a .yaml upload
into the editor to look over, duplicates any template under a free name,
edits and deletes their own; anyone can export the YAML. Built-in
templates are read-only.
Custom templates are rows in prometheus_widget_templates (migration
prometheus_002_templates), kept in backups, keyed "custom:<id>" so a
rename keeps its widgets. Saving needs a template that passes the checker
and a name no other template has. Create, update and delete are audited
and link to the template.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5691 "Generic Prometheus dashboard widget: pick a target and a template",
step 3 of milestone 508 "Prometheus widget templates".
- /plugins/prometheus/widget draws one target with one template in widget
shape: the verdict (led by "isn't answering" when the target is down),
then each block with problems first, figures beside a share bar or the
last 24h of bars, and the healthy rest folding to "+N more healthy" past
six. A missing target or template, or one that no longer checks, says so.
- library.py: bundled templates from plugins/prometheus/widgets/*.yaml,
the picker's target and template lists, and the default title.
- widgets/fabled_archives.yaml: the first bundled template, against the
series Archives reports (refined in #5693).
- Dashboard: a widget setting may take `options_from` (read when the picker
opens) with an `empty` sentence; `default_title`; a Details link may name
settings ("/plugins/prometheus/targets/{target}"). Prometheus is grouped
with the integrations.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5689 "Prometheus target page: fold state sets, show epoch times, size
bytes", step 1 of milestone 508 "Prometheus widget templates".
- A state set (name ending in _state, a state label, every series 0 or 1;
or the OpenMetrics form, the label named after the metric) reads as one
row per thing it describes, naming the current state, the others on
hover. fabledarchives_jobs{state} is a count and stays as it is.
- label_pairs / state_label / fold_states in service.py, for the widget
templates to reuse.
- After a good scrape, series it no longer has lose their newest value
(forget_gone): Archives' old build_info row read as current beside the
new one. History stays; a failed scrape forgets nothing; alert rules keep
their own last-sample time.
- Epoch times and _bytes sizes were already read by shown_value /
metric_value.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- The audit-wording guard now reads both branches of a conditional action
("backup.made" if ok else "backup.failed"), and its self-test covers that.
- The Backups save test no longer asserts the session's tz, which sign-in
state sets, not the test.
- The restore integration test uses the app's own metadata. all_metadata
without plugin dirs imports the `plugins` package as the tests do; loading
by path under the loader's names is only for `steward restore`. Loading by
path in the shared test process had mapped traefik and unifi tables a
second time for later tests.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5673 "S3 client and the backup job: settings, schedule, history, metrics":
- steward/backup/s3.py: SigV4 over httpx, path-style; put and list only
(an append-only key is enough; retention is the bucket's). Signing is held
to AWS's published examples.
- steward/backup/job.py: a 15-minute tick. The daily backup goes at the hour
chosen on the reader's clock; a missed day is made up, and a failure is
retried an hour later. History of the last 30 runs, audit
(backup.made / backup.failed), and backup_age_secs / backup_bytes under
"Steward itself", resource backup, for alert rules.
- Settings → Backups: verdict first, then the bucket (secret sealed; a
blank box keeps it), the schedule, Test the bucket, Back up now, and the
recent runs.
#5675 "Docs: backing up to Archives, and restoring": docs/core/backups.md.
Also: the audit page now words the Prometheus target actions from #5668,
which showed as codes, and a guard fails on any logged action without
wording.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The backup file (#5672 "Backup file: pg_dump of config + audit + alert
history, the secret key and a manifest"): steward-<UTC>.tar holding a
pg_dump --format=custom with metrics', check results' and Docker/Traefik/
UniFi rows left out, the secret key, and a manifest. Every table is named
KEPT or EXCLUDED in steward/backup/contents.py, and a guard test fails on a
new one. The image gains postgresql-client-17 (approved, System 180).
The restore (#5674 "Restore: `steward restore` and a CI test that a dump
brings Steward back"): pg_restore in its three sections. Between data and
constraints, a kept row pointing at a left-out row gets what its foreign
key's ON DELETE says (alert_events.ansible_run_id is cleared). It refuses
a non-empty database unless --wipe, and puts the backup's key in
/data/secret.key. Migrations run on Steward's next start.
`steward` stays the server command: it is now a click group that serves
when given no subcommand.
The publish smoke step backs up the booted image's database, restores it
into a second one, checks a marker setting came back and plugin_metrics
is empty, and starts Steward on it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The UniFi password sat in plugin.unifi in plain text: SECRET_KEYS covers
whole settings, and a plugin's settings are one dict. Saving the plugin's
settings page also wiped it, since the password box is never filled back
in and its blank value was stored.
- A plugin field whose name holds password, secret or token is a
credential (is_plugin_secret). set_setting seals it inside the dict,
every read opens it, startup seals any plaintext left from before, and
the undecryptable-secrets banner names it as plugin.<name>.<field>.
- The settings page uses the same test for its password boxes, and a blank
box keeps the saved value.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- tests/test_breadcrumbs.py: the Prometheus target list is its section's
own top page, like SNMP's, so it carries no trail.
- The integration test's exposition has label braces, which str.format read
as fields: the fake scrape raised, the target was recorded down and no
series landed. It substitutes the counter with replace() now.
- docs/plugins/prometheus.md: adding a target, filtering, how series are
named and stored, alerting, and the Fabled Archives recipe.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A bundled prometheus plugin, on by default like the other protocol plugins.
Targets are rows edited on its page: address, bearer token (encrypted),
interval, keep/leave-out name patterns and a series cap. Steward does the
filtering; the endpoint serves everything (Archives' ruling).
- A 5 s tick scrapes due targets at once, each within a 30 s deadline, and
writes the outcome to its row and its series through record_metrics, so
alert rules can watch any of them.
- Gauges are stored as they are; counters (and a histogram's or summary's
_sum/_count) as <name>_per_sec rates from the second scrape. Buckets and
quantiles are skipped. A labelled series is stored under
"<target>:k=v,…". Each target also records is_up, scrape_ms, series_kept
and series_capped.
- Pages: targets not answering first, with why; a target's series grouped
by metric with their newest values; a chart per series; the editor with
Test, which scrapes once and says what would be stored.
- The counter-to-rate keeper moves to steward/core/counter_rates.py; SNMP's
RateKeeper now scales its rate on top of it.
- metric_value reads Prometheus's _seconds as a span and _per_sec as a rate.
- The catalog guard treats an empty list as a dynamic source (snmp,
prometheus), as the rule form does.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The Alerts widget (/alerts/widget) is shaped like every widget: a verdict
("2 firing, 1 silent." / "Nothing firing: all 12 rules are quiet."), then
what is firing or silent with its reading now, what the rule fires on and
since when, then what someone acknowledged and who; a line when alert
email failed in the last 7 days or is retrying (outbox.alert_trouble),
linking admins to Settings -> System; the quiet rules counted, a covering
rule once. It sits beside the Overview in the recommended layout (6x4
each).
The Overview's "+N more" line pointed at the removed /status/; alerts now
link to /alerts/ and monitors to /monitors/.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5509 "Delivery you can see: each alert's email state in history, and
outgoing email on Settings -> System" (milestone 501).
- Alerts -> History: an alert's email reads from the outbox (#5506):
"gathering" with when it sends, "sending", "retrying" with the attempt,
next try and the server's last answer, "sent", or "failed" naming each
reader it didn't reach and why (outbox.delivery / email_states). Events
from before the outbox keep what they recorded. The Notified column
shows once an alert has a queued email.
- Settings -> System gets "Outgoing email": every email of the last 7
days not yet sent, failures first, then the longest retrying, with
recipient, kind, state and last error, and a count of what was sent.
Failures and anything still retrying after 5 minutes lead the page's
verdict (outbox.outgoing).
- get_alert_history (MCP) adds `email`: the same delivery state.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Run 8942 failed this test alone (118 passed): it read the row back in a
second asyncio.run, and the app's pooled connection belonged to the
first, closed loop. The behaviour held ("The alerts this email carried
were deleted"); the settled row is now read from attempt's own result.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5508 "Every other email through the outbox: weekly report, invites,
password resets, and the test buttons" (milestone 501).
- outbox.send_each queues one email per reader (claimed, so the sender
doesn't take it too) and tries them at once, for a page that answers
with what happened; outbox.said words the result. attempt takes a cfg
for a test of the form's unsaved settings and returns the settled rows.
- Weekly report (scheduled and Send now) and invites: queued, tried at
once; one the server couldn't take yet is retried for an hour. An invite
that didn't go shows its link with the reason, as before.
- Password reset: queued in the reset link's own transaction; the sender
delivers it on its next tick.
- Settings -> Notifications test email and a rule's Send test: queued and
recorded with retry=False, tried once (System 156 ruling).
- Removed mail.send, notifications._send_email and _smtp_send_sync;
dispatch_notifications is webhook only. A guard asserts only the outbox
(and smtp_open itself) opens an SMTP connection, with a self-test.
Also fixes run 8941 (#5507's integration test): its Also-send-to
addresses were "@test", which is_real_address rightly drops; they are
example.com now.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5507 "Alert emails through the outbox, gathered for 2 minutes"
(milestone 501). Replaces the in-memory core/alert_burst.py.
- outbox.hold_alert: an alert joins each recipient's held alert email, or
opens one due 2 minutes later (System 156 ruling). A Postgres advisory
lock keeps two alerts at once from opening two windows for one reader.
- email_outbox_events.alert_json (migration 0043) keeps the alert as it
fired, so the email written when the window closes says what it said.
- write_held writes it then, in the reader's zone: the usual alert email
for one alert, the list for several (outbox.alert_words).
- after_settle records on each AlertEvent, once every email carrying it
has settled: email_sent, or email_error naming each reader it didn't
reach and the server's reason.
- A held email survives a restart and goes out when its window closes.
- A refusal of a message's one recipient no longer repeats the address.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5506 "Email outbox: a table of outgoing email and a sender that retries
until accepted, for up to an hour" (milestone 501).
- email_outbox (migration 0042) holds each message per recipient, its
words, its state (held / pending / sending / sent / failed), attempts and
the server's last answer. email_outbox_events links alert events for
#5507.
- core/outbox.py: enqueue; a sender task every 15 s claims what is due
(SKIP LOCKED, stale claims released), sends the batch over one
connection with no session held, and settles each row:
- accepted -> sent;
- a 5xx refusal -> failed at once with the server's words;
- no connection, a timeout, a 4xx -> retried after 30 s, 1, 2, 5, 10,
15 min, the last try at the hour, then failed;
- a test (retry=False) settles on its first answer.
It runs on startup, so a row left by a restart goes out at boot.
- notifications.smtp_open is the one place a connection is opened.
- mail.message takes the html part as optional.
- cleanup ages email_outbox rows on the data window.
Senders move onto it in #5507 and #5508.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5469 "Settings → System: table sizes and what the last cleanup removed".
- Storage: each table's size on disk (pg_total_relation_size, indexes and TOAST
included) and Postgres's row estimate (reltuples), largest first, the 15
largest by name and the rest summed, under the database's total size. Read
from the catalog only, never by counting rows.
- Last cleanup: when it ran, how long it took, and what each pass removed (each
aged table, run log files, the metric and Docker rollups, the log ring), or
"Nothing was past its window". run_cleanup now returns that summary and keeps
it as the cleanup.last_run setting, as reports.last_sent_at does; cleanup_secs
(#5467) is recorded from the same figure.
- A hint links to where each retention window is set.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5468 "Log search by service name: searching "steward_app" refuses because every
deploy makes a new container name".
search_container_logs(container="steward_app") was refused, because no container
carries the service's own name and five task containers partly matched. Now a
service's name reaches every task container Steward holds anything for, live and
removed (docker_svc.service_containers, as the service page's logs already do),
so one search spans deploys. Each match carries the slot and task_id that wrote
it, and the reply names the service and how many tasks were searched. A container
named exactly what was asked still takes precedence. A miss that partly matches
task containers now names their services.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5490 "A day of one metric reads fast even cold". A cold 24 h series read
measured 18.3 s against 63 ms warm (#5485): every sample's row sat on its own
heap page. ix_plugin_metrics_series_value carries `value` (INCLUDE), so the
read is an index-only scan; it replaces the plain series index from 0023.
It is built once, with CREATE INDEX CONCURRENTLY from a task started after the
app is serving, never in a migration, so the boot isn't held while it builds
over the live table (#5220) and ingest and pages carry on. A build cut short
by a restart leaves an INVALID index; the next boot drops and rebuilds it.
Later boots find it present and only check. Postgres keeps it current on each
insert, as it did the index it replaces.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5487 "MCP: edit inventory group membership". Group membership decides which
hosts a group alert rule covers, and the MCP could only read it. The new
operator tool takes a group and lists to add and remove, each a target's
name or id or the Steward host it is linked to; an unknown name refuses the
whole change. It answers with the group's targets after and the alert rules
covering the group (whose per-host rules follow within a minute), and audits
as ansible.group.updated by "<user> via MCP", as the Inventory page does.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5485 "A rule's page waits up to 25 s for its metric preview: a cold 24 h
series read from plugin_metrics". Opening "Agent offline" took 25.6 s; a
cold day of one series measured 18.3 s over MCP, a warm one 63 ms. The
pickers still come with the page from plugin_metric_latest; the preview is
its own request (hx-trigger=load), showing "Reading the last day…" until it
lands. Picker swaps no longer read it either.
#5486 "Send test where it can be found": the button moves from inside "What
happens" to the page header beside History, its result under the verdict.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5475 "Rule form: pick the host, then its part; metrics that part reports;
a link to groups; a preview of the metric", from the operator's review of
the New rule page: hosts and their parts shared one long picker.
- Host agent rules pick a Host, then a Part: "The whole host", then its
mounts, CPU cores, disks, interfaces and sensors under a heading each,
cores in number order. A part that reports nothing a rule can watch
(an interface's traffic) isn't offered.
- The Metric list holds what the picked resource reports, in catalog order.
- Under it, a preview: the reading now, and the last 24 h low to high with
an hourly sparkline drawn from zero (UI ruling 8), read from one series.
- The group picker links to Ansible > Inventory to make a group, in a new
tab, and re-reads the groups when the tab comes back into view.
- Picking a host or part swaps the pickers alone (/alerts/api/rule_target);
picking a metric swaps the preview (/alerts/api/metric_preview).
- A rule's own resource and metric stay on offer after they go quiet.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
