#5673 "S3 client and the backup job: settings, schedule, history, metrics":
- steward/backup/s3.py: SigV4 over httpx, path-style; put and list only
(an append-only key is enough; retention is the bucket's). Signing is held
to AWS's published examples.
- steward/backup/job.py: a 15-minute tick. The daily backup goes at the hour
chosen on the reader's clock; a missed day is made up, and a failure is
retried an hour later. History of the last 30 runs, audit
(backup.made / backup.failed), and backup_age_secs / backup_bytes under
"Steward itself", resource backup, for alert rules.
- Settings → Backups: verdict first, then the bucket (secret sealed; a
blank box keeps it), the schedule, Test the bucket, Back up now, and the
recent runs.
#5675 "Docs: backing up to Archives, and restoring": docs/core/backups.md.
Also: the audit page now words the Prometheus target actions from #5668,
which showed as codes, and a guard fails on any logged action without
wording.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The Alerts widget (/alerts/widget) is shaped like every widget: a verdict
("2 firing, 1 silent." / "Nothing firing: all 12 rules are quiet."), then
what is firing or silent with its reading now, what the rule fires on and
since when, then what someone acknowledged and who; a line when alert
email failed in the last 7 days or is retrying (outbox.alert_trouble),
linking admins to Settings -> System; the quiet rules counted, a covering
rule once. It sits beside the Overview in the recommended layout (6x4
each).
The Overview's "+N more" line pointed at the removed /status/; alerts now
link to /alerts/ and monitors to /monitors/.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5509 "Delivery you can see: each alert's email state in history, and
outgoing email on Settings -> System" (milestone 501).
- Alerts -> History: an alert's email reads from the outbox (#5506):
"gathering" with when it sends, "sending", "retrying" with the attempt,
next try and the server's last answer, "sent", or "failed" naming each
reader it didn't reach and why (outbox.delivery / email_states). Events
from before the outbox keep what they recorded. The Notified column
shows once an alert has a queued email.
- Settings -> System gets "Outgoing email": every email of the last 7
days not yet sent, failures first, then the longest retrying, with
recipient, kind, state and last error, and a count of what was sent.
Failures and anything still retrying after 5 minutes lead the page's
verdict (outbox.outgoing).
- get_alert_history (MCP) adds `email`: the same delivery state.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5508 "Every other email through the outbox: weekly report, invites,
password resets, and the test buttons" (milestone 501).
- outbox.send_each queues one email per reader (claimed, so the sender
doesn't take it too) and tries them at once, for a page that answers
with what happened; outbox.said words the result. attempt takes a cfg
for a test of the form's unsaved settings and returns the settled rows.
- Weekly report (scheduled and Send now) and invites: queued, tried at
once; one the server couldn't take yet is retried for an hour. An invite
that didn't go shows its link with the reason, as before.
- Password reset: queued in the reset link's own transaction; the sender
delivers it on its next tick.
- Settings -> Notifications test email and a rule's Send test: queued and
recorded with retry=False, tried once (System 156 ruling).
- Removed mail.send, notifications._send_email and _smtp_send_sync;
dispatch_notifications is webhook only. A guard asserts only the outbox
(and smtp_open itself) opens an SMTP connection, with a self-test.
Also fixes run 8941 (#5507's integration test): its Also-send-to
addresses were "@test", which is_real_address rightly drops; they are
example.com now.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5507 "Alert emails through the outbox, gathered for 2 minutes"
(milestone 501). Replaces the in-memory core/alert_burst.py.
- outbox.hold_alert: an alert joins each recipient's held alert email, or
opens one due 2 minutes later (System 156 ruling). A Postgres advisory
lock keeps two alerts at once from opening two windows for one reader.
- email_outbox_events.alert_json (migration 0043) keeps the alert as it
fired, so the email written when the window closes says what it said.
- write_held writes it then, in the reader's zone: the usual alert email
for one alert, the list for several (outbox.alert_words).
- after_settle records on each AlertEvent, once every email carrying it
has settled: email_sent, or email_error naming each reader it didn't
reach and the server's reason.
- A held email survives a restart and goes out when its window closes.
- A refusal of a message's one recipient no longer repeats the address.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5467 "Steward's own health: metrics, rules can watch them, a page that shows them".
The self-monitor tick now records, under source "steward", resource "process":
ingest_posts / ingest_failed / ingest_slowest_ms (the host-agent posts since the
last tick, tallied in steward/core/health.py), db_roundtrip_ms (a timed SELECT 1),
and unwatched_secs (the gap before this tick past watch.GAP_S, from Steward's own
last sample). The hourly cleanup records cleanup_secs. The open-file pair is still
recorded, and is now only skipped on its own where /proc is absent. All of these
are in CORE_METRICS, so an alert rule can watch them under "Steward itself".
Settings → System (/settings/system/, admin) shows a verdict first (failed posts,
a gap in watching, open files near the limit), then each measure now and over the
last day, with a sparkline scaled to its own peak.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5490 "A day of one metric reads fast even cold". A cold 24 h series read
measured 18.3 s against 63 ms warm (#5485): every sample's row sat on its own
heap page. ix_plugin_metrics_series_value carries `value` (INCLUDE), so the
read is an index-only scan; it replaces the plain series index from 0023.
It is built once, with CREATE INDEX CONCURRENTLY from a task started after the
app is serving, never in a migration, so the boot isn't held while it builds
over the live table (#5220) and ingest and pages carry on. A build cut short
by a restart leaves an INVALID index; the next boot drops and rebuilds it.
Later boots find it present and only check. Postgres keeps it current on each
insert, as it did the index it replaces.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5487 "MCP: edit inventory group membership". Group membership decides which
hosts a group alert rule covers, and the MCP could only read it. The new
operator tool takes a group and lists to add and remove, each a target's
name or id or the Steward host it is linked to; an unknown name refuses the
whole change. It answers with the group's targets after and the alert rules
covering the group (whose per-host rules follow within a minute), and audits
as ansible.group.updated by "<user> via MCP", as the Inventory page does.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5485 "A rule's page waits up to 25 s for its metric preview: a cold 24 h
series read from plugin_metrics". Opening "Agent offline" took 25.6 s; a
cold day of one series measured 18.3 s over MCP, a warm one 63 ms. The
pickers still come with the page from plugin_metric_latest; the preview is
its own request (hx-trigger=load), showing "Reading the last day…" until it
lands. Picker swaps no longer read it either.
#5486 "Send test where it can be found": the button moves from inside "What
happens" to the page header beside History, its result under the verdict.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5475 "Rule form: pick the host, then its part; metrics that part reports;
a link to groups; a preview of the metric", from the operator's review of
the New rule page: hosts and their parts shared one long picker.
- Host agent rules pick a Host, then a Part: "The whole host", then its
mounts, CPU cores, disks, interfaces and sensors under a heading each,
cores in number order. A part that reports nothing a rule can watch
(an interface's traffic) isn't offered.
- The Metric list holds what the picked resource reports, in catalog order.
- Under it, a preview: the reading now, and the last 24 h low to high with
an hourly sparkline drawn from zero (UI ruling 8), read from one series.
- The group picker links to Ansible > Inventory to make a group, in a new
tab, and re-reads the groups when the tab comes back into view.
- Picking a host or part swaps the pickers alone (/alerts/api/rule_target);
picking a metric swaps the preview (/alerts/api/metric_preview).
- A rule's own resource and metric stay on offer after they go quiet.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5466 "Send test: check a rule's delivery from the rule". A rule's page gets
a Send test button (operator role), and MCP gets send_test_alert. Each sends
a TEST alert for that rule now, by email to everyone who gets alerts and by
webhook. The alert carries the rule's name and condition, at the latest
reading of what it watches; a rule covering hosts uses its first host. The
result for each channel shows under the button, or "Nothing is set up to
send to". Nothing fires, no event is written, and the test is audited as
alert_rule.tested. An alert with no reading now says "has no reading yet"
instead of "is None".
Docs: alerting.md also gains the burst email from #5465 "One email per burst
of alerts".
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5464 "Group rules: one rule covers every host, or every host in a group",
step 2 of milestone 499 "Alerts you can trust". Each rule named one host, so
the live instance carried six rules for three hosts, and a new host had no
agent-offline or disk alert until someone copied them.
- alert_rules.covers / cover_group / parent_id (migration 0041). A rule on
host_agent may cover "all_hosts" or a "group" (an inventory group: the
Hosts ruling, #5294). A covering rule has no state; Steward keeps one
ordinary child rule per covered host, named "<rule>: <host>", so each
host's alert keeps its own state, history and acknowledgement.
- alerts.service.reconcile_children: a covered host that has reported in
the last 7 days gets a child; a host no longer covered loses its child;
children take the parent's condition, checks, notify setting, action and
on/off. Run on save and each minute (alert_reconcile).
- A child is changed through its parent: edit, toggle and delete of a child
are refused (409 / PartOfCoveringRule); its edit page opens the parent.
- Rule form: a Covers choice for host sources, with the group picker; fixed
once the rule exists. Rules list: a covering rule reads "every host in
swarm · 3 hosts" with its hosts' rules nested under it, each opening its
history. The history picker and the rule count leave covering rules out.
- MCP: create_alert_rule takes covers/group; update_alert_rule takes group
and passes changes down; list_alert_rules shows covers, hosts and part_of.
- docs/core/alerting.md: "Rules that cover hosts".
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5463 "Acknowledgements on record: an event and who did it", step 1 of
milestone 499 "Alerts you can trust". The live walk in #5461 found that
acknowledging changed only the alert's current state: once it resolved,
nothing said who had acknowledged it or when.
- alert_events.actor_id (migration 0040): the person behind a transition.
- Acknowledging writes a firing → acknowledged event by the acknowledger.
- A rule edit, disable or enable that resets an alert carries its actor
(web and MCP alike).
- The history page notes "acknowledged by X" and "rule disabled by X"; MCP
get_alert_history returns `by`.
- One usernames() helper in alerts.service, used by the active list, the
history page and the MCP tools.
- docs/core/alerting.md: the state diagram no longer shows an acknowledged
alert re-firing, and says what an event records.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The dashboard's host widgets took 18-60 s. Every "latest reading" query
(hosts overview, host-agent fleet, host page, vitals, SNMP, MCP get_host)
searched plugin_metrics for the newest row per metric. Postgres has no skip
scan, so each one read every raw sample in the 7-day window.
- plugin_metric_latest: one row per (source, resource, metric).
record_metrics upserts it in the same transaction, moving forward only and
in sorted key order. Migration 0030 seeds it by stepping the existing index.
- Retention drops a latest row once its sample leaves the raw window, so a
resource that stopped reporting disappears when its history does, as before.
Renames carry it along.
- A quiet host keeps its last values, marked stale. A time-bounded query
would have shown it as missing.
- Vitals sparklines span the 6 h they claim, averaged in SQL to ~120 points,
instead of fetching 6 h raw and drawing the last 120 samples.
- Dashboard and share-view cells: "Loading…" until the first answer; a 20 s
deadline per fetch; "Couldn't load (why)" with Retry now and an automatic
retry for cells that don't poll; "as of HH:MM" in the header, and
"couldn't refresh · as of" when a refresh fails over shown data.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- One run per host: start_run claims every host it touches (addresses read
from the inventory Steward wrote; a source's own inventory file is claimed
as that file) before it takes a run slot. The claim is all-or-nothing. A run
waiting on a host shows queued and names the run it waits for, honours a
cancel, and fails after HOST_WAIT_SECONDS (1 h) with which host and run.
- A due schedule whose previous run is still queued or running skips the
firing: last_run_at moves on, last_run_id still names the run going, and
last_error says "Skipped: the previous run was still going". The Schedules
page shows that beside the run (muted), and a launch error in red.
- An alert rule that fires while the action run from an earlier firing is
still going starts no new run; the event says so and names the run. An
in-process set closes the gap between the check and the run being linked.
- docs/core/ansible.md: queued/cancelled statuses and the overlap rules.
Tests: tests/core/test_ansible_overlap.py (claim keys, waiting, all-or-nothing,
cancel, deadline) and tests/integration/test_ansible_overlap.py (two real
runs on one host take turns; schedule skip then fire; alert action skip then
start).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- git ships in the image; checkouts live under /data/ansible/sources
(STEWARD_ANSIBLE_SOURCE_DIR), on the persistent volume.
- One task, ansible_git_pulls, reads the sources every 30 s and pulls each
as its interval passes: a source added in Settings is pulled without a
restart, a removed one is forgotten.
- git_pull: a 120 s timeout ending the process group, GIT_TERMINAL_PROMPT=0,
pull --ff-only, a failed clone removed; it raises GitPullError with git's
message, so the Sync button stops saying Synced after a failure (and the
message is now escaped).
- An alert action's vars and the firing's steward_alert_* vars reach
ansible-playbook as one JSON extra-vars file, not -e key=value, which
Ansible split on whitespace (a rule named 'High CPU on web').
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
MCP-8 #4972 "Connect card, docs, and end-to-end integration test".
- AI access page: a "Connect a client" card with the /mcp URL (from
Settings → General → Public base URL, else the request host) and recipes
for Claude Code, Claude Desktop (mcp-remote) and other Streamable HTTP
clients, plus the reverse-proxy note. The token-created page shows the
same recipes with the new token filled in, ready to paste.
- docs/core/mcp.md: connecting, the reverse-proxy requirements, the
security model (token ceiling, default deny, check-mode default, secrets,
audit), tools by role, response conventions, adding tools.
docs/architecture.md describes the /mcp dispatch outside the blueprint
pipeline; docs/reference/code-map.md gains an MCP section.
- tests/integration/test_mcp.py: over HTTP, a viewer token lists no write
tools and is refused one by name; an operator token reads get_overview
and its maintenance-window create/end land in the audit log as
"<user> via MCP" with the token id. (Revoked → 401 was already covered.)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Renames the Python package directory, CLI command, env var prefix,
docker-compose service/container/image, Postgres role/db, and all
visible branding. Marketing form is "Fabled Steward".
Clean break from the previous rebrand: drops the fabledscryer→roundtable
import shim in __init__.py and the FABLEDSCRYER_* env var fallback in
config.py and migrations/env.py. Env vars are now STEWARD_* only.
Heads-up for existing deployments:
- Postgres user/db renamed fabledscryer → steward in docker-compose.yml.
Existing volumes need the role/db renamed inside Postgres, or override
POSTGRES_USER/POSTGRES_DB to keep the old names.
- Host-agent systemd unit is now steward-agent.service. Existing agents
keep running under the old name; reinstall to switch.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Sweeps README, docs/, example configs, code docstrings, report and
notification subject lines, plugin catalog URL default, and stray
comments in Dockerfile/entrypoint.sh/.gitignore. Adds a legacy note
to docs/core/configuration.md explaining the FABLEDSCRYER_* fallback.
docker-compose DB credentials intentionally left as fabledscryer to
preserve the existing pgdata volume.
