#5741 "Traefik routers are graded against their own normal", milestone 513.
- A router's 4xx share and p95 learn their normal (learn_normal in the
plugin's setup). Once a router has 5 days of history, a reading over the
fixed cutoff but inside its normal is healthy, and the healthy table says
it is usual for that router. One out of its normal is a warning that says
what is usual, and a metric anomaly.
- The 5xx share is a failure: fixed cutoff always, whatever the history or
traffic (System 182's ruling).
- A 4xx share over fewer requests than Settings -> Thresholds' new "Fewest
requests to judge 4xx" (30 per 15 minutes) isn't graded on the page, and
isn't judged for anomalies.
- Core: learn_normal(gate=(metric, least)) judges a key only while its
resource's gate metric averages at least `least`; `least` may be a
function, so it follows a setting. learn_normal(since=...) learns only
from history after a moment: Traefik latency learns from #5755's
per-scrape figures only.
- The Traefik page lists the routers' metric anomalies of the last 30 days
(_anomaly_card.html). The dashboard widget keeps its why to problems.
- Thresholds and docs say when the cutoffs give way to a normal.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5752 "Container CPU and memory history lives in the one metric store;
docker_metrics and docker_metrics_hourly are retired". Container cpu_pct and
mem_pct already went to plugin_metrics for alerting; now mem_usage_bytes does
too, and every snapshot in an ingest (backlog included) is recorded at its
own time. The container history chart, Busiest Containers, a service's
history and the MCP container tools read plugin_metrics under resource
"<host>/<container>", rolled up and pruned by the shared metric retention.
docker_012_drop_metrics drops both tables. Their history is not copied
(lesson #5221 "A data migration's cost is set by the live table"); the
store already holds 7 days of container CPU and memory %. Docker retention
now prunes only lifecycle events, and Settings -> Thresholds has one
"Metric retention" card for every source.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5740 "A metric that departs from its normal is recorded as an anomaly to
review, and is findable after it ends", milestone 513.
- Every minute each learned key's last 15 minutes is judged against its
hour's normal: above the usual hold x1.25 and by the metric's floor opens
an occurrence, which ends when the level comes back.
- metric_anomalies holds one row per item to review; a repeat while it
still waits joins it as an occurrence (metric_anomaly_spans).
- An ended row nobody reviews within anomalies.review_days (14, Settings ->
Thresholds) lapses; an ongoing one never does.
- Learning skips every hour an occurrence covers unless its anomaly was
marked normal: no normal is learned without review.
- Alerts -> Anomalies lists those to review first, then the reviewed and
lapsed under day dividers; marking is operator/admin and audited.
- The Overview widget and get_overview warn of open ones and count those
to review; the host page shows its own; the weekly report lists the
week's and the lapsed; MCP list_anomalies and review_anomaly.
- Reviewed and lapsed rows are pruned at data.retention_days, and kept
in backups like alert history.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5673 "S3 client and the backup job: settings, schedule, history, metrics":
- steward/backup/s3.py: SigV4 over httpx, path-style; put and list only
(an append-only key is enough; retention is the bucket's). Signing is held
to AWS's published examples.
- steward/backup/job.py: a 15-minute tick. The daily backup goes at the hour
chosen on the reader's clock; a missed day is made up, and a failure is
retried an hour later. History of the last 30 runs, audit
(backup.made / backup.failed), and backup_age_secs / backup_bytes under
"Steward itself", resource backup, for alert rules.
- Settings → Backups: verdict first, then the bucket (secret sealed; a
blank box keeps it), the schedule, Test the bucket, Back up now, and the
recent runs.
#5675 "Docs: backing up to Archives, and restoring": docs/core/backups.md.
Also: the audit page now words the Prometheus target actions from #5668,
which showed as codes, and a guard fails on any logged action without
wording.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The UniFi password sat in plugin.unifi in plain text: SECRET_KEYS covers
whole settings, and a plugin's settings are one dict. Saving the plugin's
settings page also wiped it, since the password box is never filled back
in and its blank value was stored.
- A plugin field whose name holds password, secret or token is a
credential (is_plugin_secret). set_setting seals it inside the dict,
every read opens it, startup seals any plaintext left from before, and
the undecryptable-secrets banner names it as plugin.<name>.<field>.
- The settings page uses the same test for its password boxes, and a blank
box keeps the saved value.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5509 "Delivery you can see: each alert's email state in history, and
outgoing email on Settings -> System" (milestone 501).
- Alerts -> History: an alert's email reads from the outbox (#5506):
"gathering" with when it sends, "sending", "retrying" with the attempt,
next try and the server's last answer, "sent", or "failed" naming each
reader it didn't reach and why (outbox.delivery / email_states). Events
from before the outbox keep what they recorded. The Notified column
shows once an alert has a queued email.
- Settings -> System gets "Outgoing email": every email of the last 7
days not yet sent, failures first, then the longest retrying, with
recipient, kind, state and last error, and a count of what was sent.
Failures and anything still retrying after 5 minutes lead the page's
verdict (outbox.outgoing).
- get_alert_history (MCP) adds `email`: the same delivery state.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5508 "Every other email through the outbox: weekly report, invites,
password resets, and the test buttons" (milestone 501).
- outbox.send_each queues one email per reader (claimed, so the sender
doesn't take it too) and tries them at once, for a page that answers
with what happened; outbox.said words the result. attempt takes a cfg
for a test of the form's unsaved settings and returns the settled rows.
- Weekly report (scheduled and Send now) and invites: queued, tried at
once; one the server couldn't take yet is retried for an hour. An invite
that didn't go shows its link with the reason, as before.
- Password reset: queued in the reset link's own transaction; the sender
delivers it on its next tick.
- Settings -> Notifications test email and a rule's Send test: queued and
recorded with retry=False, tried once (System 156 ruling).
- Removed mail.send, notifications._send_email and _smtp_send_sync;
dispatch_notifications is webhook only. A guard asserts only the outbox
(and smtp_open itself) opens an SMTP connection, with a self-test.
Also fixes run 8941 (#5507's integration test): its Also-send-to
addresses were "@test", which is_real_address rightly drops; they are
example.com now.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5469 "Settings → System: table sizes and what the last cleanup removed".
- Storage: each table's size on disk (pg_total_relation_size, indexes and TOAST
included) and Postgres's row estimate (reltuples), largest first, the 15
largest by name and the rest summed, under the database's total size. Read
from the catalog only, never by counting rows.
- Last cleanup: when it ran, how long it took, and what each pass removed (each
aged table, run log files, the metric and Docker rollups, the log ring), or
"Nothing was past its window". run_cleanup now returns that summary and keeps
it as the cleanup.last_run setting, as reports.last_sent_at does; cleanup_secs
(#5467) is recorded from the same figure.
- A hint links to where each retention window is set.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5467 "Steward's own health: metrics, rules can watch them, a page that shows them".
The self-monitor tick now records, under source "steward", resource "process":
ingest_posts / ingest_failed / ingest_slowest_ms (the host-agent posts since the
last tick, tallied in steward/core/health.py), db_roundtrip_ms (a timed SELECT 1),
and unwatched_secs (the gap before this tick past watch.GAP_S, from Steward's own
last sample). The hourly cleanup records cleanup_secs. The open-file pair is still
recorded, and is now only skipped on its own where /proc is absent. All of these
are in CORE_METRICS, so an alert rule can watch them under "Steward itself".
Settings → System (/settings/system/, admin) shows a verdict first (failed posts,
a gap in watching, open files near the limit), then each measure now and over the
last day, with a sparkline scaled to its own peak.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5427 "A password reset doesn't end the sessions already signed in with
the old password": users.sessions_since (migration 0038) is set on reset;
refresh_session_user ends a session whose login_at is earlier. An admin
resetting their own password carries on signed in.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The operator's ask: an invite link sent by mail, the key in the URL, and
the new user choosing their own username, password and email.
- user_invites (migration 0037): the key's sha256 only, the address, the
role and days the admin chose, who sent it, used/revoked.
- core/invites: create (revokes an older open invite to the address),
renew (new key, new clock), find_pending, the four states, signup checks.
- /invite/<key>, outside sign-in and throttled like login: the invitee's
username, email (prefilled, theirs to change) and password; the account
takes the admin's role, the key is used up in the same transaction and
they are signed in. A bad key gets one plain message.
- Settings › Users: Invite by email (address, role, 1/7/30 days), sent from
the admin in their zone; without a mail server, or if sending fails, the
link is shown once to copy. Invites not yet used are listed with Resend
and Withdraw.
- Audit: user.invited, user.invite_resent, user.invite_revoked, user.joined.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Operator review of the Account page: two sections with two different
save buttons, one stuck to the next card and one beside a squeezed hint.
- Account: time zone and Email are one form; POST /account/ saves both,
nothing unless all of it is valid. /account/email is gone.
- Users › account: Name and Email are one form; the rename route also
saves the email switches. /settings/users/<id>/email is gone.
- The Email section is a plain section now (email_section); its hint sits
under the switches, names the zone as the picker does, and links
Notifications only for an admin.
- apply_email_form leaves a switch the form doesn't carry as it is.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Reading the settings autobegins the session's transaction, so the
nested db.begin() that saves the first admin's zone as smtp.tz raised
InvalidRequestError and every visit 500'd until a zone was saved. The
integration test signed in without a zone, so that path never ran;
it now has a test of its own.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Milestone 495 "Email goes to each account, in its own time zone", on the
operator's ruling (System 156): notifications go to each user, times in
their zone, defaults on, toggled per user or per role.
- users.notify_alerts / notify_report (migration 0036): None follows the
role's default, notify.<kind>.<role> in settings, all on to start.
- mail.pick_recipients: active accounts that want the email and have an
address mail can reach (an SSO placeholder like name@oidc is skipped and
flagged), each in its own zone with no UTC beside it, then the Also send
to addresses in their chosen zone with UTC beside. One copy per address,
all over one SMTP connection.
- Alerts, the weekly report and the test email (to you, in your zone, plus
the typed addresses) send that way. The save-time smtp.tz capture is gone.
- Notifications: a role × email table of defaults, who each email reaches
and who can't be reached, and Also send to with its own zone, which starts
as the first admin's to open the page.
- Account page: your address and your two switches. Users: an Emailed
column, and an admin sets an account's switches on its page.
- The Alerts page counts the accounts it emails. zone_select macro and
formatting.zones_by_region shared by Account and Notifications.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5336 "Users page: list accounts, change role, disable, reset a local
password, create a local user".
- Settings › Users (admin only, a Settings tab): every account with how it
signs in, its role, last sign-in (from the audit log) and working AI
tokens; disabled accounts last. A row opens the account.
- An account's page: rename (any account), role and password (local
accounts; an LDAP or single sign-on role comes from the provider at
every sign-in, and the page says so), disable or enable, and Delete
behind the confirm dialog.
- Refusals are said beside the control: nobody disables, demotes or
deletes themselves, the last active admin stays, and an account that
made alert rules, maintenance windows or playbook runs is disabled
rather than deleted (those records keep who made them).
- Add user: a local account with a role and a password of 8+ characters.
- Every change is audited as user.*, and the audit log names and links the
account.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5335 "Emails Steward sends read as designed messages: an HTML alert email
and weekly report, with a plain-text twin".
- core/mail.py renders each email as multipart/alternative from
templates/email/: a table-and-inline-style HTML part and a plain-text
twin carrying the same facts. Colours are the design system's light
theme, resolved from theme.css; nothing is fetched when a mail opens.
- Alert email: state badge, a headline sentence ("Disk used on web01 is
94%."), rule, resource, reading, the condition in words, and the time in
the zone Notifications was saved in with UTC beside it; "View in
Steward" when Steward's address is set. Subject "[Steward] Firing: <rule>
— <resource> 94%". Alerts now carry their operator.
- Weekly report: monitors down, active alerts, the week's alert events,
uptime and busiest Traefik routers as small tables with links. Sent from
the configured From address; the "Generated … UTC" line and the stale
repo footer are gone.
- condition_text() states a condition from a rule's parts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The last page of #5214's walk. Settings still described Steward as it was
before monitors were unified and SNMP was rebuilt, drew its tabs below the
breadcrumb, and showed the report schedule in UTC.
- Tabs (#5377): settings/_tabs.html is a settings_tabs() macro called in
each page's section_tabs block, above the trail, as Ansible's are. The
breadcrumb guard also fails any page that {% include %}s a tab strip.
- Report time (#5378): reports.schedule_tz keeps the zone of whoever saved
the schedule; check_and_send compares in it (report_due), and the form shows
the day and hour on the reader's clock (schedule_in) under their zone's
name. "Last sent" uses |when.
- Wording (#5379): the report contents list what the report holds (monitors,
not ping-enabled hosts); "Ping latency" is "Response time" and says what it
grades; plugin detail says enabling loads now and disabling needs a
restart; the monitor interval says it applies on the next check (the
scheduler reads it every pass); the dead list-config branch is gone.
- Traefik (#5380): the 5xx and 4xx cutoffs are settings
(traefik.threshold.error_5xx_pct 0.1, error_4xx_pct 1.0) in a "Traefik
routers" group on Thresholds while the plugin runs, read by router_state
through error_cutoffs(). The Traefik page has a Thresholds button for
admins, to /settings/thresholds/#traefik.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
First-party plugins have shipped inside the image since a7a281c, and the
catalog's only repository (Steward-plugins) answers 404, so Steward polled a
dead URL every minute for nothing. The operator chose to remove the catalog
entirely (#5214 "Review: readable and fast on the live instance").
Removed:
- core/plugin_index.py and the plugin_catalog_refresh startup task;
- the plugins.index_url default, and migration 0035 deletes the stored
plugins.index_url / plugins.repositories rows;
- the Settings -> Plugins repository editor, catalog, refresh, install and
reload routes and their templates;
- the download/checksum/extract install path in plugin_manager, and the
PLUGIN_INSTALL_DIR config it wrote to;
- plugins/index.yaml, docs/plugins/index.yaml.example and the guide's
"Publishing to the Catalog" section (now "Shipping a Plugin").
Kept: enabling and configuring plugins, hot-reload on enable, restart, orphan
cleanup, and the external root (/data/plugins) for operator-mounted plugins.
tests/test_no_plugin_catalog.py guards the routes, default and task.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The operator's ruling (Background Jobs, 2026-10-08): a Steward page reads
stored records; the sources are contacted by scheduled collection. An audit
of every GET handler, HTMX fragment, widget poll and MCP read tool found one
page load that reached out: Settings -> Plugins' catalog fragment fetched
each plugin repository (5 min cache, 10 s timeout each, retried on every
load when unreachable).
plugin_catalog_refresh ticks every minute and fetches a repository whose
copy is missing or older than an hour, waiting 5 min after a failure;
saving repositories clears the cache, so the next tick fetches. The
fragment only reads the cache and says which repositories are not fetched
yet. Refresh is now a POST that fetches at once, as does install.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Agent 1.12.0 stamps each post with sent_at; ingest compares it with the
moment the post arrived: clock_offset_s / clock_checked_at on the
registration (host_agent_003), and a clock_off_secs metric (either way)
that alert rules can watch. Replaces the old skew log, which measured the
newest sample and was off by up to an interval.
- Settings -> Thresholds: Clock drift, 30 s warn / 5 min critical.
- Host page: 'clock 0.3s ahead' in the vitals line, toned past the warning.
- Hosts list: names hosts whose clock is off; when every reporting host is
off by about the same amount, says Steward's own clock is the likely one.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Dockerfile: python:3.14-slim, the Python CI tests on (was 3.13). tini from
apt is PID 1 (ENTRYPOINT tini -- entrypoint.sh): it reaps the zombies ssh
and ansible leave and forwards SIGTERM.
- cli: Hypercorn's serve() with one worker (run, stream and scheduler state is
in-process), a 10 s graceful timeout, and SIGTERM/SIGINT as the shutdown
trigger. Access and error lines go through Steward's logging. --debug keeps
Quart's reloading dev server.
- Logging: steward/core/logging_setup.py installs one root handler before
anything logs (until now a library's basicConfig on import was the only
configuration). Its level is the `log.level` setting (DEBUG/INFO/WARNING/
ERROR, default INFO), applied at boot and on save in Settings → General.
- CI publish: build, then "The image boots with no internet" (rule 164,
snippet #3552), then push, so a broken image never reaches a tag (rule 177).
On an --internal network with its own Postgres, the image must migrate and
answer /health, run tini as PID 1, serve first-run setup with its assets
from /static and no third-party host, and exit 0 on SIGTERM. Logs and
clean-up run always.
Tests: tests/core/test_logging_setup.py; the smoke step itself.
Not in this commit: the dependency lockfile (it conflicts with rule 154 and
is waiting on the operator).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- git ships in the image; checkouts live under /data/ansible/sources
(STEWARD_ANSIBLE_SOURCE_DIR), on the persistent volume.
- One task, ansible_git_pulls, reads the sources every 30 s and pulls each
as its interval passes: a source added in Settings is pulled without a
restart, a removed one is forgotten.
- git_pull: a 120 s timeout ending the process group, GIT_TERMINAL_PROMPT=0,
pull --ff-only, a failed clone removed; it raises GitPullError with git's
message, so the Sync button stops saying Synced after a failure (and the
message is now escaped).
- An alert action's vars and the firing's steward_alert_* vars reach
ansible-playbook as one JSON extra-vars file, not -e key=value, which
Ansible split on whitespace (a rule named 'High CPU on web').
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- prune_by_age(column) registers a time-series column with the hourly cleanup;
Traefik (metrics, access summaries, global stats) and UniFi (WAN, client,
WLAN, DPI, events, alarms, speedtests) register theirs in setup().
- Alert events join monitor results and Ansible runs on data.retention_days,
read from Settings each run; Ansible run log files older than it are deleted.
- audit.retention_days (365) keeps the audit log longer, set on Settings > General.
- metric_series.points reads metrics.retention.raw_days, so the chart's
raw/rollup seam is where the rollup put it (RAW_DAYS removed).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5064 "Notifications that arrive":
- render_webhook: alert.* is escaped to sit inside a JSON string (a quote
or newline in a rule name no longer breaks the body); raw.* is the plain
value for |tojson; a missing value reads "no data"; the template runs
sandboxed and the body is checked to be JSON before it is sent.
- SMTP: 15s timeout; port 465 connects with implicit TLS, otherwise the
box means STARTTLS; a From address setting (falls back to the username).
- A webhook error no longer carries the URL, which is a secret.
- Settings → Notifications: "Send a test email" / "Send a test webhook"
send a sample alert with the form's values (unsaved) and show the result
and the body that went out; template help names alert.* and raw.*.
- Notification audits go through audit_request; the test send is audited
as settings.notification.tested, with the channel and outcome only.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5037 "Secrets at rest and in run output: webhook URL and git token
encrypted, askpass via env, secret values scrubbed from logs".
- webhook.url joins SECRET_KEYS; existing values are sealed at startup by
migrate_plaintext_secrets. Settings -> Notifications no longer echoes the
URL: it names the saved host, keeps it when left blank, and has a Remove
box.
- Git source http_token is sealed inside ansible.sources on save and at
startup, opened in get_sources, and counted by the undecryptable-secrets
scan as ansible.sources[<name>].http_token.
- The askpass script holds no token: git runs it with STEWARD_GIT_TOKEN in
its environment and printf prints it as data.
- The executor masks every secret the run is handed (become/vault
passwords, bootstrap passwords, sensitive run vars, and the per-host agent
tokens the host_agent plugin now passes as redact) in each line before it
is broadcast, logged or stored.
- Bundled "Write agent config" tasks are no_log; a test fails if any bundled
task templating steward_token lacks it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5036 "SSO and LDAP: users keep their auth source, no merging into local
accounts, is_active checked, LDAP TLS real and filter escaped".
- users gain auth_source (local/ldap/oidc) and external_id (migration 0027).
A provider account is found by (source, subject) only: the LDAP entry DN
or the OIDC sub. A first sign-in whose username or email another account
holds is refused with a message naming why; a disabled user stays out.
- Local passwords sign in local accounts only.
- LDAP: port 636 is LDAPS, otherwise StartTLS before any bind; the server
certificate is checked against the system CAs or a CA pasted in
Settings -> Auth. The login name is escaped into the filter (RFC 4515),
and an empty password is refused before the directory sees it.
- OIDC: the ID token must carry this sign-in's nonce and the userinfo sub.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5035 "Sessions: lifetime applied, role and is_active re-read per request,
logout is a POST, failed logins throttled".
- steward/core/proxy.py: trusted_value / is_https / client_ip read
X-Forwarded-Proto and -For only as far as the trusted proxy hops
(Settings → General, default 1, 0 = trust none).
- The session cookie is Secure, and HSTS is sent, only on requests that
arrived over HTTPS by that rule; plain-HTTP and LAN use are unchanged
and nothing redirects.
- A login lasts Settings → General's session lifetime from when it began
(session.permanent + a login_at stamp checked every request — the
cookie's own age slides on every response). A deleted or deactivated
user is logged out on their next request; a role change applies at once.
- Logout is a POST (a form in the sidebar); GET /logout is gone.
- Failed logins pause an address after login.max_failures within
login.lockout_minutes (Settings → General; defaults 5 in 15 min),
keyed by address so nobody can lock a user out by name. Failures are
audited as auth.login_failed with the address.
- to_access_cfg projects the new settings into app.config at startup and
on save, beside the existing projections.
- Integration tests seed real users and a login_at, since every request
now re-reads the session's user.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5032 "Plugin install takes its URL and checksum from the catalog, requires
the checksum, and extracts only inside the plugin folder". The install form
carried the download URL and checksum as hidden fields, so any admin POST
could make Steward fetch and import code from anywhere, unchecked, and a zip
member like ../x.py was written wherever it pointed.
- The route takes only the plugin name and reads download_url and
checksum_sha256 from the configured catalogs; a name no repository lists
is refused. Installs are audited (plugin.install / plugin.install_failed).
- The checksum is required; the name must be a plain identifier.
- extract_plugin refuses any member that resolves outside the plugin
folder, caps the download at 50 MB and the unpacked size at 200 MB.
- The zip unpacks into a staging folder and is swapped in whole: a refused
or failed install leaves the installed copy untouched, and an update no
longer leaves the old version's deleted files behind.
- The catalog shows "Can't install: the catalog lists no checksum" in place
of the button for an entry without one.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
MCP-5 of milestone 461 "MCP server: Steward as an LLM-inspectable,
LLM-operable hub". Contributed through each plugin's register_mcp_tools
hook, so they exist only while the plugin is enabled.
docker (viewer, read-only):
- list_containers: filter by host, status, compose project, or problem
(restarting, crash_looping, unhealthy, oom_killed, exited_nonzero);
problems sort first and each row names what is wrong.
- get_container: state, CPU/memory history across the raw/hourly seam,
lifecycle events. A unique part of a swarm task name is enough.
- list_swarm_services: replicas per node (agent-less placements as ghosts)
and the swarm's nodes.
- search_container_logs: text (% and _ literal), stream, range, context
lines, paging by a full-precision `before` cursor; over-long lines keep
both ends and say how much of the middle was cut.
- get_docker_disk_usage: bytes per host, reclaimable first.
- mcp_overview: failing containers and under-replicated swarm services
join get_overview.
host_agent: list_agents (reporting / stale / pending, version, OS), and an
overview finding for agents registered over 30 minutes ago that never
checked in.
Settings > Thresholds gains the TLS-expiry warn/crit days (were 14/3 in
code) and the crash-loop deaths-per-hour count (3), which get_overview
and the docker tools now read.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The admin banner counts every plugin load failure, but settings/plugins.html
only rendered failure reasons inside the plugin card macro -- which iterates
discovered plugins. A plugin that failed BECAUSE it has no directory was
therefore counted in the banner and shown nowhere on the page the banner links
to. That dead end is what the operator hit: "1 plugin failed to load" on the
dashboard, nothing wrong on the details page, and no way to clean it up.
Folds the reason into the "Configured but not installed" rows rather than
adding a competing section, and drives that list from the UNION of stored
plugin.* settings and recorded load failures instead of stored settings alone.
A failure with no stored row -- a plugin enabled by a DEFAULTS entry, which is
exactly what plugin.http was -- would otherwise still be counted and still be
invisible.
Rows carry `removable`: only a plugin with a real stored row can be cleaned up,
so a default-declared one shows the reason and an explanation instead of a
Remove button that would delete nothing while the default reasserts it. Rows
with a reason read as an error (red, "Failed to load"); rows that merely have
leftover config stay a warning. A discovered plugin that failed is excluded --
its own card already shows the reason.
Replaces find_orphaned_plugin_names outright rather than adding a second
overlapping helper; it had one caller.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Operator saw the `http` plugin -- deleted when ping/dns/http were unified into
the Monitor entity -- still reported as enabled, with no way to clear it.
It was never an orphaned row. `plugin.http` was hardcoded in core DEFAULTS, so
deleting the app_settings row did nothing: the default reasserted it on the
next settings load. That is precisely why no cleanup UI could have fixed it.
Rule 22 says the removed subsystem should have taken its setting with it.
Purged the surviving references -- the DEFAULTS key, "http" in
CAPABILITY_PLUGINS, the stale plugin_manager docstring example, and the
capabilities blurb still advertising HTTP/uptime as a bundled capability --
plus a migration dropping any stored plugin.http row. Untouched: `http` as a
MONITOR TYPE (icmp/tcp/dns/http) everywhere it appears, and http_001_initial,
which is kept deliberately so existing DBs resolve the revision graph.
For the general case, cleanup splits by provenance rather than being uniformly
automatic or uniformly manual:
Bundled plugins ship in the image and version atomically with core, so they
cannot be transiently missing -- a plugin.* default with no bundled directory
is unambiguously a bug. Guarded by a unit test that fails CI, which is the
only part safe to automate.
External plugins live in operator-mounted /data/plugins, where absence is
ambiguous: unmounted volume, failed install, mid-upgrade. Auto-deleting their
config would silently destroy unrecoverable credentials on a transient
condition, so Settings now lists them under "Configured but not installed"
with an explicit, confirmed, audit-logged Remove. The section hides entirely
when empty, and the remove route refuses if the plugin is actually installed.
Orphan detection reads STORED rows, never the DEFAULTS-merged view -- offering
to remove a default-only key would be a lie -- and returns names only, since
plugin config can hold credentials and listing orphans never needs their values.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The user-facing half (rule 27). Per-container log viewer + admin controls, so
container logs are something the operator can actually touch.
Viewer (plugins/docker):
- /container/<host>/<name>/logs full page + /logs/lines HTMX fragment polled
every 5s (near-live follow); stream filter (all/stdout/stderr) + case-insensitive
text search; newest-first so the latest lines survive the poll's scroll reset;
empty/loading states; a "View logs" link from the container detail page.
Jinja auto-escapes log content (no markup injection).
Settings (Thresholds & Retention tab):
- global on/off toggle, comma-separated exclude list, retention days + max MB
per container — DB-backed, no restart. The toggle + exclude are enforced
authoritatively at ingest (_persist_logs drops disabled/excluded lines), since
the push model has no channel to tell an agent to stop.
Tests: route-defined smoke; template-parse covers the new templates; integration
test for the ingest-time toggle + exclude enforcement.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CAGR73DUowdVFVvYzLXC5C
plugin_metrics grows by (sources × resources × ~30s cadence); keeping 90d of raw
is a large table. Add a raw→hourly rollup (mirroring the Docker plugin) so only a
short raw window is kept at full resolution, with hourly averages archived longer.
- PluginMetricHourly model + core migration 0024 (plugin_metrics_hourly: avg/max/
count per source/resource/metric/hour, unique bucket constraint + lookup index).
- steward/core/metrics_retention.rollup_plugin_metrics: date_trunc('hour') agg of
raw older than the hour-aligned raw window, idempotent pg upsert into hourly,
delete the rolled raw, prune hourly beyond the rollup window.
- cleanup.py: plugin_metrics is no longer blanket-deleted at data.retention_days;
_run_metrics_retention drives the rollup with windows read live from settings.
- Settings: metrics.retention.raw_days (7) + rollup_days (90), tunable on the
Thresholds & Retention page (new "Host metrics retention" card).
- Chart read: _history_for_host merges the hourly rollup (older part of the range)
with raw date_bin (recent part, capped ≤1h), so 30d charts keep working —
recent at full resolution, older at hourly. Route passes raw_days from settings.
- Tests: unit (cutoff helpers) + integration (rollup aggregates/prunes; history
merges hourly + raw) against Postgres.
Speed was already handled by the indexes + SQL aggregation; this is the storage
lever (raw window ~10x smaller).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC
Bounds Docker time-series growth (the main scaling concern). New
docker_metrics_hourly table + docker_006 migration; a plugin retention module
(docker.run_retention capability) rolls raw docker_metrics older than the raw
window into hourly averages (idempotent upsert), deletes the rolled raw rows,
then prunes stale rollups + lifecycle events. Core cleanup.py drives it each
hourly run via the capability (no plugin-model import), reading the three
retention windows fresh from settings so changes apply without restart (rule 25).
Settings → "Thresholds & Retention" gains a Docker retention card (raw /
rolled-up / events windows, working defaults 7/90/30 days). Unit tests cover the
hour-aligned cutoff/bucketing helpers; integration test exercises the real
rollup-average + prune across both windows.
Milestone 77 task #941.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC
Move the hardcoded warn/crit cutoffs into Settings -> Thresholds (DB-backed,
live, no restart). New thresholds.* keys + to_thresholds_cfg() + a
threshold_style(value, kind) jinja global that reads them; latency reuses the
existing ping good/warn keys, uptime is direction-aware (floors).
Replace the _macros metric_style/uptime_style macros (now removed) with the
global across Hosts-Overview, host_agent fleet + panel, Uptime/SLA widget, and
the ping page uptime column — all now honor the configured cutoffs. Uptime keeps
its green 'good' look when not degraded.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Final phase of the host-IA unification (milestone 70).
- Settings → Plugins split into two tiers: "Monitoring capabilities"
(host_agent, http, snmp, docker — built-in host facets, surfaced via
Hosts/Status, on by default) and "Integrations" (traefik, unifi — external
systems, off until configured). Presentation only: a CAPABILITY_PLUGINS set
(overridable by plugin.yaml `kind:`) tags each plugin; module loading,
optional deps, and migrations are untouched.
- Drop the "default-enable a plugin" framing in the UI copy — capabilities are
described as built-in, not optional add-ons.
- Nav: remove the standalone "Uptime" item (folded into Hosts; still reachable
via the SLA button on the Hosts list).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Provisioning review corrections + the matching frontend, plus breadcrumb
header integration.
- provision.yml: authorize the managed pubkey with a regexp match on the
' steward-managed' comment so rotating the key REPLACES the host's steward
key in place instead of stacking a second authorized entry. Hand-added keys
(other comments) are untouched.
- update.yml (new): refresh agent.py + restart only. Does NOT rotate the token
or rewrite /etc/steward-agent.conf — the host keeps its identity. Asserts the
agent is already installed and fails clearly otherwise.
- host_agent /update route: runs update.yml as the managed steward user (no
token minting). Token rotation stays a deliberate action.
- settings/ansible/generate-key honors a safe relative `next` redirect, so an
inline trigger elsewhere returns to its page.
- Host Agents settings: reworked into a clear lifecycle — an intro card that
explains it runs Ansible to deploy the agent (+ inline "generate managed key"
warning/trigger when none exists), then three labelled cards: 1 Provision,
2 Install/enroll, 3 Update. Each explains what it does.
- base.html: breadcrumb now renders as a kicker line directly above the page
title (moved below alerts, tightened margin) so nested and top-level views
share one consistent header.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Turn the agent-install playbook into a full provisioning + maintenance
path. Solves the bootstrap chicken-and-egg: first contact uses an
operator-supplied password (one run, never stored), which creates a
dedicated `steward` login account with NOPASSWD sudo + Steward's managed
public key. Every run thereafter connects as `steward` with the managed
key — fully unattended (scheduled prune, agent updates).
- core/crypto: generate_ssh_keypair() — ed25519, OpenSSH formats.
- settings: ansible.ssh_public_key (non-secret, displayed) + ansible.ssh_user
(default steward); to_ansible_cfg extended.
- settings UI + route: "Generate managed key" (private encrypted, public
shown to copy) + SSH-user field.
- executor: build_bootstrap() writes a 0600 vars file (-e @file) for the
per-run user/password — never argv, never DB, never logged; drops the
managed key when a bootstrap password is given; --user floor from the
global ssh_user when no override.
- runner.trigger_run: pass-through `connection` kwarg, deliberately NOT
persisted on AnsibleRun.params (password stays out of the DB).
- bundled/host_agent/provision.yml: create steward user + authorized_keys
+ /etc/sudoers.d/steward (visudo-validated) + agent install.
- host_agent: /provision route + "Provision a fresh host" card (bootstrap
user/password; injects pubkey + user + token as space-safe JSON hostvars).
- Dockerfile: add sshpass (Ansible shells out to it for password SSH).
- tests: keypair generation + build_bootstrap (secret stays off argv).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Closes#550 (all four):
- Cancellation: track live subprocesses; POST /ansible/runs/<id>/cancel
(operator) SIGTERMs then SIGKILLs after a grace; new 'cancelled' status
(+ migration 0019, ALTER TYPE in autocommit). Queued runs cancel cleanly
before launch. Cancel button on run detail.
- Concurrency: global semaphore (ansible.max_concurrent_runs, default 3,
Settings→Ansible) caps simultaneous runs; excess show 'queued' (new status)
until a slot frees. Semaphore bound lazily per running loop.
- Structured results: parse PLAY RECAP into per-host ok/changed/unreachable/
failed/skipped + capture failed-task lines, stored in new results JSON
column (migration 0020); rendered as a host-summary table on run detail.
Keeps live streaming (no json-callback swap).
- Retention: full output written to a persistent log artifact
(/data/ansible/runs/<id>.log, env-overridable) beyond the 1 MB DB cap and
across restarts; in-memory replay buffer bounded + GC'd after completion;
Download-log route. Boot reconciliation now also sweeps stale 'queued'.
Unit tests for recap parsing + cancel flagging. Status colors updated across
run list / detail / schedules.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Global Ansible credentials, applied to every run (manual + alert-triggered):
- core/settings.py: ansible.ssh_private_key / become_password / vault_password
(plaintext at rest, masked in UI — encryption tracked in #580) + host_key_checking
(default off); surfaced via to_ansible_cfg into app.config[ANSIBLE]
- executor.py: pure build_credentials() materializes creds into a 0600 temp dir
(--private-key, --vault-password-file, become via -e @vars-file so the password
never hits argv) cleaned up in finally; pure ansible_env() sets
ANSIBLE_HOST_KEY_CHECKING. build_ansible_command stays param-only
- settings/routes.py + ansible.html: admin-only Credentials section, masked-update
(blank keeps current, explicit Clear checkbox), reload app config on save
- tests: unit (build_credentials per cred + none; ansible_env toggle); integration
vault round-trip (ansible-vault encrypt a vars file, run via executor with the
vault password, assert it decrypted)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The fabledscryer->steward rename had only ever reached host_agent. The other
five bundled plugins (http, snmp, traefik, unifi, docker) still imported
`from fabledscryer.*` (package no longer exists) and read FABLEDSCRYER_* env
vars — so every one of them was broken at import since the original rebrand.
CI stayed green only because none are enabled by default and migrations don't
import plugin modules. Now that they version in-tree, complete the rename:
- fabledscryer.* -> steward.* imports across all five plugins
- FABLEDSCRYER_* -> STEWARD_* in plugin migration env.py files
- author/repository/homepage + user-facing 'Fabled Scryer' strings -> Steward
- snmp/scheduler.py: also drop dead `now`/datetime; record_metric from steward
Adds tests/test_no_legacy_names.py — fails if 'scryer'/'roundtable' ever
reappear in shipped code (the drift bit twice; this stops a third time).
Also clears pre-existing ruff lint debt (unused imports, semicolon statements,
mid-file import) surfaced by the new lint lane.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
First-party plugins (host_agent, http, snmp, traefik, unifi, docker) are now
tracked under plugins/ and baked into the image, so they version atomically
with core — ending the cross-repo import drift the roundtable->steward rename
exposed. History for these files is preserved in the archived Roundtable-plugins
repo.
Plugin discovery becomes multi-root: PLUGIN_DIR (single) -> PLUGIN_DIRS
(bundled first, then external) + PLUGIN_INSTALL_DIR. Bundled ships in the image;
third-party plugins still mount at runtime into the external root
(STEWARD_PLUGIN_DIR, default /data/plugins) and downloads/installs land there.
Bundled shadows external on a name collision.
- config.py: load_bootstrap returns plugin_dirs + plugin_install_dir
- app.py: iterate PLUGIN_DIRS at the migration + load sites
- migration_runner.py: discover_all_in() unions every plugin root
- plugin_manager.py: resolve_plugin_path() (pure, first-root-wins); load /
install / hot-reload span all roots; installs target the external root
- settings/routes.py: _discover_plugins scans all roots, dedup bundled-first
- Dockerfile: COPY plugins/ ; docker-compose: drop host bind, document external
- tests/test_plugin_dirs.py: resolution, multi-root discovery, bootstrap split
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Renames the Python package directory, CLI command, env var prefix,
docker-compose service/container/image, Postgres role/db, and all
visible branding. Marketing form is "Fabled Steward".
Clean break from the previous rebrand: drops the fabledscryer→roundtable
import shim in __init__.py and the FABLEDSCRYER_* env var fallback in
config.py and migrations/env.py. Env vars are now STEWARD_* only.
Heads-up for existing deployments:
- Postgres user/db renamed fabledscryer → steward in docker-compose.yml.
Existing volumes need the role/db renamed inside Postgres, or override
POSTGRES_USER/POSTGRES_DB to keep the old names.
- Host-agent systemd unit is now steward-agent.service. Existing agents
keep running under the old name; reinstall to switch.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
