#5463 "Acknowledgements on record: an event and who did it", step 1 of
milestone 499 "Alerts you can trust". The live walk in #5461 found that
acknowledging changed only the alert's current state: once it resolved,
nothing said who had acknowledged it or when.
- alert_events.actor_id (migration 0040): the person behind a transition.
- Acknowledging writes a firing → acknowledged event by the acknowledger.
- A rule edit, disable or enable that resets an alert carries its actor
(web and MCP alike).
- The history page notes "acknowledged by X" and "rule disabled by X"; MCP
get_alert_history returns `by`.
- One usernames() helper in alerts.service, used by the active list, the
history page and the MCP tools.
- docs/core/alerting.md: the state diagram no longer shows an acknowledged
alert re-firing, and says what an event records.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5449 "The hourly Docker log prune ranks every log line in the table to find
the few over their cap": the size ring put a running-total window over every
row of docker_logs each hour, sorting the whole table on NFS-backed storage
to find the handful of containers past docker.logs.max_bytes_per_container.
- prune_docker_logs (capability docker.prune_logs) does the age cutoff, then
a per-container byte total picks the containers at or over the cap, and
only their rows are ranked and trimmed. Same keep rule as before: the
newest line always survives.
- run_docker_retention keeps the metrics rollup and the event prune.
- Cleanup runs the log ring in its own transaction after them.
Arose from #5431 "Ingest dropped for ~3 minutes after each deploy while
startup retention runs".
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
_DueTracker counted a task that had never run from 0 on the loop clock,
which is host uptime, so every run_on_startup=False task (hourly data
cleanup with the Docker log prune, the weekly report check, alert
reconcile, agent age, Ansible schedules) fired on the first tick after
each deploy, while agents were flushing their backlog. Agent ingest was
slow or dropped for a few minutes after every deploy.
start_scheduler now calls tracker.begin(): such a task is first due one
interval after the loop starts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
When a silent series' sample also moved the rule (fired, resolved), the
resume and the transition were two events: the resume from the
pre-evaluation state at ingest time, the transition at the sample's
time. Ties sort by a random id, and the resume's "→ inactive" landed
after "→ firing", so the history read "silent, firing, back to normal"
at once and the episode closed while the alert was firing.
_track_samples now only clears the mark and returns its note; the first
sample's evaluation records one event from no_data to wherever that
sample took the rule ("firing" or "resolved" notification), or back to
the state it was in ("data_resumed", still only when notify_no_data).
Also fixes the #5441 integration test: its rule needs a recorded
resource to target (CI 8878).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
After Steward or its database was down, every agent's age and every
series' last sample were old, so all agent-offline rules fired and all
rules went no-data at once, clearing a minute later. With per-account
alert email that would have mailed everyone a dozen notices per restart.
A periodic job now ticks a Watch (steward/core/watch.py) once it has
reached the database; the Watch says since when runs have been unbroken
(process start, or the end of a gap over 120 s). seconds_since_seen and
the no-data sweep count silence from no earlier than that, so a source
must stay quiet while Steward is watching before it counts as silent.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Step #5429 "Reset by email: request, link and new-password page" of
milestone 497 "Passwords: change your own, and reset it by email".
- Sign-in page: "Forgot your password?" while a mail server is set.
- /forgot takes a username or email and answers the same either way; only
an active local account with a real address gets mail. Requests count
per address on their own throttle, so they never lock out sign-in.
- password_resets (migration 0039): a one-time key, stored as its sha256,
working for an hour; asking again replaces it.
- /reset/<key>: the new password, then signed straight in; every other
browser is signed out (sessions_since). Bad keys count as invite keys do.
- Email "[Steward] Reset your password", in the account's own zone.
- Audited: user.password_reset_requested, user.password_reset_by_email.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5207 "UniFi Overview tab: func.strftime is SQLite-only": the WAN and
client sparklines bucketed with strftime('%s'), which Postgres lacks, so
/plugins/unifi/rows failed. Traefik's date_bin bucket moves to
core/time_range.time_bucket and both plugins use it. Devices list offline
and failing first, and a state other than online is named (Upgrading,
Adopting, Heartbeat missed...) instead of drawn as Down.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The operator's ask: an invite link sent by mail, the key in the URL, and
the new user choosing their own username, password and email.
- user_invites (migration 0037): the key's sha256 only, the address, the
role and days the admin chose, who sent it, used/revoked.
- core/invites: create (revokes an older open invite to the address),
renew (new key, new clock), find_pending, the four states, signup checks.
- /invite/<key>, outside sign-in and throttled like login: the invitee's
username, email (prefilled, theirs to change) and password; the account
takes the admin's role, the key is used up in the same transaction and
they are signed in. A bad key gets one plain message.
- Settings › Users: Invite by email (address, role, 1/7/30 days), sent from
the admin in their zone; without a mail server, or if sending fails, the
link is shown once to copy. Invites not yet used are listed with Resend
and Withdraw.
- Audit: user.invited, user.invite_resent, user.invite_revoked, user.joined.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Operator review of the Account page: two sections with two different
save buttons, one stuck to the next card and one beside a squeezed hint.
- Account: time zone and Email are one form; POST /account/ saves both,
nothing unless all of it is valid. /account/email is gone.
- Users › account: Name and Email are one form; the rename route also
saves the email switches. /settings/users/<id>/email is gone.
- The Email section is a plain section now (email_section); its hint sits
under the switches, names the zone as the picker does, and links
Notifications only for an admin.
- apply_email_form leaves a switch the form doesn't carry as it is.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Milestone 495 "Email goes to each account, in its own time zone", on the
operator's ruling (System 156): notifications go to each user, times in
their zone, defaults on, toggled per user or per role.
- users.notify_alerts / notify_report (migration 0036): None follows the
role's default, notify.<kind>.<role> in settings, all on to start.
- mail.pick_recipients: active accounts that want the email and have an
address mail can reach (an SSO placeholder like name@oidc is skipped and
flagged), each in its own zone with no UTC beside it, then the Also send
to addresses in their chosen zone with UTC beside. One copy per address,
all over one SMTP connection.
- Alerts, the weekly report and the test email (to you, in your zone, plus
the typed addresses) send that way. The save-time smtp.tz capture is gone.
- Notifications: a role × email table of defaults, who each email reaches
and who can't be reached, and Also send to with its own zone, which starts
as the first admin's to open the page.
- Account page: your address and your two switches. Users: an Emailed
column, and an admin sets an account's switches on its page.
- The Alerts page counts the accounts it emails. zone_select macro and
formatting.zones_by_region shared by Account and Notifications.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5335 "Emails Steward sends read as designed messages: an HTML alert email
and weekly report, with a plain-text twin".
- core/mail.py renders each email as multipart/alternative from
templates/email/: a table-and-inline-style HTML part and a plain-text
twin carrying the same facts. Colours are the design system's light
theme, resolved from theme.css; nothing is fetched when a mail opens.
- Alert email: state badge, a headline sentence ("Disk used on web01 is
94%."), rule, resource, reading, the condition in words, and the time in
the zone Notifications was saved in with UTC beside it; "View in
Steward" when Steward's address is set. Subject "[Steward] Firing: <rule>
— <resource> 94%". Alerts now carry their operator.
- Weekly report: monitors down, active alerts, the week's alert events,
uptime and busiest Traefik routers as small tables with links. Sent from
the configured From address; the "Generated … UTC" line and the stale
repo footer are gone.
- condition_text() states a condition from a rule's parts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The last page of #5214's walk. Settings still described Steward as it was
before monitors were unified and SNMP was rebuilt, drew its tabs below the
breadcrumb, and showed the report schedule in UTC.
- Tabs (#5377): settings/_tabs.html is a settings_tabs() macro called in
each page's section_tabs block, above the trail, as Ansible's are. The
breadcrumb guard also fails any page that {% include %}s a tab strip.
- Report time (#5378): reports.schedule_tz keeps the zone of whoever saved
the schedule; check_and_send compares in it (report_due), and the form shows
the day and hour on the reader's clock (schedule_in) under their zone's
name. "Last sent" uses |when.
- Wording (#5379): the report contents list what the report holds (monitors,
not ping-enabled hosts); "Ping latency" is "Response time" and says what it
grades; plugin detail says enabling loads now and disabling needs a
restart; the monitor interval says it applies on the next check (the
scheduler reads it every pass); the dead list-config branch is gone.
- Traefik (#5380): the 5xx and 4xx cutoffs are settings
(traefik.threshold.error_5xx_pct 0.1, error_4xx_pct 1.0) in a "Traefik
routers" group on Thresholds while the plugin runs, read by router_state
through error_cutoffs(). The Traefik page has a Thresholds button for
admins, to /settings/thresholds/#traefik.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
First-party plugins have shipped inside the image since a7a281c, and the
catalog's only repository (Steward-plugins) answers 404, so Steward polled a
dead URL every minute for nothing. The operator chose to remove the catalog
entirely (#5214 "Review: readable and fast on the live instance").
Removed:
- core/plugin_index.py and the plugin_catalog_refresh startup task;
- the plugins.index_url default, and migration 0035 deletes the stored
plugins.index_url / plugins.repositories rows;
- the Settings -> Plugins repository editor, catalog, refresh, install and
reload routes and their templates;
- the download/checksum/extract install path in plugin_manager, and the
PLUGIN_INSTALL_DIR config it wrote to;
- plugins/index.yaml, docs/plugins/index.yaml.example and the guide's
"Publishing to the Catalog" section (now "Shipping a Plugin").
Kept: enabling and configuring plugins, hot-reload on enable, restart, orphan
cleanup, and the external root (/data/plugins) for operator-mounted plugins.
tests/test_no_plugin_catalog.py guards the routes, default and task.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The plugin_catalog_refresh tick retries a failed repository every 5 minutes;
the configured index URL answers 404, which logged a full traceback each time.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The operator's ruling (Background Jobs, 2026-10-08): a Steward page reads
stored records; the sources are contacted by scheduled collection. An audit
of every GET handler, HTMX fragment, widget poll and MCP read tool found one
page load that reached out: Settings -> Plugins' catalog fragment fetched
each plugin repository (5 min cache, 10 s timeout each, retried on every
load when unreachable).
plugin_catalog_refresh ticks every minute and fetches a repository whose
copy is missing or older than an hour, waiting 5 min after a failure;
saving repositories clears the cache, so the next tick fetches. The
fragment only reads the cache and says which repositories are not fetched
yet. Refresh is now a POST that fetches at once, as does install.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The operator found "Today 08:03" in place of a date odd on the audit log,
and asked for the recency groups as dividers instead. |when now always
writes the date ("Thu 8 Oct 08:03"); mark_day_groups gives each entry its
group in the reader's zone, and a divider row opens each group. A scrolled
page carries the group the last one ended in, so it is not opened twice.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Agent 1.12.0 stamps each post with sent_at; ingest compares it with the
moment the post arrived: clock_offset_s / clock_checked_at on the
registration (host_agent_003), and a clock_off_secs metric (either way)
that alert rules can watch. Replaces the old skew log, which measured the
newest sample and was off by up to an interval.
- Settings -> Thresholds: Clock drift, 30 s warn / 5 min critical.
- Host page: 'clock 0.3s ahead' in the vitals line, toned past the warning.
- Hosts list: names hosts whose clock is off; when every reporting host is
off by about the same amount, says Steward's own clock is the likely one.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- users.time_zone (0034). An account without one takes the browser's zone
on its first page (POST /account/time-zone/adopt, never over a choice made
by hand); the session follows the account on every request.
- /account: who you are and the zone every page writes its times in,
reached from your name in the sidebar.
- formatting.when: 'Today 06:58', 'Yesterday 23:41', 'Tue 6 Oct 14:12',
'6 Oct 2025 14:12', 24-hour, in the reader's zone; |ago's hover is the
exact local moment instead of UTC; tz_label() for column headers.
- Audit log: When (EDT) with exact times, one detail fact a line, on classes.
- tzdata in the image, and the airgapped smoke checks a zone resolves.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Service page:
- The header carries the verdict, mode, stack, image and when it was reported.
'In trouble since' shows only while the service is in trouble; once recovered,
'Last failed X ago · its logs'.
- The failure card shows only while the service is down, degraded or crash-looping.
- The tile strip and the kv card are gone.
- One Tasks table: running tasks first with CPU, memory and since when, then
Swarm's record. The node is named as its Steward host throughout, and
'N failed in 24h' sits in the caption.
Container page:
- Same shape: header verdict, a 'Last exit' card only while it is stopped
after a failing exit, set-up and I/O in one kv card, and lifecycle as a table.
- A swarm task's breadcrumb goes through its service.
Both pages:
- Resource history is the one fragment (_usage_history.html): now and peak in
cores and bytes, drawn with a floor of 1 core / 1 GiB (sparkline_svg floor=),
so idle noise draws flat.
- The container page's old % sparklines, its private _sparkline and the unused
bucket()/container_history() helpers are removed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- #5304 "Docker disk totals recorded over time per host, with a growth chart and alertable metrics"
- #5305 "Disk page: images grouped by repository, versions folded under it"
- #5306 "Docker overview: one disk line per host linking to the Disk page"
Each `docker system df` report is now also recorded as metrics, with the host
as the resource: disk_total_bytes, its parts, and disk_reclaimable_bytes. An
alert rule can watch them. The Disk page starts with the hosts side by side,
showing how much each grew over the selected range with a line of it, then
gives each host its own section. An image's versions fold under its
repository, which is charged only for the layers its versions don't share.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5290 "Host page: the health tiles span the card's width; even spacing above Ansible":
the four tiles take equal quarters (halves on a phone) and their sparklines stretch
across them; the Monitors + Agent row drops the Agent card's own margin, so the gap
above Ansible is 1rem like every other.
#5291 "Add host opens as a dialog over the Hosts list": a native <dialog> on /hosts/,
opened by Add host or /hosts/?add=1. A name clash re-renders the list with it open,
the error and what was typed. GET /hosts/new is gone; hosts/form.html is Edit only.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Status repeated the Monitors page's counts, check strip and uptime. Monitors
now has one header row and every column Status had: the last 30 checks,
latest response with its trend, uptime over 24h / 7d / 30d (the range picker
goes) and TLS days on every row. /status and /status/rows and the nav link
are removed outright; the dashboard Overview (/status/widget) stays and its
Details link goes to Monitors.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A chart built while its cell was still being laid out could stay blank until
a window resize; it now measures and draws again after layout settles. Each
refresh rebuilds the chart (the flash), and its buckets are minutes wide, so
the widget refreshes every 5 minutes instead of every poll.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The fill bars were a share of the host's capacity, a sliver on hosts with
80 cores. Each figure now sits beside 24 bars of the last 6 hours (1h/24h
per widget), each row scaled to its own peak but never below 1 core or
1 GiB (configurable per widget), so a 0.05-core container reads flat and
a bar's height means real load. One grouped date_bin query feeds them.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The Monitors widget opens with a verdict ("1 of 5 monitors down."), then one
row per monitor under a single header: latency and uptime over 24h, 7d and
30d, down monitors first with why and since when, the name linking to the
monitor. Counts follow the type filter. Problems always show; the limit
trims only the healthy rest ("+N more up").
Uptime / SLA (uptime_summary) repeated those numbers per host; migration
0033 moves dashboards onto Monitors. Shared widget styles (.w-verdict,
.w-table, .w-sub, .w-more) carry the widget-shape ruling for the widgets
that follow.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- New installs start from widgets.RECOMMENDED, packed row by row for the
enabled plugins; the edit drawer can reset a dashboard to it.
- Host Agent — Resources folds into Hosts — Overview (which gains load 1m),
UniFi — Devices into UniFi — Overview; migration 0032 moves existing
dashboards onto the survivors.
- The Overview labels what it counts (hosts, monitors up/down/pending,
alerts firing) and lists firing alerts and down monitors with since-times.
- Uptime polls every 5 minutes (poll_seconds) instead of every cycle.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5212 "Plain labels: no raw ids, scopes, metric keys or unexplained jargon",
batch 3:
- Disk page: each tile says what it holds. "Writable layers" becomes
"Container files" and Build cache says it is safe to clear. Shared is
"Shared with other images" with a tooltip, and "dangling" is gone.
Untagged images show by id instead of "<none>", a capped list says
"The 50 largest of N images", and each host shows "as of" its report.
- Traefik: an expired certificate reads "expired 12 days ago", not "-12
days" (cert_left). The Traffic empty state knows whether the access log
is on: off points to Settings → Plugins → Traefik; on says to check the
mounted log path.
- Moments use |ago with an exact UTC tooltip: UniFi last seen, alarms,
events, speed test and DPI; AI token last used.
- Settings: the Auth tab is "Authentication" like its breadcrumb, plugin
fields read "Metrics URL" not metrics_url (field_label), a source's pull
interval is "every 1h", and the last report time drops the ISO "T".
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5212 "Plain labels: no raw ids, scopes, metric keys or unexplained jargon",
batch 2:
- A run records where it came from: AnsibleRun.origin ("schedule", "alert"
or "mcp") and origin_id, set by the scheduler, Run now, alert actions and
the MCP tools. Migration 0031 backfills past runs from alert events,
schedules' last runs, and MCP audit entries.
- The runs list has a Started by column: "schedule nightly", "alert rule
nas down", "alice via MCP" or "alice", linked where there is a page.
The raw scope tooltip is gone.
- Run detail: no run id in the subtitle or body, Started by in words, and
the host summary lists failed and unreachable hosts first. The live
output only follows new lines while the reader is at the bottom, and
when the run finishes the page reloads once so the badge and summary
are final.
- list_ansible_runs and get_ansible_run report origin and origin_id.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5212 "Plain labels: no raw ids, scopes, metric keys or unexplained jargon",
batch 1:
- Audit log: each action reads as a sentence ("paused monitor"), with the
code as its tooltip. Detail shows flags as yes/no, lists joined, a run's
scope by name, spans as spans, and no ids the entity column already gives.
The filter is now a picker of areas.
- Alert rules, history and Status say the condition in words: "is down",
"Fullest disk >= 85%". Readings use metric_value, and history times use
|ago.
- Rule form: sources and metrics by name, the threshold's unit, "fires
when it is down/up" for an up/down metric instead of 0.5, and a metric
reference built from the catalog.
- Schedules and Maintenance: viewers no longer see Run now, Pause, Edit,
Delete or New. Intervals and next runs read as "every 2h" and "in 3h";
a scope reads "Every alert on Host agent".
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5211 "Lists load as you scroll: audit, alert history, Ansible runs"
(milestone 477), per preference 172.
- core/paging.py: keyset paging over (time, id), newest first, with an
opaque "<iso>|<id>" cursor; a page reads one row past itself to know
whether another follows. No OFFSET, so rows added while someone scrolls
never shift the pages after them.
- _macros.html scroll_more / scroll_end and static/scroll-more.js, written
once: an IntersectionObserver with a 600px margin fetches the next page
before the reader reaches the end; "Loading more…" while in flight,
"End of the list." at the end, and a failed page says so with a Retry
button instead of re-firing on every scroll.
- Audit log: no "Load more" link and no 500-row cap.
- Alert history: no 200-row stop with "pick a shorter range";
alert_svc.history takes the cursor, so the MCP tool and the page share
one query.
- Ansible runs: no silent cap of 50.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5209 "Docker views: readable names, CPU/memory in cores and bytes, failures
listed with since-time" (milestone 477 "UI problems first").
- A swarm task reads as its service and slot ("codex_db · 1", or the node
for a global service), with the raw name in the title: Docker page, host
panel, busiest widget, container detail.
- CPU reads in cores (Docker's 100% = one core), toned by its share of the
host's CPUs from the agent's per-core series. Memory reads in bytes, against
a limit only when one is really set (Docker reports host RAM otherwise).
- Exit 137/143 without an OOM kill is a stop, not a failure: no
exited_nonzero, no crash-loop death, no "failed (24h)", dim exit code.
- "Failed (24h)" is a list (Docker page #failures, linked from the widget
and the tile): plain containers by their deaths, swarm services by Swarm's
failed tasks; a second manager's copy of a task death counts once.
- Failing swarm services lead Needs attention; services sort problems first
with since-times on the page, the widget and the Swarm page; nodes that
are down or drained sort first.
- Busiest widget orders by a 5-minute CPU average so the list holds still.
- Host panel: a table, problems first with what is wrong and since when;
replaced swarm tasks hidden and counted.
- Container detail: node hostname, I/O labelled "since start", Restarts
hidden for swarm tasks with a note on where task failures are.
- Widget names: "Docker — Services & failures", "Docker — Busiest containers".
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Migration 0030 seeded plugin_metric_latest from the whole history: two index
probes per distinct metric key. On the live instance that held the first
boot for 7+ minutes with Steward down. The migration now only creates the
table. Live sources fill it on their next sample, and backfill_latest runs
after startup for resources that have history but no latest rows. Per
resource it does one backward probe for the last sample, then takes the
newest value per metric within the hour before it. Resources that already
have rows are skipped, so later boots only walk the resource names.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
_last_n_results (Status, the monitors list, host monitors), host_status
(hosts overview, list_hosts) and the report's down-monitor list each
ranked every result a monitor ever recorded to keep the newest few. Each
monitor now gets ORDER BY checked_at DESC LIMIT n in a LATERAL subquery,
which walks ix_monitor_results_monitor_checked backwards and stops.
Also fixes the latest-table test, which read between two begin() blocks
and so autobegan a transaction the second block then refused.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The dashboard's host widgets took 18-60 s. Every "latest reading" query
(hosts overview, host-agent fleet, host page, vitals, SNMP, MCP get_host)
searched plugin_metrics for the newest row per metric. Postgres has no skip
scan, so each one read every raw sample in the 7-day window.
- plugin_metric_latest: one row per (source, resource, metric).
record_metrics upserts it in the same transaction, moving forward only and
in sorted key order. Migration 0030 seeds it by stepping the existing index.
- Retention drops a latest row once its sample leaves the raw window, so a
resource that stopped reporting disappears when its history does, as before.
Renames carry it along.
- A quiet host keeps its last values, marked stale. A time-bounded query
would have shown it as missing.
- Vitals sparklines span the 6 h they claim, averaged in SQL to ~120 points,
instead of fetching 6 h raw and drawing the last 120 samples.
- Dashboard and share-view cells: "Loading…" until the first answer; a 20 s
deadline per fetch; "Couldn't load (why)" with Retry now and an automatic
retry for cells that don't poll; "as of HH:MM" in the header, and
"couldn't refresh · as of" when a refresh fails over shown data.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Dockerfile: python:3.14-slim, the Python CI tests on (was 3.13). tini from
apt is PID 1 (ENTRYPOINT tini -- entrypoint.sh): it reaps the zombies ssh
and ansible leave and forwards SIGTERM.
- cli: Hypercorn's serve() with one worker (run, stream and scheduler state is
in-process), a 10 s graceful timeout, and SIGTERM/SIGINT as the shutdown
trigger. Access and error lines go through Steward's logging. --debug keeps
Quart's reloading dev server.
- Logging: steward/core/logging_setup.py installs one root handler before
anything logs (until now a library's basicConfig on import was the only
configuration). Its level is the `log.level` setting (DEBUG/INFO/WARNING/
ERROR, default INFO), applied at boot and on save in Settings → General.
- CI publish: build, then "The image boots with no internet" (rule 164,
snippet #3552), then push, so a broken image never reaches a tag (rule 177).
On an --internal network with its own Postgres, the image must migrate and
answer /health, run tini as PID 1, serve first-run setup with its assets
from /static and no third-party host, and exit 0 on SIGTERM. Logs and
clean-up run always.
Tests: tests/core/test_logging_setup.py; the smoke step itself.
Not in this commit: the dependency lockfile (it conflicts with rule 154 and
is waiting on the operator).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- One run per host: start_run claims every host it touches (addresses read
from the inventory Steward wrote; a source's own inventory file is claimed
as that file) before it takes a run slot. The claim is all-or-nothing. A run
waiting on a host shows queued and names the run it waits for, honours a
cancel, and fails after HOST_WAIT_SECONDS (1 h) with which host and run.
- A due schedule whose previous run is still queued or running skips the
firing: last_run_at moves on, last_run_id still names the run going, and
last_error says "Skipped: the previous run was still going". The Schedules
page shows that beside the run (muted), and a launch error in red.
- An alert rule that fires while the action run from an earlier firing is
still going starts no new run; the event says so and names the run. An
in-process set closes the gap between the check and the run being linked.
- docs/core/ansible.md: queued/cancelled statuses and the overlap rules.
Tests: tests/core/test_ansible_overlap.py (claim keys, waiting, all-or-nothing,
cancel, deadline) and tests/integration/test_ansible_overlap.py (two real
runs on one host take turns; schedule skip then fire; alert action skip then
start).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- git ships in the image; checkouts live under /data/ansible/sources
(STEWARD_ANSIBLE_SOURCE_DIR), on the persistent volume.
- One task, ansible_git_pulls, reads the sources every 30 s and pulls each
as its interval passes: a source added in Settings is pulled without a
restart, a removed one is forgotten.
- git_pull: a 120 s timeout ending the process group, GIT_TERMINAL_PROMPT=0,
pull --ff-only, a failed clone removed; it raises GitPullError with git's
message, so the Sync button stops saying Synced after a failure (and the
message is now escaped).
- An alert action's vars and the firing's steward_alert_* vars reach
ansible-playbook as one JSON extra-vars file, not -e key=value, which
Ansible split on whitespace (a rule named 'High CPU on web').
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- prune_by_age(column) registers a time-series column with the hourly cleanup;
Traefik (metrics, access summaries, global stats) and UniFi (WAN, client,
WLAN, DPI, events, alarms, speedtests) register theirs in setup().
- Alert events join monitor results and Ansible runs on data.retention_days,
read from Settings each run; Ansible run log files older than it are deleted.
- audit.retention_days (365) keeps the audit log longer, set on Settings > General.
- metric_series.points reads metrics.retention.raw_days, so the chart's
raw/rollup seam is where the rollup put it (RAW_DAYS removed).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Measured on dev (385a579), agent posts cut to five samples still dropped:
flush=8.8s for 990 rows, 6.6s for 680, 8.4s for 1000 — about 110 rows/s,
~9 ms a row, with rules and evaluation near zero. One random read per
row fits a random uuid4 primary key whose index no longer fits in memory:
every new row lands on a random page of it. (Inferred from the rate; the
table and index sizes have not been read.)
- steward.core.ids.time_ordered_id: a UUIDv7-layout string (48-bit ms
timestamp, then random), same 36-char shape as the stored uuid4 ids, so
no migration. Python's uuid7() arrives in 3.14; the image runs 3.13.
- Used for plugin_metrics and its hourly rollup, monitor_results,
audit_events, the docker metrics/hourly/events/logs tables, and the
traefik and unifi time series.
- MAX_SAMPLES_PER_INGEST 5 → 2 (~400 rows) for now: it fits the agent's
10 s even at the old rate, so hosts land data whichever way the new rate
comes out. Raise it once the new rate is measured.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The first instrumented drop on dev read "dropped after 10.0s … (completed:
parse=0.1s)": the whole stall is inside record_metrics. It now marks its
three steps (flush of the rows, the rule and state reads, the per-sample
evaluation) on the caller's phases, and the drop line carries the request's
samples and metric row count.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Runs read "update.yml on web-1" (run_title over the scope label, beside
describe_scope) on the runs list, the run page's title and breadcrumb, and
the run-started panel. The list shows started |ago (full time on hover),
|duration (counting "so far" while a run is going) and a dry-run tag.
Run statuses join STATES, so the list and the run page use status_badge:
queued dim, running info (blue, so work in progress doesn't read as a
problem), success ok "Succeeded", failed crit, interrupted warn, cancelled
dim. run_started.html leaves its hard-coded colours for tokens; the run
page's facts move to the .kv list and its failures to var(--red). Template
hex colours: 54 -> 47.
Also fixes#5124's sidebar count, which CI run 8389 failed on: the stylesheet
guard allows no weight above 500, and .nav-count asked for 600.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The Alerts page's active list now says each alert in one line around the
resource it watches, e.g. "cpu_pct on web-1 is 94% (> 90%), firing for 12m",
with web-1 linking to its host, monitor or container page (or the plugin's
page for Traefik, UniFi and SNMP). Values take their unit from the metric
name through formatting.metric_value. A silent series reads "No cpu_pct from
web-1 for 20m"; an up/down metric says the word.
Actions follow the routes' roles: acknowledge, edit, toggle and New rule for
operators and admins, Delete for admins only, nothing for viewers.
The sidebar shows how many alerts need attention beside Alerts, from an
async context processor that skips htmx fragments, logged-out pages and the
database-down page, and renders without the count if the query fails.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The host page's vitals strip becomes its health header: Live or Stale,
then one sentence — "Memory 93% is past its critical threshold", "Every
reading is within its thresholds", or, for a stale host, "No report since
20m ago; the readings below are from then" — and last seen as |ago. CPU,
memory, root disk and load per core follow as stat tiles graded by
Settings → Thresholds, with their sparklines.
The full-metrics page drew its bars and gauges at a hard-coded 70/90
(temperatures 70/85) while every other page used the configured cutoffs
(80/90 by default), so a 75 % CPU was amber here and fine everywhere else.
bar() now takes the metric kind and colours its .meter-fill through
status_class; gauges, temperatures and per-core bars do the same.
The host's monitor rows move onto the Monitors list's shapes: a status dot,
the name linking to the monitor's page, Ping/TCP/DNS/HTTP, pill-up/
pill-down, latency and uptime through status_class, labelled icon buttons.
HEX_CEILING drops to 54.
New formatting.worst_reading(readings, thresholds) picks the reading
furthest past its cutoff, the first of equals winning.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The Monitors list now opens with a count strip (down, up with how many
degraded, pending, paused) and sorts down, degraded, pending, up, paused,
by name within each. Degraded is up but worth a look: uptime at or below
its warning floor, or the latest response past its critical cutoff, both
from Settings → Thresholds; a slow-but-tolerable reply is coloured, not
sorted.
A monitor had no page of its own: every link went to the list. Now
/monitors/<id> shows its state and since when, latency, uptime for 24h/7d/
30d, TLS days, the last checks, recent failures with their errors, and its
settings (HTTP header names, never their values). Status links to it, and
adding or editing returns to it.
Adding is its own page (/monitors/new, form.html shared with edit) behind
an "Add monitor" button; a refused add re-renders the form with the error
and what was typed instead of a bare 400 text. Row actions are labelled
icon buttons, type names read Ping/TCP/DNS/HTTP (|monitor_type), and the
heartbeat uses pill-up/pill-down, so HEX_CEILING drops to 56.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5120 "Status page: what is wrong first".
- A "Needs attention" card heads the Status board: each down monitor with
how long it has been down, and each active alert (firing, acknowledged,
pending, no data) with its condition and how long, linking to its history.
- StatusEntry.since / since_at_least: when the current up/down run began,
from the recent checks (streak_start); "for at least" when every check we
have agrees.
- Counts as stat tiles (Down, Active alerts, Pending, Up); names wrap
instead of being cut off; dots carry their label.
- Uptime and TLS are coloured by status_class against Settings → Thresholds
instead of literal cutoffs; heartbeat pills and DOWN/UP use tokens and
status_badge rather than hex and bold.
- tests/core/test_status_page.py: streak_start cases, and the rows render
with the problem card before the list, each since-time, threshold tones
and no hex colour.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5094 "Status and layout macros, and form classes".
- formatting.STATES: one state -> (tone, label) map for up, down, stale,
firing, pending, acknowledged, no_data, ok, warn, crit, resolved and
inactive; status_info() reads it (an unmapped state is drawn dim under its
own name). required() lets a macro refuse a missing argument.
- _macros.html: status_badge and status_dot (a dot always carries its label,
sr-only unless shown), stat_tile, empty_state, page_header (actions via a
call block) and icon_button, which raises without a label.
- steward.css: tone classes (badge-/dot-/stat- ok, warn, crit, info, dim),
.sr-only, .page-header, .stat-tile, .empty-state, .btn-icon and the form
classes .field, .field-label, .field-hint and .input. Info text is a 40%
mix (--blue): the house --fs-info-fg falls to 4.18:1 on its tint over a
hovered row, 40% holds 5.09:1.
- First users: the Alerts list, history and maintenance pages (headers,
state badges, empty states) and the alert forms' hints; the inline inp/lbl
style sets on the monitor and host forms become .input / .field-label.
- tests/core/test_macros.py renders each macro through the app environment.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5093 "Shared format filters and status_class driven by Settings → Thresholds".
- steward/core/formatting.py: human_bytes (binary units), human_rate,
human_duration (two largest units), ago (a <time> titled with the absolute
UTC time) and status_class(value, kind, thresholds) -> ok/warn/crit/unknown,
which raises on a kind the thresholds don't name.
- app.py registers them as the bytes, rate, duration and ago filters and the
status_class global; threshold_style is now graded by status_class (and caps
its weight at 500).
- Replaces the copies: docker's _human_bytes/_human_uptime, core alerts'
_duration, the host-agent fmt_bps/fmt_bytes macros, the Traefik and UniFi
byte-rate arithmetic, the swarm-failures ago macro and two uptime spans.
- Tests: a table per helper, and every template compiled against the app's
filters, so an unknown filter fails CI rather than a page.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5065 "Renames carry their alerts". Alert rules, maintenance-window scopes
and metric history key a thing by (source, name), so a rename used to leave
the rules watching a name nothing reported under, and split the history.
- core/renames.carry_rename moves AlertRule.resource_name,
MaintenanceWindow.scope_resource, plugin_metrics and plugin_metrics_hourly
to the new name in the caller's transaction. A host rename also carries
its parts: host_agent "<host>:…" and docker "<host>/…" (HOST_SOURCES).
Matching is exact (a prefix compare, not LIKE or a collation range).
History left under the new name by something deleted is dropped first
rather than merged in.
- Web monitor edit, MCP update_monitor and web host edit call it; MCP
answers with `carried` (it used to return a warning naming stranded rules).
- A name another monitor of the same type, or another host, already has is
refused on create and rename: two would share one history.
- The rename is audited (monitor.updated / host.renamed) with what moved.
Tests: an integration test renames a firing monitor through the web route
and finds its rule still firing and cleared by a sample on the new name,
its window and raw+hourly history moved and an orphan's history dropped; a
host rename moves agent and container parts but not a longer-named host's
or another source's; a guard that every host_agent resource is "<host>" or
"<host>:…".
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The rule name goes into the Subject header, and EmailMessage refuses a
header holding a line break, so the whole email failed. The web form can't
enter one but MCP takes any string. The subject now folds whitespace.
Found by the new From-fallback test in CI run 8349.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5064 "Notifications that arrive":
- render_webhook: alert.* is escaped to sit inside a JSON string (a quote
or newline in a rule name no longer breaks the body); raw.* is the plain
value for |tojson; a missing value reads "no data"; the template runs
sandboxed and the body is checked to be JSON before it is sent.
- SMTP: 15s timeout; port 465 connects with implicit TLS, otherwise the
box means STARTTLS; a From address setting (falls back to the username).
- A webhook error no longer carries the URL, which is a secret.
- Settings → Notifications: "Send a test email" / "Send a test webhook"
send a sample alert with the form's values (unsaved) and show the result
and the body that went out; template help names alert.* and raw.*.
- Notification audits go through audit_request; the test send is audited
as settings.notification.tested, with the channel and outcome only.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
