#5490 "A day of one metric reads fast even cold". A cold 24 h series read
measured 18.3 s against 63 ms warm (#5485): every sample's row sat on its own
heap page. ix_plugin_metrics_series_value carries `value` (INCLUDE), so the
read is an index-only scan; it replaces the plain series index from 0023.
It is built once, with CREATE INDEX CONCURRENTLY from a task started after the
app is serving, never in a migration, so the boot isn't held while it builds
over the live table (#5220) and ingest and pages carry on. A build cut short
by a restart leaves an INVALID index; the next boot drops and rebuilds it.
Later boots find it present and only check. Postgres keeps it current on each
insert, as it did the index it replaces.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5487 "MCP: edit inventory group membership". Group membership decides which
hosts a group alert rule covers, and the MCP could only read it. The new
operator tool takes a group and lists to add and remove, each a target's
name or id or the Steward host it is linked to; an unknown name refuses the
whole change. It answers with the group's targets after and the alert rules
covering the group (whose per-host rules follow within a minute), and audits
as ansible.group.updated by "<user> via MCP", as the Inventory page does.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5485 "A rule's page waits up to 25 s for its metric preview: a cold 24 h
series read from plugin_metrics". Opening "Agent offline" took 25.6 s; a
cold day of one series measured 18.3 s over MCP, a warm one 63 ms. The
pickers still come with the page from plugin_metric_latest; the preview is
its own request (hx-trigger=load), showing "Reading the last day…" until it
lands. Picker swaps no longer read it either.
#5486 "Send test where it can be found": the button moves from inside "What
happens" to the page header beside History, its result under the verdict.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5483 "Alerts list: a covering rule's hosts start collapsed behind a toggle;
no dashed rows", from the operator's review of the Alerts list. Each rule is
its own tbody; its hosts sit in another, hidden until "N hosts" is pressed.
A rule with a host that isn't quiet opens on its own and names the host where
"last fired" was, so a problem never hides behind the fold. Host rows lose the
dashed rules for a sunken band under the indent.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5475 "Rule form: pick the host, then its part…". CI run 8909 caught it:
moving from the whole host to a mount kept "CPU" in the mount's Metric list,
because a chosen metric always stayed on offer. Only a rule's own page keeps
its resource and metric after they go quiet (keep=True from edit_rule); the
pickers' swaps drop a metric the new part doesn't report.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5475 "Rule form: pick the host, then its part; metrics that part reports;
a link to groups; a preview of the metric", from the operator's review of
the New rule page: hosts and their parts shared one long picker.
- Host agent rules pick a Host, then a Part: "The whole host", then its
mounts, CPU cores, disks, interfaces and sensors under a heading each,
cores in number order. A part that reports nothing a rule can watch
(an interface's traffic) isn't offered.
- The Metric list holds what the picked resource reports, in catalog order.
- Under it, a preview: the reading now, and the last 24 h low to high with
an hourly sparkline drawn from zero (UI ruling 8), read from one series.
- The group picker links to Ansible > Inventory to make a group, in a new
tab, and re-reads the groups when the tab comes back into view.
- Picking a host or part swaps the pickers alone (/alerts/api/rule_target);
picking a metric swaps the preview (/alerts/api/metric_preview).
- A rule's own resource and metric stay on offer after they go quiet.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5473 "Opening an alert rule takes up to 15 s". With the rule pickers now
reading plugin_metric_latest, nine integration tests failed on rule_fields:
"host_agent has no resource 'box-…'". They seeded a resource by adding a raw
PluginMetric row, which no production path does: host-agent ingest and
record_metric both go through record_metrics, which keeps the latest-value
table. The seeds now go through record_metrics too.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5473 "Opening an alert rule takes up to 15 s: the Resource picker scans
every raw host-agent sample". The access log showed the edit page at 6-15 s
(one load gave up) while /alerts/ took 0.2 s. resources_for ran SELECT
DISTINCT over plugin_metrics, and Postgres has no skip scan, so it read every
host-agent sample; every rule page, source swap, rule save and MCP rule call
paid it. resources_for and metrics_for now read plugin_metric_latest, one
row per series, which lists the same names. A test fails if either reads
plugin_metrics again.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5466 "Send test: check a rule's delivery from the rule". A rule's page gets
a Send test button (operator role), and MCP gets send_test_alert. Each sends
a TEST alert for that rule now, by email to everyone who gets alerts and by
webhook. The alert carries the rule's name and condition, at the latest
reading of what it watches; a rule covering hosts uses its first host. The
result for each channel shows under the button, or "Nothing is set up to
send to". Nothing fires, no event is written, and the test is audited as
alert_rule.tested. An alert with no reading now says "has no reading yet"
instead of "is None".
Docs: alerting.md also gains the burst email from #5465 "One email per burst
of alerts".
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5465 "One email per burst of alerts". When a switch reboots, every host's
agent-offline rule fires at once and each account got one email per rule.
Alert email now waits 30 s from the first alert: each recipient then gets one
message listing every alert in the window, oldest first, in their own zone
("[Steward] 3 alerts: 2 firing, 1 resolved"). One alert alone keeps its
usual email. Webhooks still post per alert, at once. Every alert's event
records the email's outcome. A burst held when Steward stops is not sent;
the alerts stay on the record and on the Alerts page.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5464 "Group rules: one rule covers every host, or every host in a group",
step 2 of milestone 499 "Alerts you can trust". Each rule named one host, so
the live instance carried six rules for three hosts, and a new host had no
agent-offline or disk alert until someone copied them.
- alert_rules.covers / cover_group / parent_id (migration 0041). A rule on
host_agent may cover "all_hosts" or a "group" (an inventory group: the
Hosts ruling, #5294). A covering rule has no state; Steward keeps one
ordinary child rule per covered host, named "<rule>: <host>", so each
host's alert keeps its own state, history and acknowledgement.
- alerts.service.reconcile_children: a covered host that has reported in
the last 7 days gets a child; a host no longer covered loses its child;
children take the parent's condition, checks, notify setting, action and
on/off. Run on save and each minute (alert_reconcile).
- A child is changed through its parent: edit, toggle and delete of a child
are refused (409 / PartOfCoveringRule); its edit page opens the parent.
- Rule form: a Covers choice for host sources, with the group picker; fixed
once the rule exists. Rules list: a covering rule reads "every host in
swarm · 3 hosts" with its hosts' rules nested under it, each opening its
history. The history picker and the rule count leave covering rules out.
- MCP: create_alert_rule takes covers/group; update_alert_rule takes group
and passes changes down; list_alert_rules shows covers, hosts and part_of.
- docs/core/alerting.md: "Rules that cover hosts".
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5463 "Acknowledgements on record: an event and who did it", step 1 of
milestone 499 "Alerts you can trust". The live walk in #5461 found that
acknowledging changed only the alert's current state: once it resolved,
nothing said who had acknowledged it or when.
- alert_events.actor_id (migration 0040): the person behind a transition.
- Acknowledging writes a firing → acknowledged event by the acknowledger.
- A rule edit, disable or enable that resets an alert carries its actor
(web and MCP alike).
- The history page notes "acknowledged by X" and "rule disabled by X"; MCP
get_alert_history returns `by`.
- One usernames() helper in alerts.service, used by the active list, the
history page and the MCP tools.
- docs/core/alerting.md: the state diagram no longer shows an acknowledged
alert re-firing, and says what an event records.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5449 "The hourly Docker log prune ranks every log line in the table to find
the few over their cap": the size ring put a running-total window over every
row of docker_logs each hour, sorting the whole table on NFS-backed storage
to find the handful of containers past docker.logs.max_bytes_per_container.
- prune_docker_logs (capability docker.prune_logs) does the age cutoff, then
a per-container byte total picks the containers at or over the cap, and
only their rows are ranked and trimmed. Same keep rule as before: the
newest line always survives.
- run_docker_retention keeps the metrics rollup and the event prune.
- Cleanup runs the log ring in its own transaction after them.
Arose from #5431 "Ingest dropped for ~3 minutes after each deploy while
startup retention runs".
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Run now set last_run_at to the click, and a schedule's next turn counts
from last_run_at, so a manual run of the nightly prune at 14:00 moved
every later run to 14:00. Operator ruling: "run now shouldn't change the
schedule it should just run the playbook and leave the schedule as it
was."
Run now no longer touches last_run_at. It still records the run as
last_run_id, so the list shows it and a firing during it is skipped.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
_DueTracker counted a task that had never run from 0 on the loop clock,
which is host uptime, so every run_on_startup=False task (hourly data
cleanup with the Docker log prune, the weekly report check, alert
reconcile, agent age, Ansible schedules) fired on the first tick after
each deploy, while agents were flushing their backlog. Agent ingest was
slow or dropped for a few minutes after every deploy.
start_scheduler now calls tracker.begin(): such a task is first due one
interval after the loop starts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each firing stored last_run_at = the moment the ~60 s loop noticed it,
and the next was due a whole interval after that, so every firing's
lateness carried into the next: Nightly Docker Prune moved from 05:44:45
to 05:47:05 over six days.
A firing now stores its due slot, the latest last_run_at + k * interval
not after now (due_slot), for both a launch and a skipped firing. Missed
slots after an outage are skipped, not run in a burst.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
check_interval_s was `m.check_interval_seconds or None`, so a monitor
following monitors.poll_interval_seconds read as null and a reader
couldn't tell how often it ran. Monitor results now carry the effective
interval and `interval_from: "monitor" | "global"`.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
tcp_check swallowed every exception and icmp_check returned on a
non-zero exit without a reason, so every down ICMP/TCP monitor stored
error_msg NULL and the Monitors page and MCP showed it down with no why.
Each failure now names its cause: "No reply within 2 s", "Connection
refused on port N", "No answer on port N within 5 s", "Can't resolve
<name>: ...", the OS error, or ping's own first stderr line. An ICMP
check that fell back to TCP says so, and a ping that outlives its
timeout is killed rather than left running.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
When a silent series' sample also moved the rule (fired, resolved), the
resume and the transition were two events: the resume from the
pre-evaluation state at ingest time, the transition at the sample's
time. Ties sort by a random id, and the resume's "→ inactive" landed
after "→ firing", so the history read "silent, firing, back to normal"
at once and the episode closed while the alert was firing.
_track_samples now only clears the mark and returns its note; the first
sample's evaluation records one event from no_data to wherever that
sample took the rule ("firing" or "resolved" notification), or back to
the state it was in ("data_resumed", still only when notify_no_data).
Also fixes the #5441 integration test: its rule needs a recorded
resource to target (CI 8878).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
After Steward or its database was down, every agent's age and every
series' last sample were old, so all agent-offline rules fired and all
rules went no-data at once, clearing a minute later. With per-account
alert email that would have mailed everyone a dozen notices per restart.
A periodic job now ticks a Watch (steward/core/watch.py) once it has
reached the database; the Watch says since when runs have been unbroken
(process start, or the end of a gap over 120 s). seconds_since_seen and
the no-data sweep count silence from no earlier than that, so a source
must stay quiet while Steward is watching before it counts as silent.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Step #5429 "Reset by email: request, link and new-password page" of
milestone 497 "Passwords: change your own, and reset it by email".
- Sign-in page: "Forgot your password?" while a mail server is set.
- /forgot takes a username or email and answers the same either way; only
an active local account with a real address gets mail. Requests count
per address on their own throttle, so they never lock out sign-in.
- password_resets (migration 0039): a one-time key, stored as its sha256,
working for an hour; asking again replaces it.
- /reset/<key>: the new password, then signed straight in; every other
browser is signed out (sessions_since). Bad keys count as invite keys do.
- Email "[Steward] Reset your password", in the account's own zone.
- Audited: user.password_reset_requested, user.password_reset_by_email.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Step #5428 "Account page: change your own password" of milestone 497
"Passwords: change your own, and reset it by email". A Password section
in the page's one form, for local accounts: current, new, again. Blank
keeps the password; a new one signs out every other browser
(sessions_since) and keeps this one. A wrong current password counts
toward the login throttle. Audited as account.password_changed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5207 "UniFi Overview tab: func.strftime is SQLite-only": the WAN and
client sparklines bucketed with strftime('%s'), which Postgres lacks, so
/plugins/unifi/rows failed. Traefik's date_bin bucket moves to
core/time_range.time_bucket and both plugins use it. Devices list offline
and failing first, and a state other than online is named (Upgrading,
Adopting, Heartbeat missed...) instead of drawn as Down.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5427 "A password reset doesn't end the sessions already signed in with
the old password": users.sessions_since (migration 0038) is set on reset;
refresh_session_user ends a session whose login_at is earlier. An admin
resetting their own password carries on signed in.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The operator's ask: an invite link sent by mail, the key in the URL, and
the new user choosing their own username, password and email.
- user_invites (migration 0037): the key's sha256 only, the address, the
role and days the admin chose, who sent it, used/revoked.
- core/invites: create (revokes an older open invite to the address),
renew (new key, new clock), find_pending, the four states, signup checks.
- /invite/<key>, outside sign-in and throttled like login: the invitee's
username, email (prefilled, theirs to change) and password; the account
takes the admin's role, the key is used up in the same transaction and
they are signed in. A bad key gets one plain message.
- Settings › Users: Invite by email (address, role, 1/7/30 days), sent from
the admin in their zone; without a mail server, or if sending fails, the
link is shown once to copy. Invites not yet used are listed with Resend
and Withdraw.
- Audit: user.invited, user.invite_resent, user.invite_revoked, user.joined.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Operator review of the Account page: two sections with two different
save buttons, one stuck to the next card and one beside a squeezed hint.
- Account: time zone and Email are one form; POST /account/ saves both,
nothing unless all of it is valid. /account/email is gone.
- Users › account: Name and Email are one form; the rename route also
saves the email switches. /settings/users/<id>/email is gone.
- The Email section is a plain section now (email_section); its hint sits
under the switches, names the zone as the picker does, and links
Notifications only for an admin.
- apply_email_form leaves a switch the form doesn't carry as it is.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Reading the settings autobegins the session's transaction, so the
nested db.begin() that saves the first admin's zone as smtp.tz raised
InvalidRequestError and every visit 500'd until a zone was saved. The
integration test signed in without a zone, so that path never ran;
it now has a test of its own.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Milestone 495 "Email goes to each account, in its own time zone", on the
operator's ruling (System 156): notifications go to each user, times in
their zone, defaults on, toggled per user or per role.
- users.notify_alerts / notify_report (migration 0036): None follows the
role's default, notify.<kind>.<role> in settings, all on to start.
- mail.pick_recipients: active accounts that want the email and have an
address mail can reach (an SSO placeholder like name@oidc is skipped and
flagged), each in its own zone with no UTC beside it, then the Also send
to addresses in their chosen zone with UTC beside. One copy per address,
all over one SMTP connection.
- Alerts, the weekly report and the test email (to you, in your zone, plus
the typed addresses) send that way. The save-time smtp.tz capture is gone.
- Notifications: a role × email table of defaults, who each email reaches
and who can't be reached, and Also send to with its own zone, which starts
as the first admin's to open the page.
- Account page: your address and your two switches. Users: an Emailed
column, and an admin sets an account's switches on its page.
- The Alerts page counts the accounts it emails. zone_select macro and
formatting.zones_by_region shared by Account and Notifications.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5336 "Users page: list accounts, change role, disable, reset a local
password, create a local user".
- Settings › Users (admin only, a Settings tab): every account with how it
signs in, its role, last sign-in (from the audit log) and working AI
tokens; disabled accounts last. A row opens the account.
- An account's page: rename (any account), role and password (local
accounts; an LDAP or single sign-on role comes from the provider at
every sign-in, and the page says so), disable or enable, and Delete
behind the confirm dialog.
- Refusals are said beside the control: nobody disables, demotes or
deletes themselves, the last active admin stays, and an account that
made alert rules, maintenance windows or playbook runs is disabled
rather than deleted (those records keep who made them).
- Add user: a local account with a role and a password of 8+ characters.
- Every change is audited as user.*, and the audit log names and links the
account.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5335 "Emails Steward sends read as designed messages: an HTML alert email
and weekly report, with a plain-text twin".
- core/mail.py renders each email as multipart/alternative from
templates/email/: a table-and-inline-style HTML part and a plain-text
twin carrying the same facts. Colours are the design system's light
theme, resolved from theme.css; nothing is fetched when a mail opens.
- Alert email: state badge, a headline sentence ("Disk used on web01 is
94%."), rule, resource, reading, the condition in words, and the time in
the zone Notifications was saved in with UTC beside it; "View in
Steward" when Steward's address is set. Subject "[Steward] Firing: <rule>
— <resource> 94%". Alerts now carry their operator.
- Weekly report: monitors down, active alerts, the week's alert events,
uptime and busiest Traefik routers as small tables with links. Sent from
the configured From address; the "Generated … UTC" line and the stale
repo footer are gone.
- condition_text() states a condition from a rule's parts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The last page of #5214's walk. Settings still described Steward as it was
before monitors were unified and SNMP was rebuilt, drew its tabs below the
breadcrumb, and showed the report schedule in UTC.
- Tabs (#5377): settings/_tabs.html is a settings_tabs() macro called in
each page's section_tabs block, above the trail, as Ansible's are. The
breadcrumb guard also fails any page that {% include %}s a tab strip.
- Report time (#5378): reports.schedule_tz keeps the zone of whoever saved
the schedule; check_and_send compares in it (report_due), and the form shows
the day and hour on the reader's clock (schedule_in) under their zone's
name. "Last sent" uses |when.
- Wording (#5379): the report contents list what the report holds (monitors,
not ping-enabled hosts); "Ping latency" is "Response time" and says what it
grades; plugin detail says enabling loads now and disabling needs a
restart; the monitor interval says it applies on the next check (the
scheduler reads it every pass); the dead list-config branch is gone.
- Traefik (#5380): the 5xx and 4xx cutoffs are settings
(traefik.threshold.error_5xx_pct 0.1, error_4xx_pct 1.0) in a "Traefik
routers" group on Thresholds while the plugin runs, read by router_state
through error_cutoffs(). The Traefik page has a Thresholds button for
admins, to /settings/thresholds/#traefik.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
"0035_drop_plugin_catalog_settings" is 33 characters, and
alembic_version.version_num is VARCHAR(32), so the upgrade failed with
"value too long" (CI run 8800, integration). Renamed to
0035_drop_plugin_catalog. tests/test_migration_ids.py now checks every core
and plugin revision id fits, in the unit lane, which never migrates.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
UPS-MIB upsSecondsOnBattery is zero whenever the UPS is on mains power, so
"On battery for 0s" read as a timer that had just started. The reading gets
its own unit, battery_seconds: zero says "No, on mains", anything else
"for <span>". The metric key stays on_battery_for, so stored history keeps.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
First-party plugins have shipped inside the image since a7a281c, and the
catalog's only repository (Steward-plugins) answers 404, so Steward polled a
dead URL every minute for nothing. The operator chose to remove the catalog
entirely (#5214 "Review: readable and fast on the live instance").
Removed:
- core/plugin_index.py and the plugin_catalog_refresh startup task;
- the plugins.index_url default, and migration 0035 deletes the stored
plugins.index_url / plugins.repositories rows;
- the Settings -> Plugins repository editor, catalog, refresh, install and
reload routes and their templates;
- the download/checksum/extract install path in plugin_manager, and the
PLUGIN_INSTALL_DIR config it wrote to;
- plugins/index.yaml, docs/plugins/index.yaml.example and the guide's
"Publishing to the Catalog" section (now "Shipping a Plugin").
Kept: enabling and configuring plugins, hot-reload on enable, restart, orphan
cleanup, and the external root (/data/plugins) for operator-mounted plugins.
tests/test_no_plugin_catalog.py guards the routes, default and task.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
SNMP is on by default, so the loader has already mapped snmp_devices under
_steward_plugin_snmp; importing plugins.snmp.models defined the table twice
(integration lane, run 8788).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The operator, reviewing SNMP on #5214: "this system looks incomplete like
there's something missing from it". The one device shown was plugin.yaml's
example (core-switch 192.168.1.1, community public), polled every minute;
devices could only be defined by editing YAML in the image; a failed poll was
logged at DEBUG and stored nowhere, so the page said "waiting for the next
poll" forever; v3 ran as v2c; counters showed as lifetime totals.
- Devices are rows (snmp_devices, migration snmp_001): address, port,
v1/v2c community or a v3 user with auth/priv protocols and keys (secrets
encrypted), presets plus custom readings, an optional host binding, and
the newest poll's outcome. The YAML device list and its example are gone.
- The SNMP page adds, edits and deletes them (admin, audited, with
get_audit_entities so the audit log names and links a device). Readings
come from presets (system uptime, interface traffic by number, UPS-MIB,
Linux net-snmp) and custom "OID name [unit]" lines.
- One GET per device with every OID, devices polled at once. The outcome is
written to the row in words ("No reply from … within 5 s. Check the address
and port, that SNMP is on, and the community"), with readings it lacks.
v1's whole-request error drops that OID and asks again. SNMPv3 signs in
with USM (MD5/SHA-1/SHA-2, DES/3DES/AES).
- Test asks the unsaved form's device for its name, description, uptime and
interfaces by number. It is an admin's explicit action; every page reads
only stored rows (System 178's ruling).
- Each reading has a unit; a counter is stored as a per-second rate (bytes as
bit/s), a reset or wrap skipped. The page, widget and host panel say values
in units and why a device is silent; the device page draws a chart per
reading on a dated axis.
Tests: readings and presets, the form and its secrets, v3 checks, rates,
the poller against a stand-in pysnmp (one GET, v1 retry, timeout wording,
USM), the pages, and an integration test of the poll writing outcomes and
rates on Postgres.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The operator's ruling (Background Jobs, 2026-10-08): a Steward page reads
stored records; the sources are contacted by scheduled collection. An audit
of every GET handler, HTMX fragment, widget poll and MCP read tool found one
page load that reached out: Settings -> Plugins' catalog fragment fetched
each plugin repository (5 min cache, 10 s timeout each, retried on every
load when unreachable).
plugin_catalog_refresh ticks every minute and fetches a repository whose
copy is missing or older than an hour, waiting 5 min after a failure;
saving repositories clears the cache, so the next tick fetches. The
fragment only reads the cache and says which repositories are not fetched
yet. Refresh is now a POST that fetches at once, as does install.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The phase log on the live instance: 'traefik rows: slow 7.4s (latest=7.04s
history=0.11s window=0.04s ...)'. latest_per_router walked traefik_metrics'
index one router at a time; each step lands on the router's oldest
entries, which retention deletes every minute (likely dead ones). The
scraper already records every router figure through MetricBatch, which
keeps one row per router and figure in plugin_metric_latest (#5208's
table, read as latest_by_resource does); latest_per_router reads that
now, and the router figures carry the scrape's own time.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The operator: "load was still slow" (the access log: 7.6-8.0 s on every
poll, warm), "why do the bandwidth and 4xx sections not also get
sparklines", and "no gap between the widgets". router_history reads the
4xx share and bandwidth too, and the card draws all five lines. The
problem cards sit in .node-grid with a bottom margin; a CSS column box
dropped the last card's margin. rows() and widget() log each phase when a
request passes 1 s, so the slow query names itself on the next load.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The operator: problem cards "can disappear so quickly". Each router was
graded on its latest scrape, so a 4xx share near the 1% cutoff crossed it
on one scrape and fell back on the next. recent_window reads every
router's last 15 minutes in one query, the error shares weighted by
traffic; grade_routers uses it on the page and the dashboard widget, and
the why says "over the last 15 min".
traefik_config_last_reload_success is the Unix time of the last good
reload, not a 0/1 flag, so the tile said "Reload failed" on every scrape.
traefik_005 stores it as config_last_reload_at, and the tile reads
"Config reloaded 3m ago". The router index also carries the 4xx share.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The operator found "Today 08:03" in place of a date odd on the audit log,
and asked for the recency groups as dividers instead. |when now always
writes the date ("Thu 8 Oct 08:03"); mark_day_groups gives each entry its
group in the reader's zone, and a divider row opens each group. A scrolled
page carries the group the last one ended in, so it is not opened twice.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5355 "Traefik page doesn't load": /plugins/traefik/rows took 12-28 s.
It ran a query per router ever seen, and each raw row in the range was a
heap fetch. traefik_004 rebuilds the router index to cover the charted
columns and indexes scraped_at for the request chart; router_history reads
the current routers' buckets in one query.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Agent 1.12.0 stamps each post with sent_at; ingest compares it with the
moment the post arrived: clock_offset_s / clock_checked_at on the
registration (host_agent_003), and a clock_off_secs metric (either way)
that alert rules can watch. Replaces the old skew log, which measured the
newest sample and was off by up to an interval.
- Settings -> Thresholds: Clock drift, 30 s warn / 5 min critical.
- Host page: 'clock 0.3s ahead' in the vitals line, toned past the warning.
- Hosts list: names hosts whose clock is off; when every reporting host is
off by about the same amount, says Steward's own clock is the likely one.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Run 8736: test_user_model_fields listed the columns without time_zone, and
the session-refresh fixture's users had no time_zone attribute. The refresh
test now also asserts a zone set on the Account page reaches the session.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
