Importing plugins.docker.models by path defines its tables a second time,
because the plugin manager loaded them as _steward_plugin_docker.models.
Inside the app that raises InvalidRequestError, not ImportError. The cadence
integration test hit it, and #5767's _swarm (used when a job is planned)
carried the same import. Both now go through maintenance.docker_models().
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5768 "A maintenance cadence: a group's updates and reboots, one host at a
time, on a schedule". A plan names an inventory group, a weekday and time
in a time zone, a window and a reboot policy (only when required by
default). At its slot a run takes the group's hosts in order and gives each
a #5767 maintenance job. The next host starts only when the last is back in
service: its agent is posting and, for a swarm node, the node is active and
every replicated service is at its replicas. A failure stops the run, and
no host is started once the window has passed. A host's latest failure
shows as host_agent/<host>/maintenance_failed, for an ordinary alert rule.
The page is Hosts → Maintenance.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
CI 9146 on 22b0be2:
- integration: host_agent_007 ran before core 0010 created maintenance_windows
(its window_id foreign key), so the schema never built. It now
depends_on 0010_maintenance_windows.
- unit: host_maintenance_jobs is named in the backup's EXCLUDED (run
history, like ansible_runs and host_downtime), and the
host_agent.maintenance audit action has its wording.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5767 "Steward hands a host a maintenance job and follows it to the end,
through its own restarts", milestone 514.
A host_maintenance_jobs row per job: queued → updating → [draining →]
rebooting → [undraining →] done, or failed / timed_out. Steward's answer to
an ingest post carries the step that host's agent should carry out now; the
agent (1.18.0) leaves each step id for the root helper once, and the helper
refuses an id it already ran, so repeats are harmless. Results come back as
the #5766 maintenance events, matched by step id; the reboot completes when
the host's `started` event has a boot_at after the reboot was handed out.
The job reboots only when the update says one is needed (or never, if asked).
A swarm node is drained first by itself if it's a manager or by a manager
with the helper, and made active after; with no manager to drain it the job
is refused. A minute tick times out a job past its state's deadline. A
maintenance window over host_agent/<host> lasts the rest of the job.
Everything is on the row, so a Steward restart, or the reboot of the host
Steward runs on, resumes from the next post. Ingest runs it in its own
savepoint so a job going wrong never costs the samples.
The Agent panel leads with a running job and says what a finished one did;
Manage agent has Update now (with "never reboot"), or says why a job can't
start. Integration test: hand off, Steward restarts, the post-boot events
close the job.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5766 "A root helper the agent can start, for a fixed set of maintenance
actions and nothing else", milestone 514.
The agent runs unprivileged under NoNewPrivileges, so it can't update or
reboot its host. It now leaves a request in /var/lib/steward-agent/requests,
the one path its unit lets it write; steward-maint.path starts maint.py as
root, which reads it without following links, accepts only update, reboot,
drain or undrain, deletes it, and writes the outcome to
/var/lib/steward-maint/results. The agent (1.17.0) reports each new result
state once as a `maintenance` event, re-checks pending updates when an update
finishes, and says in its metadata whether the helper is installed.
The units and the helper are served from one copy at
/plugins/host_agent/files/<name> (a fixed list), and install.sh plus the
install, provision and update playbooks all fetch them, replacing three
inline copies of the unit. Update agent installs the helper on existing
hosts; the Agent panel says under Manage agent whether it is there.
Steward handing out jobs is #5767; the cadence is #5768.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Step 1 of milestone 514 "Updates and reboots on a cadence, carried out by
each host's agent…": #5765 "The agent reports pending updates and whether
a reboot is required".
- Agent 1.16.0: an hourly background check (apt-check, apt list, or dnf -C
with updateinfo and needs-restarting), each command bounded at 120 s, no
root and no network. Debian's reboot-required marker and its packages
are read every sample.
- Ingest records updates_pending, updates_security and reboot_required for
rules, and keeps the newest on the registration (host_agent_005_updates).
- The Agent card leads with "Reboot required since …", then the updates
waiting or "Up to date", with when the check ran. list_agents carries
them too.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5764 "The agent reads a container's whole log backlog into memory, on its
main loop, with no overall deadline". Docker03's 1.13.0 agent went silent
for a day at 2.1 GB and 46 h of CPU, then ignored SIGTERM.
- One shared reader, _docker_get: each answer has an overall deadline and a
byte cap, not only a per-recv timeout. A trickling or endless body is cut
off (15 s / 32 MiB for JSON, 5 s / 4 MiB for a log read).
- Container logs: since the cursor but only the newest 2,000 lines, all
within 10 s. A note says when older lines were skipped, and a container
whose answer is too big skips ahead to now instead of re-reading its
backlog every interval.
- The agent reports its own RSS and CPU; Steward records agent_rss_bytes
and agent_cpu_pct, so a runaway agent shows before it goes silent.
Agent 1.15.0.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5759 "VDNT-Util01 reads 114 °C: its "hottest sensor" is a Super-I/O input
with nothing wired to it". b3885c3 left out AUXTIN*, but Util01's SYSTIN
reads 114 °C too, so temp_c_max stayed at 114. A board's Super-I/O inputs
(SYSTIN, CPUTIN, AUXTIN*, PCH_*) now count only below 100 °C: a wired one
reads the board, far below that, and still counts on other boards.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5760 "One stalled NFS mount freezes the whole agent: statvfs has no
deadline, so the host goes silent while it still answers ping". The agent
read every mount inline in its only loop; a hard-mounted share whose server
stops blocks statvfs in the kernel with no timeout. Each mount is now read
in a daemon thread under one 5 s deadline. One that misses it is sent as
stalled, and isn't probed again while its last probe is still blocked. A
mount that errors (a stale handle) is left out instead of emptying the list.
Steward counts them as mounts_stalled (alertable: "an NFS share stopped"),
and skips the fullest-disk figure when every mount stalled.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5759 "VDNT-Util01 reads 114 °C: its "hottest sensor" is a Super-I/O input
with nothing wired to it". AUXTIN0/1 on the board's Nuvoton chip read
110-112 °C flat at every load while the CPU package moved 49-87, so
temp_c_max read critical all week. The max now counts only readings that
can be real: not an AUXTIN input, above 0 and below 125 °C. Each sensor
keeps its own temp_c row.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5756 "The agent says when its host rebooted and when it lost contact, so
downtime is recorded as what it was". Agent 1.14.0 posts events with the
next post that gets through: started (with the host's boot time), stopping
(on SIGTERM, with whether systemd says the host is going down) and
lost_contact (from its first unanswered post to the one carrying it).
Steward keeps them as the host's downtime (host_downtime): reboot,
agent_stop, unclean (a boot after the agent was last heard that nobody
announced) or lost_contact. Retried posts apply once.
An announced stop explains the silence for reboot_grace_seconds (Settings
-> Plugins -> host_agent, 10 minutes): seconds_since_seen reads 0 until
then, and the host's other rules go quiet with it (agents_away beside
#5721's registry). Past the grace, agent offline fires.
The host's Agent card lists the last 7 days of downtime. Rows are pruned
at data.retention_days.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
VDNT-Util01's / ran out of inodes (956 free of 60.8M) with 274G of space
left; every file create failed and Steward's page read healthy at 66%.
- Agent 1.13.0 reports each mount's inodes (statvfs; a filesystem reporting
none, like btrfs, is left without).
- Ingest records inodes_used_pct per mount, and disk_used_pct_worst is the
fullest mount by space or inodes, so existing disk-full rules and the
overview see it.
- The host's health strip, the Hosts widget and the host glance grade / by
whichever is fuller and name inodes when they are ("Disk / 66% · inodes
100%", "Inodes on / ... past its critical threshold").
Hosts pick up the inode figures once their agent is updated to 1.13.0.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Nine services that lost tasks in one node's bad three minutes stayed on the
overview as warnings for a day, 17 hours after they recovered. Against the
UI ruling (only what is current on the dashboard), a service running all its
replicas is now history: its failed tasks stay on the Docker page, not on
the overview or the dashboard's failed count. Services still short of
replicas that started failing within five minutes of each other read as one
finding naming them, dated from the first.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
VDNT-Docker03's silent agent read as three overview findings (agent offline,
agent stale, the disk rule's no-data) and every drop emailed the disk rule's
silence and return on top of agent offline.
- The host agent registers its heartbeat (seconds_since_seen) and the
sources it carries (host_agent, docker). While a host's agent is past its
offline window, its other rules' no_data is recorded "while its agent is
offline" and not emailed, nor is their quiet return; a return past a
threshold still fires.
- The overview folds those rules under the host's agent-offline finding
("also silent with it: ..."), and a stale agent whose agent-offline alert
is active is that alert's finding, not a second one.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Counting every agent's silence from when Steward started watching again
(#5441) read an agent that was already gone as just seen: its agent-offline
alert resolved at 0 s and fired again three minutes later, two false emails
per restart (VDNT-Docker03, four times in a day). An agent whose last
recorded seconds_since_seen was already past the offline window now keeps
counting its whole silence; one that was fine when Steward stopped still
counts from the restart, so Steward's own downtime still fires nothing.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
CI 9128: the host vitals and monitor templates required the new grades, so
callers (and tests) that render them without one failed; they now fall back
to status_class. The new integration test read outside begin(), which
autobegan the session.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Host CPU, memory and load per core, container CPU and memory, and every
monitor's response time learn a normal per hour (milestone 513, System 182).
A reading past its fixed cutoff but within its normal reads healthy and says
what is usual at this hour; one well out of it keeps its grade and is a
metric anomaly. Disk, a monitor down and an agent offline keep their cutoffs.
- Each normal's floor, the smallest departure that counts, is a setting:
Settings -> Thresholds -> Normals (norms.floor.*). Defaults: host CPU 10
points, host memory 5, load per core 10, container CPU 10 points of one
core, container memory 5, monitor response 100 ms; Traefik's 2-point 4xx
and 50 ms p95 floors move there from constants. learn_normal takes a
floor function (floor_setting) read at each pass.
- learn_normal(parts=False): a host learns, its cores don't.
- Ingest records load_pct (load_1m / cores x 100), the figure pages grade.
- Shared grading: anomalies.against / graded, worst_reading(normals=),
hosts' host_grades, docker usage(thresholds=, norms=), monitors'
latency_grade. Applied to host vitals, the Hosts widget, MCP overview,
Docker usage, and the monitor list, widget and page.
- The monitor and container pages list their anomalies.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5741 "Traefik routers are graded against their own normal", milestone 513.
- A router's 4xx share and p95 learn their normal (learn_normal in the
plugin's setup). Once a router has 5 days of history, a reading over the
fixed cutoff but inside its normal is healthy, and the healthy table says
it is usual for that router. One out of its normal is a warning that says
what is usual, and a metric anomaly.
- The 5xx share is a failure: fixed cutoff always, whatever the history or
traffic (System 182's ruling).
- A 4xx share over fewer requests than Settings -> Thresholds' new "Fewest
requests to judge 4xx" (30 per 15 minutes) isn't graded on the page, and
isn't judged for anomalies.
- Core: learn_normal(gate=(metric, least)) judges a key only while its
resource's gate metric averages at least `least`; `least` may be a
function, so it follows a setting. learn_normal(since=...) learns only
from history after a moment: Traefik latency learns from #5755's
per-scrape figures only.
- The Traefik page lists the routers' metric anomalies of the last 30 days
(_anomaly_card.html). The dashboard widget keeps its why to problems.
- Thresholds and docs say when the cutoffs give way to a normal.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5755 "Traefik latency was a percentile of every response since Traefik
started, over unsummed label sets". _percentile_ms read the duration
histogram as it stood, which counts from Traefik's start, so p50/p95/p99 were
lifetime figures that barely moved. It also walked every code/method label
set's buckets as one unsummed list, so equal counts from two sets returned
an edge exactly: nextcloud's flat 100.0 ms p95.
service_buckets() now takes each bucket's rise since the previous scrape,
summed per le over every label set. A scrape with no responses, and the
first scrape, record no latency: traefik_007_latency_nullable lets the
columns be NULL, the batch skips the figure, and the page shows "—".
Found while preparing #5741 "Traefik routers are graded against their own
normal", which grades p95 against a learned normal.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5752 "Container CPU and memory history lives in the one metric store;
docker_metrics and docker_metrics_hourly are retired". Container cpu_pct and
mem_pct already went to plugin_metrics for alerting; now mem_usage_bytes does
too, and every snapshot in an ingest (backlog included) is recorded at its
own time. The container history chart, Busiest Containers, a service's
history and the MCP container tools read plugin_metrics under resource
"<host>/<container>", rolled up and pruned by the shared metric retention.
docker_012_drop_metrics drops both tables. Their history is not copied
(lesson #5221 "A data migration's cost is set by the live table"); the
store already holds 7 days of container CPU and memory %. Docker retention
now prunes only lifecycle events, and Settings -> Thresholds has one
"Metric retention" card for every source.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The scraper folded traefik_service_requests_total{code} into two shares and
dropped the code, so a problem couldn't say whether it was scanners (404), a
login wall (401/403) or the app down (502/503/504). Each scrape now stores the
responses per router and code that occurred (traefik_status_codes, migration
traefik_006), pruned with the router metrics and left out of backups (System
180 ruling: Traefik history is not backed up). A problem card lists its codes
over the last 15 minutes, most first, or says none occurred, or that codes
aren't recorded yet.
A counter series new since the previous scrape now counts from 0 instead of
being skipped: Traefik creates a router's 502 series at its first 502, so a
first burst of 5xx read as 0% until it happened again.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Moving the figures below the problem cards put them inside the routers
block, so with no routers yet the config-reload, connection and memory tiles
vanished (CI 9100, test_the_config_tile_says_when_it_last_reloaded). They are
a macro now, drawn in both branches.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The Traefik page now reads state first: a verdict naming each router that
needs a look, why, and when it started (from its history buckets over the
page's range), then the request chart following the page's time range, then
the problem cards (which say since when too), the proxy's own figures, and
the healthy rest. The verdict is filled out of band by /rows each poll; the
"Needs attention" and "Healthy routers" tiles it replaces are gone.
The request chart takes ?range= from the page; past a day its times carry the
day, and its clock is 24-hour (hourCycle h23).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
docs/plugins/prometheus-widgets.md covers:
- where templates live and how to copy one in, duplicate, export and share;
- a first template, walked through;
- each block (states with its deadlines, values, times, table, info) with an
example;
- the format reference: keys, units, spans, limits, what the checker refuses;
- the Fabled Archives template, block by block, with what it leaves out and
why.
Linked from the README, the plugin guide, the editor and template_format.
tests/plugins/prometheus/test_widget_guide.py checks every YAML example in the
guide against the checker, so the guide can't drift from the format.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Read from what Steward stored for Archives on 2026-10-14: the nightly
Restic replication job had said "running" since the first scrape two days
earlier, with no good copy yet and past both its late and missing times.
Archives ranks "running" above lateness, so the widget called it healthy.
- states blocks take warn_past / crit_past: a deadline metric per row (a
*_timestamp_seconds). Once it has passed, a healthy row reads as a
warning or a problem: "FullNightly-Restic is running, 11h 53m overdue".
Quiet and already-failing rows keep their own reading.
- The Archives template (version 2) uses the job's late_at / missing_at
deadlines and shows its last good copy beside the next run. It says why
section_up, the queue and the HTTP/audit counters are left out.
- tests/plugins/prometheus/archives_live.py is the series as captured
live, and test_archives_template.py renders the template over them.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Run 9093: test_every_logged_action_is_worded wanted the three
prometheus.template_* actions in ACTIONS, and the list's verdict wrote
"doesn't" as a Jinja string literal, which autoescape turns into '.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Prometheus gains a "Widget templates" tab beside Targets. It lists the
built-in templates and an admin's own, any that no longer pass the checker
first. A template opens as its YAML beside a live preview drawn over a
chosen target exactly as the dashboard widget draws it (routes.draw, now
shared by both). An admin writes or pastes a new one, reads a .yaml upload
into the editor to look over, duplicates any template under a free name,
edits and deletes their own; anyone can export the YAML. Built-in
templates are read-only.
Custom templates are rows in prometheus_widget_templates (migration
prometheus_002_templates), kept in backups, keyed "custom:<id>" so a
rename keeps its widgets. Saving needs a template that passes the checker
and a name no other template has. Create, update and delete are audited
and link to the template.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5691 "Generic Prometheus dashboard widget: pick a target and a template",
step 3 of milestone 508 "Prometheus widget templates".
- /plugins/prometheus/widget draws one target with one template in widget
shape: the verdict (led by "isn't answering" when the target is down),
then each block with problems first, figures beside a share bar or the
last 24h of bars, and the healthy rest folding to "+N more healthy" past
six. A missing target or template, or one that no longer checks, says so.
- library.py: bundled templates from plugins/prometheus/widgets/*.yaml,
the picker's target and template lists, and the default title.
- widgets/fabled_archives.yaml: the first bundled template, against the
series Archives reports (refined in #5693).
- Dashboard: a widget setting may take `options_from` (read when the picker
opens) with an `empty` sentence; `default_title`; a Details link may name
settings ("/plugins/prometheus/targets/{target}"). Prometheus is grouped
with the integrations.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5690 "Widget template format: YAML schema, validator and renderer", step 2
of milestone 508 "Prometheus widget templates".
- template_format.check(text) -> Template, or TemplateError with every
problem, each with its line and what to change. Blocks: states, values
(of -> share and bar, thresholds as numbers or shares, sparkline),
times (warn_after / crit_after), table (two labels pivoted), info.
Anchors/aliases refused, 64 KB cap, YAML's on/off/yes/no flagged.
- template_view.render(template, target, latest, now, history) -> View:
a verdict naming the worst problem, then each block with its problems
first (crit, warn, ok, then quiet states). Missing series show as "—"
and say so; a template whose `expects` prefix the target lacks says so.
- shown_value moves to service.py so the renderer can use it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5689 "Prometheus target page: fold state sets, show epoch times, size
bytes", step 1 of milestone 508 "Prometheus widget templates".
- A state set (name ending in _state, a state label, every series 0 or 1;
or the OpenMetrics form, the label named after the metric) reads as one
row per thing it describes, naming the current state, the others on
hover. fabledarchives_jobs{state} is a count and stays as it is.
- label_pairs / state_label / fold_states in service.py, for the widget
templates to reuse.
- After a good scrape, series it no longer has lose their newest value
(forget_gone): Archives' old build_info row read as current beside the
new one. History stays; a failed scrape forgets nothing; alert rules keep
their own last-sample time.
- Epoch times and _bytes sizes were already read by shown_value /
metric_value.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A bundled prometheus plugin, on by default like the other protocol plugins.
Targets are rows edited on its page: address, bearer token (encrypted),
interval, keep/leave-out name patterns and a series cap. Steward does the
filtering; the endpoint serves everything (Archives' ruling).
- A 5 s tick scrapes due targets at once, each within a 30 s deadline, and
writes the outcome to its row and its series through record_metrics, so
alert rules can watch any of them.
- Gauges are stored as they are; counters (and a histogram's or summary's
_sum/_count) as <name>_per_sec rates from the second scrape. Buckets and
quantiles are skipped. A labelled series is stored under
"<target>:k=v,…". Each target also records is_up, scrape_ms, series_kept
and series_capped.
- Pages: targets not answering first, with why; a target's series grouped
by metric with their newest values; a chart per series; the editor with
Test, which scrapes once and says what would be stored.
- The counter-to-rate keeper moves to steward/core/counter_rates.py; SNMP's
RateKeeper now scales its rate on top of it.
- metric_value reads Prometheus's _seconds as a span and _per_sec as a rate.
- The catalog guard treats an empty list as a dynamic source (snmp,
prometheus), as the rule form does.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
steward/core/prometheus.py parses the 0.0.4 exposition into families keyed
by name, each with its declared TYPE, so a scraper can tell a counter from a
gauge. A histogram or summary owns its _bucket/_sum/_count samples. Label
values are unescaped, and commas or braces inside quotes stay part of the
value. NaN and +/-Inf are read, timestamps are ignored, and a malformed
line is skipped. Traefik's parse_prometheus is now a thin reshaping of it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
At 20:25-20:39Z the NFS storage under steward_db slowed badly (checkpoint
sync 4.6 s, then 70.7 s with one file at 21.9 s, against the usual 0.002 s).
Ingest commits stalled past the agent's 10 s timeout and were dropped, and
last_seen_at, written in that same transaction, went with them. Agents
posting every few seconds read as silent for 3 minutes, and "Agent offline"
fired and emailed for all three hosts.
Ingest now notes each agent's arrival in memory right after the (read-only)
registration lookup, before any write; seconds_since_seen counts from the
later of last_seen_at and that arrival.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5470 "pysnmp-lextudio is deprecated: every startup logs a RuntimeWarning".
Dependency change: the [snmp] extra now installs `pysnmp>=7.1` instead of
`pysnmp-lextudio>=6.2`. pysnmp is the same lextudio lineage under its maintained
name; the 7.1 floor is because below 7 the name is the abandoned 4.x line with
no asyncio API. The poller already finds either API shape (_api(), the engine
close probes), so no code changes; its docstring now names what the image ships.
Approved by the operator 2026-10-09 ("yes please do the swap").
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5468 "Log search by service name: searching "steward_app" refuses because every
deploy makes a new container name".
search_container_logs(container="steward_app") was refused, because no container
carries the service's own name and five task containers partly matched. Now a
service's name reaches every task container Steward holds anything for, live and
removed (docker_svc.service_containers, as the service page's logs already do),
so one search spans deploys. Each match carries the slot and task_id that wrote
it, and the reply names the service and how many tasks were searched. A container
named exactly what was asked still takes precedence. A miss that partly matches
task containers now names their services.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5467 "Steward's own health: metrics, rules can watch them, a page that shows them".
The self-monitor tick now records, under source "steward", resource "process":
ingest_posts / ingest_failed / ingest_slowest_ms (the host-agent posts since the
last tick, tallied in steward/core/health.py), db_roundtrip_ms (a timed SELECT 1),
and unwatched_secs (the gap before this tick past watch.GAP_S, from Steward's own
last sample). The hourly cleanup records cleanup_secs. The open-file pair is still
recorded, and is now only skipped on its own where /proc is absent. All of these
are in CORE_METRICS, so an alert rule can watch them under "Steward itself".
Settings → System (/settings/system/, admin) shows a verdict first (failed posts,
a gap in watching, open files near the limit), then each measure now and over the
last day, with a sparkline scaled to its own peak.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5449 "The hourly Docker log prune ranks every log line in the table to find
the few over their cap": the size ring put a running-total window over every
row of docker_logs each hour, sorting the whole table on NFS-backed storage
to find the handful of containers past docker.logs.max_bytes_per_container.
- prune_docker_logs (capability docker.prune_logs) does the age cutoff, then
a per-container byte total picks the containers at or over the cap, and
only their rows are ranked and trimmed. Same keep rule as before: the
newest line always survives.
- run_docker_retention keeps the metrics rollup and the event prune.
- Cleanup runs the log ring in its own transaction after them.
Arose from #5431 "Ingest dropped for ~3 minutes after each deploy while
startup retention runs".
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
After Steward or its database was down, every agent's age and every
series' last sample were old, so all agent-offline rules fired and all
rules went no-data at once, clearing a minute later. With per-account
alert email that would have mailed everyone a dozen notices per restart.
A periodic job now ticks a Watch (steward/core/watch.py) once it has
reached the database; the Watch says since when runs have been unbroken
(process start, or the end of a gap over 120 s). seconds_since_seen and
the no-data sweep count silence from no earlier than that, so a source
must stay quiet while Steward is watching before it counts as silent.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5207 "UniFi Overview tab: func.strftime is SQLite-only": the WAN and
client sparklines bucketed with strftime('%s'), which Postgres lacks, so
/plugins/unifi/rows failed. Traefik's date_bin bucket moves to
core/time_range.time_bucket and both plugins use it. Devices list offline
and failing first, and a state other than online is named (Upgrading,
Adopting, Heartbeat missed...) instead of drawn as Down.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The last page of #5214's walk. Settings still described Steward as it was
before monitors were unified and SNMP was rebuilt, drew its tabs below the
breadcrumb, and showed the report schedule in UTC.
- Tabs (#5377): settings/_tabs.html is a settings_tabs() macro called in
each page's section_tabs block, above the trail, as Ansible's are. The
breadcrumb guard also fails any page that {% include %}s a tab strip.
- Report time (#5378): reports.schedule_tz keeps the zone of whoever saved
the schedule; check_and_send compares in it (report_due), and the form shows
the day and hour on the reader's clock (schedule_in) under their zone's
name. "Last sent" uses |when.
- Wording (#5379): the report contents list what the report holds (monitors,
not ping-enabled hosts); "Ping latency" is "Response time" and says what it
grades; plugin detail says enabling loads now and disabling needs a
restart; the monitor interval says it applies on the next check (the
scheduler reads it every pass); the dead list-config branch is gone.
- Traefik (#5380): the 5xx and 4xx cutoffs are settings
(traefik.threshold.error_5xx_pct 0.1, error_4xx_pct 1.0) in a "Traefik
routers" group on Thresholds while the plugin runs, read by router_state
through error_cutoffs(). The Traefik page has a Thresholds button for
admins, to /settings/thresholds/#traefik.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
UPS-MIB upsSecondsOnBattery is zero whenever the UPS is on mains power, so
"On battery for 0s" read as a timer that had just started. The reading gets
its own unit, battery_seconds: zero says "No, on mains", anything else
"for <span>". The metric key stays on_battery_for, so stored history keeps.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
First-party plugins have shipped inside the image since a7a281c, and the
catalog's only repository (Steward-plugins) answers 404, so Steward polled a
dead URL every minute for nothing. The operator chose to remove the catalog
entirely (#5214 "Review: readable and fast on the live instance").
Removed:
- core/plugin_index.py and the plugin_catalog_refresh startup task;
- the plugins.index_url default, and migration 0035 deletes the stored
plugins.index_url / plugins.repositories rows;
- the Settings -> Plugins repository editor, catalog, refresh, install and
reload routes and their templates;
- the download/checksum/extract install path in plugin_manager, and the
PLUGIN_INSTALL_DIR config it wrote to;
- plugins/index.yaml, docs/plugins/index.yaml.example and the guide's
"Publishing to the Catalog" section (now "Shipping a Plugin").
Kept: enabling and configuring plugins, hot-reload on enable, restart, orphan
cleanup, and the external root (/data/plugins) for operator-mounted plugins.
tests/test_no_plugin_catalog.py guards the routes, default and task.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The operator, reviewing SNMP on #5214: "this system looks incomplete like
there's something missing from it". The one device shown was plugin.yaml's
example (core-switch 192.168.1.1, community public), polled every minute;
devices could only be defined by editing YAML in the image; a failed poll was
logged at DEBUG and stored nowhere, so the page said "waiting for the next
poll" forever; v3 ran as v2c; counters showed as lifetime totals.
- Devices are rows (snmp_devices, migration snmp_001): address, port,
v1/v2c community or a v3 user with auth/priv protocols and keys (secrets
encrypted), presets plus custom readings, an optional host binding, and
the newest poll's outcome. The YAML device list and its example are gone.
- The SNMP page adds, edits and deletes them (admin, audited, with
get_audit_entities so the audit log names and links a device). Readings
come from presets (system uptime, interface traffic by number, UPS-MIB,
Linux net-snmp) and custom "OID name [unit]" lines.
- One GET per device with every OID, devices polled at once. The outcome is
written to the row in words ("No reply from … within 5 s. Check the address
and port, that SNMP is on, and the community"), with readings it lacks.
v1's whole-request error drops that OID and asks again. SNMPv3 signs in
with USM (MD5/SHA-1/SHA-2, DES/3DES/AES).
- Test asks the unsaved form's device for its name, description, uptime and
interfaces by number. It is an admin's explicit action; every page reads
only stored rows (System 178's ruling).
- Each reading has a unit; a counter is stored as a per-second rate (bytes as
bit/s), a reset or wrap skipped. The page, widget and host panel say values
in units and why a device is silent; the device page draws a chart per
reading on a dated axis.
Tests: readings and presets, the form and its secrets, v3 checks, rates,
the poller against a stand-in pysnmp (one GET, v1 retry, timeout wording,
USM), the pages, and an integration test of the poll writing outcomes and
rates on Postgres.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The phase log on the live instance: 'traefik rows: slow 7.4s (latest=7.04s
history=0.11s window=0.04s ...)'. latest_per_router walked traefik_metrics'
index one router at a time; each step lands on the router's oldest
entries, which retention deletes every minute (likely dead ones). The
scraper already records every router figure through MetricBatch, which
keeps one row per router and figure in plugin_metric_latest (#5208's
table, read as latest_by_resource does); latest_per_router reads that
now, and the router figures carry the scrape's own time.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The operator: "load was still slow" (the access log: 7.6-8.0 s on every
poll, warm), "why do the bandwidth and 4xx sections not also get
sparklines", and "no gap between the widgets". router_history reads the
4xx share and bandwidth too, and the card draws all five lines. The
problem cards sit in .node-grid with a bottom margin; a CSS column box
dropped the last card's margin. rows() and widget() log each phase when a
request passes 1 s, so the slow query names itself on the next load.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The operator: problem cards "can disappear so quickly". Each router was
graded on its latest scrape, so a 4xx share near the 1% cutoff crossed it
on one scrape and fell back on the next. recent_window reads every
router's last 15 minutes in one query, the error shares weighted by
traffic; grade_routers uses it on the page and the dashboard widget, and
the why says "over the last 15 min".
traefik_config_last_reload_success is the Unix time of the last good
reload, not a 0/1 flag, so the tile said "Reload failed" on every scrape.
traefik_005 stores it as config_last_reload_at, and the tile reads
"Config reloaded 3m ago". The router index also carries the 4xx share.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5355 "Traefik page doesn't load": /plugins/traefik/rows took 12-28 s.
It ran a query per router ever seen, and each raw row in the range was a
heap fetch. traefik_004 rebuilds the router index to cover the charted
columns and indexes scraped_at for the request chart; router_history reads
the current routers' buckets in one query.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Agent 1.12.0 stamps each post with sent_at; ingest compares it with the
moment the post arrived: clock_offset_s / clock_checked_at on the
registration (host_agent_003), and a clock_off_secs metric (either way)
that alert rules can watch. Replaces the old skew log, which measured the
newest sample and was off by up to an interval.
- Settings -> Thresholds: Clock drift, 30 s warn / 5 min critical.
- Host page: 'clock 0.3s ahead' in the vitals line, toned past the warning.
- Hosts list: names hosts whose clock is off; when every reporting host is
off by about the same amount, says Steward's own clock is the likely one.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
/ansible/inventory/ shows every target (address, its linked Steward host,
its groups) and every group (members, the schedules aimed at it) under a
verdict line. The whole row opens the target or group. Add target and
Add group open small dialogs; a failed add comes back with its dialog
open and what was typed. The red x leaves the rows: Delete is on the
target's or group's page, behind the confirm dialog, saying which
schedules would be left with nothing to run on.
The two list pages (/inventory/targets, /inventory/groups) are gone
with no redirect (rule 122); their URLs keep only the Add POST.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Confirm dialog (#5314):
- Every data-confirm form now asks in one native <dialog> (base.html + static/confirm.js)
instead of window.confirm.
- The question becomes the heading and what follows it the detail; the act's
button is named after the form's own button; Cancel has the focus and Esc
backs out. Optional data-confirm-title/-detail/-action/-dismiss attributes.
- All 27 existing uses work as they are. Cancelling a run says 'Keep running'.
Alerts list (#5315):
- A verdict line (what is firing, how many rules, where notifications go,
or 'notifies nobody yet' with a link) replaces the empty Active card.
- Each rule's name opens it; rows show 'last fired' and fold fire/clear into
the condition; History and Disable stay; Delete leaves the list.
- .docker-verdict becomes the shared .page-verdict.
Rule page (#5317):
- Named for the rule (title and crumb), with when it last fired and how many
changes of state it holds.
- Three sections: what it watches, when it fires (checks side by side,
seconds echoed as a duration), what happens (where notifications go, no-data,
the playbook, whose fields stay folded until ticked).
- Delete at the foot, behind the dialog, saying its history goes with it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
