On 2026-10-17 the NFS under Postgres stalled for ~15 minutes. Two things
went wrong for the readings that would have explained it:
- The SNMP poll stored before it polled again, so while the database hung
it stopped polling. It now collects on schedule and hands each round to a
Holding (steward/core/holding.py): one writer stores what is held, oldest
first, each reading at the time it was collected. Bounded at two hours of
polls, with a 30-minute deadline on a store; the device list read falls
back to the last one read after 10 s.
- "Mount not answering" (10 checks in a row) went pending on Docker02 and
Docker03 and never fired: the agent sent each post's backlog with its
newest, healthy sample, and that sample ended every post's run. Agent
1.22.0 queues the new sample behind the backlog, so Steward receives a
host's samples in the order they were taken.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A new plan's last_slot_at holds the moment it was made, so its first
slot is the next one; the tool returned that moment, a Wednesday
evening for a Sunday plan, beside latest_run: null. cadence.fired_slot
returns it only when it falls on one of the plan's slots.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each source in an anomaly's `around` was cut to its oldest 40 with
nothing in the reply to say so; on a busy host the events nearest the
occurrence were the ones dropped. Each source now keeps the 40 nearest
the occurrence's start, and `around_left_out` counts the rest. Sources
return everything their bounded query read (Docker and the host agent
no longer cut at 40 and 20 themselves), so the count is honest.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
get_anomaly gives each occurrence against what is usual at that hour, the
metric around the newest one with its normal, what else happened on the
resource's host around each occurrence (alert transitions, Ansible runs,
other anomalies there and a count elsewhere, plus plugins' own events
through a new mcp_events hook: the host agent's maintenance jobs and
downtime, Docker's container starts, deaths and OOM kills), and how earlier
anomalies on the same metric were judged.
get_metric_normal reads the learned slots and the level that would be
flagged. review_anomaly takes one id or several, all or nothing, with a
note. get_anomaly_settings and (admin) set_anomaly_settings read and change
the review window and each normal's floor, as Settings → Thresholds does.
metric_series.points takes an optional `until`; core.alerts gains
host_of_resource; floor_setting names its floor.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5834 "The host agent is always on, and the MCP sees a host's
maintenance", step 1 of milestone 520 "The MCP sees what Steward has
grown".
The host agent is a core system (operator, 2026-10-15: "it's very core
to the functionality of Steward as a whole and shouldn't be considered
something that can be toggled off"). ALWAYS_ON_PLUGINS in core settings
makes to_plugins_cfg report it enabled whatever its stored row says, so
it always loads and its MCP tools are always listed. Settings -> Plugins
shows it as "Built in: always on" with no switch, and a save keeps it
enabled. Its code and URLs stay under plugins/host_agent: every
installed agent posts to /plugins/host_agent/ingest.
Maintenance through the MCP:
- list_maintenance_plans: each plan's group, weekly slot, reboot policy,
next slot, and its latest run (hosts, how far it got, who it skipped
and why, how it ended).
- get_maintenance_job(host | job): a host's jobs, newest and running
first, or one job: state, packages updated, reboot, drained by, why it
failed, when its current state times out.
- run_maintenance(host, reboot="if_required" | "never"): admin, as the
host page's Update now; refuses a busy host or one with no helper;
audited "<user> via MCP".
- The overview names a host whose latest job in the last week failed or
timed out.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent 1.21.0. The hourly update check now covers the package managers
beyond apt and dnf (#5803 "The agent's update check reads pacman,
zypper and yum"):
- yum: `yum -C check-update`, `updateinfo list security`,
`needs-restarting -r`, as dnf.
- zypper: `--no-refresh list-updates` (`--dup` on Tumbleweed),
needed security patches from `list-patches --category security`,
and `zypper needs-rebooting`.
- pacman: `checkupdates` from pacman-contrib, which install-deps.sh now
installs. pacman has no security notion; a reboot is required once
the running kernel's /usr/lib/modules/<release> is gone.
The checks sit in one UPDATE_CHECKS table, and a failure names its
manager.
The root helper's update runs on the same hosts (#5804 "The scheduled
update runs on pacman, zypper and yum hosts"): yum upgrade, zypper
refresh + update (dup on Tumbleweed, with 102/103 counted as success),
pacman -Syu. Each step carries a label so a failure says which one,
and the reboot check reads zypper and the pacman kernel too.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Overview: a verdict line when something is wrong ('1 alert firing, 1
monitor down.'); its picker description now says what it lists (alerts,
monitors down, anomalies). Host figures have their home in Hosts.
Comments no longer name the Status page that was folded into Monitors.
- Monitors widget: a monitor not checked yet says when it was added.
- Host page: filesystems fullest first, by the worse of space and inodes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
smartctl reports every disk behind a PERC/MegaRAID controller as /dev/bus/0
with type megaraid,N; the helper named a drive by its path alone, so all of
VDNT-Docker02's and Docker03's disks read as one drive "0", each overwriting
the last. The slot is now part of the name (bus0-megaraid3), and Steward gives
a repeated name an ordinal so two drives never share a series. Agent 1.20.1.
fix(ansible): Steward stopping mid-run logs no traceback (#5749)
The run's done-callback called exception() on a cancelled task, which raises;
it now logs a run cut short by shutdown at INFO. A finished run's in-memory
state is dropped by a timer rather than a sleeping task, which shutdown found
still pending.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The root helper gains one read-only action, `drives`: it finds the drives
itself (smartctl --scan-open), reads each as JSON, and writes a summary the
agent carries on every sample (agent 1.20.0). Steward judges the figures in
drives.py (failing / worn / watch / ok / no SMART / unreadable), records them
per drive and as host roll-ups for covering rules, keeps the newest report on
the registration and lists it on the host page.
install-deps.sh installs smartmontools with apt, dnf, yum, zypper, pacman or
apk, run by install.sh and every agent playbook; one it can't install is
reported, never fatal.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
On 2026-10-14 nine swarm services sat at 0/1 and nothing fired. From each
manager's swarm report the docker plugin now records three figures on
resource "swarm": swarm_services_short, swarm_nodes_down and
swarm_managers_unreachable. Drained nodes are left out of the last two,
since a drain is deliberate. The figures are recorded at most once every
30 s whichever manager reported, and are offered to alert rules.
Agent 1.19.0 sends each manager's ManagerStatus.Reachability. Older agents
fall back to the node's status.
A maintenance job on a swarm node also keeps a window over docker/swarm,
so its drain and reboot hold those alerts until the job ends.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
temp_c_max was recorded on every ingest, but host_agent didn't offer it
to rules, so "host running hot" couldn't be watched. The catalog guard
didn't see the row(...) helper ingest builds its rows with. Now offered:
temp_c_max, temp_c per sensor, and network and disk throughput per host,
interface and disk. A new guard holds every literal row(...) name to the
catalog, except NOT_ALERTABLE, where each exclusion is given a reason.
temp_c_max reads in °C.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
On the live instance the hourly refresh took 3 min 8 s, averaging a week of
raw temp_c samples each time. The hourly averages are now kept in memory.
The first refresh after startup reads the whole window. Each later one reads
raw samples from the start of the hour the last read ended in, so the hour
that was still filling is read again whole. merge_hours folds the new rows
in and drops hours older than the window.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Rollups exist only for hours older than the raw window (7 days by default),
so the first deploy read an empty week and judged nothing. refresh now
averages the raw samples by the hour and adds any rolled-up hours in the
window. Sensors are named from plugin_metric_latest, so each series is read
through its index instead of a pattern over every host_agent row. An
integration test covers both halves against Postgres.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A board's spare thermistor inputs report whatever an open input reads, and
the value rule from #5759 only catches the impossible ones. Once an hour,
sensors.refresh judges each sensor against the host's CPU (Package id N,
Tctl/Tdie/TccdN, PECI Agent N) over a week of hourly averages. A sensor is
ruled out if it falls as the CPU warms (r <= -0.5) or holds within 0.5 °C
while the CPU moved 10 °C or more. A judgment needs at least 72 shared hours.
The host temperature leaves judged sensors out at ingest. The host page
lists them under "Not counted", with each reason given once.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- #5780 "The error-pages service is graded on other routers' errors". The
Traefik scrape reads which services an errors middleware sends to, and
grades them healthy: their 4xx and 5xx are the codes of the errors they
cover for, already graded on the routers that failed.
- #5781 "The Maintenance failed rules sit at no data until a host has had
a maintenance job". Every host with an agent reads 0 until a job fails.
- #5782 "VDNT-Util01's host page lists SYSTIN at 115 °C in red". The host
page lists sensors that counts_as_hottest leaves out (unwired board
inputs, 0 °C or below) on one muted line, not as heat.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5712 "Each router problem says what it means, whether it's real, and what
to do". A problem card now names its kind and reads its codes:
- Server errors: 500 means the app is failing; 502/503/504 means Traefik
couldn't reach it or it timed out.
- Client errors: 404 scanners, a 401/403 login wall, 429 rate limiting, or
499 clients giving up on a slow app.
- Slow: p95 against p50.
Each card says whether the problem is likely real, and ends with one next
step. A fault links to the swarm service behind the router, matched by
name, and traffic links to Settings → Thresholds. The wording lives in
plugins/traefik/meaning.py.
The plugin manager now hands out a loaded plugin's submodule
(loaded_plugin_submodule), so Traefik and host_agent reach Docker's
models the same way (#5775).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5777 "Maintenance would make a node the operator keeps drained active after
its reboot". A swarm node already set to drain or pause, such as a manager
kept free of work, is no longer drained before its reboot or made active
after it. Its job only updates and reboots it, and back-in-service needs it
ready but not active.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Importing plugins.docker.models by path defines its tables a second time,
because the plugin manager loaded them as _steward_plugin_docker.models.
Inside the app that raises InvalidRequestError, not ImportError. The cadence
integration test hit it, and #5767's _swarm (used when a job is planned)
carried the same import. Both now go through maintenance.docker_models().
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5768 "A maintenance cadence: a group's updates and reboots, one host at a
time, on a schedule". A plan names an inventory group, a weekday and time
in a time zone, a window and a reboot policy (only when required by
default). At its slot a run takes the group's hosts in order and gives each
a #5767 maintenance job. The next host starts only when the last is back in
service: its agent is posting and, for a swarm node, the node is active and
every replicated service is at its replicas. A failure stops the run, and
no host is started once the window has passed. A host's latest failure
shows as host_agent/<host>/maintenance_failed, for an ordinary alert rule.
The page is Hosts → Maintenance.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
CI 9146 on 22b0be2:
- integration: host_agent_007 ran before core 0010 created maintenance_windows
(its window_id foreign key), so the schema never built. It now
depends_on 0010_maintenance_windows.
- unit: host_maintenance_jobs is named in the backup's EXCLUDED (run
history, like ansible_runs and host_downtime), and the
host_agent.maintenance audit action has its wording.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5767 "Steward hands a host a maintenance job and follows it to the end,
through its own restarts", milestone 514.
A host_maintenance_jobs row per job: queued → updating → [draining →]
rebooting → [undraining →] done, or failed / timed_out. Steward's answer to
an ingest post carries the step that host's agent should carry out now; the
agent (1.18.0) leaves each step id for the root helper once, and the helper
refuses an id it already ran, so repeats are harmless. Results come back as
the #5766 maintenance events, matched by step id; the reboot completes when
the host's `started` event has a boot_at after the reboot was handed out.
The job reboots only when the update says one is needed (or never, if asked).
A swarm node is drained first by itself if it's a manager or by a manager
with the helper, and made active after; with no manager to drain it the job
is refused. A minute tick times out a job past its state's deadline. A
maintenance window over host_agent/<host> lasts the rest of the job.
Everything is on the row, so a Steward restart, or the reboot of the host
Steward runs on, resumes from the next post. Ingest runs it in its own
savepoint so a job going wrong never costs the samples.
The Agent panel leads with a running job and says what a finished one did;
Manage agent has Update now (with "never reboot"), or says why a job can't
start. Integration test: hand off, Steward restarts, the post-boot events
close the job.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5766 "A root helper the agent can start, for a fixed set of maintenance
actions and nothing else", milestone 514.
The agent runs unprivileged under NoNewPrivileges, so it can't update or
reboot its host. It now leaves a request in /var/lib/steward-agent/requests,
the one path its unit lets it write; steward-maint.path starts maint.py as
root, which reads it without following links, accepts only update, reboot,
drain or undrain, deletes it, and writes the outcome to
/var/lib/steward-maint/results. The agent (1.17.0) reports each new result
state once as a `maintenance` event, re-checks pending updates when an update
finishes, and says in its metadata whether the helper is installed.
The units and the helper are served from one copy at
/plugins/host_agent/files/<name> (a fixed list), and install.sh plus the
install, provision and update playbooks all fetch them, replacing three
inline copies of the unit. Update agent installs the helper on existing
hosts; the Agent panel says under Manage agent whether it is there.
Steward handing out jobs is #5767; the cadence is #5768.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Step 1 of milestone 514 "Updates and reboots on a cadence, carried out by
each host's agent…": #5765 "The agent reports pending updates and whether
a reboot is required".
- Agent 1.16.0: an hourly background check (apt-check, apt list, or dnf -C
with updateinfo and needs-restarting), each command bounded at 120 s, no
root and no network. Debian's reboot-required marker and its packages
are read every sample.
- Ingest records updates_pending, updates_security and reboot_required for
rules, and keeps the newest on the registration (host_agent_005_updates).
- The Agent card leads with "Reboot required since …", then the updates
waiting or "Up to date", with when the check ran. list_agents carries
them too.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5764 "The agent reads a container's whole log backlog into memory, on its
main loop, with no overall deadline". Docker03's 1.13.0 agent went silent
for a day at 2.1 GB and 46 h of CPU, then ignored SIGTERM.
- One shared reader, _docker_get: each answer has an overall deadline and a
byte cap, not only a per-recv timeout. A trickling or endless body is cut
off (15 s / 32 MiB for JSON, 5 s / 4 MiB for a log read).
- Container logs: since the cursor but only the newest 2,000 lines, all
within 10 s. A note says when older lines were skipped, and a container
whose answer is too big skips ahead to now instead of re-reading its
backlog every interval.
- The agent reports its own RSS and CPU; Steward records agent_rss_bytes
and agent_cpu_pct, so a runaway agent shows before it goes silent.
Agent 1.15.0.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5759 "VDNT-Util01 reads 114 °C: its "hottest sensor" is a Super-I/O input
with nothing wired to it". b3885c3 left out AUXTIN*, but Util01's SYSTIN
reads 114 °C too, so temp_c_max stayed at 114. A board's Super-I/O inputs
(SYSTIN, CPUTIN, AUXTIN*, PCH_*) now count only below 100 °C: a wired one
reads the board, far below that, and still counts on other boards.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5760 "One stalled NFS mount freezes the whole agent: statvfs has no
deadline, so the host goes silent while it still answers ping". The agent
read every mount inline in its only loop; a hard-mounted share whose server
stops blocks statvfs in the kernel with no timeout. Each mount is now read
in a daemon thread under one 5 s deadline. One that misses it is sent as
stalled, and isn't probed again while its last probe is still blocked. A
mount that errors (a stale handle) is left out instead of emptying the list.
Steward counts them as mounts_stalled (alertable: "an NFS share stopped"),
and skips the fullest-disk figure when every mount stalled.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5759 "VDNT-Util01 reads 114 °C: its "hottest sensor" is a Super-I/O input
with nothing wired to it". AUXTIN0/1 on the board's Nuvoton chip read
110-112 °C flat at every load while the CPU package moved 49-87, so
temp_c_max read critical all week. The max now counts only readings that
can be real: not an AUXTIN input, above 0 and below 125 °C. Each sensor
keeps its own temp_c row.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5756 "The agent says when its host rebooted and when it lost contact, so
downtime is recorded as what it was". Agent 1.14.0 posts events with the
next post that gets through: started (with the host's boot time), stopping
(on SIGTERM, with whether systemd says the host is going down) and
lost_contact (from its first unanswered post to the one carrying it).
Steward keeps them as the host's downtime (host_downtime): reboot,
agent_stop, unclean (a boot after the agent was last heard that nobody
announced) or lost_contact. Retried posts apply once.
An announced stop explains the silence for reboot_grace_seconds (Settings
-> Plugins -> host_agent, 10 minutes): seconds_since_seen reads 0 until
then, and the host's other rules go quiet with it (agents_away beside
#5721's registry). Past the grace, agent offline fires.
The host's Agent card lists the last 7 days of downtime. Rows are pruned
at data.retention_days.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
VDNT-Util01's / ran out of inodes (956 free of 60.8M) with 274G of space
left; every file create failed and Steward's page read healthy at 66%.
- Agent 1.13.0 reports each mount's inodes (statvfs; a filesystem reporting
none, like btrfs, is left without).
- Ingest records inodes_used_pct per mount, and disk_used_pct_worst is the
fullest mount by space or inodes, so existing disk-full rules and the
overview see it.
- The host's health strip, the Hosts widget and the host glance grade / by
whichever is fuller and name inodes when they are ("Disk / 66% · inodes
100%", "Inodes on / ... past its critical threshold").
Hosts pick up the inode figures once their agent is updated to 1.13.0.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
VDNT-Docker03's silent agent read as three overview findings (agent offline,
agent stale, the disk rule's no-data) and every drop emailed the disk rule's
silence and return on top of agent offline.
- The host agent registers its heartbeat (seconds_since_seen) and the
sources it carries (host_agent, docker). While a host's agent is past its
offline window, its other rules' no_data is recorded "while its agent is
offline" and not emailed, nor is their quiet return; a return past a
threshold still fires.
- The overview folds those rules under the host's agent-offline finding
("also silent with it: ..."), and a stale agent whose agent-offline alert
is active is that alert's finding, not a second one.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Counting every agent's silence from when Steward started watching again
(#5441) read an agent that was already gone as just seen: its agent-offline
alert resolved at 0 s and fired again three minutes later, two false emails
per restart (VDNT-Docker03, four times in a day). An agent whose last
recorded seconds_since_seen was already past the offline window now keeps
counting its whole silence; one that was fine when Steward stopped still
counts from the restart, so Steward's own downtime still fires nothing.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
CI 9128: the host vitals and monitor templates required the new grades, so
callers (and tests) that render them without one failed; they now fall back
to status_class. The new integration test read outside begin(), which
autobegan the session.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Host CPU, memory and load per core, container CPU and memory, and every
monitor's response time learn a normal per hour (milestone 513, System 182).
A reading past its fixed cutoff but within its normal reads healthy and says
what is usual at this hour; one well out of it keeps its grade and is a
metric anomaly. Disk, a monitor down and an agent offline keep their cutoffs.
- Each normal's floor, the smallest departure that counts, is a setting:
Settings -> Thresholds -> Normals (norms.floor.*). Defaults: host CPU 10
points, host memory 5, load per core 10, container CPU 10 points of one
core, container memory 5, monitor response 100 ms; Traefik's 2-point 4xx
and 50 ms p95 floors move there from constants. learn_normal takes a
floor function (floor_setting) read at each pass.
- learn_normal(parts=False): a host learns, its cores don't.
- Ingest records load_pct (load_1m / cores x 100), the figure pages grade.
- Shared grading: anomalies.against / graded, worst_reading(normals=),
hosts' host_grades, docker usage(thresholds=, norms=), monitors'
latency_grade. Applied to host vitals, the Hosts widget, MCP overview,
Docker usage, and the monitor list, widget and page.
- The monitor and container pages list their anomalies.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
At 20:25-20:39Z the NFS storage under steward_db slowed badly (checkpoint
sync 4.6 s, then 70.7 s with one file at 21.9 s, against the usual 0.002 s).
Ingest commits stalled past the agent's 10 s timeout and were dropped, and
last_seen_at, written in that same transaction, went with them. Agents
posting every few seconds read as silent for 3 minutes, and "Agent offline"
fired and emailed for all three hosts.
Ingest now notes each agent's arrival in memory right after the (read-only)
registration lookup, before any write; seconds_since_seen counts from the
later of last_seen_at and that arrival.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5467 "Steward's own health: metrics, rules can watch them, a page that shows them".
The self-monitor tick now records, under source "steward", resource "process":
ingest_posts / ingest_failed / ingest_slowest_ms (the host-agent posts since the
last tick, tallied in steward/core/health.py), db_roundtrip_ms (a timed SELECT 1),
and unwatched_secs (the gap before this tick past watch.GAP_S, from Steward's own
last sample). The hourly cleanup records cleanup_secs. The open-file pair is still
recorded, and is now only skipped on its own where /proc is absent. All of these
are in CORE_METRICS, so an alert rule can watch them under "Steward itself".
Settings → System (/settings/system/, admin) shows a verdict first (failed posts,
a gap in watching, open files near the limit), then each measure now and over the
last day, with a sparkline scaled to its own peak.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
After Steward or its database was down, every agent's age and every
series' last sample were old, so all agent-offline rules fired and all
rules went no-data at once, clearing a minute later. With per-account
alert email that would have mailed everyone a dozen notices per restart.
A periodic job now ticks a Watch (steward/core/watch.py) once it has
reached the database; the Watch says since when runs have been unbroken
(process start, or the end of a gap over 120 s). seconds_since_seen and
the no-data sweep count silence from no earlier than that, so a source
must stay quiet while Steward is watching before it counts as silent.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Agent 1.12.0 stamps each post with sent_at; ingest compares it with the
moment the post arrived: clock_offset_s / clock_checked_at on the
registration (host_agent_003), and a clock_off_secs metric (either way)
that alert rules can watch. Replaces the old skew log, which measured the
newest sample and was off by up to an interval.
- Settings -> Thresholds: Clock drift, 30 s warn / 5 min critical.
- Host page: 'clock 0.3s ahead' in the vitals line, toned past the warning.
- Hosts list: names hosts whose clock is off; when every reporting host is
off by about the same amount, says Steward's own clock is the likely one.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
/ansible/inventory/ shows every target (address, its linked Steward host,
its groups) and every group (members, the schedules aimed at it) under a
verdict line. The whole row opens the target or group. Add target and
Add group open small dialogs; a failed add comes back with its dialog
open and what was typed. The red x leaves the rows: Delete is on the
target's or group's page, behind the confirm dialog, saying which
schedules would be left with nothing to run on.
The two list pages (/inventory/targets, /inventory/groups) are gone
with no redirect (rule 122); their URLs keep only the Add POST.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5290 "Host page: the health tiles span the card's width; even spacing above Ansible":
the four tiles take equal quarters (halves on a phone) and their sparklines stretch
across them; the Monitors + Agent row drops the Agent card's own margin, so the gap
above Ansible is 1rem like every other.
#5291 "Add host opens as a dialog over the Hosts list": a native <dialog> on /hosts/,
opened by Add host or /hosts/?add=1. A name clash re-renders the list with it open,
the error and what was typed. GET /hosts/new is gone; hosts/form.html is Edit only.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Hosts list:
- CPU and Memory are their own labelled columns (#5279); a silent
agent's last readings are greyed, and Status says "agent silent for 7m".
- One uptime figure, 30 days, with 24h and 7d on hover. The SLA tab
(/hosts/uptime) repeated it and is removed (#5280; one home per fact).
- Status is a word with how long ("Down for 12m"); host_status now carries
down_since. Edit and Delete live on the host's page only (#5281).
- Figures are right-aligned with tabular digits (.table .num), not a
monospace face.
Agent fleet (#5282): its add-host form is gone, so hosts are added only on
Hosts -> Add host. That form was also the only way to get a token for a
manual curl install, so that moves to the host's Agent card as "Install
manually" (POST /fleet/<host_id>/register), shown on the fleet page as
before. "Delete" there reads "Unregister".
Add host (#5283): laid out like the add-monitor form; a name clash comes
back inline with what was typed instead of a bare text 400; the address
is optional, as the host page already allowed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Docker panel (#5276): problems first, then the 8 busiest running
containers by CPU; the other running and the stopped fold behind one
"+N more running · M stopped" line. Containers that are not swarm tasks
read the way the Docker page reads them: CI jobs as "CI: workflow · job",
buildx builders as "BuildKit builder · name".
Monitors card (#5277): the edit/delete icon buttons and the inline add
form are gone. Each row links to its monitor's page; "Add monitor" opens
the add form filed under this host, starting from its address, with the
host in its breadcrumbs and Cancel back to it.
Agent card (#5278): Full metrics stays in view; Update, Rotate token,
Unregister agent (was "Remove") and Re-provision sit under "Manage agent".
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- New installs start from widgets.RECOMMENDED, packed row by row for the
enabled plugins; the edit drawer can reset a dashboard to it.
- Host Agent — Resources folds into Hosts — Overview (which gains load 1m),
UniFi — Devices into UniFi — Overview; migration 0032 moves existing
dashboards onto the survivors.
- The Overview labels what it counts (hosts, monitors up/down/pending,
alerts firing) and lists firing alerts and down monitors with since-times.
- Uptime polls every 5 minutes (poll_seconds) instead of every cycle.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5210 "Problems first, with since-time, across every list" (milestone 477),
second batch.
- Host page: an "Alerts on this host" strip first (rules on its agent
series, its containers or its monitors: alert_svc.watches_host); its
monitors problems first with "Down 12m · reason"; Docker above Ansible.
- Uptime page and widget: worst 24h uptime first, then 7d and 30d; hosts
with no uptime yet last.
- Agent panel and fleet widget: last seen as "3m ago", not a clock time.
- SNMP: a device is "Reachable" only while it answered within the last 3
polls; older readings say "No answer" with their age, and every device
says when it was last read (was: reachable whenever any reading existed).
- Traefik: a router missing from 3 scrapes in a row is listed apart as no
longer reported, by when it was last seen, instead of keeping its last
state (a removed router stayed Failing).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The dashboard's host widgets took 18-60 s. Every "latest reading" query
(hosts overview, host-agent fleet, host page, vitals, SNMP, MCP get_host)
searched plugin_metrics for the newest row per metric. Postgres has no skip
scan, so each one read every raw sample in the 7-day window.
- plugin_metric_latest: one row per (source, resource, metric).
record_metrics upserts it in the same transaction, moving forward only and
in sorted key order. Migration 0030 seeds it by stepping the existing index.
- Retention drops a latest row once its sample leaves the raw window, so a
resource that stopped reporting disappears when its history does, as before.
Renames carry it along.
- A quiet host keeps its last values, marked stale. A time-bounded query
would have shown it as missing.
- Vitals sparklines span the 6 h they claim, averaged in SQL to ~120 points,
instead of fetching 6 h raw and drawing the last 120 samples.
- Dashboard and share-view cells: "Loading…" until the first answer; a 20 s
deadline per fetch; "Couldn't load (why)" with Retry now and an automatic
retry for cells that don't poll; "as of HH:MM" in the header, and
"couldn't refresh · as of" when a refresh fails over shown data.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
MAX_SAMPLES_PER_INGEST was 2 while agent 1.10 drains a backlog five samples a
post, so the server dropped the oldest three of every such post. Posts now
answer in 0.4-1.3 s on dev, including the boot cleanup window, so five fit
the agent's 10 s. A test holds the server cap at or above the agent's.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent 1.11.0 sends finished_at (State.FinishedAt) and reads a stopped
container's logs once per stop: since its cursor if it was being read, or
its last 200 lines when first seen already crashed (non-zero exit, stopped
in the last 10 min). Cursors and finals are forgotten once a container is no
longer listed.
Ingest gives a container first seen already exited non-zero a die event, and
dates die/stop events by finished_at when the agent sends it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
persist_host_docker marks docker.status, .swarm, .disk, .logs, .metrics,
.events, .containers, .alerts and .reap on the ingest's phase timer, each
flushed before its mark so a part's writes are charged to it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Measured on dev (385a579), agent posts cut to five samples still dropped:
flush=8.8s for 990 rows, 6.6s for 680, 8.4s for 1000 — about 110 rows/s,
~9 ms a row, with rules and evaluation near zero. One random read per
row fits a random uuid4 primary key whose index no longer fits in memory:
every new row lands on a random page of it. (Inferred from the rate; the
table and index sizes have not been read.)
- steward.core.ids.time_ordered_id: a UUIDv7-layout string (48-bit ms
timestamp, then random), same 36-char shape as the stored uuid4 ids, so
no migration. Python's uuid7() arrives in 3.14; the image runs 3.13.
- Used for plugin_metrics and its hourly rollup, monitor_results,
audit_events, the docker metrics/hourly/events/logs tables, and the
traefik and unifi time series.
- MAX_SAMPLES_PER_INGEST 5 → 2 (~400 rows) for now: it fits the agent's
10 s even at the old rate, so hosts land data whichever way the new rate
comes out. Raise it once the new rate is measured.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- BACKOFF_CAP 300 → 60: Steward reads a host stale after 180 s, so an agent
backing off to five minutes read stopped long after Steward came back
(lesson #4355: a check-in inherits its call's backoff).
- A POST carries at most five samples: the oldest four buffered plus the
new one, the rest held for the next post. A backlog drains a few at a
time instead of in one post Steward cannot write before the timeout
(the loop measured in #5149).
- Per-container Docker work has a 20 s deadline over the whole fan-out.
Past it the sample reports docker_status "error" ("N of M containers
not read within 20 s") and no list: a partial list would read as
removals, and a record without its inspect as restarts.
- /system/df is asked every 600 s rather than every sample; Steward keeps
the last answer.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
