Three 404s on a quiet router read 7.5% and were raised as an anomaly; most of
the 4xx anomalies on review stood for 3 to 7 errors in 15 minutes. A new
Settings → Thresholds minimum, traefik.threshold.min_4xx (default 20, more
than one a minute held for the window), applies to both the router's grade
on its page and to anomaly detection, as min_5xx does for 5xx.
learn_normal takes `fewest`: for a share gated on a per-second rate, the
fewest events (rate × window × share) it must stand for to be judged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The host page gains a "Who's sending" panel, and the MCP a tool, that
put the host's outbound (or inbound) traffic down to its containers and
NFS mounts: read in shared time buckets, so the host's busiest bucket
says who was sending in it. A swarm service's tasks add up under its
name; what none of them accounts for is the host itself.
Also fixes CI 9488: per-container net_rx_bps / net_tx_bps join the
docker alert catalog, so a rule can catch a container flooding the
uplink.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Docker02's outbound traffic tops out the internet connection and Steward
couldn't say who sends it: a container's counters were kept only as a
running total, and NFS traffic belongs to no container.
- Docker ingest turns each running container's cumulative network counters
into net_rx_bps / net_tx_bps on "<host>/<container>" (net_rates), counting
from the stored row so a restart of Steward loses nothing; a container
new, recreated or restarted has no rate until its next snapshot.
- Agent 1.25.0 reads the bytes sent and received per NFS mount from
mountstats (already parsed for the delays) as tx_bps / rx_bps; Steward
stores them as nfs_tx_bps / nfs_rx_bps on "<host>:<mount>".
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The SNMP poll asks each device's sysName in the same GET and keeps it
(snmp_003_sys_name). A syslog source told apart by name finds its own
address from the SNMP device (reported or given name) or host of that
name, so its lines reach the SNMP device's and host's pages again; the
edit page gains Device address, which always wins (syslog_004).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Through ingress every device arrives from Docker's own address, so a
source there is keyed by the hostname its lines carry (new column
sender, unique with address). The set-up help no longer guesses the
address from the web hostname: it shows the plugin's send_to setting,
covers ingress and host mode, ASUSTOR ADM, and both syslog formats.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
On 2026-10-18 an update on VDNT-Docker02 restarted the Docker daemon six
seconds in, before the drain that came only with the reboot, and every
container on the node stopped. Docker's own packages restart the daemon from
their install scripts; needrestart's list-only mode doesn't stop that.
- The helper gains a `check` action (agent 1.24.0): a dry run of the update
for each package manager, whether Docker is among it, and the running
containers outside the swarm with their restart policies.
- A swarm node is drained before an update that upgrades Docker, else only
before a reboot, and at most once a job.
- After a drain the job settles 5 minutes before the update or reboot.
- Once the work is over, the containers that ran outside the swarm must be
seen running again within 10 minutes, or the job fails naming them.
- An older agent gets no check and its job runs as before.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The Syslog page says where to send (this server, its port and protocols) and
where the setting is on Synology, QNAP, TrueNAS, OpenWrt, UniFi and rsyslog,
open until a device sends. A sender whose address is on Steward's own
container network (Swarm ingress publishing hides the real sender) is
flagged, with the host-mode ports to publish instead. compose files and the
Dockerfile publish 5514/udp and /tcp; docs/plugins/syslog.md describes the
plugin and the deploy.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A pattern (phrases, a severity, a source, a window) is counted every minute
into plugin_metrics as lines_matching_<key> per added source, zero included,
so an ordinary alert rule fires on a match and clears once the window passes.
Each count reads only the lines stored since the last, by line id; a new or
changed pattern reads its window once.
Patterns page with ready-made storage-trouble presets and "Alert on it",
which opens the rule form aimed at the count. Source names are kept unique,
and a rename moves its counts and the rules on them.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- /plugins/syslog/sources/<id>: the lines newest first, loading as you
scroll, filtered by severity and worse, app, text and time, with day
dividers and times to the second in the reader's zone.
- The Syslog page links each source and shows its lines in the last hour.
- A Syslog panel on a host's page and an SNMP device's: the newest lines of
the sources bound to it or at its address.
- MCP: list_syslog_sources and search_syslog (source, host, text, severity
and worse, app, range, paging by next_before).
- Shared: escape_like moves to steward/core/like.py and cut_middle to the
MCP shaping module, replacing four inline copies; |when takes seconds=.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Audit: syslog.source_* worded, and a Syslog filter area.
- Backups keep syslog_sources (added, named, ignored; what it last sent
isn't compared) and leave syslog_lines out, as logs.
- The integration test reaches the plugin as loaded (#5775), so its
tables aren't defined twice; ruff E741.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- plugins/syslog: RFC 5424 and RFC 3164 parsing, RFC 6587 TCP framing.
- Lines are queued as they arrive and stored every 5 s through a Holding,
so a storage stall delays them rather than losing them.
- A new sender is an offered source, counted and not kept, until an admin
adds it on the Syslog page (name, host); it can be ignored instead.
- The listener follows the plugin's settings (port, UDP, TCP), rebinding
within seconds of a save; a port it can't open is said on the page.
- Hourly ring per source: age window and byte cap from Settings.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The agent sent Docker's working set (usage less inactive_file), which still
counts recently used file cache. A file-heavy container with no limit read
191 GB while its processes held 2.2 GB, and the host's own figure, which
already leaves cache out, disagreed with it.
Agent 1.23.0 sends mem_used_bytes / mem_used_pct (anon + shmem, so
Postgres's shared buffers count; v1: rss + shmem) and mem_cache_bytes
apart. The new names keep the two meanings apart in history and in learned
normals: docker_013 renames the columns, clears the old values and adds
mem_cache_bytes. The Docker page, swarm by-node sums, Busiest Containers,
container detail, normals and the MCP tools all read the new figure. File
cache is shown only on the container's page and in get_container.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Synology: what each disk and volume reads and writes, bytes and requests
a second and the load DSM gives it (SYNOLOGY-STORAGEIO- and -SPACEIO-MIB,
64-bit byte counters), and each disk's reconnects and sectors not found.
- Linux server: CPU waiting on disk, the wait tick counter's share of the
CPU's, and memory available (free + buffers + cache). A reading may now be
worked out from others: the poll asks for its inputs, rates each counter,
and stores only the result.
- Interface traffic: errors and discards in and out, per second.
- byte_count unit: a byte counter shown as bytes a second.
Asustor's MIB has nothing more for this; its Linux server readings gain the
same CPU wait and available memory.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each sample carries each NFS mount's requests since the last one: how many
a second, their round trip and execute time (READ and WRITE on their own),
and how many were sent again or hit a major timeout. mountstats is the
kernel's own count and never touches the share, so it reads while one is
stuck. Steward stores them per mount and the slowest on the host as
nfs_exe_max_ms, for an "NFS slow" rule and the host page's NFS chart beside
mounts_stalled; each mount's bar says what a request took.
Agent 1.22.0 (which also carries the backlog order from 41fc807).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
On 2026-10-17 the NFS under Postgres stalled for ~15 minutes. Two things
went wrong for the readings that would have explained it:
- The SNMP poll stored before it polled again, so while the database hung
it stopped polling. It now collects on schedule and hands each round to a
Holding (steward/core/holding.py): one writer stores what is held, oldest
first, each reading at the time it was collected. Bounded at two hours of
polls, with a 30-minute deadline on a store; the device list read falls
back to the last one read after 10 s.
- "Mount not answering" (10 checks in a row) went pending on Docker02 and
Docker03 and never fired: the agent sent each post's backlog with its
newest, healthy sample, and that sample ended every post's run. Agent
1.22.0 queues the new sample behind the backlog, so Steward receives a
host's samples in the order they were taken.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The table walk stopped after 64 raw entries, counting the RAM disks, loops,
partitions and md arrays that rows_like then drops, so a Synology's eighth
disk was never reached. The walk is now bounded at 1,024 entries per column;
64 still caps the rows kept.
ADM labels its disks "Disk0".."Disk4" while their indexes run 1..5; the
Asustor disk table now names each row by its index ("Disk 1"). Keys are
unchanged, so history carries on.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A preset can now read a vendor's tables: one row per disk, volume, fan or
sensor. The scheduled poll discovers the rows by walking the device: on its
first poll, after a save, and then hourly, and only while it answers. Each row
becomes readings named by the device's own labels ("Disk 1 status"). Only the
columns a row answered are asked for.
A vendor status, whether a code (Synology) or a word (Asustor), is a state
reading. It is stored as a level (0 OK, 1 busy, 2 warning, 3 failed), so one
alert rule covers any vendor, and shown in the device's own word.
- Synology: system, power and fan status and temperature; per disk: status,
health, temperature and bad sectors; per volume: RAID state (scrubbing,
parity check, rebuild, degraded), free space and size.
- Asustor: per disk: status, SMART and temperature; per volume: status, free
space and size; CPU and system temperature, free memory and fans.
From ADM's ASUSTOR-DISK-MIB and ASUSTOR-SYSTEM-MIB.
- Linux server: plus each physical disk's 1-minute busy % (UCD-DISKIO-MIB).
Migration snmp_002_table_rows adds tables, tables_at and texts to
snmp_devices. Milestone 530.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A new plan's last_slot_at holds the moment it was made, so its first
slot is the next one; the tool returned that moment, a Wednesday
evening for a Sunday plan, beside latest_run: null. cadence.fired_slot
returns it only when it falls on one of the plan's slots.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A Traefik that hasn't restarted keeps reporting the certificate it
replaced, so codex.fabledsword.com's old one (21 days left) was listed
beside its renewal (81 days) and would have raised an expiry warning on
2026-10-22. A reported certificate whose every name a later-expiring
one covers is now old, on the page, the widget, the overview and MCP.
list_certificates takes `limit` (default 100) with total/truncated, and
says how many served certificates no HTTP router uses (`unrouted`).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
One 502 among about 70 requests read steward@swarm critical at 1.4%:
past the 30-request minimum and far past the 0.1% cutoff, yet one
error. A new Settings → Thresholds field, "Fewest server errors to
fail" (traefik.threshold.min_5xx, default 3), must also be reached in
the 15-minute window; the 0.1% cutoff still catches a busy router.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
router_findings never passed `since`, so every failing router read as
brand new and sorted last among the criticals. The overview now reads
the failing routers' last day of history and dates each one with
problem_since, as the dashboard widget does.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Two 500s of about sixteen requests graded roundtable@swarm critical at
9.1%. The Settings → Thresholds minimum of requests in 15 minutes (30 by
default) now guards both shares; the 5xx share still keeps its fixed
cutoff whatever the router's history.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Through one address Steward read whichever replica answered, and with
host-mode ports behind a keepalived VIP that can be the replica serving
nothing: since the 18:43 restart every scrape landed on Docker03's idle
Traefik, which has no traefik_service_* series, so no router was stored.
A metrics_url host name that resolves to several addresses (a swarm
service's tasks.<service>, a Kubernetes headless Service, round-robin DNS)
is now scraped address by address, with the original Host header and TLS
name. Answers from one process are kept once; a replica that doesn't answer
is skipped with a warning. Requests, bytes and latency buckets are summed
over the replicas that have a baseline, so the sum's rise is the sum of each
one's own rise; certificates are listed once; Traefik's own figures add
connections and memory and keep the newest reload. A name with one address
is one scrape, as before.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
One metrics URL can reach several Traefik processes: a swarm service's
replicas behind the routing mesh or its VIP, a Kubernetes Service's pods,
a keepalived pair after failover, or the same Traefik after a restart.
Each keeps its own counters, so comparing one with another read the second
one's lifetime totals as new traffic: retired routers flashed 100% 502 and
an idle router showed 834 req/s whenever the scrape moved to the other
replica.
Each scrape is now keyed by Traefik's process_start_time_seconds and
compared only with the same process's previous scrape. A process seen for
the first time (or not in the last hour) sets its baseline and counts
nothing. Without a start time every scrape is one process, as before.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The snmp plugin registers list_snmp_devices and get_snmp_device; the
prometheus plugin registers list_prometheus_targets and
get_prometheus_target. Both are viewer-level, read what the poller and
scraper stored, and exist only while their plugin is enabled.
get_overview names a device or target whose last poll or scrape failed;
stale, pending and off are left out.
A device's community and v3 keys and a target's token are never
returned, and a password in a target URL is masked.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
get_network gives the WAN, clients, Wi-Fi networks, speed tests and devices
by status; list_devices, list_clients (everyone seen, or the newest poll's
top talkers), list_network_events, list_alarms and get_network_config
(networks, port forwards, firewall rules) read what the poller stored.
get_overview names the WAN down while the poll is current, a device offline
or degraded that the controller still reports, and active alarms.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
list_routers grades every router as the Traefik page does (the last 15
minutes, 5xx on its cutoff, 4xx and p95 against the router's normal) and
gives a troubled one its status codes, what they mean and the swarm
service behind it. get_router adds its history, codes, normal and
anomalies; list_certificates the served certificates and the routers they
cover; get_traffic the access log's clients, countries and routers.
get_overview names a router answering 5xx and a certificate near expiry.
The access-log aggregation moves into routes.summarize_access, shared by
the traffic page and get_traffic.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5835 "MCP: swarm nodes", step 2 of milestone 520. list_swarm_services
already listed every node (role, leader, availability, status); the
audit missed it. What was missing:
- Each node row carries `host_id` (the Steward host of that name) and
`maintenance` (the state of a maintenance job running on it), so an
agent can go node -> get_host / get_maintenance_job.
- get_overview names a node whose status isn't ready, unless its host
has a maintenance job running (rebooting on purpose).
- Drained alone is not a finding: VDNT-Util01 is a manager kept drained
on purpose (#5777), and a drain a failed job left behind is already
that job's maintenance_failed finding.
unique_nodes() keeps the newest report of each node, as the list did.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5834 "The host agent is always on, and the MCP sees a host's
maintenance", step 1 of milestone 520 "The MCP sees what Steward has
grown".
The host agent is a core system (operator, 2026-10-15: "it's very core
to the functionality of Steward as a whole and shouldn't be considered
something that can be toggled off"). ALWAYS_ON_PLUGINS in core settings
makes to_plugins_cfg report it enabled whatever its stored row says, so
it always loads and its MCP tools are always listed. Settings -> Plugins
shows it as "Built in: always on" with no switch, and a save keeps it
enabled. Its code and URLs stay under plugins/host_agent: every
installed agent posts to /plugins/host_agent/ingest.
Maintenance through the MCP:
- list_maintenance_plans: each plan's group, weekly slot, reboot policy,
next slot, and its latest run (hosts, how far it got, who it skipped
and why, how it ended).
- get_maintenance_job(host | job): a host's jobs, newest and running
first, or one job: state, packages updated, reboot, drained by, why it
failed, when its current state times out.
- run_maintenance(host, reboot="if_required" | "never"): admin, as the
host page's Update now; refuses a busy host or one with no helper;
audited "<user> via MCP".
- The overview names a host whose latest job in the last week failed or
timed out.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent 1.21.0. The hourly update check now covers the package managers
beyond apt and dnf (#5803 "The agent's update check reads pacman,
zypper and yum"):
- yum: `yum -C check-update`, `updateinfo list security`,
`needs-restarting -r`, as dnf.
- zypper: `--no-refresh list-updates` (`--dup` on Tumbleweed),
needed security patches from `list-patches --category security`,
and `zypper needs-rebooting`.
- pacman: `checkupdates` from pacman-contrib, which install-deps.sh now
installs. pacman has no security notion; a reboot is required once
the running kernel's /usr/lib/modules/<release> is gone.
The checks sit in one UPDATE_CHECKS table, and a failure names its
manager.
The root helper's update runs on the same hosts (#5804 "The scheduled
update runs on pacman, zypper and yum hosts"): yum upgrade, zypper
refresh + update (dup on Tumbleweed, with 102/103 counted as success),
pacman -Syu. Each step carries a label so a failure says which one,
and the reboot check reads zypper and the pacman kernel too.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Overview: a verdict line when something is wrong ('1 alert firing, 1
monitor down.'); its picker description now says what it lists (alerts,
monitors down, anomalies). Host figures have their home in Hosts.
Comments no longer name the Status page that was folded into Monitors.
- Monitors widget: a monitor not checked yet says when it was added.
- Host page: filesystems fullest first, by the worse of space and inodes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Its problems and since come from the same container_problems and
problem_since the Docker page uses, read from the day's deaths and
health changes; troubled containers sort worst problem first, then name.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Routers: a router in trouble reads 'started 2h ago' or 'all of the last
24 h', from problem_since over the day's history, read for the troubled
routers only (a certificate's problem carries its own date). Hosts: a
down host reads 'a monitor is down for 3h'; host_glance keeps the
down_since host_status already had.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
smartctl reports every disk behind a PERC/MegaRAID controller as /dev/bus/0
with type megaraid,N; the helper named a drive by its path alone, so all of
VDNT-Docker02's and Docker03's disks read as one drive "0", each overwriting
the last. The slot is now part of the name (bus0-megaraid3), and Steward gives
a repeated name an ordinal so two drives never share a series. Agent 1.20.1.
fix(ansible): Steward stopping mid-run logs no traceback (#5749)
The run's done-callback called exception() on a cancelled task, which raises;
it now logs a run cut short by shutdown at INFO. A finished run's in-memory
state is dropped by a timer rather than a sleeping task, which shutdown found
still pending.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The root helper gains one read-only action, `drives`: it finds the drives
itself (smartctl --scan-open), reads each as JSON, and writes a summary the
agent carries on every sample (agent 1.20.0). Steward judges the figures in
drives.py (failing / worn / watch / ok / no SMART / unreadable), records them
per drive and as host roll-ups for covering rules, keeps the newest report on
the registration and lists it on the host page.
install-deps.sh installs smartmontools with apt, dnf, yum, zypper, pacman or
apk, run by install.sh and every agent playbook; one it can't install is
reported, never fatal.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
On 2026-10-14 nine swarm services sat at 0/1 and nothing fired. From each
manager's swarm report the docker plugin now records three figures on
resource "swarm": swarm_services_short, swarm_nodes_down and
swarm_managers_unreachable. Drained nodes are left out of the last two,
since a drain is deliberate. The figures are recorded at most once every
30 s whichever manager reported, and are offered to alert rules.
Agent 1.19.0 sends each manager's ManagerStatus.Reachability. Older agents
fall back to the node's status.
A maintenance job on a swarm node also keeps a window over docker/swarm,
so its drain and reboot hold those alerts until the job ends.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
On the live instance the hourly refresh took 3 min 8 s, averaging a week of
raw temp_c samples each time. The hourly averages are now kept in memory.
The first refresh after startup reads the whole window. Each later one reads
raw samples from the start of the hour the last read ended in, so the hour
that was still filling is read again whole. merge_hours folds the new rows
in and drops hours older than the window.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A board's spare thermistor inputs report whatever an open input reads, and
the value rule from #5759 only catches the impossible ones. Once an hour,
sensors.refresh judges each sensor against the host's CPU (Package id N,
Tctl/Tdie/TccdN, PECI Agent N) over a week of hourly averages. A sensor is
ruled out if it falls as the CPU warms (r <= -0.5) or holds within 0.5 °C
while the CPU moved 10 °C or more. A judgment needs at least 72 shared hours.
The host temperature leaves judged sensors out at ingest. The host page
lists them under "Not counted", with each reason given once.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- #5780 "The error-pages service is graded on other routers' errors". The
Traefik scrape reads which services an errors middleware sends to, and
grades them healthy: their 4xx and 5xx are the codes of the errors they
cover for, already graded on the routers that failed.
- #5781 "The Maintenance failed rules sit at no data until a host has had
a maintenance job". Every host with an agent reads 0 until a job fails.
- #5782 "VDNT-Util01's host page lists SYSTIN at 115 °C in red". The host
page lists sensors that counts_as_hottest leaves out (unwired board
inputs, 0 °C or below) on one muted line, not as heat.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5712 "Each router problem says what it means, whether it's real, and what
to do". A problem card now names its kind and reads its codes:
- Server errors: 500 means the app is failing; 502/503/504 means Traefik
couldn't reach it or it timed out.
- Client errors: 404 scanners, a 401/403 login wall, 429 rate limiting, or
499 clients giving up on a slow app.
- Slow: p95 against p50.
Each card says whether the problem is likely real, and ends with one next
step. A fault links to the swarm service behind the router, matched by
name, and traffic links to Settings → Thresholds. The wording lives in
plugins/traefik/meaning.py.
The plugin manager now hands out a loaded plugin's submodule
(loaded_plugin_submodule), so Traefik and host_agent reach Docker's
models the same way (#5775).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5777 "Maintenance would make a node the operator keeps drained active after
its reboot". A swarm node already set to drain or pause, such as a manager
kept free of work, is no longer drained before its reboot or made active
after it. Its job only updates and reboots it, and back-in-service needs it
ready but not active.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5768 "A maintenance cadence: a group's updates and reboots, one host at a
time, on a schedule". A plan names an inventory group, a weekday and time
in a time zone, a window and a reboot policy (only when required by
default). At its slot a run takes the group's hosts in order and gives each
a #5767 maintenance job. The next host starts only when the last is back in
service: its agent is posting and, for a swarm node, the node is active and
every replicated service is at its replicas. A failure stops the run, and
no host is started once the window has passed. A host's latest failure
shows as host_agent/<host>/maintenance_failed, for an ordinary alert rule.
The page is Hosts → Maintenance.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5767 "Steward hands a host a maintenance job and follows it to the end,
through its own restarts", milestone 514.
A host_maintenance_jobs row per job: queued → updating → [draining →]
rebooting → [undraining →] done, or failed / timed_out. Steward's answer to
an ingest post carries the step that host's agent should carry out now; the
agent (1.18.0) leaves each step id for the root helper once, and the helper
refuses an id it already ran, so repeats are harmless. Results come back as
the #5766 maintenance events, matched by step id; the reboot completes when
the host's `started` event has a boot_at after the reboot was handed out.
The job reboots only when the update says one is needed (or never, if asked).
A swarm node is drained first by itself if it's a manager or by a manager
with the helper, and made active after; with no manager to drain it the job
is refused. A minute tick times out a job past its state's deadline. A
maintenance window over host_agent/<host> lasts the rest of the job.
Everything is on the row, so a Steward restart, or the reboot of the host
Steward runs on, resumes from the next post. Ingest runs it in its own
savepoint so a job going wrong never costs the samples.
The Agent panel leads with a running job and says what a finished one did;
Manage agent has Update now (with "never reboot"), or says why a job can't
start. Integration test: hand off, Steward restarts, the post-boot events
close the job.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5766 "A root helper the agent can start, for a fixed set of maintenance
actions and nothing else", milestone 514.
The agent runs unprivileged under NoNewPrivileges, so it can't update or
reboot its host. It now leaves a request in /var/lib/steward-agent/requests,
the one path its unit lets it write; steward-maint.path starts maint.py as
root, which reads it without following links, accepts only update, reboot,
drain or undrain, deletes it, and writes the outcome to
/var/lib/steward-maint/results. The agent (1.17.0) reports each new result
state once as a `maintenance` event, re-checks pending updates when an update
finishes, and says in its metadata whether the helper is installed.
The units and the helper are served from one copy at
/plugins/host_agent/files/<name> (a fixed list), and install.sh plus the
install, provision and update playbooks all fetch them, replacing three
inline copies of the unit. Update agent installs the helper on existing
hosts; the Agent panel says under Manage agent whether it is there.
Steward handing out jobs is #5767; the cadence is #5768.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Step 1 of milestone 514 "Updates and reboots on a cadence, carried out by
each host's agent…": #5765 "The agent reports pending updates and whether
a reboot is required".
- Agent 1.16.0: an hourly background check (apt-check, apt list, or dnf -C
with updateinfo and needs-restarting), each command bounded at 120 s, no
root and no network. Debian's reboot-required marker and its packages
are read every sample.
- Ingest records updates_pending, updates_security and reboot_required for
rules, and keeps the newest on the registration (host_agent_005_updates).
- The Agent card leads with "Reboot required since …", then the updates
waiting or "Up to date", with when the check ran. list_agents carries
them too.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
