Replaces the read-only token and the webhook automation, which every install
would have had to paste into its own Home Assistant. Connect now goes to Home
Assistant's own sign-in page, and Steward keeps the refresh token that comes
back, encrypted, and calls backup.create_automatic and update.install directly.
The connection page warns that the stored credential has the same access as
the user who signed in, and recommends a service account so Steward's actions
are easy to tell apart in Home Assistant's logs. A user who isn't an
administrator gets the status page and no Install.
Those actions answer only when finished, so a request cut off once sent counts
as sent and the outcome is read from the entities. Removing the connection, or
signing in again, ends the old sign-in on Home Assistant.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A new plugin reads Home Assistant with a read-only token every 5 minutes and
shows which Core, OS and add-on updates are waiting and how old the last backup
is. Install takes a backup through a local-only webhook, confirms Home
Assistant's last-backup time moved, installs, and follows the update to its end,
through Home Assistant's and Steward's own restarts. One runs at a time. Each
ends as one timeline entry.
Steward holds nothing that can act on the house: the webhook's automation can
only take a backup or install an allowed update (System 231's ruling). A refused
token stops polling, as Home Assistant bans an address after a few refusals.
Milestone 576 steps 1 to 5.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#6727 "A metric that bursts now and then opens anomalies at hours whose usual
peak reads 0". usual_peak is the p90 of a slot's hourly max, so a metric that
bursts in under about 1 hour in 10 reads 0 there however often it bursts across
the day. #6723 "An anomaly opens on a burst of minutes inside the metric's usual
peak, so most rows aren't outside the norm" therefore still opened on those.
usual_anywhere is the median, across the hours of the day, of each hour's
highest peak. Height is now judged against the greater of it and the slot's
usual peak. Bursts at any hour lift the median; a job at a time of its own
raises a few hours and leaves it low, so the same level at another time of day
is still an anomaly, as System 182 rules. A held level still opens.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Patterns could only be added on the Syslog page, so an agent asked to map
them on the NASes couldn't. list_syslog_patterns gives each pattern's metric
and the ready-made ones not added; add_syslog_preset (admin, audited) adds one.
The key-picking helper moves beside the patterns for both callers.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A rule compared a reading with a fixed number, so Synology Disk 1's bad
sectors going from 3,160 to 3,392 could say nothing: a count that only grows
is soon past any line. "rose by" compares each reading with the one before it,
kept on the alert's state and only for such a rule. A fall (a replaced disk)
doesn't fire. On the rule form, through MCP, and worded in emails, the alert
sentence, history and the timeline.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Rows on the live instance were mostly bursts of a few minutes: 28 of 52
peaked within the metric's own usual peak, and 42 of 52 latest occurrences
lasted 30 minutes or less, which is what a burst shorter than the 15-minute
average looks like once its span is back-dated. Nothing asked how high or how
long, as the System 182 ruling says a normal records.
A departure now also has to be past the slot's usual peak, or held: the
median of the last hour over the same line. Those spans were also keeping the
burstiest hours out of learning, lowering the usual lines further.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Docker's ingress address depends on the node Steward's task is on, and sources
were keyed by (address, sender). After a task move the same device arrived under
a new key and became a new "offered" row, so its lines were counted, not kept.
A named sender is now one row whatever address it last came from, enforced by a
partial unique index. Migration syslog_005 folds rows sharing a name into the
added one and moves their lines and patterns across.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
"swarm services short on swarm is 6." becomes "6 swarm services are
short of their replicas.", with a Which line naming each service and how
many of its tasks run. The swarm figures get labels, and the counting
figures (swarm, mounts, drives, updates) read as sentences in the email,
its subject and the Alerts page. A plugin names what its count counts
through core.alerts.names_behind; the Docker plugin does for the three
swarm figures, from the stored swarm. Webhooks get alert.which too.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each weekly run drains Docker02 then Docker03, and the swarm never moves a
task back by itself, so every run ended with nearly every service on one
node. On 2026-10-25 the operator had that spread out by hand.
- Agent 1.26.0 reports each swarm service's pinned digest, and every
repo@sha256 its host holds (from /system/df, every 10 min).
- Once a run's last host is back in service, Steward picks services on the
fullest node to move to the emptiest until they are within one task. It
takes services with several replicas on one node first. A service moves only
when the emptier node already holds its exact image. One the plan keeps in
place, one below its replicas, and one with no pinned digest stay put, and
the run says why.
- A swarm manager's helper carries it out through a new action, rebalance,
with the service names (validated, at most 48). It force-updates each in
turn with the spec unchanged and no registry lookup, skips any a
constraint places, and gives each 5 min to settle.
- The run ends done with what moved, or failed if a move didn't settle or
the manager didn't answer in 60 min. A docker-scoped window covers the
restarts, and the step goes on the timeline.
- Plans gain "Spread swarm services back out" (on by default) and "Keep in
place", on the create form and in an "After the run" form on each plan.
list_maintenance_plans shows both and the latest run's spread.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
On 2026-10-25 the 09:00 job on VDNT-Docker02 drained the node, which ran
Traefik and Steward. The agent's result reaches Steward through Traefik,
and Traefik took 11.5 minutes to start on Docker03 (its image had to be
pulled). The job gave up at the 10-minute drain deadline and finished
timed_out where it stood, so the node stayed drained for hours.
- A job that fails or times out while its node may be drained (the drain
was handed out, or went through and wasn't undone) goes to undraining
first. The outcome is kept in the new column ends_as, and the job
finishes as that once the node is active again. If the undrain goes
wrong too, detail says both.
- The drain deadline is 30 minutes instead of 10, so a result held up while
the proxy moves still lands.
- Migration host_agent_012_ends_as. get_maintenance_job returns ends_as.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A ticket records a known problem: title, area, hosts, subject, description,
links to alert rules and anomalies, and a log of comments, edits, resolving
and reopening. Opening, resolving (with the resolution as the note) and
reopening go on the timeline; every step is audited, and the audit rows stay
off the timeline.
An alert or anomaly an open ticket covers (by link, by subject, or a
host-wide ticket on the host itself) shows a Known issue badge, and offers
"Open a ticket" prefilled from its rule or anomaly when none does. Hosts,
monitors and containers get a Known issues card.
MCP: list_tickets, get_ticket (viewer); open_ticket, update_ticket,
comment_ticket, resolve_ticket, reopen_ticket (operator). list_alerts and
list_anomalies carry known_issue; get_overview, get_host, get_monitor and
get_container carry open_tickets. Also update_change and delete_change for
changes recorded by hand (author or admin).
Tickets are left out of backups, like the timeline (2026-10-15 ruling).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A change Steward can't see happen (a config edited, a disk swapped) is
recorded by whoever made it: what was done, its area, when (in the reader's
own zone), the hosts, what it was done to and a detail. Its author or an
admin edits or deletes it; recording, editing and deleting are audited and
those audit rows stay off the timeline. log_change records one as
"<user> via MCP", editable by the token's user.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The host agent writes each maintenance job once it ends, and each stretch a
host was away (a reboot, a crash, lost contact, the agent restarting), kept up
to date until it is back. Docker judges each snapshot: a unit on a new image,
or a container recreated, is a deploy; deaths past the Thresholds crash-loop
count are one crash-loop entry that later deaths extend; single restarts are
recorded but hidden. An anomaly has one entry whose type follows its review:
only those judged real are shown by default.
A source whose record changes keeps one entry for it (record_or_update).
Plugins register the records their entries point to, so an aged-out job or
downtime reads "Details no longer kept". The first fill runs once per source
(timeline.filled), the host agent's jobs and downtime included.
get_anomaly's "what else was going on" reads the timeline for the host or
resource instead of querying alerts, runs, maintenance, downtime and container
deaths itself; Docker's own source keeps only starts, stops and health
changes. A new kind, Observed, is for what Steward saw happen with no rule or
person behind it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A stored log, timeline_entries, with one row per thing that happened. Each
source writes its row in the same transaction as its own record, so the
history outlives the 30/90/365-day retention of the tables it points to:
- alert events (core/alerts._add_event): fired, resolved, acknowledged,
went silent, readings back;
- finished Ansible runs (executor, and the startup sweep of interrupted
runs): changed something, changed nothing, failed, dry run;
- audit rows (core/audit.log_audit): Steward's settings, users and access,
sign-ins, backups. Actions another writer records are left to it.
Entries carry a kind (fixed) and an area (Service, Host OS, Hardware,
Storage, Network, Security, Monitoring). Settings → Timeline edits the
areas and each type's area and visibility; both are applied when the
timeline is read, so a change re-sorts past entries and a hidden type
comes back when shown. Kept forever unless a retention is set.
The first start fills the timeline from the last 90 days of records once,
skipping any already covered. A host rename carries its entries.
UI: Timeline in the sidebar (filter by host, area, kind; scrolls on), a
History card on each host page. MCP: get_timeline.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The host page gains a "Who's sending" panel, and the MCP a tool, that
put the host's outbound (or inbound) traffic down to its containers and
NFS mounts: read in shared time buckets, so the host's busiest bucket
says who was sending in it. A swarm service's tasks add up under its
name; what none of them accounts for is the host itself.
Also fixes CI 9488: per-container net_rx_bps / net_tx_bps join the
docker alert catalog, so a rule can catch a container flooding the
uplink.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Docker02's outbound traffic tops out the internet connection and Steward
couldn't say who sends it: a container's counters were kept only as a
running total, and NFS traffic belongs to no container.
- Docker ingest turns each running container's cumulative network counters
into net_rx_bps / net_tx_bps on "<host>/<container>" (net_rates), counting
from the stored row so a restart of Steward loses nothing; a container
new, recreated or restarted has no rate until its next snapshot.
- Agent 1.25.0 reads the bytes sent and received per NFS mount from
mountstats (already parsed for the delays) as tx_bps / rx_bps; Steward
stores them as nfs_tx_bps / nfs_rx_bps on "<host>:<mount>".
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The SNMP poll asks each device's sysName in the same GET and keeps it
(snmp_003_sys_name). A syslog source told apart by name finds its own
address from the SNMP device (reported or given name) or host of that
name, so its lines reach the SNMP device's and host's pages again; the
edit page gains Device address, which always wins (syslog_004).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Through ingress every device arrives from Docker's own address, so a
source there is keyed by the hostname its lines carry (new column
sender, unique with address). The set-up help no longer guesses the
address from the web hostname: it shows the plugin's send_to setting,
covers ingress and host mode, ASUSTOR ADM, and both syslog formats.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
On 2026-10-18 an update on VDNT-Docker02 restarted the Docker daemon six
seconds in, before the drain that came only with the reboot, and every
container on the node stopped. Docker's own packages restart the daemon from
their install scripts; needrestart's list-only mode doesn't stop that.
- The helper gains a `check` action (agent 1.24.0): a dry run of the update
for each package manager, whether Docker is among it, and the running
containers outside the swarm with their restart policies.
- A swarm node is drained before an update that upgrades Docker, else only
before a reboot, and at most once a job.
- After a drain the job settles 5 minutes before the update or reboot.
- Once the work is over, the containers that ran outside the swarm must be
seen running again within 10 minutes, or the job fails naming them.
- An older agent gets no check and its job runs as before.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The Syslog page says where to send (this server, its port and protocols) and
where the setting is on Synology, QNAP, TrueNAS, OpenWrt, UniFi and rsyslog,
open until a device sends. A sender whose address is on Steward's own
container network (Swarm ingress publishing hides the real sender) is
flagged, with the host-mode ports to publish instead. compose files and the
Dockerfile publish 5514/udp and /tcp; docs/plugins/syslog.md describes the
plugin and the deploy.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- /plugins/syslog/sources/<id>: the lines newest first, loading as you
scroll, filtered by severity and worse, app, text and time, with day
dividers and times to the second in the reader's zone.
- The Syslog page links each source and shows its lines in the last hour.
- A Syslog panel on a host's page and an SNMP device's: the newest lines of
the sources bound to it or at its address.
- MCP: list_syslog_sources and search_syslog (source, host, text, severity
and worse, app, range, paging by next_before).
- Shared: escape_like moves to steward/core/like.py and cut_middle to the
MCP shaping module, replacing four inline copies; |when takes seconds=.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The agent sent Docker's working set (usage less inactive_file), which still
counts recently used file cache. A file-heavy container with no limit read
191 GB while its processes held 2.2 GB, and the host's own figure, which
already leaves cache out, disagreed with it.
Agent 1.23.0 sends mem_used_bytes / mem_used_pct (anon + shmem, so
Postgres's shared buffers count; v1: rss + shmem) and mem_cache_bytes
apart. The new names keep the two meanings apart in history and in learned
normals: docker_013 renames the columns, clears the old values and adds
mem_cache_bytes. The Docker page, swarm by-node sums, Busiest Containers,
container detail, normals and the MCP tools all read the new figure. File
cache is shown only on the container's page and in get_container.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each sample carries each NFS mount's requests since the last one: how many
a second, their round trip and execute time (READ and WRITE on their own),
and how many were sent again or hit a major timeout. mountstats is the
kernel's own count and never touches the share, so it reads while one is
stuck. Steward stores them per mount and the slowest on the host as
nfs_exe_max_ms, for an "NFS slow" rule and the host page's NFS chart beside
mounts_stalled; each mount's bar says what a request took.
Agent 1.22.0 (which also carries the backlog order from 41fc807).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
On 2026-10-17 the NFS under Postgres stalled for ~15 minutes. Two things
went wrong for the readings that would have explained it:
- The SNMP poll stored before it polled again, so while the database hung
it stopped polling. It now collects on schedule and hands each round to a
Holding (steward/core/holding.py): one writer stores what is held, oldest
first, each reading at the time it was collected. Bounded at two hours of
polls, with a 30-minute deadline on a store; the device list read falls
back to the last one read after 10 s.
- "Mount not answering" (10 checks in a row) went pending on Docker02 and
Docker03 and never fired: the agent sent each post's backlog with its
newest, healthy sample, and that sample ended every post's run. Agent
1.22.0 queues the new sample behind the backlog, so Steward receives a
host's samples in the order they were taken.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A redeploy's new process ticks at once, so its once-a-minute samples never
broke for 90 s and get_anomaly showed none of the day's restarts. The first
self-monitor tick in a process now records restart_gap_secs (time since the
last process's last sample), and steward_gaps shows each one as a restart.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A Traefik that hasn't restarted keeps reporting the certificate it
replaced, so codex.fabledsword.com's old one (21 days left) was listed
beside its renewal (81 days) and would have raised an expiry warning on
2026-10-22. A reported certificate whose every name a later-expiring
one covers is now old, on the page, the widget, the overview and MCP.
list_certificates takes `limit` (default 100) with total/truncated, and
says how many served certificates no HTTP router uses (`unrouted`).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each source in an anomaly's `around` was cut to its oldest 40 with
nothing in the reply to say so; on a busy host the events nearest the
occurrence were the ones dropped. Each source now keeps the 40 nearest
the occurrence's start, and `around_left_out` counts the rest. Sources
return everything their bounded query read (Docker and the host agent
no longer cut at 40 and 20 themselves), so the count is honest.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
All 16 Traefik 4xx anomalies on 2026-10-15 came back with nothing
around them, though 15 were Steward's own restarts. A router has no
host, and every events source was read by host.
- Core: each break of more than 90 s in Steward's once-a-minute health
sample is a "steward" event beside every anomaly, placed or not.
- Plugins: an optional mcp_resource_events(db, source, resource, since,
until) for a resource with no host. Traefik's gives the container
events of the swarm service behind a router, on any host.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
One 502 among about 70 requests read steward@swarm critical at 1.4%:
past the 30-request minimum and far past the 0.1% cutoff, yet one
error. A new Settings → Thresholds field, "Fewest server errors to
fail" (traefik.threshold.min_5xx, default 3), must also be reached in
the 15-minute window; the 0.1% cutoff still catches a busy router.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Two 500s of about sixteen requests graded roundtable@swarm critical at
9.1%. The Settings → Thresholds minimum of requests in 15 minutes (30 by
default) now guards both shares; the 5xx share still keeps its fixed
cutoff whatever the router's history.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Through one address Steward read whichever replica answered, and with
host-mode ports behind a keepalived VIP that can be the replica serving
nothing: since the 18:43 restart every scrape landed on Docker03's idle
Traefik, which has no traefik_service_* series, so no router was stored.
A metrics_url host name that resolves to several addresses (a swarm
service's tasks.<service>, a Kubernetes headless Service, round-robin DNS)
is now scraped address by address, with the original Host header and TLS
name. Answers from one process are kept once; a replica that doesn't answer
is skipped with a warning. Requests, bytes and latency buckets are summed
over the replicas that have a baseline, so the sum's rise is the sum of each
one's own rise; certificates are listed once; Traefik's own figures add
connections and memory and keep the newest reload. A name with one address
is one scrape, as before.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
One metrics URL can reach several Traefik processes: a swarm service's
replicas behind the routing mesh or its VIP, a Kubernetes Service's pods,
a keepalived pair after failover, or the same Traefik after a restart.
Each keeps its own counters, so comparing one with another read the second
one's lifetime totals as new traffic: retired routers flashed 100% 502 and
an idle router showed 834 req/s whenever the scrape moved to the other
replica.
Each scrape is now keyed by Traefik's process_start_time_seconds and
compared only with the same process's previous scrape. A process seen for
the first time (or not in the last hour) sets its baseline and counts
nothing. Without a start time every scrape is one process, as before.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The snmp plugin registers list_snmp_devices and get_snmp_device; the
prometheus plugin registers list_prometheus_targets and
get_prometheus_target. Both are viewer-level, read what the poller and
scraper stored, and exist only while their plugin is enabled.
get_overview names a device or target whose last poll or scrape failed;
stale, pending and off are left out.
A device's community and v3 keys and a target's token are never
returned, and a password in a target URL is masked.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
get_network gives the WAN, clients, Wi-Fi networks, speed tests and devices
by status; list_devices, list_clients (everyone seen, or the newest poll's
top talkers), list_network_events, list_alarms and get_network_config
(networks, port forwards, firewall rules) read what the poller stored.
get_overview names the WAN down while the poll is current, a device offline
or degraded that the controller still reports, and active alarms.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
list_routers grades every router as the Traefik page does (the last 15
minutes, 5xx on its cutoff, 4xx and p95 against the router's normal) and
gives a troubled one its status codes, what they mean and the swarm
service behind it. get_router adds its history, codes, normal and
anomalies; list_certificates the served certificates and the routers they
cover; get_traffic the access log's clients, countries and routers.
get_overview names a router answering 5xx and a certificate near expiry.
The access-log aggregation moves into routes.summarize_access, shared by
the traffic page and get_traffic.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
get_anomaly gives each occurrence against what is usual at that hour, the
metric around the newest one with its normal, what else happened on the
resource's host around each occurrence (alert transitions, Ansible runs,
other anomalies there and a count elsewhere, plus plugins' own events
through a new mcp_events hook: the host agent's maintenance jobs and
downtime, Docker's container starts, deaths and OOM kills), and how earlier
anomalies on the same metric were judged.
get_metric_normal reads the learned slots and the level that would be
flagged. review_anomaly takes one id or several, all or nothing, with a
note. get_anomaly_settings and (admin) set_anomaly_settings read and change
the review window and each normal's floor, as Settings → Thresholds does.
metric_series.points takes an optional `until`; core.alerts gains
host_of_resource; floor_setting names its floor.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
list_backups, check_backup and make_backup are Settings → Backups for an
agent, admin only like the page. make_backup runs the page's job in the
background and is audited as "<user> via MCP". Restoring stays in the
browser. get_overview names a scheduled backup that failed, is two hours
past its slot, or has no bucket.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5834 "The host agent is always on, and the MCP sees a host's
maintenance", step 1 of milestone 520 "The MCP sees what Steward has
grown".
The host agent is a core system (operator, 2026-10-15: "it's very core
to the functionality of Steward as a whole and shouldn't be considered
something that can be toggled off"). ALWAYS_ON_PLUGINS in core settings
makes to_plugins_cfg report it enabled whatever its stored row says, so
it always loads and its MCP tools are always listed. Settings -> Plugins
shows it as "Built in: always on" with no switch, and a save keeps it
enabled. Its code and URLs stay under plugins/host_agent: every
installed agent posts to /plugins/host_agent/ingest.
Maintenance through the MCP:
- list_maintenance_plans: each plan's group, weekly slot, reboot policy,
next slot, and its latest run (hosts, how far it got, who it skipped
and why, how it ended).
- get_maintenance_job(host | job): a host's jobs, newest and running
first, or one job: state, packages updated, reboot, drained by, why it
failed, when its current state times out.
- run_maintenance(host, reboot="if_required" | "never"): admin, as the
host page's Update now; refuses a busy host or one with no helper;
audited "<user> via MCP".
- The overview names a host whose latest job in the last week failed or
timed out.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5748 "MCP can write, edit and test this instance's own playbooks
(steward-local)": two admin tools beside run_playbook, so an agent can
build automation for these hosts that isn't part of Steward itself.
- save_playbook(playbook, content, source="steward-local", replace=false)
goes through the editor's own validate_playbook_yaml and
write_playbook. A playbook that isn't a list of plays comes back with
the YAML error and its line, and nothing is written. An existing file
is overwritten only with replace=true. steward-builtin is refused
(Steward's playbooks change in its repository), as are git sources
(their repo owns them).
- After writing, `ansible-playbook --syntax-check` runs against a
throwaway localhost inventory (no host contacted, 60 s), so a module
name Ansible can't resolve shows before any run. The result rides the
reply; the file is kept either way.
- delete_playbook(playbook, source="steward-local").
- Both audit as ansible.playbook.created/updated/deleted, "<user> via
MCP", with the size and never the content.
The testing loop is the existing one: run_playbook defaults to check
mode, on one target, then get_ansible_run. The integration test does
that loop with the real ansible-playbook.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent 1.21.0. The hourly update check now covers the package managers
beyond apt and dnf (#5803 "The agent's update check reads pacman,
zypper and yum"):
- yum: `yum -C check-update`, `updateinfo list security`,
`needs-restarting -r`, as dnf.
- zypper: `--no-refresh list-updates` (`--dup` on Tumbleweed),
needed security patches from `list-patches --category security`,
and `zypper needs-rebooting`.
- pacman: `checkupdates` from pacman-contrib, which install-deps.sh now
installs. pacman has no security notion; a reboot is required once
the running kernel's /usr/lib/modules/<release> is gone.
The checks sit in one UPDATE_CHECKS table, and a failure names its
manager.
The root helper's update runs on the same hosts (#5804 "The scheduled
update runs on pacman, zypper and yum hosts"): yum upgrade, zypper
refresh + update (dup on Tumbleweed, with 102/103 counted as success),
pacman -Syu. Each step carries a label so a failure says which one,
and the reboot check reads zypper and the pacman kernel too.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5818 "First-run setup: Restore from a backup, by upload": beside creating
the first admin, setup offers "Or restore from a backup". Only while no
user exists; the upload is checked (a newer backup or one whose key can't
open its secrets is refused and not kept), then restored with no safety
backup (nothing to keep), and Steward restarts into the backup's users.
#5819 "Docs and CI": backups.md leads with restoring from Settings →
Backups and from setup; `steward restore --in-place` does the same-schema
restore from the shell, and CI's image check now uses it with a real
pg_restore: a changed setting comes back and the audit log stays.
Also exempts errors/restoring.html from the breadcrumb guard, as with
the other whole-app error pages: it replaces every page while a restore
runs, so there is nowhere to trail back to.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5823 "Check a backup: compare what it would restore with what is in
place, without restoring".
- steward/backup/check.py: pg_restore writes the backup's rows as COPY
text with no database involved. The rows in place go out through COPY in
the same form. Tables are compared on their primary keys, over the columns
both sides have, and grouped by area.
- Sealed values are opened with each side's own key and compared as
plaintext digests, so a secret sealed twice still matches. They are never
shown. A backup whose key can't open its own secrets fails the check.
- What Steward records as it runs (last-seen times, a schedule's last run,
backup history, scraped UniFi fields) is left out of the comparison:
contents.ACTIVITY, STATE_SETTINGS and HAND_SET_ONLY.
- Settings → Backups:
- a Check button on each sent backup, which fetches it from the bucket
(new s3.get_file);
- "Check a file" to upload one, with the size limit raised before the
CSRF check reads the form;
- a results page.
- Each backup Steward sends checks itself once written, and its history
row says how that went.
- New CLI command `steward check-backup FILE`. CI's image smoke step checks a
real backup against its source and expects a match.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5822 "Backup contents: settings and keys only". The operator's ruling
(System 180, 2026-10-15) overturns 2026-10-12's "Config + audit + alerts":
a backup is for system recovery and carries what would otherwise have to
be set up again, with no logs or history.
- contents.py: audit_events, alert_events, alert_states, metric_anomalies,
metric_anomaly_spans and user_invites move to EXCLUDED.
- Alert state is no longer restored, and evaluation skips a rule with no
state row. ensure_rule_states (alerts/service.py) gives every
single-resource rule lacking one an inactive row; alert_reconcile runs
it each minute.
- Docs, the Backups page and the CLI help say what a backup holds now.
CI's image check also finds the audit log empty after a restore.
- tests/integration/test_backup_restore_fixes.py is removed: its case (an
alert event naming a left-out Ansible run) no longer arises. The SET NULL
fix-up is covered by a unit test on synthetic tables.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
smartctl reports every disk behind a PERC/MegaRAID controller as /dev/bus/0
with type megaraid,N; the helper named a drive by its path alone, so all of
VDNT-Docker02's and Docker03's disks read as one drive "0", each overwriting
the last. The slot is now part of the name (bus0-megaraid3), and Steward gives
a repeated name an ordinal so two drives never share a series. Agent 1.20.1.
fix(ansible): Steward stopping mid-run logs no traceback (#5749)
The run's done-callback called exception() on a cancelled task, which raises;
it now logs a run cut short by shutdown at INFO. A finished run's in-memory
state is dropped by a timer rather than a sleeping task, which shutdown found
still pending.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The root helper gains one read-only action, `drives`: it finds the drives
itself (smartctl --scan-open), reads each as JSON, and writes a summary the
agent carries on every sample (agent 1.20.0). Steward judges the figures in
drives.py (failing / worn / watch / ok / no SMART / unreadable), records them
per drive and as host roll-ups for covering rules, keeps the newest report on
the registration and lists it on the host page.
install-deps.sh installs smartmontools with apt, dnf, yum, zypper, pacman or
apk, run by install.sh and every agent playbook; one it can't install is
reported, never fatal.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
On 2026-10-14 nine swarm services sat at 0/1 and nothing fired. From each
manager's swarm report the docker plugin now records three figures on
resource "swarm": swarm_services_short, swarm_nodes_down and
swarm_managers_unreachable. Drained nodes are left out of the last two,
since a drain is deliberate. The figures are recorded at most once every
30 s whichever manager reported, and are offered to alert rules.
Agent 1.19.0 sends each manager's ManagerStatus.Reachability. Older agents
fall back to the node's status.
A maintenance job on a swarm node also keeps a window over docker/swarm,
so its drain and reboot hold those alerts until the job ends.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
temp_c_max was recorded on every ingest, but host_agent didn't offer it
to rules, so "host running hot" couldn't be watched. The catalog guard
didn't see the row(...) helper ingest builds its rows with. Now offered:
temp_c_max, temp_c per sensor, and network and disk throughput per host,
interface and disk. A new guard holds every literal row(...) name to the
catalog, except NOT_ALERTABLE, where each exclusion is given a reason.
temp_c_max reads in °C.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A board's spare thermistor inputs report whatever an open input reads, and
the value rule from #5759 only catches the impossible ones. Once an hour,
sensors.refresh judges each sensor against the host's CPU (Package id N,
Tctl/Tdie/TccdN, PECI Agent N) over a week of hourly averages. A sensor is
ruled out if it falls as the CPU warms (r <= -0.5) or holds within 0.5 °C
while the CPU moved 10 °C or more. A judgment needs at least 72 shared hours.
The host temperature leaves judged sensors out at ingest. The host page
lists them under "Not counted", with each reason given once.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5712 "Each router problem says what it means, whether it's real, and what
to do". A problem card now names its kind and reads its codes:
- Server errors: 500 means the app is failing; 502/503/504 means Traefik
couldn't reach it or it timed out.
- Client errors: 404 scanners, a 401/403 login wall, 429 rate limiting, or
499 clients giving up on a slow app.
- Slow: p95 against p50.
Each card says whether the problem is likely real, and ends with one next
step. A fault links to the swarm service behind the router, matched by
name, and traffic links to Settings → Thresholds. The wording lives in
plugins/traefik/meaning.py.
The plugin manager now hands out a loaded plugin's submodule
(loaded_plugin_submodule), so Traefik and host_agent reach Docker's
models the same way (#5775).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5768 "A maintenance cadence: a group's updates and reboots, one host at a
time, on a schedule". A plan names an inventory group, a weekday and time
in a time zone, a window and a reboot policy (only when required by
default). At its slot a run takes the group's hosts in order and gives each
a #5767 maintenance job. The next host starts only when the last is back in
service: its agent is posting and, for a swarm node, the node is active and
every replicated service is at its replicas. A failure stops the run, and
no host is started once the window has passed. A host's latest failure
shows as host_agent/<host>/maintenance_failed, for an ordinary alert rule.
The page is Hosts → Maintenance.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
