A new plugin reads Home Assistant with a read-only token every 5 minutes and
shows which Core, OS and add-on updates are waiting and how old the last backup
is. Install takes a backup through a local-only webhook, confirms Home
Assistant's last-backup time moved, installs, and follows the update to its end,
through Home Assistant's and Steward's own restarts. One runs at a time. Each
ends as one timeline entry.
Steward holds nothing that can act on the house: the webhook's automation can
only take a backup or install an allowed update (System 231's ruling). A refused
token stops polling, as Home Assistant bans an address after a few refusals.
Milestone 576 steps 1 to 5.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#6718 "A swarm container's metrics, normal and anomalies are keyed by its task
name, so they start over whenever the task moves". The move found 391,091
hourly-only task names against 51 services and about 400 keys with raw samples.
Log the services with the most keys and a few names, to read before those keys
start moving.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#6718 "A swarm container's metrics, normal and anomalies are keyed by its task
name, so they start over whenever the task moves". Live, the move took
VDNT-Docker02's I/O pressure (psi_io_full_avg60) from about 1% to about 23%,
and passes of 5000 raw samples outran their 15 s slot. Each moved sample
rewrites its row and every index on it.
Batches are 1000 samples, and after a pass the task waits four times as long
as it took, so it holds the disk for about a fifth of the time and the move
slows if the disk does.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#6718 "A swarm container's metrics, normal and anomalies are keyed by its task
name, so they start over whenever the task moves". Live, the move found 391,440
keys, about 2,500 of them with raw samples, and took one key per 15 s pass:
about 68 days, in sorted order, so recent history could wait months while
charts and normals read thin.
A pass now runs for up to 2 s, and keys with raw samples go first. Old hourly
keys take a few statements each. A key with no anomalies no longer rewrites the
service's timeline entries. The start logs how many keys each table holds, as
that count is far higher than expected and not explained yet.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#6727 "A metric that bursts now and then opens anomalies at hours whose usual
peak reads 0". usual_peak is the p90 of a slot's hourly max, so a metric that
bursts in under about 1 hour in 10 reads 0 there however often it bursts across
the day. #6723 "An anomaly opens on a burst of minutes inside the metric's usual
peak, so most rows aren't outside the norm" therefore still opened on those.
usual_anywhere is the median, across the hours of the day, of each hour's
highest peak. Height is now judged against the greater of it and the slot's
usual peak. Bursts at any hour lift the median; a job at a time of its own
raises a few hours and leaves it low, so the same level at another time of day
is still an anomaly, as System 182 rules. A held level still opens.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#6718 "A swarm container's metrics, normal and anomalies are keyed by its task
name, so they start over whenever the task moves". The operator: "the metrics
should be keys to the service name not the task name... as that's what's
consistent between updates."
CPU, memory, restart count and health now go under "swarm/<service>", so a
normal, its anomalies and an alert rule carry across a redeploy or a move
between nodes. Network rates stay "<host>/<service>", the task id gone, as they
answer who on a host is sending. Tasks of a multi-task service share the key,
so its level reads as an average across them (of 51 live services, only
authentik-worker and portainer_agent run more than one task).
History under task names moves in a background task, 5000 raw samples every
15 s, not a migration: the raw table holds about 2M such rows and the app
would wait on it at boot. Hourly rows combine where two tasks share an hour,
and rows still to review merge to one per metric.
A service's anomalies no longer reach a host's page, as it has no one host;
get_anomaly reads what happened around one from the service's own events.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Patterns could only be added on the Syslog page, so an agent asked to map
them on the NASes couldn't. list_syslog_patterns gives each pattern's metric
and the ready-made ones not added; add_syslog_preset (admin, audited) adds one.
The key-picking helper moves beside the patterns for both callers.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
"Host running hot" read the host's hottest sensor and didn't name it, so the
operator couldn't tell what hit 100 °C on VDNT-Util01. The alert email now
names the sensor and its reading, through the same names_behind a count uses.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A rule compared a reading with a fixed number, so Synology Disk 1's bad
sectors going from 3,160 to 3,392 could say nothing: a count that only grows
is soon past any line. "rose by" compares each reading with the one before it,
kept on the alert's state and only for such a rule. A fall (a replaced disk)
doesn't fire. On the rule form, through MCP, and worded in emails, the alert
sentence, history and the timeline.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Rows on the live instance were mostly bursts of a few minutes: 28 of 52
peaked within the metric's own usual peak, and 42 of 52 latest occurrences
lasted 30 minutes or less, which is what a burst shorter than the 15-minute
average looks like once its span is back-dated. Nothing asked how high or how
long, as the System 182 ruling says a normal records.
A departure now also has to be past the slot's usual peak, or held: the
median of the last hour over the same line. Those spans were also keeping the
burstiest hours out of learning, lowering the usual lines further.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Docker's ingress address depends on the node Steward's task is on, and sources
were keyed by (address, sender). After a task move the same device arrived under
a new key and became a new "offered" row, so its lines were counted, not kept.
A named sender is now one row whatever address it last came from, enforced by a
partial unique index. Migration syslog_005 folds rows sharing a name into the
added one and moves their lines and patterns across.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A pasted drop list whose line breaks had become spaces was stored as one
pattern that matched nothing, so Home Assistant's two 526-series
bookkeeping families filled the 1,000-series cap. No metric name holds
whitespace, so patterns now split on spaces and tabs as well as lines
and commas.
Home Assistant names its Unix times *_time_seconds (e.g.
homeassistant_last_updated_time_seconds), which read as "20751d 21h".
is_moment() takes both suffixes, for the value shown and the chart axis.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Monitors, schedules and maintenance plans already pause with a
Pause/Resume button and say "Paused" where they're listed. Prometheus
targets and syslog patterns only had a tick box inside the edit form,
so a target refused by its endpoint couldn't be stopped from its page.
- Both get a Pause/Resume button (target page and list; patterns list)
posting to /toggle, recorded in the audit log, and a Paused badge.
- The forms lose the tick box. A new one starts running; an edit keeps
whether it is paused.
- A target's "off" state is now "paused", in its pages and MCP tools.
A resumed target reads "not scraped yet" until its next scrape, and its
counter rates start over.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
"swarm services short on swarm is 6." becomes "6 swarm services are
short of their replicas.", with a Which line naming each service and how
many of its tasks run. The swarm figures get labels, and the counting
figures (swarm, mounts, drives, updates) read as sentences in the email,
its subject and the Alerts page. A plugin names what its count counts
through core.alerts.names_behind; the Docker plugin does for the three
swarm figures, from the stored swarm. Webhooks get alert.which too.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The first applied run lost VDNT-Docker03 29 images (10.4 GB) before its
script ran. Its `docker system prune` removes dangling images, and Docker
counts an image with no tag as dangling: every image a swarm task pulled by
digest. Docker03 was left holding only its own tasks' images, as under the
old prune. Nothing went down, since a running container's image can't be
removed.
- Step 1 now prunes containers, networks, build cache and (prune_volumes)
volumes one by one. Every image decision is left to the script, which
keeps the images services are pinned to.
- A test guards against any image-pruning command in the playbook.
- The playbook moves to maintenance/swarm_prune/, with its script in that
folder's files/, so a helper's name can't collide with another
playbook's (the operator's direction for playbooks with helpers).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`docker image ls` on Docker 29 leaves out images with no tag, which is every
image a swarm task pulled by digest: the check run on VDNT-Docker03 saw 4 of
its 58. Missing them only kept them, but superseded digests there would
never have gone. The list now comes from GET /images/json on the socket.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The nightly prune ran `docker system prune -a`, which deletes every image no
local container uses, so each swarm node kept only its own tasks' images.
When the other node failed or was drained, its services had to pull first:
on 2026-10-25 Traefik took 11.5 min to start on Docker03.
maintenance/swarm_prune.yml (built in) prunes stopped containers, unused
networks, dangling images and build cache, then removes only the images that
no container uses and no swarm service is pinned to, wherever that service
runs. Old digests left by redeploys still go.
- A container's image is kept by its id and by the reference it was made
from, since on the containerd store the id alone hasn't held a tag.
- A node that can't list the swarm's services removes no images and fails
rather than pruning blind.
- Check mode lists what would go.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each weekly run drains Docker02 then Docker03, and the swarm never moves a
task back by itself, so every run ended with nearly every service on one
node. On 2026-10-25 the operator had that spread out by hand.
- Agent 1.26.0 reports each swarm service's pinned digest, and every
repo@sha256 its host holds (from /system/df, every 10 min).
- Once a run's last host is back in service, Steward picks services on the
fullest node to move to the emptiest until they are within one task. It
takes services with several replicas on one node first. A service moves only
when the emptier node already holds its exact image. One the plan keeps in
place, one below its replicas, and one with no pinned digest stay put, and
the run says why.
- A swarm manager's helper carries it out through a new action, rebalance,
with the service names (validated, at most 48). It force-updates each in
turn with the spec unchanged and no registry lookup, skips any a
constraint places, and gives each 5 min to settle.
- The run ends done with what moved, or failed if a move didn't settle or
the manager didn't answer in 60 min. A docker-scoped window covers the
restarts, and the step goes on the timeline.
- Plans gain "Spread swarm services back out" (on by default) and "Keep in
place", on the create form and in an "After the run" form on each plan.
list_maintenance_plans shows both and the latest run's spread.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
On 2026-10-25 the 09:00 job on VDNT-Docker02 drained the node, which ran
Traefik and Steward. The agent's result reaches Steward through Traefik,
and Traefik took 11.5 minutes to start on Docker03 (its image had to be
pulled). The job gave up at the 10-minute drain deadline and finished
timed_out where it stood, so the node stayed drained for hours.
- A job that fails or times out while its node may be drained (the drain
was handed out, or went through and wasn't undone) goes to undraining
first. The outcome is kept in the new column ends_as, and the job
finishes as that once the node is active again. If the undrain goes
wrong too, detail says both.
- The drain deadline is 30 minutes instead of 10, so a result held up while
the proxy moves still lands.
- Migration host_agent_012_ends_as. get_maintenance_job returns ends_as.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A ticket records a known problem: title, area, hosts, subject, description,
links to alert rules and anomalies, and a log of comments, edits, resolving
and reopening. Opening, resolving (with the resolution as the note) and
reopening go on the timeline; every step is audited, and the audit rows stay
off the timeline.
An alert or anomaly an open ticket covers (by link, by subject, or a
host-wide ticket on the host itself) shows a Known issue badge, and offers
"Open a ticket" prefilled from its rule or anomaly when none does. Hosts,
monitors and containers get a Known issues card.
MCP: list_tickets, get_ticket (viewer); open_ticket, update_ticket,
comment_ticket, resolve_ticket, reopen_ticket (operator). list_alerts and
list_anomalies carry known_issue; get_overview, get_host, get_monitor and
get_container carry open_tickets. Also update_change and delete_change for
changes recorded by hand (author or admin).
Tickets are left out of backups, like the timeline (2026-10-15 ruling).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The change-form test set the zone on the session, which the auth middleware
resets from the account on every request. The password reset test asserted
BST, which stopped being London's zone name on 25 Oct.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A change Steward can't see happen (a config edited, a disk swapped) is
recorded by whoever made it: what was done, its area, when (in the reader's
own zone), the hosts, what it was done to and a detail. Its author or an
admin edits or deletes it; recording, editing and deleting are audited and
those audit rows stay off the timeline. log_change records one as
"<user> via MCP", editable by the token's user.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The host agent writes each maintenance job once it ends, and each stretch a
host was away (a reboot, a crash, lost contact, the agent restarting), kept up
to date until it is back. Docker judges each snapshot: a unit on a new image,
or a container recreated, is a deploy; deaths past the Thresholds crash-loop
count are one crash-loop entry that later deaths extend; single restarts are
recorded but hidden. An anomaly has one entry whose type follows its review:
only those judged real are shown by default.
A source whose record changes keeps one entry for it (record_or_update).
Plugins register the records their entries point to, so an aged-out job or
downtime reads "Details no longer kept". The first fill runs once per source
(timeline.filled), the host agent's jobs and downtime included.
get_anomaly's "what else was going on" reads the timeline for the host or
resource instead of querying alerts, runs, maintenance, downtime and container
deaths itself; Docker's own source keeps only starts, stops and health
changes. A new kind, Observed, is for what Steward saw happen with no rule or
person behind it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A stored log, timeline_entries, with one row per thing that happened. Each
source writes its row in the same transaction as its own record, so the
history outlives the 30/90/365-day retention of the tables it points to:
- alert events (core/alerts._add_event): fired, resolved, acknowledged,
went silent, readings back;
- finished Ansible runs (executor, and the startup sweep of interrupted
runs): changed something, changed nothing, failed, dry run;
- audit rows (core/audit.log_audit): Steward's settings, users and access,
sign-ins, backups. Actions another writer records are left to it.
Entries carry a kind (fixed) and an area (Service, Host OS, Hardware,
Storage, Network, Security, Monitoring). Settings → Timeline edits the
areas and each type's area and visibility; both are applied when the
timeline is read, so a change re-sorts past entries and a hidden type
comes back when shown. Kept forever unless a retention is set.
The first start fills the timeline from the last 90 days of records once,
skipping any already covered. A host rename carries its entries.
UI: Timeline in the sidebar (filter by host, area, kind; scrolls on), a
History card on each host page. MCP: get_timeline.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
18032a4 re-checked a "not back in service" job every minute with no end,
so any later swarm wobble on the host (radarr's failed task during the
downloads move, say) flipped maintenance_failed back to 1: five false
firings and ten emails for Docker02 overnight on 10-19. And the overview's
finding never learned the rule at all, so it still said Docker02's
maintenance failed three days on.
The job now keeps came_back_at, set the first time its host is found back.
maintenance.still_failed(job) is the one answer: the figure, the overview,
the cadence page's run rows and the host panel all ask it. A failed update
or a timed-out job still flags its host until a job succeeds.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Three 404s on a quiet router read 7.5% and were raised as an anomaly; most of
the 4xx anomalies on review stood for 3 to 7 errors in 15 minutes. A new
Settings → Thresholds minimum, traefik.threshold.min_4xx (default 20, more
than one a minute held for the window), applies to both the router's grade
on its page and to anomaly detection, as min_5xx does for 5xx.
learn_normal takes `fewest`: for a share gated on a per-second rate, the
fewest events (rate × window × share) it must stand for to be judged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
response_ms was the whole `ping -c 1` process, or an open_connection awaited
on the event loop: about 15 ms on a LAN whose round trip is under 1, and every
host read slow together whenever Steward was busy. A ping now reports the
round trip ping itself printed (the elapsed time only if it printed none);
a TCP connect is timed in a thread, the name resolved first and not counted.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
get_host_senders on VDNT-Docker02 ran past 120 s: a prefix LIKE on
plugin_metrics.resource_name can't use its index (the column's collation
isn't C), so each read went through the whole table, and the host page's
Who's sending panel repeats it on every refresh. The containers and NFS
mounts under a host are now named from plugin_metric_latest, and their
series read by resource IN (...) on the series index.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The host page gains a "Who's sending" panel, and the MCP a tool, that
put the host's outbound (or inbound) traffic down to its containers and
NFS mounts: read in shared time buckets, so the host's busiest bucket
says who was sending in it. A swarm service's tasks add up under its
name; what none of them accounts for is the host itself.
Also fixes CI 9488: per-container net_rx_bps / net_tx_bps join the
docker alert catalog, so a rule can catch a container flooding the
uplink.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Docker02's outbound traffic tops out the internet connection and Steward
couldn't say who sends it: a container's counters were kept only as a
running total, and NFS traffic belongs to no container.
- Docker ingest turns each running container's cumulative network counters
into net_rx_bps / net_tx_bps on "<host>/<container>" (net_rates), counting
from the stored row so a restart of Steward loses nothing; a container
new, recreated or restarted has no rate until its next snapshot.
- Agent 1.25.0 reads the bytes sent and received per NFS mount from
mountstats (already parsed for the delays) as tx_bps / rx_bps; Steward
stores them as nfs_tx_bps / nfs_rx_bps on "<host>:<mount>".
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Docker02's Sunday job updated and rebooted, then failed only because
portal_app was short of its replica past the 30-minute settle. Its
maintenance_failed figure stayed 1, and the alert kept firing, until a
later job succeeds. A job failed for not coming back into service is
checked again each minute and reads 0 once the host is back; a failed
or timed-out update still reads 1 until a job succeeds.
Also: compose.deploy.yml's syslog comment still said devices are told
apart by address under Swarm (stale since #6096).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The SNMP poll asks each device's sysName in the same GET and keeps it
(snmp_003_sys_name). A syslog source told apart by name finds its own
address from the SNMP device (reported or given name) or host of that
name, so its lines reach the SNMP device's and host's pages again; the
edit page gains Device address, which always wins (syslog_004).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Through ingress every device arrives from Docker's own address, so a
source there is keyed by the hostname its lines carry (new column
sender, unique with address). The set-up help no longer guesses the
address from the web hostname: it shows the plugin's send_to setting,
covers ingress and host mode, ASUSTOR ADM, and both syslog formats.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
On 2026-10-18 an update on VDNT-Docker02 restarted the Docker daemon six
seconds in, before the drain that came only with the reboot, and every
container on the node stopped. Docker's own packages restart the daemon from
their install scripts; needrestart's list-only mode doesn't stop that.
- The helper gains a `check` action (agent 1.24.0): a dry run of the update
for each package manager, whether Docker is among it, and the running
containers outside the swarm with their restart policies.
- A swarm node is drained before an update that upgrades Docker, else only
before a reboot, and at most once a job.
- After a drain the job settles 5 minutes before the update or reboot.
- Once the work is over, the containers that ran outside the swarm must be
seen running again within 10 minutes, or the job fails naming them.
- An older agent gets no check and its job runs as before.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
During Docker02's maintenance reboot its Agent offline alert fired muted by
the job's window, then resolved just after the window closed: the resolution
was mailed for a firing nobody had been told of. A resolution whose firing
was held and never released is now recorded with a note, not sent.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The Syslog page says where to send (this server, its port and protocols) and
where the setting is on Synology, QNAP, TrueNAS, OpenWrt, UniFi and rsyslog,
open until a device sends. A sender whose address is on Steward's own
container network (Swarm ingress publishing hides the real sender) is
flagged, with the host-mode ports to publish instead. compose files and the
Dockerfile publish 5514/udp and /tcp; docs/plugins/syslog.md describes the
plugin and the deploy.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
get_anomaly's circumstances gain the host's or SNMP device's syslog
warnings and worse (mcp_events / mcp_resource_events). An anomaly row opens
"Syslog around it", read only when opened. A shared Chart.js plugin
(chart-marks.js) draws a dashed line at each syslog error on the host
metrics charts and an SNMP device's charts, with a line saying how many.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A pattern (phrases, a severity, a source, a window) is counted every minute
into plugin_metrics as lines_matching_<key> per added source, zero included,
so an ordinary alert rule fires on a match and clears once the window passes.
Each count reads only the lines stored since the last, by line id; a new or
changed pattern reads its window once.
Patterns page with ready-made storage-trouble presets and "Alert on it",
which opens the rule form aimed at the count. Source names are kept unique,
and a rename moves its counts and the rules on them.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- /plugins/syslog/sources/<id>: the lines newest first, loading as you
scroll, filtered by severity and worse, app, text and time, with day
dividers and times to the second in the reader's zone.
- The Syslog page links each source and shows its lines in the last hour.
- A Syslog panel on a host's page and an SNMP device's: the newest lines of
the sources bound to it or at its address.
- MCP: list_syslog_sources and search_syslog (source, host, text, severity
and worse, app, range, paging by next_before).
- Shared: escape_like moves to steward/core/like.py and cut_middle to the
MCP shaping module, replacing four inline copies; |when takes seconds=.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
