18032a4 re-checked a "not back in service" job every minute with no end,
so any later swarm wobble on the host (radarr's failed task during the
downloads move, say) flipped maintenance_failed back to 1: five false
firings and ten emails for Docker02 overnight on 10-19. And the overview's
finding never learned the rule at all, so it still said Docker02's
maintenance failed three days on.
The job now keeps came_back_at, set the first time its host is found back.
maintenance.still_failed(job) is the one answer: the figure, the overview,
the cadence page's run rows and the host panel all ask it. A failed update
or a timed-out job still flags its host until a job succeeds.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Three 404s on a quiet router read 7.5% and were raised as an anomaly; most of
the 4xx anomalies on review stood for 3 to 7 errors in 15 minutes. A new
Settings → Thresholds minimum, traefik.threshold.min_4xx (default 20, more
than one a minute held for the window), applies to both the router's grade
on its page and to anomaly detection, as min_5xx does for 5xx.
learn_normal takes `fewest`: for a share gated on a per-second rate, the
fewest events (rate × window × share) it must stand for to be judged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
response_ms was the whole `ping -c 1` process, or an open_connection awaited
on the event loop: about 15 ms on a LAN whose round trip is under 1, and every
host read slow together whenever Steward was busy. A ping now reports the
round trip ping itself printed (the elapsed time only if it printed none);
a TCP connect is timed in a thread, the name resolved first and not counted.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The host page gains a "Who's sending" panel, and the MCP a tool, that
put the host's outbound (or inbound) traffic down to its containers and
NFS mounts: read in shared time buckets, so the host's busiest bucket
says who was sending in it. A swarm service's tasks add up under its
name; what none of them accounts for is the host itself.
Also fixes CI 9488: per-container net_rx_bps / net_tx_bps join the
docker alert catalog, so a rule can catch a container flooding the
uplink.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Docker02's outbound traffic tops out the internet connection and Steward
couldn't say who sends it: a container's counters were kept only as a
running total, and NFS traffic belongs to no container.
- Docker ingest turns each running container's cumulative network counters
into net_rx_bps / net_tx_bps on "<host>/<container>" (net_rates), counting
from the stored row so a restart of Steward loses nothing; a container
new, recreated or restarted has no rate until its next snapshot.
- Agent 1.25.0 reads the bytes sent and received per NFS mount from
mountstats (already parsed for the delays) as tx_bps / rx_bps; Steward
stores them as nfs_tx_bps / nfs_rx_bps on "<host>:<mount>".
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Docker02's Sunday job updated and rebooted, then failed only because
portal_app was short of its replica past the 30-minute settle. Its
maintenance_failed figure stayed 1, and the alert kept firing, until a
later job succeeds. A job failed for not coming back into service is
checked again each minute and reads 0 once the host is back; a failed
or timed-out update still reads 1 until a job succeeds.
Also: compose.deploy.yml's syslog comment still said devices are told
apart by address under Swarm (stale since #6096).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
During Docker02's maintenance reboot its Agent offline alert fired muted by
the job's window, then resolved just after the window closed: the resolution
was mailed for a firing nobody had been told of. A resolution whose firing
was held and never released is now recorded with a note, not sent.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
get_anomaly's circumstances gain the host's or SNMP device's syslog
warnings and worse (mcp_events / mcp_resource_events). An anomaly row opens
"Syslog around it", read only when opened. A shared Chart.js plugin
(chart-marks.js) draws a dashed line at each syslog error on the host
metrics charts and an SNMP device's charts, with a line saying how many.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A pattern (phrases, a severity, a source, a window) is counted every minute
into plugin_metrics as lines_matching_<key> per added source, zero included,
so an ordinary alert rule fires on a match and clears once the window passes.
Each count reads only the lines stored since the last, by line id; a new or
changed pattern reads its window once.
Patterns page with ready-made storage-trouble presets and "Alert on it",
which opens the rule form aimed at the count. Source names are kept unique,
and a rename moves its counts and the rules on them.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- /plugins/syslog/sources/<id>: the lines newest first, loading as you
scroll, filtered by severity and worse, app, text and time, with day
dividers and times to the second in the reader's zone.
- The Syslog page links each source and shows its lines in the last hour.
- A Syslog panel on a host's page and an SNMP device's: the newest lines of
the sources bound to it or at its address.
- MCP: list_syslog_sources and search_syslog (source, host, text, severity
and worse, app, range, paging by next_before).
- Shared: escape_like moves to steward/core/like.py and cut_middle to the
MCP shaping module, replacing four inline copies; |when takes seconds=.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Audit: syslog.source_* worded, and a Syslog filter area.
- Backups keep syslog_sources (added, named, ignored; what it last sent
isn't compared) and leave syslog_lines out, as logs.
- The integration test reaches the plugin as loaded (#5775), so its
tables aren't defined twice; ruff E741.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- plugins/syslog: RFC 5424 and RFC 3164 parsing, RFC 6587 TCP framing.
- Lines are queued as they arrive and stored every 5 s through a Holding,
so a storage stall delays them rather than losing them.
- A new sender is an offered source, counted and not kept, until an admin
adds it on the Syslog page (name, host); it can be ignored instead.
- The listener follows the plugin's settings (port, UDP, TCP), rebinding
within seconds of a save; a port it can't open is said on the page.
- Hourly ring per source: age window and byte cap from Settings.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Saving an enabled plugin threw away hot_reload_plugin's (ok, message), so a
plugin that failed to load live read "Enabled, not loaded" with nothing
more. Now:
- a live-load failure is recorded with its reason, as a boot failure is, and
logged; a later success clears it;
- any save of an enabled plugin that isn't running tries to start it, not
only the off-to-on toggle;
- enabling one already running is a success, not a failure;
- the plugin's page offers "Restart to load" while it is enabled and not
running.
Quart never marks the app as having served its first request, so
register_blueprint after startup is allowed: hot-loading is not refused
wholesale, as #6087 suspected.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
main's z-index makes it a stacking context, so the fixed drawer (z-index 60)
could never rise over the sticky sidebar (z-index 20): its title, group
labels and first column of widgets sat underneath. Above 900px the drawer
now starts at left:220px; below, the sidebar is off-canvas and it keeps the
full width.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The agent sent Docker's working set (usage less inactive_file), which still
counts recently used file cache. A file-heavy container with no limit read
191 GB while its processes held 2.2 GB, and the host's own figure, which
already leaves cache out, disagreed with it.
Agent 1.23.0 sends mem_used_bytes / mem_used_pct (anon + shmem, so
Postgres's shared buffers count; v1: rss + shmem) and mem_cache_bytes
apart. The new names keep the two meanings apart in history and in learned
normals: docker_013 renames the columns, clears the old values and adds
mem_cache_bytes. The Docker page, swarm by-node sums, Busiest Containers,
container detail, normals and the MCP tools all read the new figure. File
cache is shown only on the container's page and in get_container.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each sample carries each NFS mount's requests since the last one: how many
a second, their round trip and execute time (READ and WRITE on their own),
and how many were sent again or hit a major timeout. mountstats is the
kernel's own count and never touches the share, so it reads while one is
stuck. Steward stores them per mount and the slowest on the host as
nfs_exe_max_ms, for an "NFS slow" rule and the host page's NFS chart beside
mounts_stalled; each mount's bar says what a request took.
Agent 1.22.0 (which also carries the backlog order from 41fc807).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The same Holding as the SNMP poll, for the rest of Steward's own
collectors:
- Monitors: the list is read within 10 s or the last one read stands in;
a start the database doesn't take is kept in memory, so a monitor isn't
checked twice; each result is held and stored at the time it was
checked. A stall shows on the monitors as the checks it saw, not a gap.
- UniFi and Traefik: each poll's writes are held (hold_write) and run
oldest first in one transaction; every reading is stamped with its scrape.
- A held round the database refuses (as opposed to can't be reached) is
stored alone and let go, so it can't hold up every round after it.
The Prometheus scrape is left as it is: its one target, Fabled Archives,
keeps its own database on the same NFS.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
On 2026-10-17 the NFS under Postgres stalled for ~15 minutes. Two things
went wrong for the readings that would have explained it:
- The SNMP poll stored before it polled again, so while the database hung
it stopped polling. It now collects on schedule and hands each round to a
Holding (steward/core/holding.py): one writer stores what is held, oldest
first, each reading at the time it was collected. Bounded at two hours of
polls, with a 30-minute deadline on a store; the device list read falls
back to the last one read after 10 s.
- "Mount not answering" (10 checks in a row) went pending on Docker02 and
Docker03 and never fired: the agent sent each post's backlog with its
newest, healthy sample, and that sample ended every post's run. Agent
1.22.0 queues the new sample behind the backlog, so Steward receives a
host's samples in the order they were taken.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A redeploy's new process ticks at once, so its once-a-minute samples never
broke for 90 s and get_anomaly showed none of the day's restarts. The first
self-monitor tick in a process now records restart_gap_secs (time since the
last process's last sample), and steward_gaps shows each one as a restart.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each source in an anomaly's `around` was cut to its oldest 40 with
nothing in the reply to say so; on a busy host the events nearest the
occurrence were the ones dropped. Each source now keeps the 40 nearest
the occurrence's start, and `around_left_out` counts the rest. Sources
return everything their bounded query read (Docker and the host agent
no longer cut at 40 and 20 themselves), so the count is honest.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
All 16 Traefik 4xx anomalies on 2026-10-15 came back with nothing
around them, though 15 were Steward's own restarts. A router has no
host, and every events source was read by host.
- Core: each break of more than 90 s in Steward's once-a-minute health
sample is a "steward" event beside every anomaly, placed or not.
- Plugins: an optional mcp_resource_events(db, source, resource, since,
until) for a resource with no host. Traefik's gives the container
events of the swarm service behind a router, on any host.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
One 502 among about 70 requests read steward@swarm critical at 1.4%:
past the 30-request minimum and far past the 0.1% cutoff, yet one
error. A new Settings → Thresholds field, "Fewest server errors to
fail" (traefik.threshold.min_5xx, default 3), must also be reached in
the 15-minute window; the 0.1% cutoff still catches a busy router.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Two 500s of about sixteen requests graded roundtable@swarm critical at
9.1%. The Settings → Thresholds minimum of requests in 15 minutes (30 by
default) now guards both shares; the 5xx share still keeps its fixed
cutoff whatever the router's history.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
still_failing was handed the query's one-pass result. The first failed
run's any() consumed it, so a second failure in the window found no
successes left and was reported though a later run had put it right
(ci-runner-volumes came back once keepalived-survey also had a failure).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
get_overview listed every failed or interrupted run of the last 24 hours,
so a dry run that failed, was fixed and re-run green kept warning for a
day. A failure now drops out once a later success of the same playbook,
source and hosts is at least as real: an applied success clears either
kind of failure, a dry-run success only a dry-run failure.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
get_anomaly gives each occurrence against what is usual at that hour, the
metric around the newest one with its normal, what else happened on the
resource's host around each occurrence (alert transitions, Ansible runs,
other anomalies there and a count elsewhere, plus plugins' own events
through a new mcp_events hook: the host agent's maintenance jobs and
downtime, Docker's container starts, deaths and OOM kills), and how earlier
anomalies on the same metric were judged.
get_metric_normal reads the learned slots and the level that would be
flagged. review_anomaly takes one id or several, all or nothing, with a
note. get_anomaly_settings and (admin) set_anomaly_settings read and change
the review window and each normal's floor, as Settings → Thresholds does.
metric_series.points takes an optional `until`; core.alerts gains
host_of_resource; floor_setting names its floor.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
list_backups, check_backup and make_backup are Settings → Backups for an
agent, admin only like the page. make_backup runs the page's job in the
background and is audited as "<user> via MCP". Restoring stays in the
browser. get_overview names a scheduled backup that failed, is two hours
past its slot, or has no bucket.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5834 "The host agent is always on, and the MCP sees a host's
maintenance", step 1 of milestone 520 "The MCP sees what Steward has
grown".
The host agent is a core system (operator, 2026-10-15: "it's very core
to the functionality of Steward as a whole and shouldn't be considered
something that can be toggled off"). ALWAYS_ON_PLUGINS in core settings
makes to_plugins_cfg report it enabled whatever its stored row says, so
it always loads and its MCP tools are always listed. Settings -> Plugins
shows it as "Built in: always on" with no switch, and a save keeps it
enabled. Its code and URLs stay under plugins/host_agent: every
installed agent posts to /plugins/host_agent/ingest.
Maintenance through the MCP:
- list_maintenance_plans: each plan's group, weekly slot, reboot policy,
next slot, and its latest run (hosts, how far it got, who it skipped
and why, how it ended).
- get_maintenance_job(host | job): a host's jobs, newest and running
first, or one job: state, packages updated, reboot, drained by, why it
failed, when its current state times out.
- run_maintenance(host, reboot="if_required" | "never"): admin, as the
host page's Update now; refuses a busy host or one with no helper;
audited "<user> via MCP".
- The overview names a host whose latest job in the last week failed or
timed out.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5748 "MCP can write, edit and test this instance's own playbooks
(steward-local)": two admin tools beside run_playbook, so an agent can
build automation for these hosts that isn't part of Steward itself.
- save_playbook(playbook, content, source="steward-local", replace=false)
goes through the editor's own validate_playbook_yaml and
write_playbook. A playbook that isn't a list of plays comes back with
the YAML error and its line, and nothing is written. An existing file
is overwritten only with replace=true. steward-builtin is refused
(Steward's playbooks change in its repository), as are git sources
(their repo owns them).
- After writing, `ansible-playbook --syntax-check` runs against a
throwaway localhost inventory (no host contacted, 60 s), so a module
name Ansible can't resolve shows before any run. The result rides the
reply; the file is kept either way.
- delete_playbook(playbook, source="steward-local").
- Both audit as ansible.playbook.created/updated/deleted, "<user> via
MCP", with the size and never the content.
The testing loop is the existing one: run_playbook defaults to check
mode, on one target, then get_ansible_run. The integration test does
that loop with the real ansible-playbook.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
On the live restore the "before restore" backup.made row read user
"unknown": _safety_backup recorded it with no username. It now carries
the person who started the restore.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5818 "First-run setup: Restore from a backup, by upload": beside creating
the first admin, setup offers "Or restore from a backup". Only while no
user exists; the upload is checked (a newer backup or one whose key can't
open its secrets is refused and not kept), then restored with no safety
backup (nothing to keep), and Steward restarts into the backup's users.
#5819 "Docs and CI": backups.md leads with restoring from Settings →
Backups and from setup; `steward restore --in-place` does the same-schema
restore from the shell, and CI's image check now uses it with a real
pg_restore: a changed setting comes back and the audit log stays.
Also exempts errors/restoring.html from the breadcrumb guard, as with
the other whole-app error pages: it replaces every page while a restore
runs, so there is nowhere to trail back to.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5816 "Restoring mode": while a backup goes back in, the scheduler holds,
ingest and HTMX answer 503 with Retry-After (agents keep their samples),
and every page says what is happening. A safety backup of Steward as it
stands is made first, to /data/backups and the bucket. The backup's key
is put in place, the restore audited, and Steward exits with code 3 so
Swarm, compose or systemd starts it fresh. A failed in-place restore
changes nothing and says why on Backups; a full replace that fails after
the wipe restarts into first-run setup.
#5817 "Settings → Backups: Restore": a Restore button on each sent backup
and on the file card. The chosen backup is checked (the same comparison
as Check, now a shared partial) and held server-side for an hour under a
token; it starts only once the admin types "restore", and an older
backup also needs "Replace everything" ticked. A newer backup, or one
whose key can't open its secrets, gets no button.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- restore_in_place checks the backup's migration heads before pg_restore
writes anything, so a backup from another schema is refused at once.
- The integration tests read their own setting rows directly:
get_all_settings knows only Steward's own keys.
- check.py: drop an import left unused by the move to pgdata.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5815 "Restore engine: in place for a same-version backup".
restore_in_place (steward/backup/restore.py) runs in one transaction:
1. It keeps aside what Steward records as it runs: ACTIVITY columns of rows
that come back, and STATE_SETTINGS.
2. It drops every foreign key touching a replaced table.
3. It truncates the replaced tables and COPYs the backup's rows in
unparsed, through asyncpg.
4. It puts the kept state back and moves sequences past their rows.
5. It puts right the rows whose parent didn't come back, by each key's own
ON DELETE. A RESTRICT key that may be empty is cleared, so a run whose
user is gone keeps its record.
6. It re-adds the keys, which checks every row.
A failure anywhere rolls the whole restore back. A backup made at another
schema is refused with NotSameSchema ("older": a full replace is the way;
"newer": update Steward).
The COPY-text reading moves into steward/backup/pgdata.py, shared by the
check and the restore. The integration tests stand in for pg_restore with
this database's own COPY output:
- settings come back while the audit log and the history of what remains
stay;
- history of what didn't come back goes;
- a failure part-way changes nothing;
- a newer schema is refused.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5823 "Check a backup: compare what it would restore with what is in
place, without restoring".
- steward/backup/check.py: pg_restore writes the backup's rows as COPY
text with no database involved. The rows in place go out through COPY in
the same form. Tables are compared on their primary keys, over the columns
both sides have, and grouped by area.
- Sealed values are opened with each side's own key and compared as
plaintext digests, so a secret sealed twice still matches. They are never
shown. A backup whose key can't open its own secrets fails the check.
- What Steward records as it runs (last-seen times, a schedule's last run,
backup history, scraped UniFi fields) is left out of the comparison:
contents.ACTIVITY, STATE_SETTINGS and HAND_SET_ONLY.
- Settings → Backups:
- a Check button on each sent backup, which fetches it from the bucket
(new s3.get_file);
- "Check a file" to upload one, with the size limit raised before the
CSRF check reads the form;
- a results page.
- Each backup Steward sends checks itself once written, and its history
row says how that went.
- New CLI command `steward check-backup FILE`. CI's image smoke step checks a
real backup against its source and expects a match.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5822 "Backup contents: settings and keys only". The operator's ruling
(System 180, 2026-10-15) overturns 2026-10-12's "Config + audit + alerts":
a backup is for system recovery and carries what would otherwise have to
be set up again, with no logs or history.
- contents.py: audit_events, alert_events, alert_states, metric_anomalies,
metric_anomaly_spans and user_invites move to EXCLUDED.
- Alert state is no longer restored, and evaluation skips a rule with no
state row. ensure_rule_states (alerts/service.py) gives every
single-resource rule lacking one an inactive row; alert_reconcile runs
it each minute.
- Docs, the Backups page and the CLI help say what a backup holds now.
CI's image check also finds the audit log empty after a restore.
- tests/integration/test_backup_restore_fixes.py is removed: its case (an
alert event naming a left-out Ansible run) no longer arises. The SET NULL
fix-up is covered by a unit test on synthetic tables.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Overview: a verdict line when something is wrong ('1 alert firing, 1
monitor down.'); its picker description now says what it lists (alerts,
monitors down, anomalies). Host figures have their home in Hosts.
Comments no longer name the Status page that was folded into Monitors.
- Monitors widget: a monitor not checked yet says when it was added.
- Host page: filesystems fullest first, by the worse of space and inodes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Routers: a router in trouble reads 'started 2h ago' or 'all of the last
24 h', from problem_since over the day's history, read for the troubled
routers only (a certificate's problem carries its own date). Hosts: a
down host reads 'a monitor is down for 3h'; host_glance keeps the
down_since host_status already had.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
SQLAlchemy fires after_commit and after_rollback for a savepoint too. The
host-agent ingest releases one after recording its metrics, so an alert
raised by a sample started its email before the outer transaction
committed; twice on 2026-10-12 the outbox row hit the event's foreign key
first and the email was lost. A savepoint rolling back (a docker persist
failing) cleared every pending notification, the outer one's included.
_after_commit now runs its jobs only on the outermost commit, and a
rolled-back savepoint drops only the jobs added inside it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
smartctl reports every disk behind a PERC/MegaRAID controller as /dev/bus/0
with type megaraid,N; the helper named a drive by its path alone, so all of
VDNT-Docker02's and Docker03's disks read as one drive "0", each overwriting
the last. The slot is now part of the name (bus0-megaraid3), and Steward gives
a repeated name an ordinal so two drives never share a series. Agent 1.20.1.
fix(ansible): Steward stopping mid-run logs no traceback (#5749)
The run's done-callback called exception() on a cancelled task, which raises;
it now logs a run cut short by shutdown at INFO. A finished run's in-memory
state is dropped by a timer rather than a sleeping task, which shutdown found
still pending.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The root helper gains one read-only action, `drives`: it finds the drives
itself (smartctl --scan-open), reads each as JSON, and writes a summary the
agent carries on every sample (agent 1.20.0). Steward judges the figures in
drives.py (failing / worn / watch / ok / no SMART / unreadable), records them
per drive and as host roll-ups for covering rules, keeps the newest report on
the registration and lists it on the host page.
install-deps.sh installs smartmontools with apt, dnf, yum, zypper, pacman or
apk, run by install.sh and every agent playbook; one it can't install is
reported, never fatal.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
temp_c_max was recorded on every ingest, but host_agent didn't offer it
to rules, so "host running hot" couldn't be watched. The catalog guard
didn't see the row(...) helper ingest builds its rows with. Now offered:
temp_c_max, temp_c per sensor, and network and disk throughput per host,
interface and disk. A new guard holds every literal row(...) name to the
catalog, except NOT_ALERTABLE, where each exclusion is given a reason.
temp_c_max reads in °C.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5712 "Each router problem says what it means, whether it's real, and what
to do". A problem card now names its kind and reads its codes:
- Server errors: 500 means the app is failing; 502/503/504 means Traefik
couldn't reach it or it timed out.
- Client errors: 404 scanners, a 401/403 login wall, 429 rate limiting, or
499 clients giving up on a slow app.
- Slow: p95 against p50.
Each card says whether the problem is likely real, and ends with one next
step. A fault links to the swarm service behind the router, matched by
name, and traffic links to Settings → Thresholds. The wording lives in
plugins/traefik/meaning.py.
The plugin manager now hands out a loaded plugin's submodule
(loaded_plugin_submodule), so Traefik and host_agent reach Docker's
models the same way (#5775).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5768 "A maintenance cadence: a group's updates and reboots, one host at a
time, on a schedule". A plan names an inventory group, a weekday and time
in a time zone, a window and a reboot policy (only when required by
default). At its slot a run takes the group's hosts in order and gives each
a #5767 maintenance job. The next host starts only when the last is back in
service: its agent is posting and, for a swarm node, the node is active and
every replicated service is at its replicas. A failure stops the run, and
no host is started once the window has passed. A host's latest failure
shows as host_agent/<host>/maintenance_failed, for an ordinary alert rule.
The page is Hosts → Maintenance.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
CI 9146 on 22b0be2:
- integration: host_agent_007 ran before core 0010 created maintenance_windows
(its window_id foreign key), so the schema never built. It now
depends_on 0010_maintenance_windows.
- unit: host_maintenance_jobs is named in the backup's EXCLUDED (run
history, like ansible_runs and host_downtime), and the
host_agent.maintenance audit action has its wording.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5766 "A root helper the agent can start, for a fixed set of maintenance
actions and nothing else", milestone 514.
The agent runs unprivileged under NoNewPrivileges, so it can't update or
reboot its host. It now leaves a request in /var/lib/steward-agent/requests,
the one path its unit lets it write; steward-maint.path starts maint.py as
root, which reads it without following links, accepts only update, reboot,
drain or undrain, deletes it, and writes the outcome to
/var/lib/steward-maint/results. The agent (1.17.0) reports each new result
state once as a `maintenance` event, re-checks pending updates when an update
finishes, and says in its metadata whether the helper is installed.
The units and the helper are served from one copy at
/plugins/host_agent/files/<name> (a fixed list), and install.sh plus the
install, provision and update playbooks all fetch them, replacing three
inline copies of the unit. Update agent installs the helper on existing
hosts; the Agent panel says under Manage agent whether it is there.
Steward handing out jobs is #5767; the cadence is #5768.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Step 1 of milestone 514 "Updates and reboots on a cadence, carried out by
each host's agent…": #5765 "The agent reports pending updates and whether
a reboot is required".
- Agent 1.16.0: an hourly background check (apt-check, apt list, or dnf -C
with updateinfo and needs-restarting), each command bounded at 120 s, no
root and no network. Debian's reboot-required marker and its packages
are read every sample.
- Ingest records updates_pending, updates_security and reboot_required for
rules, and keeps the newest on the registration (host_agent_005_updates).
- The Agent card leads with "Reboot required since …", then the updates
waiting or "Up to date", with when the check ran. list_agents carries
them too.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5764 "The agent reads a container's whole log backlog into memory, on its
main loop, with no overall deadline". Docker03's 1.13.0 agent went silent
for a day at 2.1 GB and 46 h of CPU, then ignored SIGTERM.
- One shared reader, _docker_get: each answer has an overall deadline and a
byte cap, not only a per-recv timeout. A trickling or endless body is cut
off (15 s / 32 MiB for JSON, 5 s / 4 MiB for a log read).
- Container logs: since the cursor but only the newest 2,000 lines, all
within 10 s. A note says when older lines were skipped, and a container
whose answer is too big skips ahead to now instead of re-reading its
backlog every interval.
- The agent reports its own RSS and CPU; Steward records agent_rss_bytes
and agent_cpu_pct, so a runaway agent shows before it goes silent.
Agent 1.15.0.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5760 "One stalled NFS mount freezes the whole agent: statvfs has no
deadline, so the host goes silent while it still answers ping". The agent
read every mount inline in its only loop; a hard-mounted share whose server
stops blocks statvfs in the kernel with no timeout. Each mount is now read
in a daemon thread under one 5 s deadline. One that misses it is sent as
stalled, and isn't probed again while its last probe is still blocked. A
mount that errors (a stale handle) is left out instead of emptying the list.
Steward counts them as mounts_stalled (alertable: "an NFS share stopped"),
and skips the fullest-disk figure when every mount stalled.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
