Commit Graph
590 Commits
Author SHA1 Message Date
bvandeusenandClaude Opus 5.5 0a647bbb82 feat(syslog): devices through Swarm's ingress told apart by name; the send-to address is a setting (#6096)
CI / lint (push) Successful in 2s
CI / unit (push) Failing after 1m30s
CI / integration (push) Successful in 1m38s
CI / publish (push) Skipped
Through ingress every device arrives from Docker's own address, so a
source there is keyed by the hostname its lines carry (new column
sender, unique with address). The set-up help no longer guesses the
address from the web hostname: it shows the plugin's send_to setting,
covers ingress and host mode, ASUSTOR ADM, and both syslog formats.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-19 12:05:28 -04:00
bvandeusenandClaude Opus 5.5 d2926b6663 test(maintenance): the swarm job's boot comes after the reboot the settle hands out (#6098)
CI / lint (push) Successful in 8s
CI / unit (push) Successful in 1m33s
CI / integration (push) Successful in 1m42s
CI / publish (push) Successful in 20s
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-19 10:50:12 -04:00
bvandeusenandClaude Opus 5.5 1ba4784fa6 fix(maintenance): check first, drain before a Docker upgrade, wait 5 min after a drain, see containers back (#6098)
CI / lint (push) Successful in 2s
CI / unit (push) Failing after 1m29s
CI / integration (push) Successful in 2m7s
CI / publish (push) Skipped
On 2026-10-18 an update on VDNT-Docker02 restarted the Docker daemon six
seconds in, before the drain that came only with the reboot, and every
container on the node stopped. Docker's own packages restart the daemon from
their install scripts; needrestart's list-only mode doesn't stop that.

- The helper gains a `check` action (agent 1.24.0): a dry run of the update
  for each package manager, whether Docker is among it, and the running
  containers outside the swarm with their restart policies.
- A swarm node is drained before an update that upgrades Docker, else only
  before a reboot, and at most once a job.
- After a drain the job settles 5 minutes before the update or reboot.
- Once the work is over, the containers that ran outside the swarm must be
  seen running again within 10 minutes, or the job fails naming them.
- An older agent gets no check and its job runs as before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-19 10:44:45 -04:00
bvandeusenandClaude Opus 5.5 da81010a75 fix(alerts): an alert that fired and ended inside maintenance isn't mailed resolved (#6097)
CI / lint (push) Successful in 2s
CI / unit (push) Successful in 1m29s
CI / integration (push) Successful in 1m39s
CI / publish (push) Successful in 1m9s
During Docker02's maintenance reboot its Agent offline alert fired muted by
the job's window, then resolved just after the window closed: the resolution
was mailed for a firing nobody had been told of. A resolution whose firing
was held and never released is now recorded with a note, not sent.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-19 07:41:51 -04:00
bvandeusenandClaude Opus 5.5 ed9da56efa feat(syslog): set-up help on the page, port 5514 published, Docker-hidden senders called out (#6003)
CI / lint (push) Successful in 2s
CI / unit (push) Successful in 1m29s
CI / integration (push) Successful in 1m38s
CI / publish (push) Successful in 20s
The Syslog page says where to send (this server, its port and protocols) and
where the setting is on Synology, QNAP, TrueNAS, OpenWrt, UniFi and rsyslog,
open until a device sends. A sender whose address is on Steward's own
container network (Swarm ingress publishing hides the real sender) is
flagged, with the host-mode ports to publish instead. compose files and the
Dockerfile publish 5514/udp and /tcp; docs/plugins/syslog.md describes the
plugin and the deploy.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-19 07:01:56 -04:00
bvandeusenandClaude Opus 5.5 96f774ce24 fix(syslog): chart marks give up after 10 s rather than waiting on (#6002)
CI / lint (push) Successful in 2s
CI / integration (push) Successful in 1m38s
CI / unit (push) Successful in 1m28s
CI / publish (push) Successful in 1m12s
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-19 06:40:01 -04:00
bvandeusenandClaude Opus 5.5 521d84f318 feat(syslog): lines beside the readings: around an anomaly, in get_anomaly, and marked on host and SNMP charts (#6002)
CI / lint (push) Successful in 2s
CI / unit (push) Successful in 1m29s
CI / integration (push) Successful in 1m37s
CI / publish (push) Successful in 1m9s
get_anomaly's circumstances gain the host's or SNMP device's syslog
warnings and worse (mcp_events / mcp_resource_events). An anomaly row opens
"Syslog around it", read only when opened. A shared Chart.js plugin
(chart-marks.js) draws a dashed line at each syslog error on the host
metrics charts and an SNMP device's charts, with a line saying how many.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-19 06:36:31 -04:00
bvandeusenandClaude Opus 5.5 f37a578136 test(syslog): a preset's 15-minute window doesn't reach the 40-minute-old nfsd line (#6001)
CI / lint (push) Successful in 2s
CI / unit (push) Successful in 1m29s
CI / integration (push) Successful in 1m35s
CI / publish (push) Successful in 1m18s
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-19 06:27:23 -04:00
bvandeusenandClaude Opus 5.5 8eae3d3825 feat(syslog): pattern alerts: telling phrases counted per source each minute for alert rules, storage-trouble presets (#6001)
CI / lint (push) Successful in 2s
CI / unit (push) Successful in 1m31s
CI / integration (push) Failing after 1m40s
CI / publish (push) Skipped
A pattern (phrases, a severity, a source, a window) is counted every minute
into plugin_metrics as lines_matching_<key> per added source, zero included,
so an ordinary alert rule fires on a match and clears once the window passes.
Each count reads only the lines stored since the last, by line id; a new or
changed pattern reads its window once.

Patterns page with ready-made storage-trouble presets and "Alert on it",
which opens the rule form aimed at the count. Source names are kept unique,
and a rename moves its counts and the rules on them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-19 06:22:54 -04:00
bvandeusenandClaude Opus 5.5 dadec4e355 feat(syslog): a source's lines to read and search, on its page, its host's and its SNMP device's, and over MCP (#6000)
CI / lint (push) Successful in 2s
CI / unit (push) Successful in 1m29s
CI / integration (push) Successful in 1m36s
CI / publish (push) Successful in 1m6s
- /plugins/syslog/sources/<id>: the lines newest first, loading as you
  scroll, filtered by severity and worse, app, text and time, with day
  dividers and times to the second in the reader's zone.
- The Syslog page links each source and shows its lines in the last hour.
- A Syslog panel on a host's page and an SNMP device's: the newest lines of
  the sources bound to it or at its address.
- MCP: list_syslog_sources and search_syslog (source, host, text, severity
  and worse, app, range, paging by next_before).
- Shared: escape_like moves to steward/core/like.py and cut_middle to the
  MCP shaping module, replacing four inline copies; |when takes seconds=.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-19 06:05:52 -04:00
bvandeusenandClaude Opus 5.5 bf6019a2af test(syslog): each integration step its own session (#5999)
CI / lint (push) Successful in 2s
CI / unit (push) Successful in 1m29s
CI / integration (push) Successful in 1m34s
CI / publish (push) Successful in 1m12s
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-18 23:15:13 -04:00
bvandeusenandClaude Opus 5.5 6ff9cffed4 fix(syslog): word its audit actions, list its tables for backups, mark its page top-level, reach its modules as loaded (#5999)
CI / lint (push) Successful in 2s
CI / unit (push) Successful in 1m29s
CI / integration (push) Failing after 1m38s
CI / publish (push) Skipped
- Audit: syslog.source_* worded, and a Syslog filter area.
- Backups keep syslog_sources (added, named, ignored; what it last sent
  isn't compared) and leave syslog_lines out, as logs.
- The integration test reaches the plugin as loaded (#5775), so its
  tables aren't defined twice; ruff E741.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-18 23:12:59 -04:00
bvandeusenandClaude Opus 5.5 51892a02c0 feat(syslog): a syslog receiver: UDP and TCP, senders offered before they're kept, each source's lines in a ring (#5999)
CI / lint (push) Failing after 2s
CI / unit (push) Failing after 1m27s
CI / integration (push) Failing after 1m35s
CI / publish (push) Skipped
- plugins/syslog: RFC 5424 and RFC 3164 parsing, RFC 6587 TCP framing.
- Lines are queued as they arrive and stored every 5 s through a Holding,
  so a storage stall delays them rather than losing them.
- A new sender is an offered source, counted and not kept, until an admin
  adds it on the Syslog page (name, host); it can be ignored instead.
- The listener follows the plugin's settings (port, UDP, TCP), rebinding
  within seconds of a save; a port it can't open is said on the page.
- Hourly ring per source: age window and byte cap from Settings.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-18 23:09:07 -04:00
bvandeusenandClaude Opus 5.5 a59d1a7709 fix(plugins): a plugin that can't start live says why and offers a restart (#6087)
CI / lint (push) Successful in 2s
CI / unit (push) Successful in 1m28s
CI / integration (push) Successful in 1m33s
CI / publish (push) Successful in 1m29s
Saving an enabled plugin threw away hot_reload_plugin's (ok, message), so a
plugin that failed to load live read "Enabled, not loaded" with nothing
more. Now:
- a live-load failure is recorded with its reason, as a boot failure is, and
  logged; a later success clears it;
- any save of an enabled plugin that isn't running tries to start it, not
  only the off-to-on toggle;
- enabling one already running is a success, not a failure;
- the plugin's page offers "Restart to load" while it is enabled and not
  running.

Quart never marks the app as having served its first request, so
register_blueprint after startup is allowed: hot-loading is not refused
wholesale, as #6087 suspected.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-18 22:56:54 -04:00
bvandeusenandClaude Opus 5.5 f7616b71c3 fix(dashboard): the Add-a-widget drawer starts at the sidebar's edge, so none of it hides behind it (#6086)
main's z-index makes it a stacking context, so the fixed drawer (z-index 60)
could never rise over the sticky sidebar (z-index 20): its title, group
labels and first column of widgets sat underneath. Above 900px the drawer
now starts at left:220px; below, the sidebar is off-canvas and it keeps the
full width.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-18 22:56:54 -04:00
bvandeusenandClaude Opus 5.5 56063f27a4 fix(docker): a container's memory is its processes' own; file cache shown on its page (#6030)
CI / lint (push) Successful in 2s
CI / unit (push) Successful in 1m29s
CI / integration (push) Successful in 1m45s
CI / publish (push) Successful in 1m43s
The agent sent Docker's working set (usage less inactive_file), which still
counts recently used file cache. A file-heavy container with no limit read
191 GB while its processes held 2.2 GB, and the host's own figure, which
already leaves cache out, disagreed with it.

Agent 1.23.0 sends mem_used_bytes / mem_used_pct (anon + shmem, so
Postgres's shared buffers count; v1: rss + shmem) and mem_cache_bytes
apart. The new names keep the two meanings apart in history and in learned
normals: docker_013 renames the columns, clears the old values and adds
mem_cache_bytes. The Docker page, swarm by-node sums, Busiest Containers,
container detail, normals and the MCP tools all read the new figure. File
cache is shown only on the container's page and in get_container.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-17 21:27:29 -04:00
bvandeusenandClaude Opus 5.5 1335b9afaf test(host_agent): the NFS line test gives the host a reading, so its metrics render (#5996)
CI / lint (push) Successful in 2s
CI / unit (push) Successful in 1m31s
CI / integration (push) Successful in 1m33s
CI / publish (push) Successful in 1m8s
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-17 09:51:07 -04:00
bvandeusenandClaude Opus 5.5 ad8bbd8d69 feat(snmp): a NAS's disk and volume I/O, CPU waiting on disk, available memory, interface errors (#5997)
CI / lint (push) Successful in 2s
CI / integration (push) Successful in 1m31s
CI / unit (push) Failing after 1m32s
CI / publish (push) Skipped
- Synology: what each disk and volume reads and writes, bytes and requests
  a second and the load DSM gives it (SYNOLOGY-STORAGEIO- and -SPACEIO-MIB,
  64-bit byte counters), and each disk's reconnects and sectors not found.
- Linux server: CPU waiting on disk, the wait tick counter's share of the
  CPU's, and memory available (free + buffers + cache). A reading may now be
  worked out from others: the poll asks for its inputs, rates each counter,
  and stores only the result.
- Interface traffic: errors and discards in and out, per second.
- byte_count unit: a byte counter shown as bytes a second.

Asustor's MIB has nothing more for this; its Linux server readings gain the
same CPU wait and available memory.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-17 09:50:52 -04:00
bvandeusenandClaude Opus 5.5 3f9aae7f46 feat(host_agent): NFS delay per mount, read from /proc/self/mountstats (#5996)
CI / unit (push) Failing after 1m32s
CI / integration (push) Successful in 1m31s
CI / publish (push) Skipped
CI / lint (push) Successful in 2s
Each sample carries each NFS mount's requests since the last one: how many
a second, their round trip and execute time (READ and WRITE on their own),
and how many were sent again or hit a major timeout. mountstats is the
kernel's own count and never touches the share, so it reads while one is
stuck. Steward stores them per mount and the slowest on the host as
nfs_exe_max_ms, for an "NFS slow" rule and the host page's NFS chart beside
mounts_stalled; each mount's bar says what a request took.

Agent 1.22.0 (which also carries the backlog order from 41fc807).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-17 09:46:14 -04:00
bvandeusenandClaude Opus 5.5 2df8431de9 feat: monitors, UniFi and Traefik keep collecting through a storage stall (#5995)
CI / lint (push) Successful in 3s
CI / integration (push) Successful in 1m38s
CI / unit (push) Successful in 1m39s
CI / publish (push) Successful in 1m11s
The same Holding as the SNMP poll, for the rest of Steward's own
collectors:

- Monitors: the list is read within 10 s or the last one read stands in;
  a start the database doesn't take is kept in memory, so a monitor isn't
  checked twice; each result is held and stored at the time it was
  checked. A stall shows on the monitors as the checks it saw, not a gap.
- UniFi and Traefik: each poll's writes are held (hold_write) and run
  oldest first in one transaction; every reading is stamped with its scrape.
- A held round the database refuses (as opposed to can't be reached) is
  stored alone and let go, so it can't hold up every round after it.

The Prometheus scrape is left as it is: its one target, Fabled Archives,
keeps its own database on the same NFS.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-17 09:41:14 -04:00
bvandeusenandClaude Opus 5.5 41fc8073db feat: readings kept through a storage stall; an agent's backlog arrives in order (#5995)
CI / lint (push) Successful in 2s
CI / unit (push) Successful in 1m39s
CI / integration (push) Successful in 1m49s
CI / publish (push) Successful in 1m49s
On 2026-10-17 the NFS under Postgres stalled for ~15 minutes. Two things
went wrong for the readings that would have explained it:

- The SNMP poll stored before it polled again, so while the database hung
  it stopped polling. It now collects on schedule and hands each round to a
  Holding (steward/core/holding.py): one writer stores what is held, oldest
  first, each reading at the time it was collected. Bounded at two hours of
  polls, with a 30-minute deadline on a store; the device list read falls
  back to the last one read after 10 s.
- "Mount not answering" (10 checks in a row) went pending on Docker02 and
  Docker03 and never fired: the agent sent each post's backlog with its
  newest, healthy sample, and that sample ended every post's run. Agent
  1.22.0 queues the new sample behind the backlog, so Steward receives a
  host's samples in the order they were taken.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-17 09:36:47 -04:00
bvandeusenandClaude Opus 5.5 bd1c32d2a4 fix(snmp): a disk listed late in UCD-DISKIO is read, and Asustor disks are named by bay (#5992)
CI / lint (push) Successful in 2s
CI / unit (push) Successful in 1m37s
CI / integration (push) Successful in 1m50s
CI / publish (push) Successful in 1m29s
The table walk stopped after 64 raw entries, counting the RAM disks, loops,
partitions and md arrays that rows_like then drops, so a Synology's eighth
disk was never reached. The walk is now bounded at 1,024 entries per column;
64 still caps the rows kept.

ADM labels its disks "Disk0".."Disk4" while their indexes run 1..5; the
Asustor disk table now names each row by its index ("Disk 1"). Keys are
unchanged, so history carries on.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 23:16:25 -04:00
bvandeusenandClaude Opus 5.5 d065cafd11 feat(snmp): NAS presets for Synology and Asustor, reading every disk and volume (#5985, #5986)
CI / lint (push) Successful in 3s
CI / unit (push) Successful in 1m38s
CI / integration (push) Successful in 1m41s
CI / publish (push) Successful in 1m31s
A preset can now read a vendor's tables: one row per disk, volume, fan or
sensor. The scheduled poll discovers the rows by walking the device: on its
first poll, after a save, and then hourly, and only while it answers. Each row
becomes readings named by the device's own labels ("Disk 1 status"). Only the
columns a row answered are asked for.

A vendor status, whether a code (Synology) or a word (Asustor), is a state
reading. It is stored as a level (0 OK, 1 busy, 2 warning, 3 failed), so one
alert rule covers any vendor, and shown in the device's own word.

- Synology: system, power and fan status and temperature; per disk: status,
  health, temperature and bad sectors; per volume: RAID state (scrubbing,
  parity check, rebuild, degraded), free space and size.
- Asustor: per disk: status, SMART and temperature; per volume: status, free
  space and size; CPU and system temperature, free memory and fans.
  From ADM's ASUSTOR-DISK-MIB and ASUSTOR-SYSTEM-MIB.
- Linux server: plus each physical disk's 1-minute busy % (UCD-DISKIO-MIB).

Migration snmp_002_table_rows adds tables, tables_at and texts to
snmp_devices. Milestone 530.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 22:40:14 -04:00
bvandeusenandClaude Opus 5.5 4ded8ecee4 fix(mcp): get_anomaly shows a quick Steward restart, not only a gap over 90 s (#5862)
CI / lint (push) Successful in 2s
CI / unit (push) Successful in 1m27s
CI / integration (push) Successful in 1m33s
CI / publish (push) Successful in 1m15s
A redeploy's new process ticks at once, so its once-a-minute samples never
broke for 90 s and get_anomaly showed none of the day's restarts. The first
self-monitor tick in a process now records restart_gap_secs (time since the
last process's last sample), and steward_gaps shows each one as a restart.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-15 16:11:27 -04:00
bvandeusenandClaude Opus 5.5 22da64a284 fix(mcp): list_maintenance_plans reports only a slot that fired as last_slot_at (#5863)
CI / lint (push) Successful in 2s
CI / unit (push) Successful in 1m27s
CI / integration (push) Successful in 1m32s
CI / publish (push) Successful in 1m6s
A new plan's last_slot_at holds the moment it was made, so its first
slot is the next one; the tool returned that moment, a Wednesday
evening for a Sunday plan, beside latest_run: null. cadence.fired_slot
returns it only when it falls on one of the plan's slots.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-15 16:00:48 -04:00
bvandeusenandClaude Opus 5.5 398312b8be fix(traefik): a renewed certificate is old, and list_certificates is bounded (#5864)
A Traefik that hasn't restarted keeps reporting the certificate it
replaced, so codex.fabledsword.com's old one (21 days left) was listed
beside its renewal (81 days) and would have raised an expiry warning on
2026-10-22. A reported certificate whose every name a later-expiring
one covers is now old, on the page, the widget, the overview and MCP.

list_certificates takes `limit` (default 100) with total/truncated, and
says how many served certificates no HTTP router uses (`unrouted`).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-15 16:00:19 -04:00
bvandeusenandClaude Opus 5.5 622addc09e fix(mcp): get_anomaly keeps the events nearest the occurrence and says how many it left out (#5861)
Each source in an anomaly's `around` was cut to its oldest 40 with
nothing in the reply to say so; on a busy host the events nearest the
occurrence were the ones dropped. Each source now keeps the 40 nearest
the occurrence's start, and `around_left_out` counts the rest. Sources
return everything their bounded query read (Docker and the host agent
no longer cut at 40 and 20 themselves), so the count is honest.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-15 15:59:16 -04:00
bvandeusenandClaude Opus 5.5 91bb199d14 fix(mcp): get_anomaly shows Steward's own restarts beside every anomaly, and a router's containers (#5862)
All 16 Traefik 4xx anomalies on 2026-10-15 came back with nothing
around them, though 15 were Steward's own restarts. A router has no
host, and every events source was read by host.

- Core: each break of more than 90 s in Steward's once-a-minute health
  sample is a "steward" event beside every anomaly, placed or not.
- Plugins: an optional mcp_resource_events(db, source, resource, since,
  until) for a resource with no host. Traefik's gives the container
  events of the swarm service behind a router, on any host.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-15 15:58:28 -04:00
bvandeusenandClaude Opus 5.5 a24b9da69d fix(traefik): a router fails on 5xx only once it answered a few server errors (#5865)
CI / lint (push) Successful in 2s
CI / unit (push) Successful in 1m28s
CI / integration (push) Successful in 1m34s
CI / publish (push) Successful in 1m10s
One 502 among about 70 requests read steward@swarm critical at 1.4%:
past the 30-request minimum and far past the 0.1% cutoff, yet one
error. A new Settings → Thresholds field, "Fewest server errors to
fail" (traefik.threshold.min_5xx, default 3), must also be reached in
the 15-minute window; the 0.1% cutoff still catches a busy router.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-15 15:49:51 -04:00
bvandeusenandClaude Opus 5.5 c625528191 fix(mcp): a failing router in get_overview says since when, as the page does (#5860)
CI / lint (push) Successful in 2s
CI / unit (push) Successful in 1m27s
CI / integration (push) Successful in 1m33s
CI / publish (push) Successful in 19s
router_findings never passed `since`, so every failing router read as
brand new and sorted last among the criticals. The overview now reads
the failing routers' last day of history and dates each one with
problem_since, as the dashboard widget does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-15 15:40:35 -04:00
bvandeusenandClaude Opus 5.5 80674603bc fix(traefik): a router's 5xx share is judged only over enough requests, like its 4xx share (#5859)
CI / lint (push) Successful in 1s
CI / unit (push) Successful in 1m27s
CI / integration (push) Successful in 1m33s
CI / publish (push) Successful in 1m15s
Two 500s of about sixteen requests graded roundtable@swarm critical at
9.1%. The Settings → Thresholds minimum of requests in 15 minutes (30 by
default) now guards both shares; the 5xx share still keeps its fixed
cutoff whatever the router's history.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-15 15:35:57 -04:00
bvandeusenandClaude Opus 5.5 bf017e65c2 feat(traefik): scrape every Traefik replica behind the metrics URL and add them up (#5844)
CI / lint (push) Successful in 2s
CI / unit (push) Successful in 1m27s
CI / integration (push) Successful in 1m33s
CI / publish (push) Successful in 19s
Through one address Steward read whichever replica answered, and with
host-mode ports behind a keepalived VIP that can be the replica serving
nothing: since the 18:43 restart every scrape landed on Docker03's idle
Traefik, which has no traefik_service_* series, so no router was stored.

A metrics_url host name that resolves to several addresses (a swarm
service's tasks.<service>, a Kubernetes headless Service, round-robin DNS)
is now scraped address by address, with the original Host header and TLS
name. Answers from one process are kept once; a replica that doesn't answer
is skipped with a warning. Requests, bytes and latency buckets are summed
over the replicas that have a baseline, so the sum's rise is the sum of each
one's own rise; certificates are listed once; Traefik's own figures add
connections and memory and keep the newest reload. A name with one address
is one scrape, as before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-15 14:55:54 -04:00
bvandeusenandClaude Opus 5.5 6fa62c9520 fix(mcp): read the successes once, so every failed run is checked against all of them (#5842)
CI / lint (push) Successful in 2s
CI / unit (push) Successful in 1m26s
CI / integration (push) Successful in 1m33s
CI / publish (push) Successful in 1m7s
still_failing was handed the query's one-pass result. The first failed
run's any() consumed it, so a second failure in the window found no
successes left and was reported though a later run had put it right
(ci-runner-volumes came back once keepalived-survey also had a failure).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-15 14:52:19 -04:00
bvandeusenandClaude Opus 5.5 98440d8de6 fix(traefik): count each Traefik process only against its own last scrape (#5844)
CI / lint (push) Successful in 2s
CI / unit (push) Successful in 1m28s
CI / integration (push) Successful in 1m33s
CI / publish (push) Successful in 20s
One metrics URL can reach several Traefik processes: a swarm service's
replicas behind the routing mesh or its VIP, a Kubernetes Service's pods,
a keepalived pair after failover, or the same Traefik after a restart.
Each keeps its own counters, so comparing one with another read the second
one's lifetime totals as new traffic: retired routers flashed 100% 502 and
an idle router showed 834 req/s whenever the scrape moved to the other
replica.

Each scrape is now keyed by Traefik's process_start_time_seconds and
compared only with the same process's previous scrape. A process seen for
the first time (or not in the last hour) sets its baseline and counts
nothing. Without a start time every scrape is one process, as before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-15 14:36:10 -04:00
bvandeusenandClaude Opus 5.5 70eb253190 fix(mcp): the overview drops an Ansible failure a later run put right (#5842)
CI / lint (push) Successful in 2s
CI / unit (push) Successful in 1m29s
CI / integration (push) Successful in 1m34s
CI / publish (push) Successful in 1m29s
get_overview listed every failed or interrupted run of the last 24 hours,
so a dry run that failed, was fixed and re-run green kept warning for a
day. A failure now drops out once a later success of the same playbook,
source and hosts is at least as real: an applied success clears either
kind of failure, a dry-run success only a dry-run failure.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-15 13:58:24 -04:00
bvandeusenandClaude Opus 5.5 eee8469a17 feat(mcp): SNMP devices and Prometheus targets: their state, readings and series (#5840)
CI / lint (push) Successful in 2s
CI / unit (push) Successful in 1m26s
CI / integration (push) Successful in 1m33s
CI / publish (push) Successful in 19s
The snmp plugin registers list_snmp_devices and get_snmp_device; the
prometheus plugin registers list_prometheus_targets and
get_prometheus_target. Both are viewer-level, read what the poller and
scraper stored, and exist only while their plugin is enabled.
get_overview names a device or target whose last poll or scrape failed;
stale, pending and off are left out.

A device's community and v3 keys and a target's token are never
returned, and a password in a target URL is masked.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-15 13:31:46 -04:00
bvandeusenandClaude Opus 5.5 2d88315bb1 feat(mcp): the UniFi network: WAN, devices, clients, events, alarms and config (#5839)
get_network gives the WAN, clients, Wi-Fi networks, speed tests and devices
by status; list_devices, list_clients (everyone seen, or the newest poll's
top talkers), list_network_events, list_alarms and get_network_config
(networks, port forwards, firewall rules) read what the poller stored.
get_overview names the WAN down while the poll is current, a device offline
or degraded that the controller still reports, and active alarms.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-15 13:28:19 -04:00
bvandeusenandClaude Opus 5.5 684a47b3eb feat(mcp): Traefik routers, certificates and traffic (#5838)
CI / lint (push) Successful in 3s
CI / unit (push) Successful in 1m29s
CI / integration (push) Successful in 1m37s
CI / publish (push) Successful in 20s
list_routers grades every router as the Traefik page does (the last 15
minutes, 5xx on its cutoff, 4xx and p95 against the router's normal) and
gives a troubled one its status codes, what they mean and the swarm
service behind it. get_router adds its history, codes, normal and
anomalies; list_certificates the served certificates and the routers they
cover; get_traffic the access log's clients, countries and routers.
get_overview names a router answering 5xx and a certificate near expiry.

The access-log aggregation moves into routes.summarize_access, shared by
the traffic page and get_traffic.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-15 13:26:34 -04:00
bvandeusenandClaude Opus 5.5 0945d78290 feat(mcp): understand and judge a metric anomaly: what happened, what was usual, what else went on (#5837)
CI / lint (push) Successful in 2s
CI / unit (push) Successful in 1m30s
CI / integration (push) Successful in 1m34s
CI / publish (push) Successful in 1m8s
get_anomaly gives each occurrence against what is usual at that hour, the
metric around the newest one with its normal, what else happened on the
resource's host around each occurrence (alert transitions, Ansible runs,
other anomalies there and a count elsewhere, plus plugins' own events
through a new mcp_events hook: the host agent's maintenance jobs and
downtime, Docker's container starts, deaths and OOM kills), and how earlier
anomalies on the same metric were judged.

get_metric_normal reads the learned slots and the level that would be
flagged. review_anomaly takes one id or several, all or nothing, with a
note. get_anomaly_settings and (admin) set_anomaly_settings read and change
the review window and each normal's floor, as Settings → Thresholds does.

metric_series.points takes an optional `until`; core.alerts gains
host_of_resource; floor_setting names its floor.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-15 13:17:34 -04:00
bvandeusenandClaude Opus 5.5 8feec4dc77 feat(mcp): Steward's own backups: the runs, the next one, checking and making one (#5836)
CI / lint (push) Successful in 2s
CI / unit (push) Successful in 1m25s
CI / integration (push) Successful in 1m31s
CI / publish (push) Successful in 1m13s
list_backups, check_backup and make_backup are Settings → Backups for an
agent, admin only like the page. make_backup runs the page's job in the
background and is audited as "<user> via MCP". Restoring stays in the
browser. get_overview names a scheduled backup that failed, is two hours
past its slot, or has no bucket.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-15 13:09:12 -04:00
bvandeusenandClaude Opus 5.5 c7ca459b81 feat(mcp): swarm nodes link to their host and maintenance; the overview names a node that is down (#5835)
CI / lint (push) Successful in 2s
CI / integration (push) Successful in 1m31s
CI / unit (push) Successful in 1m38s
CI / publish (push) Successful in 19s
#5835 "MCP: swarm nodes", step 2 of milestone 520. list_swarm_services
already listed every node (role, leader, availability, status); the
audit missed it. What was missing:
- Each node row carries `host_id` (the Steward host of that name) and
  `maintenance` (the state of a maintenance job running on it), so an
  agent can go node -> get_host / get_maintenance_job.
- get_overview names a node whose status isn't ready, unless its host
  has a maintenance job running (rebooting on purpose).
- Drained alone is not a finding: VDNT-Util01 is a manager kept drained
  on purpose (#5777), and a drain a failed job left behind is already
  that job's maintenance_failed finding.
unique_nodes() keeps the newest report of each node, as the list did.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-15 13:04:02 -04:00
bvandeusenandClaude Opus 5.5 3ad8eed83d feat: the host agent is always on; the MCP sees and starts host maintenance (#5834)
CI / lint (push) Successful in 2s
CI / integration (push) Successful in 1m32s
CI / unit (push) Successful in 1m39s
CI / publish (push) Successful in 1m14s
#5834 "The host agent is always on, and the MCP sees a host's
maintenance", step 1 of milestone 520 "The MCP sees what Steward has
grown".

The host agent is a core system (operator, 2026-10-15: "it's very core
to the functionality of Steward as a whole and shouldn't be considered
something that can be toggled off"). ALWAYS_ON_PLUGINS in core settings
makes to_plugins_cfg report it enabled whatever its stored row says, so
it always loads and its MCP tools are always listed. Settings -> Plugins
shows it as "Built in: always on" with no switch, and a save keeps it
enabled. Its code and URLs stay under plugins/host_agent: every
installed agent posts to /plugins/host_agent/ingest.

Maintenance through the MCP:
- list_maintenance_plans: each plan's group, weekly slot, reboot policy,
  next slot, and its latest run (hosts, how far it got, who it skipped
  and why, how it ended).
- get_maintenance_job(host | job): a host's jobs, newest and running
  first, or one job: state, packages updated, reboot, drained by, why it
  failed, when its current state times out.
- run_maintenance(host, reboot="if_required" | "never"): admin, as the
  host page's Update now; refuses a busy host or one with no helper;
  audited "<user> via MCP".
- The overview names a host whose latest job in the last week failed or
  timed out.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-15 12:59:42 -04:00
bvandeusenandClaude Opus 5.5 e122860cdc feat(mcp): write, test and delete this instance's own playbooks (#5748)
CI / lint (push) Successful in 2s
CI / integration (push) Successful in 1m29s
CI / unit (push) Successful in 1m39s
CI / publish (push) Successful in 1m6s
#5748 "MCP can write, edit and test this instance's own playbooks
(steward-local)": two admin tools beside run_playbook, so an agent can
build automation for these hosts that isn't part of Steward itself.

- save_playbook(playbook, content, source="steward-local", replace=false)
  goes through the editor's own validate_playbook_yaml and
  write_playbook. A playbook that isn't a list of plays comes back with
  the YAML error and its line, and nothing is written. An existing file
  is overwritten only with replace=true. steward-builtin is refused
  (Steward's playbooks change in its repository), as are git sources
  (their repo owns them).
- After writing, `ansible-playbook --syntax-check` runs against a
  throwaway localhost inventory (no host contacted, 60 s), so a module
  name Ansible can't resolve shows before any run. The result rides the
  reply; the file is kept either way.
- delete_playbook(playbook, source="steward-local").
- Both audit as ansible.playbook.created/updated/deleted, "<user> via
  MCP", with the size and never the content.

The testing loop is the existing one: run_playbook defaults to check
mode, on one target, then get_ansible_run. The integration test does
that loop with the real ansible-playbook.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-15 12:01:39 -04:00
bvandeusenandClaude Opus 5.5 39685585cd chore(host-agent): say why the root helper keeps its own os-release reader
CI / lint (push) Successful in 2s
CI / integration (push) Successful in 1m27s
CI / unit (push) Successful in 1m37s
CI / publish (push) Successful in 20s
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-15 11:52:35 -04:00
bvandeusenandClaude Opus 5.5 4054bf6bbb feat(host-agent): updates on pacman, zypper and yum hosts (#5803, #5804)
CI / lint (push) Successful in 3s
CI / integration (push) Successful in 1m26s
CI / unit (push) Successful in 1m35s
CI / publish (push) Successful in 20s
Agent 1.21.0. The hourly update check now covers the package managers
beyond apt and dnf (#5803 "The agent's update check reads pacman,
zypper and yum"):
- yum: `yum -C check-update`, `updateinfo list security`,
  `needs-restarting -r`, as dnf.
- zypper: `--no-refresh list-updates` (`--dup` on Tumbleweed),
  needed security patches from `list-patches --category security`,
  and `zypper needs-rebooting`.
- pacman: `checkupdates` from pacman-contrib, which install-deps.sh now
  installs. pacman has no security notion; a reboot is required once
  the running kernel's /usr/lib/modules/<release> is gone.
The checks sit in one UPDATE_CHECKS table, and a failure names its
manager.

The root helper's update runs on the same hosts (#5804 "The scheduled
update runs on pacman, zypper and yum hosts"): yum upgrade, zypper
refresh + update (dup on Tumbleweed, with 102/103 counted as success),
pacman -Syu. Each step carries a label so a failure says which one,
and the reboot check reads zypper and the pacman kernel too.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-15 11:47:13 -04:00
bvandeusenandClaude Opus 5.5 1878f91293 chore: the systemd unit runs Steward, not FabledNetMon
CI / lint (push) Successful in 2s
CI / integration (push) Successful in 1m27s
CI / unit (push) Successful in 1m37s
CI / publish (push) Successful in 15s
systemd/fablednetmon.service still carried the project's first name in
its user, paths, command and config location, so it couldn't start
Steward as written. It is now systemd/steward.service: user steward,
/opt/steward, `steward --config /etc/steward/config.yaml`, with a note
that /data must be writable (secret.key, safety backups) and why
Restart=on-failure brings Steward back after a web UI restore (exit 3).

The legacy-name guard now reads service units too, and forbids
"netmon" beside "scryer" and "roundtable". The 2026-04 rebrand plan and
spec under docs/superpowers keep the old names: they record the rename.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-15 11:35:41 -04:00
bvandeusenandClaude Opus 5.5 8569b3d674 fix(backup): the safety backup's audit row names who restored (#5820)
CI / lint (push) Successful in 2s
CI / integration (push) Successful in 1m28s
CI / unit (push) Successful in 1m39s
CI / publish (push) Successful in 1m10s
On the live restore the "before restore" backup.made row read user
"unknown": _safety_backup recorded it with no username. It now carries
the person who started the restore.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-15 11:32:08 -04:00
bvandeusenandClaude Opus 5.5 e445c83e67 feat(backup): restore at first-run setup, restore --in-place, docs lead with the UI (#5818, #5819)
CI / integration (push) Successful in 1m28s
CI / unit (push) Successful in 1m37s
CI / publish (push) Successful in 1m17s
CI / lint (push) Successful in 2s
#5818 "First-run setup: Restore from a backup, by upload": beside creating
the first admin, setup offers "Or restore from a backup". Only while no
user exists; the upload is checked (a newer backup or one whose key can't
open its secrets is refused and not kept), then restored with no safety
backup (nothing to keep), and Steward restarts into the backup's users.

#5819 "Docs and CI": backups.md leads with restoring from Settings →
Backups and from setup; `steward restore --in-place` does the same-schema
restore from the shell, and CI's image check now uses it with a real
pg_restore: a changed setting comes back and the audit log stays.

Also exempts errors/restoring.html from the breadcrumb guard, as with
the other whole-app error pages: it replaces every page while a restore
runs, so there is nowhere to trail back to.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-15 11:05:39 -04:00
bvandeusenandClaude Opus 5.5 fb99f8b551 feat(backup): restore from Settings → Backups, with restoring mode (#5816, #5817)
CI / lint (push) Successful in 2s
CI / integration (push) Successful in 1m27s
CI / unit (push) Failing after 1m37s
CI / publish (push) Skipped
#5816 "Restoring mode": while a backup goes back in, the scheduler holds,
ingest and HTMX answer 503 with Retry-After (agents keep their samples),
and every page says what is happening. A safety backup of Steward as it
stands is made first, to /data/backups and the bucket. The backup's key
is put in place, the restore audited, and Steward exits with code 3 so
Swarm, compose or systemd starts it fresh. A failed in-place restore
changes nothing and says why on Backups; a full replace that fails after
the wipe restarts into first-run setup.

#5817 "Settings → Backups: Restore": a Restore button on each sent backup
and on the file card. The chosen backup is checked (the same comparison
as Check, now a shared partial) and held server-side for an hour under a
token; it starts only once the admin types "restore", and an older
backup also needs "Replace everything" ticked. A newer backup, or one
whose key can't open its secrets, gets no button.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-15 11:03:01 -04:00
bvandeusenandClaude Opus 5.5 9e5f6e183a fix(backup): refuse another schema before reading the dump; tests read settings directly (#5815)
CI / lint (push) Successful in 3s
CI / integration (push) Successful in 1m25s
CI / unit (push) Successful in 1m39s
CI / publish (push) Successful in 1m11s
- restore_in_place checks the backup's migration heads before pg_restore
  writes anything, so a backup from another schema is refused at once.
- The integration tests read their own setting rows directly:
  get_all_settings knows only Steward's own keys.
- check.py: drop an import left unused by the move to pgdata.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-15 10:56:57 -04:00