18032a4 re-checked a "not back in service" job every minute with no end,
so any later swarm wobble on the host (radarr's failed task during the
downloads move, say) flipped maintenance_failed back to 1: five false
firings and ten emails for Docker02 overnight on 10-19. And the overview's
finding never learned the rule at all, so it still said Docker02's
maintenance failed three days on.
The job now keeps came_back_at, set the first time its host is found back.
maintenance.still_failed(job) is the one answer: the figure, the overview,
the cadence page's run rows and the host panel all ask it. A failed update
or a timed-out job still flags its host until a job succeeds.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The host page gains a "Who's sending" panel, and the MCP a tool, that
put the host's outbound (or inbound) traffic down to its containers and
NFS mounts: read in shared time buckets, so the host's busiest bucket
says who was sending in it. A swarm service's tasks add up under its
name; what none of them accounts for is the host itself.
Also fixes CI 9488: per-container net_rx_bps / net_tx_bps join the
docker alert catalog, so a rule can catch a container flooding the
uplink.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
On 2026-10-18 an update on VDNT-Docker02 restarted the Docker daemon six
seconds in, before the drain that came only with the reboot, and every
container on the node stopped. Docker's own packages restart the daemon from
their install scripts; needrestart's list-only mode doesn't stop that.
- The helper gains a `check` action (agent 1.24.0): a dry run of the update
for each package manager, whether Docker is among it, and the running
containers outside the swarm with their restart policies.
- A swarm node is drained before an update that upgrades Docker, else only
before a reboot, and at most once a job.
- After a drain the job settles 5 minutes before the update or reboot.
- Once the work is over, the containers that ran outside the swarm must be
seen running again within 10 minutes, or the job fails naming them.
- An older agent gets no check and its job runs as before.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
get_anomaly's circumstances gain the host's or SNMP device's syslog
warnings and worse (mcp_events / mcp_resource_events). An anomaly row opens
"Syslog around it", read only when opened. A shared Chart.js plugin
(chart-marks.js) draws a dashed line at each syslog error on the host
metrics charts and an SNMP device's charts, with a line saying how many.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each sample carries each NFS mount's requests since the last one: how many
a second, their round trip and execute time (READ and WRITE on their own),
and how many were sent again or hit a major timeout. mountstats is the
kernel's own count and never touches the share, so it reads while one is
stuck. Steward stores them per mount and the slowest on the host as
nfs_exe_max_ms, for an "NFS slow" rule and the host page's NFS chart beside
mounts_stalled; each mount's bar says what a request took.
Agent 1.22.0 (which also carries the backlog order from 41fc807).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The root helper gains one read-only action, `drives`: it finds the drives
itself (smartctl --scan-open), reads each as JSON, and writes a summary the
agent carries on every sample (agent 1.20.0). Steward judges the figures in
drives.py (failing / worn / watch / ok / no SMART / unreadable), records them
per drive and as host roll-ups for covering rules, keeps the newest report on
the registration and lists it on the host page.
install-deps.sh installs smartmontools with apt, dnf, yum, zypper, pacman or
apk, run by install.sh and every agent playbook; one it can't install is
reported, never fatal.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A board's spare thermistor inputs report whatever an open input reads, and
the value rule from #5759 only catches the impossible ones. Once an hour,
sensors.refresh judges each sensor against the host's CPU (Package id N,
Tctl/Tdie/TccdN, PECI Agent N) over a week of hourly averages. A sensor is
ruled out if it falls as the CPU warms (r <= -0.5) or holds within 0.5 °C
while the CPU moved 10 °C or more. A judgment needs at least 72 shared hours.
The host temperature leaves judged sensors out at ingest. The host page
lists them under "Not counted", with each reason given once.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- #5780 "The error-pages service is graded on other routers' errors". The
Traefik scrape reads which services an errors middleware sends to, and
grades them healthy: their 4xx and 5xx are the codes of the errors they
cover for, already graded on the routers that failed.
- #5781 "The Maintenance failed rules sit at no data until a host has had
a maintenance job". Every host with an agent reads 0 until a job fails.
- #5782 "VDNT-Util01's host page lists SYSTIN at 115 °C in red". The host
page lists sensors that counts_as_hottest leaves out (unwired board
inputs, 0 °C or below) on one muted line, not as heat.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5768 "A maintenance cadence: a group's updates and reboots, one host at a
time, on a schedule". A plan names an inventory group, a weekday and time
in a time zone, a window and a reboot policy (only when required by
default). At its slot a run takes the group's hosts in order and gives each
a #5767 maintenance job. The next host starts only when the last is back in
service: its agent is posting and, for a swarm node, the node is active and
every replicated service is at its replicas. A failure stops the run, and
no host is started once the window has passed. A host's latest failure
shows as host_agent/<host>/maintenance_failed, for an ordinary alert rule.
The page is Hosts → Maintenance.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5767 "Steward hands a host a maintenance job and follows it to the end,
through its own restarts", milestone 514.
A host_maintenance_jobs row per job: queued → updating → [draining →]
rebooting → [undraining →] done, or failed / timed_out. Steward's answer to
an ingest post carries the step that host's agent should carry out now; the
agent (1.18.0) leaves each step id for the root helper once, and the helper
refuses an id it already ran, so repeats are harmless. Results come back as
the #5766 maintenance events, matched by step id; the reboot completes when
the host's `started` event has a boot_at after the reboot was handed out.
The job reboots only when the update says one is needed (or never, if asked).
A swarm node is drained first by itself if it's a manager or by a manager
with the helper, and made active after; with no manager to drain it the job
is refused. A minute tick times out a job past its state's deadline. A
maintenance window over host_agent/<host> lasts the rest of the job.
Everything is on the row, so a Steward restart, or the reboot of the host
Steward runs on, resumes from the next post. Ingest runs it in its own
savepoint so a job going wrong never costs the samples.
The Agent panel leads with a running job and says what a finished one did;
Manage agent has Update now (with "never reboot"), or says why a job can't
start. Integration test: hand off, Steward restarts, the post-boot events
close the job.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5766 "A root helper the agent can start, for a fixed set of maintenance
actions and nothing else", milestone 514.
The agent runs unprivileged under NoNewPrivileges, so it can't update or
reboot its host. It now leaves a request in /var/lib/steward-agent/requests,
the one path its unit lets it write; steward-maint.path starts maint.py as
root, which reads it without following links, accepts only update, reboot,
drain or undrain, deletes it, and writes the outcome to
/var/lib/steward-maint/results. The agent (1.17.0) reports each new result
state once as a `maintenance` event, re-checks pending updates when an update
finishes, and says in its metadata whether the helper is installed.
The units and the helper are served from one copy at
/plugins/host_agent/files/<name> (a fixed list), and install.sh plus the
install, provision and update playbooks all fetch them, replacing three
inline copies of the unit. Update agent installs the helper on existing
hosts; the Agent panel says under Manage agent whether it is there.
Steward handing out jobs is #5767; the cadence is #5768.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Step 1 of milestone 514 "Updates and reboots on a cadence, carried out by
each host's agent…": #5765 "The agent reports pending updates and whether
a reboot is required".
- Agent 1.16.0: an hourly background check (apt-check, apt list, or dnf -C
with updateinfo and needs-restarting), each command bounded at 120 s, no
root and no network. Debian's reboot-required marker and its packages
are read every sample.
- Ingest records updates_pending, updates_security and reboot_required for
rules, and keeps the newest on the registration (host_agent_005_updates).
- The Agent card leads with "Reboot required since …", then the updates
waiting or "Up to date", with when the check ran. list_agents carries
them too.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5756 "The agent says when its host rebooted and when it lost contact, so
downtime is recorded as what it was". Agent 1.14.0 posts events with the
next post that gets through: started (with the host's boot time), stopping
(on SIGTERM, with whether systemd says the host is going down) and
lost_contact (from its first unanswered post to the one carrying it).
Steward keeps them as the host's downtime (host_downtime): reboot,
agent_stop, unclean (a boot after the agent was last heard that nobody
announced) or lost_contact. Retried posts apply once.
An announced stop explains the silence for reboot_grace_seconds (Settings
-> Plugins -> host_agent, 10 minutes): seconds_since_seen reads 0 until
then, and the host's other rules go quiet with it (agents_away beside
#5721's registry). Past the grace, agent offline fires.
The host's Agent card lists the last 7 days of downtime. Rows are pruned
at data.retention_days.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
VDNT-Util01's / ran out of inodes (956 free of 60.8M) with 274G of space
left; every file create failed and Steward's page read healthy at 66%.
- Agent 1.13.0 reports each mount's inodes (statvfs; a filesystem reporting
none, like btrfs, is left without).
- Ingest records inodes_used_pct per mount, and disk_used_pct_worst is the
fullest mount by space or inodes, so existing disk-full rules and the
overview see it.
- The host's health strip, the Hosts widget and the host glance grade / by
whichever is fuller and name inodes when they are ("Disk / 66% · inodes
100%", "Inodes on / ... past its critical threshold").
Hosts pick up the inode figures once their agent is updated to 1.13.0.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
CI 9128: the host vitals and monitor templates required the new grades, so
callers (and tests) that render them without one failed; they now fall back
to status_class. The new integration test read outside begin(), which
autobegan the session.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Host CPU, memory and load per core, container CPU and memory, and every
monitor's response time learn a normal per hour (milestone 513, System 182).
A reading past its fixed cutoff but within its normal reads healthy and says
what is usual at this hour; one well out of it keeps its grade and is a
metric anomaly. Disk, a monitor down and an agent offline keep their cutoffs.
- Each normal's floor, the smallest departure that counts, is a setting:
Settings -> Thresholds -> Normals (norms.floor.*). Defaults: host CPU 10
points, host memory 5, load per core 10, container CPU 10 points of one
core, container memory 5, monitor response 100 ms; Traefik's 2-point 4xx
and 50 ms p95 floors move there from constants. learn_normal takes a
floor function (floor_setting) read at each pass.
- learn_normal(parts=False): a host learns, its cores don't.
- Ingest records load_pct (load_1m / cores x 100), the figure pages grade.
- Shared grading: anomalies.against / graded, worst_reading(normals=),
hosts' host_grades, docker usage(thresholds=, norms=), monitors'
latency_grade. Applied to host vitals, the Hosts widget, MCP overview,
Docker usage, and the monitor list, widget and page.
- The monitor and container pages list their anomalies.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Agent 1.12.0 stamps each post with sent_at; ingest compares it with the
moment the post arrived: clock_offset_s / clock_checked_at on the
registration (host_agent_003), and a clock_off_secs metric (either way)
that alert rules can watch. Replaces the old skew log, which measured the
newest sample and was off by up to an interval.
- Settings -> Thresholds: Clock drift, 30 s warn / 5 min critical.
- Host page: 'clock 0.3s ahead' in the vitals line, toned past the warning.
- Hosts list: names hosts whose clock is off; when every reporting host is
off by about the same amount, says Steward's own clock is the likely one.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
/ansible/inventory/ shows every target (address, its linked Steward host,
its groups) and every group (members, the schedules aimed at it) under a
verdict line. The whole row opens the target or group. Add target and
Add group open small dialogs; a failed add comes back with its dialog
open and what was typed. The red x leaves the rows: Delete is on the
target's or group's page, behind the confirm dialog, saying which
schedules would be left with nothing to run on.
The two list pages (/inventory/targets, /inventory/groups) are gone
with no redirect (rule 122); their URLs keep only the Add POST.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5290 "Host page: the health tiles span the card's width; even spacing above Ansible":
the four tiles take equal quarters (halves on a phone) and their sparklines stretch
across them; the Monitors + Agent row drops the Agent card's own margin, so the gap
above Ansible is 1rem like every other.
#5291 "Add host opens as a dialog over the Hosts list": a native <dialog> on /hosts/,
opened by Add host or /hosts/?add=1. A name clash re-renders the list with it open,
the error and what was typed. GET /hosts/new is gone; hosts/form.html is Edit only.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Hosts list:
- CPU and Memory are their own labelled columns (#5279); a silent
agent's last readings are greyed, and Status says "agent silent for 7m".
- One uptime figure, 30 days, with 24h and 7d on hover. The SLA tab
(/hosts/uptime) repeated it and is removed (#5280; one home per fact).
- Status is a word with how long ("Down for 12m"); host_status now carries
down_since. Edit and Delete live on the host's page only (#5281).
- Figures are right-aligned with tabular digits (.table .num), not a
monospace face.
Agent fleet (#5282): its add-host form is gone, so hosts are added only on
Hosts -> Add host. That form was also the only way to get a token for a
manual curl install, so that moves to the host's Agent card as "Install
manually" (POST /fleet/<host_id>/register), shown on the fleet page as
before. "Delete" there reads "Unregister".
Add host (#5283): laid out like the add-monitor form; a name clash comes
back inline with what was typed instead of a bare text 400; the address
is optional, as the host page already allowed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Docker panel (#5276): problems first, then the 8 busiest running
containers by CPU; the other running and the stopped fold behind one
"+N more running · M stopped" line. Containers that are not swarm tasks
read the way the Docker page reads them: CI jobs as "CI: workflow · job",
buildx builders as "BuildKit builder · name".
Monitors card (#5277): the edit/delete icon buttons and the inline add
form are gone. Each row links to its monitor's page; "Add monitor" opens
the add form filed under this host, starting from its address, with the
host in its breadcrumbs and Cancel back to it.
Agent card (#5278): Full metrics stays in view; Update, Rotate token,
Unregister agent (was "Remove") and Re-provision sit under "Manage agent".
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- New installs start from widgets.RECOMMENDED, packed row by row for the
enabled plugins; the edit drawer can reset a dashboard to it.
- Host Agent — Resources folds into Hosts — Overview (which gains load 1m),
UniFi — Devices into UniFi — Overview; migration 0032 moves existing
dashboards onto the survivors.
- The Overview labels what it counts (hosts, monitors up/down/pending,
alerts firing) and lists firing alerts and down monitors with since-times.
- Uptime polls every 5 minutes (poll_seconds) instead of every cycle.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5210 "Problems first, with since-time, across every list" (milestone 477),
second batch.
- Host page: an "Alerts on this host" strip first (rules on its agent
series, its containers or its monitors: alert_svc.watches_host); its
monitors problems first with "Down 12m · reason"; Docker above Ansible.
- Uptime page and widget: worst 24h uptime first, then 7d and 30d; hosts
with no uptime yet last.
- Agent panel and fleet widget: last seen as "3m ago", not a clock time.
- SNMP: a device is "Reachable" only while it answered within the last 3
polls; older readings say "No answer" with their age, and every device
says when it was last read (was: reachable whenever any reading existed).
- Traefik: a router missing from 3 scrapes in a row is listed apart as no
longer reported, by when it was last seen, instead of keeping its last
state (a removed router stayed Failing).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Traefik: routers are graded from their latest scrape (router_state: crit on
any real 5xx share, warn on p95 past the latency critical cutoff or a high
4xx share). Failing and slow routers lead as cards that say why; healthy
ones fold into one table that stays open or shut across polls. Proxy
counts are stat tiles; certificates are graded by Settings -> Thresholds
(tls_days).
UniFi: WAN health is a stat strip (latency graded by the thresholds),
devices carry status dots, event categories and firewall actions are
badges, the alarm count reuses .count-badge (renamed from .nav-count).
SNMP: each device wears a status badge and its readings are stat tiles.
Agent fleet: the table comes first, with a Status column (Reporting /
Silent / Never reported, by the host page's stale_after_seconds), silent
agents first, hosts linked, last seen as |ago.
Audit: entries name their entity (one query per type, steward/audit/
service.py) and link it; something deleted since keeps the name its entry
recorded and says "(deleted)".
Tokens: charts read colours through chart-colors.js (stewardColor resolves
a token to the rgb the browser computed; Chart.js defaults take the page's
text and border colours), so host, history, Traefik and SNMP charts drop
their hex. Template hex colours reach 0 (HEX_CEILING = 0), the last rgba
pills go, and heavy weights (600/700/bold, 60 of them) become 500, the
house pair being 400/500; a guard keeps templates there.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The host page's vitals strip becomes its health header: Live or Stale,
then one sentence — "Memory 93% is past its critical threshold", "Every
reading is within its thresholds", or, for a stale host, "No report since
20m ago; the readings below are from then" — and last seen as |ago. CPU,
memory, root disk and load per core follow as stat tiles graded by
Settings → Thresholds, with their sparklines.
The full-metrics page drew its bars and gauges at a hard-coded 70/90
(temperatures 70/85) while every other page used the configured cutoffs
(80/90 by default), so a 75 % CPU was amber here and fine everywhere else.
bar() now takes the metric kind and colours its .meter-fill through
status_class; gauges, temperatures and per-core bars do the same.
The host's monitor rows move onto the Monitors list's shapes: a status dot,
the name linking to the monitor's page, Ping/TCP/DNS/HTTP, pill-up/
pill-down, latency and uptime through status_class, labelled icon buttons.
HEX_CEILING drops to 54.
New formatting.worst_reading(readings, thresholds) picks the reading
furthest past its cutoff, the first of equals winning.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5099 "A name containing a quote runs script when an admin clicks Delete".
Inline handlers like onclick="return confirm('Delete {{ rule.name }}?')" are
HTML-decoded before the JavaScript is parsed, so autoescaping did not stop a
name such as x');fetch(…);// from running.
- Every prompt (22 handlers, plus the docker prune and token revoke ones)
is now data-confirm="…", read through dataset by one submit listener in
base.html; nothing rendered is parsed as JavaScript.
- The remaining inline handlers take integers only (|int), and the time
range buttons pass this.dataset.range.
- tests/test_inline_handlers.py fails on any {{ }} in an inline handler that
is not |int, with a self-test; tests/_templates.py is the one template
walker the template guards now share.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5093 "Shared format filters and status_class driven by Settings → Thresholds".
- steward/core/formatting.py: human_bytes (binary units), human_rate,
human_duration (two largest units), ago (a <time> titled with the absolute
UTC time) and status_class(value, kind, thresholds) -> ok/warn/crit/unknown,
which raises on a kind the thresholds don't name.
- app.py registers them as the bytes, rate, duration and ago filters and the
status_class global; threshold_style is now graded by status_class (and caps
its weight at 500).
- Replaces the copies: docker's _human_bytes/_human_uptime, core alerts'
_duration, the host-agent fmt_bps/fmt_bytes macros, the Traefik and UniFi
byte-rate arithmetic, the swarm-failures ago macro and two uptime spans.
- Tests: a table per helper, and every template compiled against the app's
filters, so an unknown filter fails CI rather than a page.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5092 "Tokens". base.html's inline palette was its own: white text on the
gold button (1.81:1), a red at 3.26:1 on a card, a warning yellow at hue
42 beside the gold accent at 46, a lavender page title, serif labels at
13px and weights up to 700.
- static/theme.css is design system 6's generated sheet (FabledSword + the
Steward accent). static/steward.css is base.html's stylesheet moved out,
with :root now aliases onto those tokens (--bg, --text-muted, --green…)
so every template keeps rendering; it states no colour of its own.
- Status text colours are Steward overrides recorded in the design system
(tokens 182-184): success #5FB87A, warning #E08A3C, error #EC7A6C. The
house -fg mixes read "up" as grey hint text (dE 8.2) and warning as the
accent (dE 11.5); these are >= 4.5:1 on page, card, hover and their own
15-25% tints. --orange is now the midpoint of warning and error.
- Primary button and active range button: dark text on gold (7.82:1).
Danger button on the house destructive fill. Badges and alerts take the
-fg colour on a tint; the gold badge uses --fs-accent-fg.
- Type: Inter body, Fraunces only for page titles and h1/h2, JetBrains
Mono for code; section and widget titles become the tiny caps label in
the body face; weights capped at 500.
- Removed: the lavender title and share-view gradient, #606080, and the
undefined --card-bg, --accent-faint, --accent-bg, --bg-alt, --border-low
and --row-alt (now real tokens); the dead settings/index.html.
- Guards (tests/test_style_tokens.py): every var(--x) used is defined;
steward.css states no colour or weight over 500; hex colours in
templates may only go down (ceiling 75, from 81 before this change),
each with a self-test that it can fail.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5033 "Share links open only their own dashboard; every route without a
role is on an explicit allow-list". Any live share token in ?s= made the
caller a viewer on every viewer route — host pages, container logs, any
dashboard — and the host-agent widget routes had no role check at all.
- steward/dashboard/share.py: a share token opens, read-only, exactly the
widget fragment URLs its own dashboard loads — same path, same
configured parameters; only the time window (hours/range) may change.
- require_role marks its view; public_route marks the deliberately open
ones (login, setup, health, the share page, agent ingest, install.sh,
agent.py) with the reason. A before_request refuses any view with
neither, so a route that forgets its decorator is closed, not open.
- /plugins/host_agent/widget and /widget/history now require viewer.
- The history widget's range buttons keep the rest of the query (and so
a share link's token) instead of rebuilding it; the uptime widget's
"Full SLA" link shows to logged-in users, not to share viewers.
- tests/test_route_access.py: a source scan that fails on any unmarked
route in core or bundled plugins, the default-deny refusal, and the
share matcher's cases.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5031 "Forged-request protection: SameSite cookie, per-session CSRF token
on every non-GET, HTMX header". A page on another site could make a
logged-in browser POST to Steward with the session cookie attached.
- steward/auth/csrf.py: a per-session token, checked in one before_request
for every POST/PUT/PATCH/DELETE (header X-CSRF-Token or form field
csrf_token, constant-time compare). The bearer-authenticated host-agent
ingest is exempt; /mcp never reaches Quart.
- Every POST form renders {{ csrf_field() }}; base.html carries the token
in a meta tag and adds it to every HTMX request; the dashboard's fetch()
calls send it.
- The session cookie is SameSite=Lax; login issues a fresh token.
- PUBLIC_BASE_URL is now loaded from Settings at startup (it was only set
when Settings → General was saved).
- tests/test_csrf.py: refusal, acceptance, rotation, the ingest exemption,
and a guard that every POST form in the templates carries the field.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-side of Docker container-log collection (milestone 79). Each interval the
agent fetches new log lines per running container over the local Docker socket,
demuxes the multiplexed stream, and folds sample["docker_logs"] into the same
push as metrics — no inbound channel needed.
- since-cursor per container kept in rate-state; a container's first interval
seeds from a short tail, then fetches incrementally via ?since=<cursor>
- _docker_request_raw + _demux_docker_logs + RFC3339Nano ts parsing (stdlib,
TTY/unframed fallback); one row per (stream, ts, line)
- per-batch byte cap with a truncation marker; logs are dropped (never buffered)
across a backoff so an outage keeps metrics but not stale logs
- docker_logs_enabled (default on) + docker_log_exclude per-host config keys
- AGENT_VERSION 1.6.0 -> 1.7.0
Server ignores the new sample key until the ingest step lands, so this commit is
CI-safe on its own.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CAGR73DUowdVFVvYzLXC5C
Realizes the chosen host-summary layout: a thin live vitals bar at the very top,
separate from the agent management panel (no more duplicated CPU/MEM/DISK/LOAD).
- New fragment _host_vitals.html + route /plugins/host_agent/vitals/<id>: compact
CPU / Memory / Disk(/) / Load-per-core (threshold-coloured + sparkline) +
Pressure + live/stale·version·last-seen. Polled every 15s; renders nothing
until the agent reports.
- The metric computation (latest snapshot + 6h sparkline query + load/core + PSI)
moves from host_panel into host_vitals. host_panel slims to management only
(reg/target/reporting/stale/ansible) and no longer queries metrics; panel.html
drops the gauge row + pressure block, keeping status + lifecycle actions.
- hosts/detail.html: vitals strip on top (full width, live), then a 2-col
[Monitors | Agent] grid, Ansible full width below, docker fragment last.
UI only. Templates parse; plugin-template parse test covers the new fragment.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC
The live refresh re-created the Chart.js charts each poll, which blanked the
canvases and re-ran the grow-in animation — a jarring flash/jump every cycle.
Make the canvases persistent in the page shell and poll only the data:
- The 3 chart canvases + their Chart instances are created once in the shell,
with animation disabled.
- /charts is now a data-only fragment swapped into a hidden div; it calls
window.applyHostSeries(series, range), which sets each dataset's data and
calls chart.update("none") — in-place, no re-create, no animation, fixed
height. Range labels update via .hm-chart-range spans.
- Current-state fragment keeps its atomic innerHTML poll into fixed-height
cards, so numbers update without a layout jump.
Net: live updates morph smoothly with no flicker or layout shift.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC
The full-metrics page rendered once server-side with the history query inline,
so the charts blocked first paint and nothing refreshed without a reload (it
reads the latest snapshot the server holds — the agent pushes ~every 30s).
Split it into a shell + two HTMX fragments:
- Shell (host_detail.html): header + shared time-range toggle + two containers;
paints instantly.
- Current state (/<id>/metrics → _host_metrics.html): identity + gauges +
per-core + filesystems + interfaces/disks + temps, from the DISTINCT ON latest
query. hx-trigger "load, every 15s" → live numbers at ~the agent cadence.
- History charts (/<id>/charts → _host_charts.html): the 3 charts + Chart.js,
from the date_bin history query. hx-trigger "load, every 60s, rangeChange" so
they lazy-load (never block paint), refresh slowly, and follow the range
selector. Leak-safe: previous Chart instances are destroyed before re-render.
- Range toggle switched from full-reload links to the shared _time_range.html
(setTimeRange + rangeChange), so only the fragments refetch.
- routes: host_detail (shell) + host_detail_metrics + host_detail_charts, with a
_split_host_metrics helper.
- tests/test_templates_parse.py now also parses plugin templates (these
fragments aren't rendered in the unit lane).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC
Two issues on the host full-metrics view:
- fmt_bytes capped at GB, so large filesystems read "25369.0 GB" instead of
"24.8 TB". Add TB and PB tiers.
- Interfaces/Disks/Temperatures shared one 3-column grid, so all three cards
stretched to the Temperatures height — a 40-core CPU made one very tall column
and left Interfaces/Disks with large empty space. Split them: Interfaces +
Disks sit side-by-side at natural height (align-items:start); Temperatures
becomes a full-width card whose readings flow into a compact multi-column grid
(wide-and-short instead of one tall column).
UI-only.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC
The page-top chrome was inconsistent: most nested views used the breadcrumb
kicker + page-title pattern, but plugin sub-pages used ad-hoc "← back" links,
~11 pages stacked a breadcrumb AND a redundant ancestor back button, and two
(monitors/edit, hosts/uptime) had a back button but no breadcrumb. The
settings/plugin_detail page stacked all of it at once.
Unify on the breadcrumb-led model:
- settings/_tabs.html: drop the hardcoded "Settings" h1; the breadcrumb
("Settings › …") plus the tab strip is the header.
- settings/plugin_detail: drop the "← Plugins" back button.
- docker container_detail/swarm/disk + snmp/device: replace ad-hoc back links
with the standard crumbs() breadcrumb.
- host_agent, ansible/*, alerts/maintenance: remove redundant ancestor back
buttons (the breadcrumb's parent crumbs already link there); keep lateral
shortcuts (Inventory/Schedules/Browse/Targets/Groups/New).
- monitors/edit, hosts/uptime: add the missing breadcrumb, drop the back link.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC
Move the hardcoded warn/crit cutoffs into Settings -> Thresholds (DB-backed,
live, no restart). New thresholds.* keys + to_thresholds_cfg() + a
threshold_style(value, kind) jinja global that reads them; latency reuses the
existing ping good/warn keys, uptime is direction-aware (floors).
Replace the _macros metric_style/uptime_style macros (now removed) with the
global across Hosts-Overview, host_agent fleet + panel, Uptime/SLA widget, and
the ping page uptime column — all now honor the configured cutoffs. Uptime keeps
its green 'good' look when not degraded.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add a live range switch to the history widget that re-requests the fragment
without entering edit mode. It rewrites the parent cell's hx-get so the choice
survives polling (htmx re-reads the attribute each poll), then fetches at once.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The agent already collects + ingests net throughput, disk I/O, and temps;
add them to the fleet-glance rows (fmt_bps + two-line io cells + temp with
70/85C threshold color), each shown only when the host reports it so VMs/
containers without sensors don't show blank cells.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Hosts-Overview: move the CPU sparkline inline next to the cpu % value it
represents (was floated far right on the name line, reading as unrelated).
- Host Agent history widget: caption the chart with the host it represents
(the panel title is generic) — links to the host hub.
- History widget: snap the y-axis to a stable 0..next-10%-band ceiling instead
of auto-scaling to the exact peak, so the 'zoom' no longer jumps each poll.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Restore the host_agent fleet-glance widget to one horizontal row per host
(name left, metric cells + sparklines right) instead of the multi-column
block grid, and zebra-stripe odd rows so adjacent hosts read as distinct.
Names still wrap rather than hard-truncate.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The agent reports every few seconds, so multi-hour history series were hundreds–
thousands of points — a dense, noisy line (esp. CPU). Bucket-average server-side
to ~120 points (keeps the shape, drops the noise) for both the history-graph
widget and the host-detail charts. Also fix the chart tooltip title showing the
raw epoch-ms (e.g. 1,781,720,471,459) — format it as HH:MM.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Address graphical issues raised from the dashboard screenshot:
- No more truncated host names. Hosts-Overview and Host-Agent-Resources put the
host name on its own line (full, wraps if needed) with its data grouped beneath
it; ping/dns (.ping-name) and uptime widget names wrap instead of ellipsis.
Prefer vertical overflow over cramming/truncating a row.
- History graph fills the panel: drop the fixed 0–100 y-axis ceiling (beginAtZero
+ 8% grace) so the lines use the vertical space instead of hugging the bottom;
axis labels still show real %.
- Container-query breakpoints: the widget body is now a query container, so
fragments restyle to their OWN panel width. Host-list widgets flow into 2 cols
≥520px and 3 cols ≥900px (.host-blocks) — use the width, remove vertical
deadspace — and collapse to one column when narrow. Mirrored into the share view.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Operator ask: show the graphs next to their fields in the fleet widget, like the
host page's AGENT panel. Each row now renders cpu/mem/disk/load as a value with a
trend sparkline beneath it (1h window), instead of bare numbers.
- _fleet_rows fetches a per-host recent series (cpu/mem/load host-level, disk from
the root mount) in one 1h query and attaches a sparkline per metric to each row.
- widget_table.html lays out a metric cell (label + threshold-coloured value +
sparkline) per field, mirroring panel.html. Threshold colour is computed in the
loop and passed into the cell macro (keeps Jinja macro scope clean).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Milestone 72 phase C — bring the host-view graphs onto the dashboard:
- host_resource_history widget reworked into a real host-view chart: epoch-ms
linear axis (no Chart.js date adapter), themed like the host-detail charts,
maintainAspectRatio:false so it fills the resized panel, unique canvas per
widget instance (wid), and empty states ("pick a host" / "no metrics yet").
Was previously unusable — it had a broken time axis and no way to choose a host.
- Add a "host" param type: the edit form renders a live dropdown of hosts
(dashboard routes now pass the host list to the editor); the chosen host_id is
stored in config and fed to the widget.
- Hosts-overview widget gains a per-row CPU sparkline (last hour) via the shared
sparkline_svg helper — the host-view at-a-glance trend, on the main widget.
Charts/sparklines render only in the browser, so CI can't exercise them — needs
an operator visual check.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Milestone 72 phase A (clarity wins, no schema change):
- Add shared metric_style() macro in _macros.html: colours a numeric metric
amber (>=warn) / red (>=crit) on the value ITSELF, not just the status dot.
Defaults 80/90 for percentage gauges; load /core uses warn 80 / crit 100.
Applied to CPU/mem/disk across host_agent panel + fleet widget + the unified
hosts-overview widget.
- Link every host reference to the host hub (/hosts/<id>) in ping, dns,
hosts-overview, host_agent fleet, and uptime widgets; status widget entries
link to their own detail_url. All guarded on session.user_id so the public
share view degrades to plain text (the bare href carries no share token).
- Fix truncation: title= tooltips on all names that ellipsis, plus a hover
underline affordance on the now-clickable ping-name links.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Host agent panel (reporting state) gains a "Re-provision" collapsible (admin +
linked target + managed key): bootstrap user/password → provision.yml, which
reinstalls the steward account + managed key + agent. This is the missing path
after regenerating the managed key — Update alone can't fix a broken key.
- Remove the Ansible sections (run-playbook + target link) from the host EDIT
page — they were the stale free-text version and the wrong place. They live on
the host detail hub (dropdown + discovered variables). Edit page now links to
the host page; edit_host route simplified (no ansible fetch).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Run UX: when start_run throws before/around launch (e.g. ENOSPC creating the
temp dir — the box is out of disk), the run was marked "failed" with empty
output. Now the exception is broadcast + written to the run output/results so
the run view shows e.g. "[run error] OSError: [Errno 28] No space left on
device" instead of a blank failure.
Widget audit follow-ups (no broken links were found; these are consistency):
- host_resource_history widget now charts root (/) disk, consistent with the
host panel (was the opaque "disk worst").
- host_resources widget: tooltip on the health dot explaining it warns on the
worst mount while the number shows root.
- status_overview widget detail_url /status → /status/ (avoid redirect).
- Normalize ad-hoc widget empty-states to the shared .empty style (wording,
which distinguishes "configured" vs "data yet", preserved).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Bring the dashboard in line with the unified host IA + improved displays.
- New core "Hosts — Overview" widget (/hosts/overview/widget): one row per host
combining monitor status (ping dot + latency, uptime 24h) with the agent
glance (CPU / memory / disk root + stale flag), each row linking to the host
hub. Reads agent data from the generic PluginMetric table via a core-safe
_agent_overview_by_host helper (no host_agent import); freshness vs the
plugin's stale window. The granular Ping/DNS/Uptime/Agent widgets stay.
- Group the add-widget picker into Core monitors / Monitoring capabilities /
Integrations (a `group` field on every WIDGET_REGISTRY entry + section
headings in _edit_panels.html), matching the Settings → Plugins taxonomy.
- Fix the agent fleet widget rows to link to the host hub (/hosts/<id>) instead
of the old /plugins/host_agent/<id>/ page.
Scribe #903.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Each at-a-glance metric (CPU, Memory, Disk /, Load) now shows a 6h sparkline
(reused core.status.sparkline_svg + a recent-series query; disk uses the root
mount sub-resource) so trend/consistency is visible, not just the instant.
- Load is now normalized: "Load /core" = 1m load ÷ CPU cores as % (100% = run
queue matches capacity), comparable across different hardware. Cores derived
from the per-core CPU metrics already collected — no agent change. Raw load
in the tooltip.
- Added a "Pressure 10s" line: PSI cpu/mem/io (some, avg10), the
hardware-independent saturation signal already collected by the agent.
Scribe #898.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
