CI 9128: the host vitals and monitor templates required the new grades, so
callers (and tests) that render them without one failed; they now fall back
to status_class. The new integration test read outside begin(), which
autobegan the session.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Host CPU, memory and load per core, container CPU and memory, and every
monitor's response time learn a normal per hour (milestone 513, System 182).
A reading past its fixed cutoff but within its normal reads healthy and says
what is usual at this hour; one well out of it keeps its grade and is a
metric anomaly. Disk, a monitor down and an agent offline keep their cutoffs.
- Each normal's floor, the smallest departure that counts, is a setting:
Settings -> Thresholds -> Normals (norms.floor.*). Defaults: host CPU 10
points, host memory 5, load per core 10, container CPU 10 points of one
core, container memory 5, monitor response 100 ms; Traefik's 2-point 4xx
and 50 ms p95 floors move there from constants. learn_normal takes a
floor function (floor_setting) read at each pass.
- learn_normal(parts=False): a host learns, its cores don't.
- Ingest records load_pct (load_1m / cores x 100), the figure pages grade.
- Shared grading: anomalies.against / graded, worst_reading(normals=),
hosts' host_grades, docker usage(thresholds=, norms=), monitors'
latency_grade. Applied to host vitals, the Hosts widget, MCP overview,
Docker usage, and the monitor list, widget and page.
- The monitor and container pages list their anomalies.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
At 20:25-20:39Z the NFS storage under steward_db slowed badly (checkpoint
sync 4.6 s, then 70.7 s with one file at 21.9 s, against the usual 0.002 s).
Ingest commits stalled past the agent's 10 s timeout and were dropped, and
last_seen_at, written in that same transaction, went with them. Agents
posting every few seconds read as silent for 3 minutes, and "Agent offline"
fired and emailed for all three hosts.
Ingest now notes each agent's arrival in memory right after the (read-only)
registration lookup, before any write; seconds_since_seen counts from the
later of last_seen_at and that arrival.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5467 "Steward's own health: metrics, rules can watch them, a page that shows them".
The self-monitor tick now records, under source "steward", resource "process":
ingest_posts / ingest_failed / ingest_slowest_ms (the host-agent posts since the
last tick, tallied in steward/core/health.py), db_roundtrip_ms (a timed SELECT 1),
and unwatched_secs (the gap before this tick past watch.GAP_S, from Steward's own
last sample). The hourly cleanup records cleanup_secs. The open-file pair is still
recorded, and is now only skipped on its own where /proc is absent. All of these
are in CORE_METRICS, so an alert rule can watch them under "Steward itself".
Settings → System (/settings/system/, admin) shows a verdict first (failed posts,
a gap in watching, open files near the limit), then each measure now and over the
last day, with a sparkline scaled to its own peak.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
After Steward or its database was down, every agent's age and every
series' last sample were old, so all agent-offline rules fired and all
rules went no-data at once, clearing a minute later. With per-account
alert email that would have mailed everyone a dozen notices per restart.
A periodic job now ticks a Watch (steward/core/watch.py) once it has
reached the database; the Watch says since when runs have been unbroken
(process start, or the end of a gap over 120 s). seconds_since_seen and
the no-data sweep count silence from no earlier than that, so a source
must stay quiet while Steward is watching before it counts as silent.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Agent 1.12.0 stamps each post with sent_at; ingest compares it with the
moment the post arrived: clock_offset_s / clock_checked_at on the
registration (host_agent_003), and a clock_off_secs metric (either way)
that alert rules can watch. Replaces the old skew log, which measured the
newest sample and was off by up to an interval.
- Settings -> Thresholds: Clock drift, 30 s warn / 5 min critical.
- Host page: 'clock 0.3s ahead' in the vitals line, toned past the warning.
- Hosts list: names hosts whose clock is off; when every reporting host is
off by about the same amount, says Steward's own clock is the likely one.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
/ansible/inventory/ shows every target (address, its linked Steward host,
its groups) and every group (members, the schedules aimed at it) under a
verdict line. The whole row opens the target or group. Add target and
Add group open small dialogs; a failed add comes back with its dialog
open and what was typed. The red x leaves the rows: Delete is on the
target's or group's page, behind the confirm dialog, saying which
schedules would be left with nothing to run on.
The two list pages (/inventory/targets, /inventory/groups) are gone
with no redirect (rule 122); their URLs keep only the Add POST.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5290 "Host page: the health tiles span the card's width; even spacing above Ansible":
the four tiles take equal quarters (halves on a phone) and their sparklines stretch
across them; the Monitors + Agent row drops the Agent card's own margin, so the gap
above Ansible is 1rem like every other.
#5291 "Add host opens as a dialog over the Hosts list": a native <dialog> on /hosts/,
opened by Add host or /hosts/?add=1. A name clash re-renders the list with it open,
the error and what was typed. GET /hosts/new is gone; hosts/form.html is Edit only.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Hosts list:
- CPU and Memory are their own labelled columns (#5279); a silent
agent's last readings are greyed, and Status says "agent silent for 7m".
- One uptime figure, 30 days, with 24h and 7d on hover. The SLA tab
(/hosts/uptime) repeated it and is removed (#5280; one home per fact).
- Status is a word with how long ("Down for 12m"); host_status now carries
down_since. Edit and Delete live on the host's page only (#5281).
- Figures are right-aligned with tabular digits (.table .num), not a
monospace face.
Agent fleet (#5282): its add-host form is gone, so hosts are added only on
Hosts -> Add host. That form was also the only way to get a token for a
manual curl install, so that moves to the host's Agent card as "Install
manually" (POST /fleet/<host_id>/register), shown on the fleet page as
before. "Delete" there reads "Unregister".
Add host (#5283): laid out like the add-monitor form; a name clash comes
back inline with what was typed instead of a bare text 400; the address
is optional, as the host page already allowed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Docker panel (#5276): problems first, then the 8 busiest running
containers by CPU; the other running and the stopped fold behind one
"+N more running · M stopped" line. Containers that are not swarm tasks
read the way the Docker page reads them: CI jobs as "CI: workflow · job",
buildx builders as "BuildKit builder · name".
Monitors card (#5277): the edit/delete icon buttons and the inline add
form are gone. Each row links to its monitor's page; "Add monitor" opens
the add form filed under this host, starting from its address, with the
host in its breadcrumbs and Cancel back to it.
Agent card (#5278): Full metrics stays in view; Update, Rotate token,
Unregister agent (was "Remove") and Re-provision sit under "Manage agent".
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- New installs start from widgets.RECOMMENDED, packed row by row for the
enabled plugins; the edit drawer can reset a dashboard to it.
- Host Agent — Resources folds into Hosts — Overview (which gains load 1m),
UniFi — Devices into UniFi — Overview; migration 0032 moves existing
dashboards onto the survivors.
- The Overview labels what it counts (hosts, monitors up/down/pending,
alerts firing) and lists firing alerts and down monitors with since-times.
- Uptime polls every 5 minutes (poll_seconds) instead of every cycle.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5210 "Problems first, with since-time, across every list" (milestone 477),
second batch.
- Host page: an "Alerts on this host" strip first (rules on its agent
series, its containers or its monitors: alert_svc.watches_host); its
monitors problems first with "Down 12m · reason"; Docker above Ansible.
- Uptime page and widget: worst 24h uptime first, then 7d and 30d; hosts
with no uptime yet last.
- Agent panel and fleet widget: last seen as "3m ago", not a clock time.
- SNMP: a device is "Reachable" only while it answered within the last 3
polls; older readings say "No answer" with their age, and every device
says when it was last read (was: reachable whenever any reading existed).
- Traefik: a router missing from 3 scrapes in a row is listed apart as no
longer reported, by when it was last seen, instead of keeping its last
state (a removed router stayed Failing).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The dashboard's host widgets took 18-60 s. Every "latest reading" query
(hosts overview, host-agent fleet, host page, vitals, SNMP, MCP get_host)
searched plugin_metrics for the newest row per metric. Postgres has no skip
scan, so each one read every raw sample in the 7-day window.
- plugin_metric_latest: one row per (source, resource, metric).
record_metrics upserts it in the same transaction, moving forward only and
in sorted key order. Migration 0030 seeds it by stepping the existing index.
- Retention drops a latest row once its sample leaves the raw window, so a
resource that stopped reporting disappears when its history does, as before.
Renames carry it along.
- A quiet host keeps its last values, marked stale. A time-bounded query
would have shown it as missing.
- Vitals sparklines span the 6 h they claim, averaged in SQL to ~120 points,
instead of fetching 6 h raw and drawing the last 120 samples.
- Dashboard and share-view cells: "Loading…" until the first answer; a 20 s
deadline per fetch; "Couldn't load (why)" with Retry now and an automatic
retry for cells that don't poll; "as of HH:MM" in the header, and
"couldn't refresh · as of" when a refresh fails over shown data.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
MAX_SAMPLES_PER_INGEST was 2 while agent 1.10 drains a backlog five samples a
post, so the server dropped the oldest three of every such post. Posts now
answer in 0.4-1.3 s on dev, including the boot cleanup window, so five fit
the agent's 10 s. A test holds the server cap at or above the agent's.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent 1.11.0 sends finished_at (State.FinishedAt) and reads a stopped
container's logs once per stop: since its cursor if it was being read, or
its last 200 lines when first seen already crashed (non-zero exit, stopped
in the last 10 min). Cursors and finals are forgotten once a container is no
longer listed.
Ingest gives a container first seen already exited non-zero a die event, and
dates die/stop events by finished_at when the agent sends it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
persist_host_docker marks docker.status, .swarm, .disk, .logs, .metrics,
.events, .containers, .alerts and .reap on the ingest's phase timer, each
flushed before its mark so a part's writes are charged to it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Measured on dev (385a579), agent posts cut to five samples still dropped:
flush=8.8s for 990 rows, 6.6s for 680, 8.4s for 1000 — about 110 rows/s,
~9 ms a row, with rules and evaluation near zero. One random read per
row fits a random uuid4 primary key whose index no longer fits in memory:
every new row lands on a random page of it. (Inferred from the rate; the
table and index sizes have not been read.)
- steward.core.ids.time_ordered_id: a UUIDv7-layout string (48-bit ms
timestamp, then random), same 36-char shape as the stored uuid4 ids, so
no migration. Python's uuid7() arrives in 3.14; the image runs 3.13.
- Used for plugin_metrics and its hourly rollup, monitor_results,
audit_events, the docker metrics/hourly/events/logs tables, and the
traefik and unifi time series.
- MAX_SAMPLES_PER_INGEST 5 → 2 (~400 rows) for now: it fits the agent's
10 s even at the old rate, so hosts land data whichever way the new rate
comes out. Raise it once the new rate is measured.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- BACKOFF_CAP 300 → 60: Steward reads a host stale after 180 s, so an agent
backing off to five minutes read stopped long after Steward came back
(lesson #4355: a check-in inherits its call's backoff).
- A POST carries at most five samples: the oldest four buffered plus the
new one, the rest held for the next post. A backlog drains a few at a
time instead of in one post Steward cannot write before the timeout
(the loop measured in #5149).
- Per-container Docker work has a 20 s deadline over the whole fan-out.
Past it the sample reports docker_status "error" ("N of M containers
not read within 20 s") and no list: a partial list would read as
removals, and a record without its inspect as restarts.
- /system/df is asked every 600 s rather than every sample; Steward keeps
the last answer.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Measured on dev (f802c9f): every dropped ingest carried samples=21,
metric_rows≈4,200 and never finished the INSERT flush inside the agent's
10 s ("completed: parse=0.1s"). The rollback left the agent's ring buffer
full, so each later post was the same size and failed the same way: no
host's metrics have landed since.
An ingest now keeps the newest MAX_SAMPLES_PER_INGEST (5) samples by ts,
logs how many it left out, and returns samples_dropped. The older samples
are the ones the ring buffer would have overwritten anyway. Why the
database writes so few rows a second is not yet known; the slow-ingest
line now measures it on a post that finishes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Agent: started_at comes from inspect's State.StartedAt (nanoseconds
trimmed; Docker's zero time is "never started"), not the list's Created,
so a restarted container's uptime counts from its restart.
- A rising restart_count between two snapshots emits a "restart" event
("restart #3", or "3 restarts, now #5"), said instead of a start; a
restart then a death gives both. A count that falls (recreated under the
same name) says nothing. The timeline draws it as ↻.
- Network and disk I/O count what the kernel calls physical: interfaces
not under /sys/devices/virtual/net (bridges, docker0, veth, vxlan, tun),
block devices with a device behind them (not dm, md, zram), so a byte is
not counted twice. Where /sys calls everything virtual (an agent in LXC)
the name-filtered set is kept rather than reporting no traffic.
- Agent tests updated for #5152's (status, detail, containers) and
(is_manager, topology) shapes; the no-Docker disk test now states the
case it was passing for by accident.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The agent (1.9.0) said nothing when Docker could not be asked and nothing
when Docker had no containers, so Steward could tell neither apart from the
other, and kept a host's last containers on screen as running either way.
- Agent: collect_docker returns (status, detail, containers). "ok" sends
docker: [] when there are none; "unavailable" (no socket, nothing
listening) and "error" (permission, timeout, bad answer) say why.
collect_swarm says whether /info answered, and a node that is not a
manager sends swarm_manager: false. A failed /info says nothing.
- Ingest: an empty docker list is a snapshot; the newest sample's status
goes to the Docker plugin with the rest.
- docker_host_status (docker_011): per host, status, detail, since and
last report. A host that never had Docker gets no row.
- An "ok" empty snapshot reaps the host's containers; an error keeps them.
swarm_manager: false drops the host's swarm services and nodes.
- Docker page: a "Docker not answering" card names each such host, why,
and since when; its containers read as stale. Duplicate header comment
in rows.html removed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The first instrumented drop on dev read "dropped after 10.0s … (completed:
parse=0.1s)": the whole stall is inside record_metrics. It now marks its
three steps (flush of the rows, the rule and state reads, the per-sample
evaluation) on the caller's phases, and the drop line carries the request's
samples and metric row count.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every host-agent ingest in the service logs ends at ~10 s with no status:
the agent gives up at its urlopen timeout (agent.py) before Steward answers.
Nothing says which part of ingest is slow, so this measures it:
- each phase (parse, metrics with alert evaluation, docker persist,
commit) is timed, and a request past 2 s logs one line with the phases,
sample count, metric rows, containers and log lines;
- a request the server drops when the agent hangs up (CancelledError) logs
how long it ran and which phases completed, then re-raises.
The fix follows once the logs say which phase it is.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Traefik: routers are graded from their latest scrape (router_state: crit on
any real 5xx share, warn on p95 past the latency critical cutoff or a high
4xx share). Failing and slow routers lead as cards that say why; healthy
ones fold into one table that stays open or shut across polls. Proxy
counts are stat tiles; certificates are graded by Settings -> Thresholds
(tls_days).
UniFi: WAN health is a stat strip (latency graded by the thresholds),
devices carry status dots, event categories and firewall actions are
badges, the alarm count reuses .count-badge (renamed from .nav-count).
SNMP: each device wears a status badge and its readings are stat tiles.
Agent fleet: the table comes first, with a Status column (Reporting /
Silent / Never reported, by the host page's stale_after_seconds), silent
agents first, hosts linked, last seen as |ago.
Audit: entries name their entity (one query per type, steward/audit/
service.py) and link it; something deleted since keeps the name its entry
recorded and says "(deleted)".
Tokens: charts read colours through chart-colors.js (stewardColor resolves
a token to the rgb the browser computed; Chart.js defaults take the page's
text and border colours), so host, history, Traefik and SNMP charts drop
their hex. Template hex colours reach 0 (HEX_CEILING = 0), the last rgba
pills go, and heavy weights (600/700/bold, 60 of them) become 500, the
house pair being 400/500; a guard keeps templates there.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The host page's vitals strip becomes its health header: Live or Stale,
then one sentence — "Memory 93% is past its critical threshold", "Every
reading is within its thresholds", or, for a stale host, "No report since
20m ago; the readings below are from then" — and last seen as |ago. CPU,
memory, root disk and load per core follow as stat tiles graded by
Settings → Thresholds, with their sparklines.
The full-metrics page drew its bars and gauges at a hard-coded 70/90
(temperatures 70/85) while every other page used the configured cutoffs
(80/90 by default), so a 75 % CPU was amber here and fine everywhere else.
bar() now takes the metric kind and colours its .meter-fill through
status_class; gauges, temperatures and per-core bars do the same.
The host's monitor rows move onto the Monitors list's shapes: a status dot,
the name linking to the monitor's page, Ping/TCP/DNS/HTTP, pill-up/
pill-down, latency and uptime through status_class, labelled icon buttons.
HEX_CEILING drops to 54.
New formatting.worst_reading(readings, thresholds) picks the reading
furthest past its cutoff, the first of equals winning.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5099 "A name containing a quote runs script when an admin clicks Delete".
Inline handlers like onclick="return confirm('Delete {{ rule.name }}?')" are
HTML-decoded before the JavaScript is parsed, so autoescaping did not stop a
name such as x');fetch(…);// from running.
- Every prompt (22 handlers, plus the docker prune and token revoke ones)
is now data-confirm="…", read through dataset by one submit listener in
base.html; nothing rendered is parsed as JavaScript.
- The remaining inline handlers take integers only (|int), and the time
range buttons pass this.dataset.range.
- tests/test_inline_handlers.py fails on any {{ }} in an inline handler that
is not |int, with a self-test; tests/_templates.py is the one template
walker the template guards now share.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5093 "Shared format filters and status_class driven by Settings → Thresholds".
- steward/core/formatting.py: human_bytes (binary units), human_rate,
human_duration (two largest units), ago (a <time> titled with the absolute
UTC time) and status_class(value, kind, thresholds) -> ok/warn/crit/unknown,
which raises on a kind the thresholds don't name.
- app.py registers them as the bytes, rate, duration and ago filters and the
status_class global; threshold_style is now graded by status_class (and caps
its weight at 500).
- Replaces the copies: docker's _human_bytes/_human_uptime, core alerts'
_duration, the host-agent fmt_bps/fmt_bytes macros, the Traefik and UniFi
byte-rate arithmetic, the swarm-failures ago macro and two uptime spans.
- Tests: a table per helper, and every template compiled against the app's
filters, so an unknown filter fails CI rather than a page.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5092 "Tokens". base.html's inline palette was its own: white text on the
gold button (1.81:1), a red at 3.26:1 on a card, a warning yellow at hue
42 beside the gold accent at 46, a lavender page title, serif labels at
13px and weights up to 700.
- static/theme.css is design system 6's generated sheet (FabledSword + the
Steward accent). static/steward.css is base.html's stylesheet moved out,
with :root now aliases onto those tokens (--bg, --text-muted, --green…)
so every template keeps rendering; it states no colour of its own.
- Status text colours are Steward overrides recorded in the design system
(tokens 182-184): success #5FB87A, warning #E08A3C, error #EC7A6C. The
house -fg mixes read "up" as grey hint text (dE 8.2) and warning as the
accent (dE 11.5); these are >= 4.5:1 on page, card, hover and their own
15-25% tints. --orange is now the midpoint of warning and error.
- Primary button and active range button: dark text on gold (7.82:1).
Danger button on the house destructive fill. Badges and alerts take the
-fg colour on a tint; the gold badge uses --fs-accent-fg.
- Type: Inter body, Fraunces only for page titles and h1/h2, JetBrains
Mono for code; section and widget titles become the tiny caps label in
the body face; weights capped at 500.
- Removed: the lavender title and share-view gradient, #606080, and the
undefined --card-bg, --accent-faint, --accent-bg, --bg-alt, --border-low
and --row-alt (now real tokens); the dead settings/index.html.
- Guards (tests/test_style_tokens.py): every var(--x) used is defined;
steward.css states no colour or weight over 500; hex colours in
templates may only go down (ceiling 75, from 81 before this change),
each with a self-test that it can fail.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
CI 8325: NOW - 10 min is 11:50, not 09:50; and rule_fields' refusal lists
the catalog in order, so seconds_since_seen goes last.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent offline is an ordinary rule now. The host_agent scheduler records
host_agent/<host>/seconds_since_seen every minute (from last_seen_at, or
registration time for an agent that never reported), replacing the task
that only logged stale agents. The catalog offers the metric, and picking
it on a new rule fills "> 180".
No data: each rule's state tracks its last sample and its usual gap,
smoothed so that an outage counts as at most three gaps. A core task,
alert_silence_check, runs every 60s and marks a rule no_data_since once its
series has had no sample for three usual gaps (at least 5 minutes; an hour
while the cadence is unknown). A never-sampled rule counts from when it
started watching. The mark sits beside the state, so a firing alert that
goes silent stays firing. Each silence records one event and sends one
notification (rule option notify_no_data, default on; muted in maintenance
windows). The next sample clears it with a "data resumed" event and
notification. Creating, changing, disabling or re-enabling a rule restarts
its watch.
Migration 0029 adds the columns and seeds each existing rule's last sample
and gap from its last day of data, so an upgrade does not flag every rule.
Shown on the Alerts page (a NO DATA badge with its start time), in MCP
list_alerts, list_alert_rules and get_host (state "no_data", no_data_since),
in get_overview findings (alert_no_data, or a firing alert marked silent)
and in the weekly report. The rule form has the notify option, and its
metric reference now matches what traefik records and lists host_agent.
Emails say "no data" and carry the event's note.
The metric-catalog guard also reads positional MetricBatch.add arguments.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5061 "Metric catalog belongs to what emits: plugins register their
metrics; names match what is recorded".
- The static METRIC_CATALOG is gone. metric_catalog() is core's own sources
(the four monitor types and "steward", the app's open_fds/open_fds_pct,
which were recorded but never offered) plus each loaded plugin's
get_alert_metrics() export. A disabled plugin's sources leave the picker.
- traefik offers what it records: error_rate_4xx_pct and error_rate_5xx_pct.
The old "error_rate" was never recorded, so a rule on it could not fire.
- tests/core/test_metric_catalog.py walks every MetricBatch.add /
record_metric call (traefik's tuple loop included) and fails on a recorded
metric the catalog lacks; it shows on a synthetic writer that it can fail.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5060 "One evaluation path: record_metrics batch API; host-agent ingest
evaluates rules".
- record_metrics(session, rows) writes metric rows and evaluates the rules
that watch them, reading rules, states and maintenance windows once per
batch. Each rule sees its key's samples oldest first, so an agent catching
up after an outage counts consecutive checks as they happened.
- Host-agent ingest now goes through it. Before, it wrote plugin_metrics
directly and no host_agent rule could ever fire. The catalog gains the
per-mount disk_used_pct ("<host>:<mount>").
- MetricBatch gathers a transaction's samples; the docker ingest, traefik,
unifi, snmp, monitor and self-monitor writers record one batch each instead
of a rule query per sample.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5037 "Secrets at rest and in run output: webhook URL and git token
encrypted, askpass via env, secret values scrubbed from logs".
- webhook.url joins SECRET_KEYS; existing values are sealed at startup by
migrate_plaintext_secrets. Settings -> Notifications no longer echoes the
URL: it names the saved host, keeps it when left blank, and has a Remove
box.
- Git source http_token is sealed inside ansible.sources on save and at
startup, opened in get_sources, and counted by the undecryptable-secrets
scan as ansible.sources[<name>].http_token.
- The askpass script holds no token: git runs it with STEWARD_GIT_TOKEN in
its environment and printf prints it as data.
- The executor masks every secret the run is handed (become/vault
passwords, bootstrap passwords, sensitive run vars, and the per-host agent
tokens the host_agent plugin now passes as redact) in each line before it
is broadcast, logged or stored.
- Bundled "Write agent config" tasks are no_log; a test fails if any bundled
task templating steward_token lacks it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5033 "Share links open only their own dashboard; every route without a
role is on an explicit allow-list". Any live share token in ?s= made the
caller a viewer on every viewer route — host pages, container logs, any
dashboard — and the host-agent widget routes had no role check at all.
- steward/dashboard/share.py: a share token opens, read-only, exactly the
widget fragment URLs its own dashboard loads — same path, same
configured parameters; only the time window (hours/range) may change.
- require_role marks its view; public_route marks the deliberately open
ones (login, setup, health, the share page, agent ingest, install.sh,
agent.py) with the reason. A before_request refuses any view with
neither, so a route that forgets its decorator is closed, not open.
- /plugins/host_agent/widget and /widget/history now require viewer.
- The history widget's range buttons keep the rest of the query (and so
a share link's token) instead of rebuilding it; the uptime widget's
"Full SLA" link shows to logged-in users, not to share viewers.
- tests/test_route_access.py: a source scan that fails on any unmarked
route in core or bundled plugins, the default-deny refusal, and the
share matcher's cases.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5031 "Forged-request protection: SameSite cookie, per-session CSRF token
on every non-GET, HTMX header". A page on another site could make a
logged-in browser POST to Steward with the session cookie attached.
- steward/auth/csrf.py: a per-session token, checked in one before_request
for every POST/PUT/PATCH/DELETE (header X-CSRF-Token or form field
csrf_token, constant-time compare). The bearer-authenticated host-agent
ingest is exempt; /mcp never reaches Quart.
- Every POST form renders {{ csrf_field() }}; base.html carries the token
in a meta tag and adds it to every HTMX request; the dashboard's fetch()
calls send it.
- The session cookie is SameSite=Lax; login issues a fresh token.
- PUBLIC_BASE_URL is now loaded from Settings at startup (it was only set
when Settings → General was saved).
- tests/test_csrf.py: refusal, acceptance, rotation, the ingest exemption,
and a guard that every POST form in the templates carries the field.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Milestone 465 "Swarm task awareness". Fixes three defects from the live MCP
review #4973:
- #5008 "Swarm task history read as live state"
- #5009 "Crash-loop detection is blind to Swarm services"
- #5011 "Swarm task errors are not captured"
Agent and ingest (agent 1.8.0):
- A manager's agent now reports each service's newest tasks: slot, node,
desired and current state, Swarm's Err/Message, exit code and time.
- They are stored in docker_swarm_services.tasks_json (migration
docker_010_swarm_tasks).
Swarm view (pure):
- A task container is "replaced" when Swarm set its task's desired state off
`running`. Without the task on record, a newer or running task in the same
slot decides.
- Replaced containers are no longer replicas or crashes.
- Each service gets, from the task records:
- failures_1h and failures_24h (failed or rejected tasks);
- crash_looping (Settings → Thresholds count);
- last_error, in Swarm's own words.
MCP:
- New findings: swarm_service_crash_looping (critical) and swarm_task_failed
(warning, on a healthy service with a failure in 24h).
- Down and degraded findings name the last task error.
- list_swarm_services carries the failure fields.
- Container rows mark swarm_replaced and give a replaced task no problems.
Web:
- The containers page swarm cards and the Swarm topology page show crash
loops, recent failures and the last task error.
- The dashboard widget flags a crash loop, and its "failed (24h)" count no
longer counts a rolling update's killed task.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
MCP-5 of milestone 461 "MCP server: Steward as an LLM-inspectable,
LLM-operable hub". Contributed through each plugin's register_mcp_tools
hook, so they exist only while the plugin is enabled.
docker (viewer, read-only):
- list_containers: filter by host, status, compose project, or problem
(restarting, crash_looping, unhealthy, oom_killed, exited_nonzero);
problems sort first and each row names what is wrong.
- get_container: state, CPU/memory history across the raw/hourly seam,
lifecycle events. A unique part of a swarm task name is enough.
- list_swarm_services: replicas per node (agent-less placements as ghosts)
and the swarm's nodes.
- search_container_logs: text (% and _ literal), stream, range, context
lines, paging by a full-precision `before` cursor; over-long lines keep
both ends and say how much of the middle was cut.
- get_docker_disk_usage: bytes per host, reclaimable first.
- mcp_overview: failing containers and under-replicated swarm services
join get_overview.
host_agent: list_agents (reporting / stale / pending, version, OS), and an
overview finding for agents registered over 30 minutes ago that never
checked in.
Settings > Thresholds gains the TLS-expiry warn/crit days (were 14/3 in
code) and the crash-loop deaths-per-hour count (3), which get_overview
and the docker tools now read.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Ingest side of container logs. The host_agent ingest route extracts
sample["docker_logs"] as time-series batches (append every line, each carrying
its own Docker ts with recorded_at as fallback) and hands them to the docker
capability alongside the existing container/swarm/disk data — still under the
begin_nested SAVEPOINT so a logs failure can't sink host metrics.
- ingest.py: _persist_logs + pure _log_rows(batches, host_id) that shapes/filters
records (drops the _steward truncation marker, malformed + lineless records,
normalises unknown stream → stdout); persist_host_docker gains a logs= param
- routes.py: accumulate docker_log_batches, add to the guard + capability call
- unit test for _log_rows shaping/filtering; integration push→store→query test
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CAGR73DUowdVFVvYzLXC5C
Agent-side of Docker container-log collection (milestone 79). Each interval the
agent fetches new log lines per running container over the local Docker socket,
demuxes the multiplexed stream, and folds sample["docker_logs"] into the same
push as metrics — no inbound channel needed.
- since-cursor per container kept in rate-state; a container's first interval
seeds from a short tail, then fetches incrementally via ?since=<cursor>
- _docker_request_raw + _demux_docker_logs + RFC3339Nano ts parsing (stdlib,
TTY/unframed fallback); one row per (stream, ts, line)
- per-batch byte cap with a truncation marker; logs are dropped (never buffered)
across a backoff so an outage keeps metrics but not stale logs
- docker_logs_enabled (default on) + docker_log_exclude per-host config keys
- AGENT_VERSION 1.6.0 -> 1.7.0
Server ignores the new sample key until the ingest step lands, so this commit is
CI-safe on its own.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CAGR73DUowdVFVvYzLXC5C
plugin_metrics grows by (sources × resources × ~30s cadence); keeping 90d of raw
is a large table. Add a raw→hourly rollup (mirroring the Docker plugin) so only a
short raw window is kept at full resolution, with hourly averages archived longer.
- PluginMetricHourly model + core migration 0024 (plugin_metrics_hourly: avg/max/
count per source/resource/metric/hour, unique bucket constraint + lookup index).
- steward/core/metrics_retention.rollup_plugin_metrics: date_trunc('hour') agg of
raw older than the hour-aligned raw window, idempotent pg upsert into hourly,
delete the rolled raw, prune hourly beyond the rollup window.
- cleanup.py: plugin_metrics is no longer blanket-deleted at data.retention_days;
_run_metrics_retention drives the rollup with windows read live from settings.
- Settings: metrics.retention.raw_days (7) + rollup_days (90), tunable on the
Thresholds & Retention page (new "Host metrics retention" card).
- Chart read: _history_for_host merges the hourly rollup (older part of the range)
with raw date_bin (recent part, capped ≤1h), so 30d charts keep working —
recent at full resolution, older at hourly. Route passes raw_days from settings.
- Tests: unit (cutoff helpers) + integration (rollup aggregates/prunes; history
merges hourly + raw) against Postgres.
Speed was already handled by the indexes + SQL aggregation; this is the storage
lever (raw window ~10x smaller).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC
Realizes the chosen host-summary layout: a thin live vitals bar at the very top,
separate from the agent management panel (no more duplicated CPU/MEM/DISK/LOAD).
- New fragment _host_vitals.html + route /plugins/host_agent/vitals/<id>: compact
CPU / Memory / Disk(/) / Load-per-core (threshold-coloured + sparkline) +
Pressure + live/stale·version·last-seen. Polled every 15s; renders nothing
until the agent reports.
- The metric computation (latest snapshot + 6h sparkline query + load/core + PSI)
moves from host_panel into host_vitals. host_panel slims to management only
(reg/target/reporting/stale/ansible) and no longer queries metrics; panel.html
drops the gauge row + pressure block, keeping status + lifecycle actions.
- hosts/detail.html: vitals strip on top (full width, live), then a 2-col
[Monitors | Agent] grid, Ansible full width below, docker fragment last.
UI only. Templates parse; plugin-template parse test covers the new fragment.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC
The live refresh re-created the Chart.js charts each poll, which blanked the
canvases and re-ran the grow-in animation — a jarring flash/jump every cycle.
Make the canvases persistent in the page shell and poll only the data:
- The 3 chart canvases + their Chart instances are created once in the shell,
with animation disabled.
- /charts is now a data-only fragment swapped into a hidden div; it calls
window.applyHostSeries(series, range), which sets each dataset's data and
calls chart.update("none") — in-place, no re-create, no animation, fixed
height. Range labels update via .hm-chart-range spans.
- Current-state fragment keeps its atomic innerHTML poll into fixed-height
cards, so numbers update without a layout jump.
Net: live updates morph smoothly with no flicker or layout shift.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC
The full-metrics page rendered once server-side with the history query inline,
so the charts blocked first paint and nothing refreshed without a reload (it
reads the latest snapshot the server holds — the agent pushes ~every 30s).
Split it into a shell + two HTMX fragments:
- Shell (host_detail.html): header + shared time-range toggle + two containers;
paints instantly.
- Current state (/<id>/metrics → _host_metrics.html): identity + gauges +
per-core + filesystems + interfaces/disks + temps, from the DISTINCT ON latest
query. hx-trigger "load, every 15s" → live numbers at ~the agent cadence.
- History charts (/<id>/charts → _host_charts.html): the 3 charts + Chart.js,
from the date_bin history query. hx-trigger "load, every 60s, rangeChange" so
they lazy-load (never block paint), refresh slowly, and follow the range
selector. Leak-safe: previous Chart instances are destroyed before re-render.
- Range toggle switched from full-reload links to the shared _time_range.html
(setTimeRange + rangeChange), so only the fragments refetch.
- routes: host_detail (shell) + host_detail_metrics + host_detail_charts, with a
_split_host_metrics helper.
- tests/test_templates_parse.py now also parses plugin templates (these
fragments aren't rendered in the unit lane).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC
The previous commit's integration test failed at collection-time import:
importing plugins.host_agent.routes (which imports the host_agent ORM models at
top level) double-registers host_agent_registrations against the app-loaded
plugin's metadata ("Table already defined").
Move the two pure read helpers (_latest_metrics_for_host, _history_for_host)
plus SOURCE_MODULE / HISTORY_METRICS into plugins/host_agent/metrics_query.py,
which imports only the core PluginMetric model — no plugin models. routes.py
imports them back. The integration test now imports from metrics_query and no
longer trips the loader's registration guard. No behavior change.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC
Follow-on to the plugin_metrics indexes. plugin_metrics is already retention-
bounded (core cleanup prunes > data.retention_days, default 90d) and charts top
out at 30d, so the cost wasn't growth — it was the read path shipping raw rows
to Python.
- _history_for_host: bucket + average in SQL via date_bin (epoch-aligned, ~120
buckets) instead of fetching every raw sample (a 30d range was hundreds of
thousands of rows) and downsampling in Python. Uses the new
(source_module, resource_name, recorded_at) index.
- _latest_metrics_for_host: DISTINCT ON (resource_name, metric_name) ORDER BY
recorded_at DESC — newest row per group in one index-ordered pass, replacing
the GROUP-BY-max subquery self-joined back to the whole history.
- Integration test validates both against Postgres.
Deliberately not a materialized raw→hourly rollup: these query-side changes
deliver the speed; a rollup would additionally cut storage and remains a future
option if scale demands it.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC
Two issues on the host full-metrics view:
- fmt_bytes capped at GB, so large filesystems read "25369.0 GB" instead of
"24.8 TB". Add TB and PB tiers.
- Interfaces/Disks/Temperatures shared one 3-column grid, so all three cards
stretched to the Temperatures height — a 40-core CPU made one very tall column
and left Interfaces/Disks with large empty space. Split them: Interfaces +
Disks sit side-by-side at natural height (align-items:start); Temperatures
becomes a full-width card whose readings flow into a compact multi-column grid
(wide-and-short instead of one tall column).
UI-only.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC
The page-top chrome was inconsistent: most nested views used the breadcrumb
kicker + page-title pattern, but plugin sub-pages used ad-hoc "← back" links,
~11 pages stacked a breadcrumb AND a redundant ancestor back button, and two
(monitors/edit, hosts/uptime) had a back button but no breadcrumb. The
settings/plugin_detail page stacked all of it at once.
Unify on the breadcrumb-led model:
- settings/_tabs.html: drop the hardcoded "Settings" h1; the breadcrumb
("Settings › …") plus the tab strip is the header.
- settings/plugin_detail: drop the "← Plugins" back button.
- docker container_detail/swarm/disk + snmp/device: replace ad-hoc back links
with the standard crumbs() breadcrumb.
- host_agent, ansible/*, alerts/maintenance: remove redundant ancestor back
buttons (the breadcrumb's parent crumbs already link there); keep lateral
shortcuts (Inventory/Schedules/Browse/Targets/Groups/New).
- monitors/edit, hosts/uptime: add the missing breadcrumb, drop the back link.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC
Backend for the image/disk panel (milestone 77 #942). Agent gains
collect_disk_usage() — one /system/df call (gated on containers existing, so
Docker-less hosts pay nothing), surfacing reclaimable bytes (image size held by
unreferenced images), layers/containers/volumes/build-cache sizes, and the top
50 images by size. Emitted as sample["docker_disk"]; host_agent ingest tracks
the newest sample's copy and hands it to the docker capability as a 5th arg.
New current-state tables docker_disk_usage (one row/host) + docker_images
(per-host image rows), docker_007 migration; ingest upserts the summary and
replaces the image set per host (stale images pruned). Unit tests for the df
parsing/reclaimable math + build_sample gating; integration test for
persistence + image-set replacement.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC
Wires the agent's enriched + swarm payloads through the docker.persist_host_
samples capability:
* Swarm topology — persist sample["swarm"] into docker_swarm_services /
docker_swarm_nodes (upsert + prune stale, host-scoped so two managers don't
clobber). Migration docker_005 adds services.placement_json for the
task→node placement the agent now reports.
* Lifecycle events — _derive_events (pure, unit-tested) diffs the newest
snapshot against stored per-container state: start / stop / die (non-zero
exit) / oom / health_change → docker_events rows. Skipped on a host's first
snapshot so the baseline doesn't emit a start per existing container.
* Alerts — record restart_count (always) and is_healthy (1.0/0.0, only when a
HEALTHCHECK exists) alongside cpu/mem, under host-scoped resource names;
METRIC_CATALOG[docker] gains restart_count + is_healthy so they're alertable.
host_agent ingest captures the newest sample's swarm object and threads it to
the capability (now persist_host_docker(session, host, snapshots, swarm=None));
invoked when containers OR swarm are present, under the same SAVEPOINT. Unit
tests cover the event-diff matrix; integration tests cover event derivation
across two snapshots and swarm topology round-trip (incl. placement).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC
Adds collect_swarm(socket_path) to the host agent. Self-detects a Swarm
manager via /info (Swarm.ControlAvailable) — workers and non-swarm daemons
return None and never touch the manager-only endpoints (one cheap /info call,
no 503s). On a manager it queries /services, /tasks, /nodes and emits
sample["swarm"] = {services, nodes}:
* services roll desired-vs-running replicas up from the task list (replica
health isn't on the service object), handle replicated + global mode, strip
the @sha256 image digest, and carry cross-node task→node placement.
* nodes normalise role / availability / status + the manager leader flag.
build_sample omits the swarm key entirely off managers, same silent contract
as collect_docker. Unit tests cover manager detection, replica roll-up +
placement (replicated & global), node normalisation, worker silent-skip, and
build_sample wiring.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC
