View mode splits each saved row into seven, so a widget no longer carries up
to 69px of empty space below its content (Overview, SNMP). Edit mode and the
saved layout keep 70px rows. The Busiest Containers host column aligns left
like its header, its note fits one line, and the Hosts column reads Disk.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
CI 8551 integration: "integer out of range" for a 2099 timestamp.
The SQL in the log showed a second problem: `CAST(epoch AS INTEGER) / 60`
compiled to `/ CAST(60 AS NUMERIC)`, i.e. decimal division. Every scrape got
its own "bucket", so the request chart and the Traefik page's sparklines
never averaged anything. The int4 cast would also overflow in 2038.
Both now use date_bin from the epoch, as host_agent/metrics_query.py does.
The integration test adds a second scrape 30 s later in the same minute and
checks that the bucket averages the totals and weights the 5xx share over
both scrapes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5256 "Traefik — Request chart: blank on the live dashboard; time axis and honest labels".
- "Req/s" was the average of the routers, not the total. Each scrape is now
summed over routers, then averaged per bucket (request_history).
- The 5xx share is now weighted by each router's traffic. The axis is no
longer pinned at 0–100 (suggestedMax 5).
- The x axis shows times, in the viewer's local time. The tooltip shows
both series at that time.
- A verdict line comes first: now / peak req/s and the peak 5xx share. The
figures read even if the chart doesn't.
- The blank live chart's cause is still unknown. Chart.js and
chart-colors.js both load on the dashboard. A missing library or a thrown
error is now written into the widget instead of leaving it empty, so the
cause will show on the page.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5255 "SNMP — Devices: status in words and key readings".
- Verdict first: "All N devices answering", "ups: no reply since 14m ago",
"N of M devices not answering", or "N not polled yet".
- Devices that stopped answering come first, then those never polled, then
the rest.
- One header row: Device / Readings / Polled.
- The name is the link; the separate "detail →" is gone.
- The hollow dot is replaced by words: "no data yet", or "no reply since …;
readings are from then".
- Key readings: the first two in config order, gauges ahead of counters.
Units come from the label suffix: _centisecs as a duration, _pct, _dV as
volts, _octets as bytes total.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
CI 8548: '<th' also matched '<thead', and the certificate test's clock was
read before its certificates were built, so 10 days came out as 10.00…
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5254 "Traefik — Routers: current routers only, header row, certs on the router they serve".
- Router hosts come from Traefik's API (/api/http|tcp/routers), read
best-effort on each scrape. An optional api_url overrides it; the default
is /api on the metrics host. A certificate's expiry then shows in a TLS
column on the router whose host it covers. An expiring one makes that
router a problem, graded by Settings → Thresholds.
- Only current: routers and certificates must have appeared in the last
three scrapes. Certificates are upserted by serial and never deleted, so
renewed or retired ones lingered on the widget.
- Widget order: verdict, problem routers with the reason, then the busiest
healthy routers under one header (Router / Req/s / Bandwidth / p95 / TLS).
A missing value shows "—".
- If the API is unreachable, expiring certificates are listed after the
routers, not above them.
- The page splits certificates into served (with the routers using each,
plus SANs) and "No longer served".
- Latest row per router: an index walk (recursive CTE + LATERAL) replaces
DISTINCT plus a query per router.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The widget names the busiest container in a verdict ("Busiest: tdarr on
VDNT-Docker03, 1.6 of 8 cores."), then a fixed-layout table (container,
host, CPU in cores, memory in bytes) with each figure's share drawn as
shading behind it: of the host's cores, and of the memory limit or the
host's memory.
The numbers were already rendered, but each row grew as wide as its longest
name. A grid item won't shrink below its content, so Gitea CI container
names pushed the figures out of view and the cell scrolled sideways. Names
now come from display_name(): a swarm task by its service (no slot number),
an Actions job container as "CI: <workflow> · <job>". They cut off with …
and show whole on hover.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A verdict line (services healthy · other containers running · failed in
24h), then "Needs a look": troubled services with why (crash-looping, N of
M running), since when and the last error, and troubled containers
(unhealthy, restarting, exited with an error, out of memory). Then "Failed
in the last 24h" by name, with how often, when and where. Then the healthy
rest one line each (name · hosts · replicas) in as many columns as the
widget is wide, collapsing after 40 to "+N more healthy".
Per-container CPU is gone from this widget: Busiest Containers owns it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The Hosts — Overview widget opens with a verdict ("VDNT-Util01: CPU high."
or "All 3 hosts healthy."), then one row per host under one header: CPU,
memory, disk / and load. Rows sort by host_glance: a down monitor, a
reading past critical, a quiet agent, a warning, then the healthy. Load
reads "16.8 / 8 cores", graded per core as Settings → Thresholds intends
(cores counted from the per-core CPU sub-resources, as on the host page).
A quiet agent's last figures are shown uncoloured, with when it last reported.
Ping and 24h uptime are gone from this widget: they belong to Monitors. So
are the unlabelled CPU sparkline and the 30-day uptime query it no longer
needs.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The Monitors widget opens with a verdict ("1 of 5 monitors down."), then one
row per monitor under a single header: latency and uptime over 24h, 7d and
30d, down monitors first with why and since when, the name linking to the
monitor. Counts follow the type filter. Problems always show; the limit
trims only the healthy rest ("+N more up").
Uptime / SLA (uptime_summary) repeated those numbers per host; migration
0033 moves dashboards onto Monitors. Shared widget styles (.w-verdict,
.w-table, .w-sub, .w-more) carry the widget-shape ruling for the widgets
that follow.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each widget is as tall as its content, never taller than its saved height
(where it scrolls). The dashboard is capped at 2400px instead of 1400px.
Past 12 columns' worth of width it adds columns (one per 110px, up to 24)
and places widgets again in reading order at their saved widths, so more sit
side by side instead of stretching. Edit mode shows and saves the
12-column layout, so a layout doesn't depend on the screen it was edited on.
The share view uses the same layout code (static/dash-layout.js).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- New installs start from widgets.RECOMMENDED, packed row by row for the
enabled plugins; the edit drawer can reset a dashboard to it.
- Host Agent — Resources folds into Hosts — Overview (which gains load 1m),
UniFi — Devices into UniFi — Overview; migration 0032 moves existing
dashboards onto the survivors.
- The Overview labels what it counts (hosts, monitors up/down/pending,
alerts firing) and lists firing alerts and down monitors with since-times.
- Uptime polls every 5 minutes (poll_seconds) instead of every cycle.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5212 "Plain labels: no raw ids, scopes, metric keys or unexplained jargon",
batch 3:
- Disk page: each tile says what it holds. "Writable layers" becomes
"Container files" and Build cache says it is safe to clear. Shared is
"Shared with other images" with a tooltip, and "dangling" is gone.
Untagged images show by id instead of "<none>", a capped list says
"The 50 largest of N images", and each host shows "as of" its report.
- Traefik: an expired certificate reads "expired 12 days ago", not "-12
days" (cert_left). The Traffic empty state knows whether the access log
is on: off points to Settings → Plugins → Traefik; on says to check the
mounted log path.
- Moments use |ago with an exact UTC tooltip: UniFi last seen, alarms,
events, speed test and DPI; AI token last used.
- Settings: the Auth tab is "Authentication" like its breadcrumb, plugin
fields read "Metrics URL" not metrics_url (field_label), a source's pull
interval is "every 1h", and the last report time drops the ISO "T".
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5212 "Plain labels: no raw ids, scopes, metric keys or unexplained jargon",
batch 2:
- A run records where it came from: AnsibleRun.origin ("schedule", "alert"
or "mcp") and origin_id, set by the scheduler, Run now, alert actions and
the MCP tools. Migration 0031 backfills past runs from alert events,
schedules' last runs, and MCP audit entries.
- The runs list has a Started by column: "schedule nightly", "alert rule
nas down", "alice via MCP" or "alice", linked where there is a page.
The raw scope tooltip is gone.
- Run detail: no run id in the subtitle or body, Started by in words, and
the host summary lists failed and unreachable hosts first. The live
output only follows new lines while the reader is at the bottom, and
when the run finishes the page reloads once so the badge and summary
are final.
- list_ansible_runs and get_ansible_run report origin and origin_id.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Run 8504 failed unit: tests/core/test_alert_history_row.py built a rule
with no metric_name, which _history_row now reads to say the reading in
words. The rule now names cpu_pct, and the value and condition asserts
expect "CPU 98%", "No CPU reading" and "CPU > 90%", plus an up/down case.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5212 "Plain labels: no raw ids, scopes, metric keys or unexplained jargon",
batch 1:
- Audit log: each action reads as a sentence ("paused monitor"), with the
code as its tooltip. Detail shows flags as yes/no, lists joined, a run's
scope by name, spans as spans, and no ids the entity column already gives.
The filter is now a picker of areas.
- Alert rules, history and Status say the condition in words: "is down",
"Fullest disk >= 85%". Readings use metric_value, and history times use
|ago.
- Rule form: sources and metrics by name, the threshold's unit, "fires
when it is down/up" for an up/down metric instead of 0.5, and a metric
reference built from the catalog.
- Schedules and Maintenance: viewers no longer see Run now, Pause, Edit,
Delete or New. Intervals and next runs read as "every 2h" and "in 3h";
a scope reads "Every alert on Host agent".
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5211 "Lists load as you scroll: audit, alert history, Ansible runs"
(milestone 477), per preference 172.
- core/paging.py: keyset paging over (time, id), newest first, with an
opaque "<iso>|<id>" cursor; a page reads one row past itself to know
whether another follows. No OFFSET, so rows added while someone scrolls
never shift the pages after them.
- _macros.html scroll_more / scroll_end and static/scroll-more.js, written
once: an IntersectionObserver with a 600px margin fetches the next page
before the reader reaches the end; "Loading more…" while in flight,
"End of the list." at the end, and a failed page says so with a Retry
button instead of re-firing on every scroll.
- Audit log: no "Load more" link and no 500-row cap.
- Alert history: no 200-row stop with "pick a shorter range";
alert_svc.history takes the cursor, so the MCP tool and the page share
one query.
- Ansible runs: no silent cap of 50.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5210 "Problems first, with since-time, across every list" (milestone 477),
second batch.
- Host page: an "Alerts on this host" strip first (rules on its agent
series, its containers or its monitors: alert_svc.watches_host); its
monitors problems first with "Down 12m · reason"; Docker above Ansible.
- Uptime page and widget: worst 24h uptime first, then 7d and 30d; hosts
with no uptime yet last.
- Agent panel and fleet widget: last seen as "3m ago", not a clock time.
- SNMP: a device is "Reachable" only while it answered within the last 3
polls; older readings say "No answer" with their age, and every device
says when it was last read (was: reachable whenever any reading existed).
- Traefik: a router missing from 3 scrapes in a row is listed apart as no
longer reported, by when it was last seen, instead of keeping its last
state (a removed router stayed Failing).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5210 "Problems first, with since-time, across every list" (milestone 477),
first batch.
- Alerts: the Active list reads firing, no data, acknowledged, pending, the
longest-running first, and re-polls itself. An acknowledged alert says how
long it has been breaching (from the alert's events; acknowledging moves
last_transition_at) and, apart from that, who acknowledged it and when.
Rules show what each is doing now; disabled rules sort last. The current
value reads the latest-value table (#5208) instead of plugin_metrics.
- Monitors: a down monitor says "Down 12m · connection refused" in the list.
- Hosts: sorted down, agent gone quiet, pending, healthy. The agent column
shows "—" for a missing reading (was 0%) and a Stale badge with its age.
agent_metrics_by_host is gone; the list reads agent_overview_by_host.
- Schedules: failing first (failed or interrupted run, or a firing that could
not start one), then running, then the rest, paused last; status badge
from the shared states.
- Maintenance windows: active (ending soonest), then upcoming (starting
soonest), with expired folded away; each says when it ends/starts/ended.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
failing_services filtered on `(service_trouble(s) or 9) <= 2`; a crash loop
ranks 0, which is falsy, so the worst kind of trouble was left out. Test the
rank against None. The page test's task error also gains the age_s the
_swarm_failures include reads.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5209 "Docker views: readable names, CPU/memory in cores and bytes, failures
listed with since-time" (milestone 477 "UI problems first").
- A swarm task reads as its service and slot ("codex_db · 1", or the node
for a global service), with the raw name in the title: Docker page, host
panel, busiest widget, container detail.
- CPU reads in cores (Docker's 100% = one core), toned by its share of the
host's CPUs from the agent's per-core series. Memory reads in bytes, against
a limit only when one is really set (Docker reports host RAM otherwise).
- Exit 137/143 without an OOM kill is a stop, not a failure: no
exited_nonzero, no crash-loop death, no "failed (24h)", dim exit code.
- "Failed (24h)" is a list (Docker page #failures, linked from the widget
and the tile): plain containers by their deaths, swarm services by Swarm's
failed tasks; a second manager's copy of a task death counts once.
- Failing swarm services lead Needs attention; services sort problems first
with since-times on the page, the widget and the Swarm page; nodes that
are down or drained sort first.
- Busiest widget orders by a 5-minute CPU average so the list holds still.
- Host panel: a table, problems first with what is wrong and since when;
replaced swarm tasks hidden and counted.
- Container detail: node hostname, I/O labelled "since start", Restarts
hidden for swarm tasks with a note on where task failures are.
- Widget names: "Docker — Services & failures", "Docker — Busiest containers".
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Migration 0030 seeded plugin_metric_latest from the whole history: two index
probes per distinct metric key. On the live instance that held the first
boot for 7+ minutes with Steward down. The migration now only creates the
table. Live sources fill it on their next sample, and backfill_latest runs
after startup for resources that have history but no latest rows. Per
resource it does one backward probe for the last sample, then takes the
newest value per metric within the hour before it. Resources that already
have rows are skipped, so later boots only walk the resource names.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
_last_n_results (Status, the monitors list, host monitors), host_status
(hosts overview, list_hosts) and the report's down-monitor list each
ranked every result a monitor ever recorded to keep the newest few. Each
monitor now gets ORDER BY checked_at DESC LIMIT n in a LATERAL subquery,
which walks ix_monitor_results_monitor_checked backwards and stops.
Also fixes the latest-table test, which read between two begin() blocks
and so autobegan a transaction the second block then refused.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The dashboard's host widgets took 18-60 s. Every "latest reading" query
(hosts overview, host-agent fleet, host page, vitals, SNMP, MCP get_host)
searched plugin_metrics for the newest row per metric. Postgres has no skip
scan, so each one read every raw sample in the 7-day window.
- plugin_metric_latest: one row per (source, resource, metric).
record_metrics upserts it in the same transaction, moving forward only and
in sorted key order. Migration 0030 seeds it by stepping the existing index.
- Retention drops a latest row once its sample leaves the raw window, so a
resource that stopped reporting disappears when its history does, as before.
Renames carry it along.
- A quiet host keeps its last values, marked stale. A time-bounded query
would have shown it as missing.
- Vitals sparklines span the 6 h they claim, averaged in SQL to ~120 points,
instead of fetching 6 h raw and drawing the last 120 samples.
- Dashboard and share-view cells: "Loading…" until the first answer; a 20 s
deadline per fetch; "Couldn't load (why)" with Retry now and an automatic
retry for cells that don't poll; "as of HH:MM" in the header, and
"couldn't refresh · as of" when a refresh fails over shown data.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The operator's concept, redrawn as a single-colour SVG in the accent gold:
a key whose bow is a cloverleaf holding three connected nodes, hanging on a
loop threaded through a keyring and swinging into the ring's middle, with a
small sparkle.
- static/logo/steward-logo.svg: the full mark (node graph in the bow).
- static/logo/steward-mark.svg: the small mark for 16-32 px (heavier lines,
one centre node, two teeth); the node graph turns to mush below ~40 px.
Used in the sidebar brand (was an inline crown) and as the favicon on every
page and on the public share view.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Run 8447 on b307f53: the boot check failed at the forced assertion, Push was
skipped, clean-up ran, :dev stayed on sha256:718d2ca (98854d9) and no
:b307f53 image exists in the registry.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Dockerfile: python:3.14-slim, the Python CI tests on (was 3.13). tini from
apt is PID 1 (ENTRYPOINT tini -- entrypoint.sh): it reaps the zombies ssh
and ansible leave and forwards SIGTERM.
- cli: Hypercorn's serve() with one worker (run, stream and scheduler state is
in-process), a 10 s graceful timeout, and SIGTERM/SIGINT as the shutdown
trigger. Access and error lines go through Steward's logging. --debug keeps
Quart's reloading dev server.
- Logging: steward/core/logging_setup.py installs one root handler before
anything logs (until now a library's basicConfig on import was the only
configuration). Its level is the `log.level` setting (DEBUG/INFO/WARNING/
ERROR, default INFO), applied at boot and on save in Settings → General.
- CI publish: build, then "The image boots with no internet" (rule 164,
snippet #3552), then push, so a broken image never reaches a tag (rule 177).
On an --internal network with its own Postgres, the image must migrate and
answer /health, run tini as PID 1, serve first-run setup with its assets
from /static and no third-party host, and exit 0 on SIGTERM. Logs and
clean-up run always.
Tests: tests/core/test_logging_setup.py; the smoke step itself.
Not in this commit: the dependency lockfile (it conflicts with rule 154 and
is waiting on the operator).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The "[queued — waiting for run X on the same host]" and run-slot lines were
broadcast to live viewers and never stored, so a run that sat queued had no
record of why once the live view was gone (CI 8439: the overlap test found
the order right and the explanation missing). _claim_hosts now returns the
line it waited on and start_run seeds the run's output with every wait.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- One run per host: start_run claims every host it touches (addresses read
from the inventory Steward wrote; a source's own inventory file is claimed
as that file) before it takes a run slot. The claim is all-or-nothing. A run
waiting on a host shows queued and names the run it waits for, honours a
cancel, and fails after HOST_WAIT_SECONDS (1 h) with which host and run.
- A due schedule whose previous run is still queued or running skips the
firing: last_run_at moves on, last_run_id still names the run going, and
last_error says "Skipped: the previous run was still going". The Schedules
page shows that beside the run (muted), and a launch error in red.
- An alert rule that fires while the action run from an earlier firing is
still going starts no new run; the event says so and names the run. An
in-process set closes the gap between the check and the run being linked.
- docs/core/ansible.md: queued/cancelled statuses and the overlap rules.
Tests: tests/core/test_ansible_overlap.py (claim keys, waiting, all-or-nothing,
cancel, deadline) and tests/integration/test_ansible_overlap.py (two real
runs on one host take turns; schedule skip then fire; alert action skip then
start).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- git ships in the image; checkouts live under /data/ansible/sources
(STEWARD_ANSIBLE_SOURCE_DIR), on the persistent volume.
- One task, ansible_git_pulls, reads the sources every 30 s and pulls each
as its interval passes: a source added in Settings is pulled without a
restart, a removed one is forgotten.
- git_pull: a 120 s timeout ending the process group, GIT_TERMINAL_PROMPT=0,
pull --ff-only, a failed clone removed; it raises GitPullError with git's
message, so the Sync button stops saying Synced after a failure (and the
message is now escaped).
- An alert action's vars and the firing's steward_alert_* vars reach
ansible-playbook as one JSON extra-vars file, not -e key=value, which
Ansible split on whitespace (a rule named 'High CPU on web').
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Output is read in 64 KiB chunks and split into lines here; a line past
64 KiB is cut and marked instead of raising out of readline's limit.
- ansible-playbook runs in its own session; whatever ends the read (finish,
an error, the task being cancelled) SIGTERMs then SIGKILLs its whole group,
so no ssh or worker it forked outlives the run.
- A run whose task is cancelled (shutdown) records 'cancelled' with
finished_at before re-raising, instead of staying 'running'.
- A cancel that lands after the process exited 0 no longer turns a
successful run into 'cancelled'.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- prune_by_age(column) registers a time-series column with the hourly cleanup;
Traefik (metrics, access summaries, global stats) and UniFi (WAN, client,
WLAN, DPI, events, alarms, speedtests) register theirs in setup().
- Alert events join monitor results and Ansible runs on data.retention_days,
read from Settings each run; Ansible run log files older than it are deleted.
- audit.retention_days (365) keeps the audit log longer, set on Settings > General.
- metric_series.points reads metrics.retention.raw_days, so the chart's
raw/rollup seam is where the rollup put it (RAW_DAYS removed).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
MAX_SAMPLES_PER_INGEST was 2 while agent 1.10 drains a backlog five samples a
post, so the server dropped the oldest three of every such post. Posts now
answer in 0.4-1.3 s on dev, including the boot cleanup window, so five fit
the agent's 10 s. A test holds the server cap at or above the agent's.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent 1.11.0 sends finished_at (State.FinishedAt) and reads a stopped
container's logs once per stop: since its cursor if it was being read, or
its last 200 lines when first seen already crashed (non-zero exit, stopped
in the last 10 min). Cursors and finals are forgotten once a container is no
longer listed.
Ingest gives a container first seen already exited non-zero a die event, and
dates die/stop events by finished_at when the agent sends it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
persist_host_docker marks docker.status, .swarm, .disk, .logs, .metrics,
.events, .containers, .alerts and .reap on the ingest's phase timer, each
flushed before its mark so a part's writes are charged to it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An audit of 1,980 text/surface pairs (every steward.css rule and every inline
template style, tints composited over page, card and hovered row) found:
- Red, yellow and orange status text on its own 20% tint over a hovered row
at 3.97-4.06:1 (badges, count badge, alert-error, btn-warn). --red and
--yellow are lifted 15% toward the primary text, --orange 18%, as --blue
already was: worst case now 4.54:1.
- Inputs, selects, ghost and range buttons outlined in --border-mid at
1.4-1.9:1, on surfaces 1.1:1 apart, so the outline is what marks them.
New --border-control (45% toward secondary text): 3.13:1 worst. The
dashboard switcher and widget-drawer fields drop their inline styles for
the shared .input class.
- The breadcrumb separator painted in a border colour (1.4-1.9:1): now
--text-dim (4.64:1) and aria-hidden, since it is decoration.
tests/test_contrast.py resolves the tokens (color-mix and tints included)
and holds every drawn pair to 4.5:1, status text on its tint to 4.5:1 and
control outlines to 3:1, with a test that the guard catches the old pairs.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Measured on dev (385a579), agent posts cut to five samples still dropped:
flush=8.8s for 990 rows, 6.6s for 680, 8.4s for 1000 — about 110 rows/s,
~9 ms a row, with rules and evaluation near zero. One random read per
row fits a random uuid4 primary key whose index no longer fits in memory:
every new row lands on a random page of it. (Inferred from the rate; the
table and index sizes have not been read.)
- steward.core.ids.time_ordered_id: a UUIDv7-layout string (48-bit ms
timestamp, then random), same 36-char shape as the stored uuid4 ids, so
no migration. Python's uuid7() arrives in 3.14; the image runs 3.13.
- Used for plugin_metrics and its hourly rollup, monitor_results,
audit_events, the docker metrics/hourly/events/logs tables, and the
traefik and unifi time series.
- MAX_SAMPLES_PER_INGEST 5 → 2 (~400 rows) for now: it fits the agent's
10 s even at the old rate, so hosts land data whichever way the new rate
comes out. Raise it once the new rate is measured.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- BACKOFF_CAP 300 → 60: Steward reads a host stale after 180 s, so an agent
backing off to five minutes read stopped long after Steward came back
(lesson #4355: a check-in inherits its call's backoff).
- A POST carries at most five samples: the oldest four buffered plus the
new one, the rest held for the next post. A backlog drains a few at a
time instead of in one post Steward cannot write before the timeout
(the loop measured in #5149).
- Per-container Docker work has a 20 s deadline over the whole fan-out.
Past it the sample reports docker_status "error" ("N of M containers
not read within 20 s") and no list: a partial list would read as
removals, and a record without its inspect as restarts.
- /system/df is asked every 600 s rather than every sample; Steward keeps
the last answer.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Measured on dev (f802c9f): every dropped ingest carried samples=21,
metric_rows≈4,200 and never finished the INSERT flush inside the agent's
10 s ("completed: parse=0.1s"). The rollback left the agent's ring buffer
full, so each later post was the same size and failed the same way: no
host's metrics have landed since.
An ingest now keeps the newest MAX_SAMPLES_PER_INGEST (5) samples by ts,
logs how many it left out, and returns samples_dropped. The older samples
are the ones the ring buffer would have overwritten anyway. Why the
database writes so few rows a second is not yet known; the slow-ingest
line now measures it on a post that finishes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- The status test imported the plugin's models after the app had mapped
them ("already defined"); it reads docker_host_status with SQL.
- The swarm-expiry count awaited inside a generator expression.
- The page test's since came from the module's NOW, so |ago said "5m 4s";
it is built at test time and the assertion checks the minutes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Agent: started_at comes from inspect's State.StartedAt (nanoseconds
trimmed; Docker's zero time is "never started"), not the list's Created,
so a restarted container's uptime counts from its restart.
- A rising restart_count between two snapshots emits a "restart" event
("restart #3", or "3 restarts, now #5"), said instead of a start; a
restart then a death gives both. A count that falls (recreated under the
same name) says nothing. The timeline draws it as ↻.
- Network and disk I/O count what the kernel calls physical: interfaces
not under /sys/devices/virtual/net (bridges, docker0, veth, vxlan, tun),
block devices with a device behind them (not dm, md, zram), so a byte is
not counted twice. Where /sys calls everything virtual (an agent in LXC)
the name-filtered set is kept rather than reporting no traffic.
- Agent tests updated for #5152's (status, detail, containers) and
(is_manager, topology) shapes; the no-Docker disk test now states the
case it was passing for by accident.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The agent (1.9.0) said nothing when Docker could not be asked and nothing
when Docker had no containers, so Steward could tell neither apart from the
other, and kept a host's last containers on screen as running either way.
- Agent: collect_docker returns (status, detail, containers). "ok" sends
docker: [] when there are none; "unavailable" (no socket, nothing
listening) and "error" (permission, timeout, bad answer) say why.
collect_swarm says whether /info answered, and a node that is not a
manager sends swarm_manager: false. A failed /info says nothing.
- Ingest: an empty docker list is a snapshot; the newest sample's status
goes to the Docker plugin with the rest.
- docker_host_status (docker_011): per host, status, detail, since and
last report. A host that never had Docker gets no row.
- An "ok" empty snapshot reaps the host's containers; an error keeps them.
swarm_manager: false drops the host's swarm services and nodes.
- Docker page: a "Docker not answering" card names each such host, why,
and since when; its containers read as stale. Duplicate header comment
in rows.html removed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The first instrumented drop on dev read "dropped after 10.0s … (completed:
parse=0.1s)": the whole stall is inside record_metrics. It now marks its
three steps (flush of the rows, the rule and state reads, the per-sample
evaluation) on the caller's phases, and the drop line carries the request's
samples and metric row count.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The dispatcher ticked once per MONITORS_POLL_INTERVAL and stamped
last_checked_at when the whole batch finished. With a 60 s tick and a 60 s
monitor the next tick saw 59.x s elapsed and skipped, so the monitor ran
every 120 s; a 30 s monitor could never run faster than the tick.
- The dispatcher ticks every 5 s (DISPATCH_TICK_SECONDS) and reads the
monitors and intervals fresh each tick, so a changed interval applies at
the next one.
- Due is measured from the scheduled start, stamped before the check runs,
with half a tick of slack so tick jitter never costs a whole tick. A
long-overdue monitor runs once.
- Each check runs in its own task and records its own result, so a slow
probe holds up nobody; a monitor still being checked is not started
again, and it is freed in a finally.
- A check past CHECK_DEADLINE_SECONDS (120) is recorded down with "Check did
not finish within 120 s" rather than holding its monitor forever.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every host-agent ingest in the service logs ends at ~10 s with no status:
the agent gives up at its urlopen timeout (agent.py) before Steward answers.
Nothing says which part of ingest is slow, so this measures it:
- each phase (parse, metrics with alert evaluation, docker persist,
commit) is timed, and a request past 2 s logs one line with the phases,
sample count, metric rows, containers and log lines;
- a request the server drops when the agent hangs up (CancelledError) logs
how long it ran and which phases completed, then re-raises.
The fix follows once the logs say which phase it is.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Traefik: routers are graded from their latest scrape (router_state: crit on
any real 5xx share, warn on p95 past the latency critical cutoff or a high
4xx share). Failing and slow routers lead as cards that say why; healthy
ones fold into one table that stays open or shut across polls. Proxy
counts are stat tiles; certificates are graded by Settings -> Thresholds
(tls_days).
UniFi: WAN health is a stat strip (latency graded by the thresholds),
devices carry status dots, event categories and firewall actions are
badges, the alarm count reuses .count-badge (renamed from .nav-count).
SNMP: each device wears a status badge and its readings are stat tiles.
Agent fleet: the table comes first, with a Status column (Reporting /
Silent / Never reported, by the host page's stale_after_seconds), silent
agents first, hosts linked, last seen as |ago.
Audit: entries name their entity (one query per type, steward/audit/
service.py) and link it; something deleted since keeps the name its entry
recorded and says "(deleted)".
Tokens: charts read colours through chart-colors.js (stewardColor resolves
a token to the rgb the browser computed; Chart.js defaults take the page's
text and border colours), so host, history, Traefik and SNMP charts drop
their hex. Template hex colours reach 0 (HEX_CEILING = 0), the last rgba
pills go, and heavy weights (600/700/bold, 60 of them) become 500, the
house pair being 400/500; a guard keeps templates there.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
