#5690 "Widget template format: YAML schema, validator and renderer", step 2
of milestone 508 "Prometheus widget templates".
- template_format.check(text) -> Template, or TemplateError with every
problem, each with its line and what to change. Blocks: states, values
(of -> share and bar, thresholds as numbers or shares, sparkline),
times (warn_after / crit_after), table (two labels pivoted), info.
Anchors/aliases refused, 64 KB cap, YAML's on/off/yes/no flagged.
- template_view.render(template, target, latest, now, history) -> View:
a verdict naming the worst problem, then each block with its problems
first (crit, warn, ok, then quiet states). Missing series show as "—"
and say so; a template whose `expects` prefix the target lacks says so.
- shown_value moves to service.py so the renderer can use it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5689 "Prometheus target page: fold state sets, show epoch times, size
bytes", step 1 of milestone 508 "Prometheus widget templates".
- A state set (name ending in _state, a state label, every series 0 or 1;
or the OpenMetrics form, the label named after the metric) reads as one
row per thing it describes, naming the current state, the others on
hover. fabledarchives_jobs{state} is a count and stays as it is.
- label_pairs / state_label / fold_states in service.py, for the widget
templates to reuse.
- After a good scrape, series it no longer has lose their newest value
(forget_gone): Archives' old build_info row read as current beside the
new one. History stays; a failed scrape forgets nothing; alert rules keep
their own last-sample time.
- Epoch times and _bytes sizes were already read by shown_value /
metric_value.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A bundled prometheus plugin, on by default like the other protocol plugins.
Targets are rows edited on its page: address, bearer token (encrypted),
interval, keep/leave-out name patterns and a series cap. Steward does the
filtering; the endpoint serves everything (Archives' ruling).
- A 5 s tick scrapes due targets at once, each within a 30 s deadline, and
writes the outcome to its row and its series through record_metrics, so
alert rules can watch any of them.
- Gauges are stored as they are; counters (and a histogram's or summary's
_sum/_count) as <name>_per_sec rates from the second scrape. Buckets and
quantiles are skipped. A labelled series is stored under
"<target>:k=v,…". Each target also records is_up, scrape_ms, series_kept
and series_capped.
- Pages: targets not answering first, with why; a target's series grouped
by metric with their newest values; a chart per series; the editor with
Test, which scrapes once and says what would be stored.
- The counter-to-rate keeper moves to steward/core/counter_rates.py; SNMP's
RateKeeper now scales its rate on top of it.
- metric_value reads Prometheus's _seconds as a span and _per_sec as a rate.
- The catalog guard treats an empty list as a dynamic source (snmp,
prometheus), as the rule form does.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
steward/core/prometheus.py parses the 0.0.4 exposition into families keyed
by name, each with its declared TYPE, so a scraper can tell a counter from a
gauge. A histogram or summary owns its _bucket/_sum/_count samples. Label
values are unescaped, and commas or braces inside quotes stay part of the
value. NaN and +/-Inf are read, timestamps are ignored, and a malformed
line is skipped. Traefik's parse_prometheus is now a thin reshaping of it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
At 20:25-20:39Z the NFS storage under steward_db slowed badly (checkpoint
sync 4.6 s, then 70.7 s with one file at 21.9 s, against the usual 0.002 s).
Ingest commits stalled past the agent's 10 s timeout and were dropped, and
last_seen_at, written in that same transaction, went with them. Agents
posting every few seconds read as silent for 3 minutes, and "Agent offline"
fired and emailed for all three hosts.
Ingest now notes each agent's arrival in memory right after the (read-only)
registration lookup, before any write; seconds_since_seen counts from the
later of last_seen_at and that arrival.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5470 "pysnmp-lextudio is deprecated: every startup logs a RuntimeWarning".
Dependency change: the [snmp] extra now installs `pysnmp>=7.1` instead of
`pysnmp-lextudio>=6.2`. pysnmp is the same lextudio lineage under its maintained
name; the 7.1 floor is because below 7 the name is the abandoned 4.x line with
no asyncio API. The poller already finds either API shape (_api(), the engine
close probes), so no code changes; its docstring now names what the image ships.
Approved by the operator 2026-10-09 ("yes please do the swap").
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5468 "Log search by service name: searching "steward_app" refuses because every
deploy makes a new container name".
search_container_logs(container="steward_app") was refused, because no container
carries the service's own name and five task containers partly matched. Now a
service's name reaches every task container Steward holds anything for, live and
removed (docker_svc.service_containers, as the service page's logs already do),
so one search spans deploys. Each match carries the slot and task_id that wrote
it, and the reply names the service and how many tasks were searched. A container
named exactly what was asked still takes precedence. A miss that partly matches
task containers now names their services.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5467 "Steward's own health: metrics, rules can watch them, a page that shows them".
The self-monitor tick now records, under source "steward", resource "process":
ingest_posts / ingest_failed / ingest_slowest_ms (the host-agent posts since the
last tick, tallied in steward/core/health.py), db_roundtrip_ms (a timed SELECT 1),
and unwatched_secs (the gap before this tick past watch.GAP_S, from Steward's own
last sample). The hourly cleanup records cleanup_secs. The open-file pair is still
recorded, and is now only skipped on its own where /proc is absent. All of these
are in CORE_METRICS, so an alert rule can watch them under "Steward itself".
Settings → System (/settings/system/, admin) shows a verdict first (failed posts,
a gap in watching, open files near the limit), then each measure now and over the
last day, with a sparkline scaled to its own peak.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5449 "The hourly Docker log prune ranks every log line in the table to find
the few over their cap": the size ring put a running-total window over every
row of docker_logs each hour, sorting the whole table on NFS-backed storage
to find the handful of containers past docker.logs.max_bytes_per_container.
- prune_docker_logs (capability docker.prune_logs) does the age cutoff, then
a per-container byte total picks the containers at or over the cap, and
only their rows are ranked and trimmed. Same keep rule as before: the
newest line always survives.
- run_docker_retention keeps the metrics rollup and the event prune.
- Cleanup runs the log ring in its own transaction after them.
Arose from #5431 "Ingest dropped for ~3 minutes after each deploy while
startup retention runs".
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
After Steward or its database was down, every agent's age and every
series' last sample were old, so all agent-offline rules fired and all
rules went no-data at once, clearing a minute later. With per-account
alert email that would have mailed everyone a dozen notices per restart.
A periodic job now ticks a Watch (steward/core/watch.py) once it has
reached the database; the Watch says since when runs have been unbroken
(process start, or the end of a gap over 120 s). seconds_since_seen and
the no-data sweep count silence from no earlier than that, so a source
must stay quiet while Steward is watching before it counts as silent.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5207 "UniFi Overview tab: func.strftime is SQLite-only": the WAN and
client sparklines bucketed with strftime('%s'), which Postgres lacks, so
/plugins/unifi/rows failed. Traefik's date_bin bucket moves to
core/time_range.time_bucket and both plugins use it. Devices list offline
and failing first, and a state other than online is named (Upgrading,
Adopting, Heartbeat missed...) instead of drawn as Down.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The last page of #5214's walk. Settings still described Steward as it was
before monitors were unified and SNMP was rebuilt, drew its tabs below the
breadcrumb, and showed the report schedule in UTC.
- Tabs (#5377): settings/_tabs.html is a settings_tabs() macro called in
each page's section_tabs block, above the trail, as Ansible's are. The
breadcrumb guard also fails any page that {% include %}s a tab strip.
- Report time (#5378): reports.schedule_tz keeps the zone of whoever saved
the schedule; check_and_send compares in it (report_due), and the form shows
the day and hour on the reader's clock (schedule_in) under their zone's
name. "Last sent" uses |when.
- Wording (#5379): the report contents list what the report holds (monitors,
not ping-enabled hosts); "Ping latency" is "Response time" and says what it
grades; plugin detail says enabling loads now and disabling needs a
restart; the monitor interval says it applies on the next check (the
scheduler reads it every pass); the dead list-config branch is gone.
- Traefik (#5380): the 5xx and 4xx cutoffs are settings
(traefik.threshold.error_5xx_pct 0.1, error_4xx_pct 1.0) in a "Traefik
routers" group on Thresholds while the plugin runs, read by router_state
through error_cutoffs(). The Traefik page has a Thresholds button for
admins, to /settings/thresholds/#traefik.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
UPS-MIB upsSecondsOnBattery is zero whenever the UPS is on mains power, so
"On battery for 0s" read as a timer that had just started. The reading gets
its own unit, battery_seconds: zero says "No, on mains", anything else
"for <span>". The metric key stays on_battery_for, so stored history keeps.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
First-party plugins have shipped inside the image since a7a281c, and the
catalog's only repository (Steward-plugins) answers 404, so Steward polled a
dead URL every minute for nothing. The operator chose to remove the catalog
entirely (#5214 "Review: readable and fast on the live instance").
Removed:
- core/plugin_index.py and the plugin_catalog_refresh startup task;
- the plugins.index_url default, and migration 0035 deletes the stored
plugins.index_url / plugins.repositories rows;
- the Settings -> Plugins repository editor, catalog, refresh, install and
reload routes and their templates;
- the download/checksum/extract install path in plugin_manager, and the
PLUGIN_INSTALL_DIR config it wrote to;
- plugins/index.yaml, docs/plugins/index.yaml.example and the guide's
"Publishing to the Catalog" section (now "Shipping a Plugin").
Kept: enabling and configuring plugins, hot-reload on enable, restart, orphan
cleanup, and the external root (/data/plugins) for operator-mounted plugins.
tests/test_no_plugin_catalog.py guards the routes, default and task.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The operator, reviewing SNMP on #5214: "this system looks incomplete like
there's something missing from it". The one device shown was plugin.yaml's
example (core-switch 192.168.1.1, community public), polled every minute;
devices could only be defined by editing YAML in the image; a failed poll was
logged at DEBUG and stored nowhere, so the page said "waiting for the next
poll" forever; v3 ran as v2c; counters showed as lifetime totals.
- Devices are rows (snmp_devices, migration snmp_001): address, port,
v1/v2c community or a v3 user with auth/priv protocols and keys (secrets
encrypted), presets plus custom readings, an optional host binding, and
the newest poll's outcome. The YAML device list and its example are gone.
- The SNMP page adds, edits and deletes them (admin, audited, with
get_audit_entities so the audit log names and links a device). Readings
come from presets (system uptime, interface traffic by number, UPS-MIB,
Linux net-snmp) and custom "OID name [unit]" lines.
- One GET per device with every OID, devices polled at once. The outcome is
written to the row in words ("No reply from … within 5 s. Check the address
and port, that SNMP is on, and the community"), with readings it lacks.
v1's whole-request error drops that OID and asks again. SNMPv3 signs in
with USM (MD5/SHA-1/SHA-2, DES/3DES/AES).
- Test asks the unsaved form's device for its name, description, uptime and
interfaces by number. It is an admin's explicit action; every page reads
only stored rows (System 178's ruling).
- Each reading has a unit; a counter is stored as a per-second rate (bytes as
bit/s), a reset or wrap skipped. The page, widget and host panel say values
in units and why a device is silent; the device page draws a chart per
reading on a dated axis.
Tests: readings and presets, the form and its secrets, v3 checks, rates,
the poller against a stand-in pysnmp (one GET, v1 retry, timeout wording,
USM), the pages, and an integration test of the poll writing outcomes and
rates on Postgres.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The phase log on the live instance: 'traefik rows: slow 7.4s (latest=7.04s
history=0.11s window=0.04s ...)'. latest_per_router walked traefik_metrics'
index one router at a time; each step lands on the router's oldest
entries, which retention deletes every minute (likely dead ones). The
scraper already records every router figure through MetricBatch, which
keeps one row per router and figure in plugin_metric_latest (#5208's
table, read as latest_by_resource does); latest_per_router reads that
now, and the router figures carry the scrape's own time.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The operator: "load was still slow" (the access log: 7.6-8.0 s on every
poll, warm), "why do the bandwidth and 4xx sections not also get
sparklines", and "no gap between the widgets". router_history reads the
4xx share and bandwidth too, and the card draws all five lines. The
problem cards sit in .node-grid with a bottom margin; a CSS column box
dropped the last card's margin. rows() and widget() log each phase when a
request passes 1 s, so the slow query names itself on the next load.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The operator: problem cards "can disappear so quickly". Each router was
graded on its latest scrape, so a 4xx share near the 1% cutoff crossed it
on one scrape and fell back on the next. recent_window reads every
router's last 15 minutes in one query, the error shares weighted by
traffic; grade_routers uses it on the page and the dashboard widget, and
the why says "over the last 15 min".
traefik_config_last_reload_success is the Unix time of the last good
reload, not a 0/1 flag, so the tile said "Reload failed" on every scrape.
traefik_005 stores it as config_last_reload_at, and the tile reads
"Config reloaded 3m ago". The router index also carries the 4xx share.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5355 "Traefik page doesn't load": /plugins/traefik/rows took 12-28 s.
It ran a query per router ever seen, and each raw row in the range was a
heap fetch. traefik_004 rebuilds the router index to cover the charted
columns and indexes scraped_at for the request chart; router_history reads
the current routers' buckets in one query.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Agent 1.12.0 stamps each post with sent_at; ingest compares it with the
moment the post arrived: clock_offset_s / clock_checked_at on the
registration (host_agent_003), and a clock_off_secs metric (either way)
that alert rules can watch. Replaces the old skew log, which measured the
newest sample and was off by up to an interval.
- Settings -> Thresholds: Clock drift, 30 s warn / 5 min critical.
- Host page: 'clock 0.3s ahead' in the vitals line, toned past the warning.
- Hosts list: names hosts whose clock is off; when every reporting host is
off by about the same amount, says Steward's own clock is the likely one.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
/ansible/inventory/ shows every target (address, its linked Steward host,
its groups) and every group (members, the schedules aimed at it) under a
verdict line. The whole row opens the target or group. Add target and
Add group open small dialogs; a failed add comes back with its dialog
open and what was typed. The red x leaves the rows: Delete is on the
target's or group's page, behind the confirm dialog, saying which
schedules would be left with nothing to run on.
The two list pages (/inventory/targets, /inventory/groups) are gone
with no redirect (rule 122); their URLs keep only the Add POST.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Confirm dialog (#5314):
- Every data-confirm form now asks in one native <dialog> (base.html + static/confirm.js)
instead of window.confirm.
- The question becomes the heading and what follows it the detail; the act's
button is named after the form's own button; Cancel has the focus and Esc
backs out. Optional data-confirm-title/-detail/-action/-dismiss attributes.
- All 27 existing uses work as they are. Cancelling a run says 'Keep running'.
Alerts list (#5315):
- A verdict line (what is firing, how many rules, where notifications go,
or 'notifies nobody yet' with a link) replaces the empty Active card.
- Each rule's name opens it; rows show 'last fired' and fold fire/clear into
the condition; History and Disable stay; Delete leaves the list.
- .docker-verdict becomes the shared .page-verdict.
Rule page (#5317):
- Named for the rule (title and crumb), with when it last fired and how many
changes of state it holds.
- Three sections: what it watches, when it fires (checks side by side,
seconds echoed as a duration), what happens (where notifications go, no-data,
the playbook, whose fields stay folded until ticked).
- Delete at the foot, behind the dialog, saying its history goes with it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Service page:
- The header carries the verdict, mode, stack, image and when it was reported.
'In trouble since' shows only while the service is in trouble; once recovered,
'Last failed X ago · its logs'.
- The failure card shows only while the service is down, degraded or crash-looping.
- The tile strip and the kv card are gone.
- One Tasks table: running tasks first with CPU, memory and since when, then
Swarm's record. The node is named as its Steward host throughout, and
'N failed in 24h' sits in the caption.
Container page:
- Same shape: header verdict, a 'Last exit' card only while it is stopped
after a failing exit, set-up and I/O in one kv card, and lifecycle as a table.
- A swarm task's breadcrumb goes through its service.
Both pages:
- Resource history is the one fragment (_usage_history.html): now and peak in
cores and bytes, drawn with a floor of 1 core / 1 GiB (sparkline_svg floor=),
so idle noise draws flat.
- The container page's old % sparklines, its private _sparkline and the unused
bucket()/container_history() helpers are removed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- #5304 "Docker disk totals recorded over time per host, with a growth chart and alertable metrics"
- #5305 "Disk page: images grouped by repository, versions folded under it"
- #5306 "Docker overview: one disk line per host linking to the Disk page"
Each `docker system df` report is now also recorded as metrics, with the host
as the resource: disk_total_bytes, its parts, and disk_reclaimable_bytes. An
alert rule can watch them. The Disk page starts with the hosts side by side,
showing how much each grew over the selected range with a line of it, then
gives each host its own section. An image's versions fold under its
repository, which is charged only for the layers its versions don't share.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5303 "Swarm page laid out by node: role, resources and the tasks placed on each"
A verdict line first, then the nodes that are down or drained and the
services short of replicas, then one card per node. A node with a Steward
agent shows what its swarm tasks use against the machine's cores and memory,
service by service. A node without one lists Swarm's placement, by count.
The services table moved to the Docker overview, where it can be filtered.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- #5298 "Docker overview: verdict line; only the problem tiles that aren't zero"
- #5299 "Docker overview: swarm services as a filterable, sortable table with a Stack column"
- #5300 "Docker overview: failures listed once, each with a link to the failed task's logs"
- #5301 "Docker overview: standalone containers in one table with a Host column"
- #5302 "Service page: replicas, image, task history, resource history and logs across all its tasks"
overview.py holds the rows, filters and sort as plain functions. The filter
bar sits outside the polled fragment and the URL keeps its state. A service's
page reads every task container Steward still holds anything for, the live
ones and removed ones known by their events, so a failed task's logs are
reachable after Docker has removed it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5290 "Host page: the health tiles span the card's width; even spacing above Ansible":
the four tiles take equal quarters (halves on a phone) and their sparklines stretch
across them; the Monitors + Agent row drops the Agent card's own margin, so the gap
above Ansible is 1rem like every other.
#5291 "Add host opens as a dialog over the Hosts list": a native <dialog> on /hosts/,
opened by Add host or /hosts/?add=1. A name clash re-renders the list with it open,
the error and what was typed. GET /hosts/new is gone; hosts/form.html is Edit only.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Hosts list:
- CPU and Memory are their own labelled columns (#5279); a silent
agent's last readings are greyed, and Status says "agent silent for 7m".
- One uptime figure, 30 days, with 24h and 7d on hover. The SLA tab
(/hosts/uptime) repeated it and is removed (#5280; one home per fact).
- Status is a word with how long ("Down for 12m"); host_status now carries
down_since. Edit and Delete live on the host's page only (#5281).
- Figures are right-aligned with tabular digits (.table .num), not a
monospace face.
Agent fleet (#5282): its add-host form is gone, so hosts are added only on
Hosts -> Add host. That form was also the only way to get a token for a
manual curl install, so that moves to the host's Agent card as "Install
manually" (POST /fleet/<host_id>/register), shown on the fleet page as
before. "Delete" there reads "Unregister".
Add host (#5283): laid out like the add-monitor form; a name clash comes
back inline with what was typed instead of a bare text 400; the address
is optional, as the host page already allowed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Docker panel (#5276): problems first, then the 8 busiest running
containers by CPU; the other running and the stopped fold behind one
"+N more running · M stopped" line. Containers that are not swarm tasks
read the way the Docker page reads them: CI jobs as "CI: workflow · job",
buildx builders as "BuildKit builder · name".
Monitors card (#5277): the edit/delete icon buttons and the inline add
form are gone. Each row links to its monitor's page; "Add monitor" opens
the add form filed under this host, starting from its address, with the
host in its breadcrumbs and Cancel back to it.
Agent card (#5278): Full metrics stays in view; Update, Rotate token,
Unregister agent (was "Remove") and Re-provision sit under "Manage agent".
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A chart built while its cell was still being laid out could stay blank until
a window resize; it now measures and draws again after layout settles. Each
refresh rebuilds the chart (the flash), and its buckets are minutes wide, so
the widget refreshes every 5 minutes instead of every poll.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each figure after its bars takes a fixed width, so memory figures of
different lengths no longer shift their bars. CPU and Memory headers say
"last 6h" and the note gives the slice length.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The fill bars were a share of the host's capacity, a sliver on hosts with
80 cores. Each figure now sits beside 24 bars of the last 6 hours (1h/24h
per widget), each row scaled to its own peak but never below 1 core or
1 GiB (configurable per widget), so a 0.05-core container reads flat and
a bar's height means real load. One grouped date_bin query feeds them.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
View mode splits each saved row into seven, so a widget no longer carries up
to 69px of empty space below its content (Overview, SNMP). Edit mode and the
saved layout keep 70px rows. The Busiest Containers host column aligns left
like its header, its note fits one line, and the Hosts column reads Disk.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
CI 8551 integration: "integer out of range" for a 2099 timestamp.
The SQL in the log showed a second problem: `CAST(epoch AS INTEGER) / 60`
compiled to `/ CAST(60 AS NUMERIC)`, i.e. decimal division. Every scrape got
its own "bucket", so the request chart and the Traefik page's sparklines
never averaged anything. The int4 cast would also overflow in 2038.
Both now use date_bin from the epoch, as host_agent/metrics_query.py does.
The integration test adds a second scrape 30 s later in the same minute and
checks that the bucket averages the totals and weights the 5xx share over
both scrapes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5256 "Traefik — Request chart: blank on the live dashboard; time axis and honest labels".
- "Req/s" was the average of the routers, not the total. Each scrape is now
summed over routers, then averaged per bucket (request_history).
- The 5xx share is now weighted by each router's traffic. The axis is no
longer pinned at 0–100 (suggestedMax 5).
- The x axis shows times, in the viewer's local time. The tooltip shows
both series at that time.
- A verdict line comes first: now / peak req/s and the peak 5xx share. The
figures read even if the chart doesn't.
- The blank live chart's cause is still unknown. Chart.js and
chart-colors.js both load on the dashboard. A missing library or a thrown
error is now written into the widget instead of leaving it empty, so the
cause will show on the page.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5255 "SNMP — Devices: status in words and key readings".
- Verdict first: "All N devices answering", "ups: no reply since 14m ago",
"N of M devices not answering", or "N not polled yet".
- Devices that stopped answering come first, then those never polled, then
the rest.
- One header row: Device / Readings / Polled.
- The name is the link; the separate "detail →" is gone.
- The hollow dot is replaced by words: "no data yet", or "no reply since …;
readings are from then".
- Key readings: the first two in config order, gauges ahead of counters.
Units come from the label suffix: _centisecs as a duration, _pct, _dV as
volts, _octets as bytes total.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5254 "Traefik — Routers: current routers only, header row, certs on the router they serve".
- Router hosts come from Traefik's API (/api/http|tcp/routers), read
best-effort on each scrape. An optional api_url overrides it; the default
is /api on the metrics host. A certificate's expiry then shows in a TLS
column on the router whose host it covers. An expiring one makes that
router a problem, graded by Settings → Thresholds.
- Only current: routers and certificates must have appeared in the last
three scrapes. Certificates are upserted by serial and never deleted, so
renewed or retired ones lingered on the widget.
- Widget order: verdict, problem routers with the reason, then the busiest
healthy routers under one header (Router / Req/s / Bandwidth / p95 / TLS).
A missing value shows "—".
- If the API is unreachable, expiring certificates are listed after the
routers, not above them.
- The page splits certificates into served (with the routers using each,
plus SANs) and "No longer served".
- Latest row per router: an index walk (recursive CTE + LATERAL) replaces
DISTINCT plus a query per router.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The widget names the busiest container in a verdict ("Busiest: tdarr on
VDNT-Docker03, 1.6 of 8 cores."), then a fixed-layout table (container,
host, CPU in cores, memory in bytes) with each figure's share drawn as
shading behind it: of the host's cores, and of the memory limit or the
host's memory.
The numbers were already rendered, but each row grew as wide as its longest
name. A grid item won't shrink below its content, so Gitea CI container
names pushed the figures out of view and the cell scrolled sideways. Names
now come from display_name(): a swarm task by its service (no slot number),
an Actions job container as "CI: <workflow> · <job>". They cut off with …
and show whole on hover.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A verdict line (services healthy · other containers running · failed in
24h), then "Needs a look": troubled services with why (crash-looping, N of
M running), since when and the last error, and troubled containers
(unhealthy, restarting, exited with an error, out of memory). Then "Failed
in the last 24h" by name, with how often, when and where. Then the healthy
rest one line each (name · hosts · replicas) in as many columns as the
widget is wide, collapsing after 40 to "+N more healthy".
Per-container CPU is gone from this widget: Busiest Containers owns it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- New installs start from widgets.RECOMMENDED, packed row by row for the
enabled plugins; the edit drawer can reset a dashboard to it.
- Host Agent — Resources folds into Hosts — Overview (which gains load 1m),
UniFi — Devices into UniFi — Overview; migration 0032 moves existing
dashboards onto the survivors.
- The Overview labels what it counts (hosts, monitors up/down/pending,
alerts firing) and lists firing alerts and down monitors with since-times.
- Uptime polls every 5 minutes (poll_seconds) instead of every cycle.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5212 "Plain labels: no raw ids, scopes, metric keys or unexplained jargon",
batch 3:
- Disk page: each tile says what it holds. "Writable layers" becomes
"Container files" and Build cache says it is safe to clear. Shared is
"Shared with other images" with a tooltip, and "dangling" is gone.
Untagged images show by id instead of "<none>", a capped list says
"The 50 largest of N images", and each host shows "as of" its report.
- Traefik: an expired certificate reads "expired 12 days ago", not "-12
days" (cert_left). The Traffic empty state knows whether the access log
is on: off points to Settings → Plugins → Traefik; on says to check the
mounted log path.
- Moments use |ago with an exact UTC tooltip: UniFi last seen, alarms,
events, speed test and DPI; AI token last used.
- Settings: the Auth tab is "Authentication" like its breadcrumb, plugin
fields read "Metrics URL" not metrics_url (field_label), a source's pull
interval is "every 1h", and the last report time drops the ISO "T".
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5212 "Plain labels: no raw ids, scopes, metric keys or unexplained jargon",
batch 2:
- A run records where it came from: AnsibleRun.origin ("schedule", "alert"
or "mcp") and origin_id, set by the scheduler, Run now, alert actions and
the MCP tools. Migration 0031 backfills past runs from alert events,
schedules' last runs, and MCP audit entries.
- The runs list has a Started by column: "schedule nightly", "alert rule
nas down", "alice via MCP" or "alice", linked where there is a page.
The raw scope tooltip is gone.
- Run detail: no run id in the subtitle or body, Started by in words, and
the host summary lists failed and unreachable hosts first. The live
output only follows new lines while the reader is at the bottom, and
when the run finishes the page reloads once so the badge and summary
are final.
- list_ansible_runs and get_ansible_run report origin and origin_id.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5210 "Problems first, with since-time, across every list" (milestone 477),
second batch.
- Host page: an "Alerts on this host" strip first (rules on its agent
series, its containers or its monitors: alert_svc.watches_host); its
monitors problems first with "Down 12m · reason"; Docker above Ansible.
- Uptime page and widget: worst 24h uptime first, then 7d and 30d; hosts
with no uptime yet last.
- Agent panel and fleet widget: last seen as "3m ago", not a clock time.
- SNMP: a device is "Reachable" only while it answered within the last 3
polls; older readings say "No answer" with their age, and every device
says when it was last read (was: reachable whenever any reading existed).
- Traefik: a router missing from 3 scrapes in a row is listed apart as no
longer reported, by when it was last seen, instead of keeping its last
state (a removed router stayed Failing).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
failing_services filtered on `(service_trouble(s) or 9) <= 2`; a crash loop
ranks 0, which is falsy, so the worst kind of trouble was left out. Test the
rank against None. The page test's task error also gains the age_s the
_swarm_failures include reads.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5209 "Docker views: readable names, CPU/memory in cores and bytes, failures
listed with since-time" (milestone 477 "UI problems first").
- A swarm task reads as its service and slot ("codex_db · 1", or the node
for a global service), with the raw name in the title: Docker page, host
panel, busiest widget, container detail.
- CPU reads in cores (Docker's 100% = one core), toned by its share of the
host's CPUs from the agent's per-core series. Memory reads in bytes, against
a limit only when one is really set (Docker reports host RAM otherwise).
- Exit 137/143 without an OOM kill is a stop, not a failure: no
exited_nonzero, no crash-loop death, no "failed (24h)", dim exit code.
- "Failed (24h)" is a list (Docker page #failures, linked from the widget
and the tile): plain containers by their deaths, swarm services by Swarm's
failed tasks; a second manager's copy of a task death counts once.
- Failing swarm services lead Needs attention; services sort problems first
with since-times on the page, the widget and the Swarm page; nodes that
are down or drained sort first.
- Busiest widget orders by a 5-minute CPU average so the list holds still.
- Host panel: a table, problems first with what is wrong and since when;
replaced swarm tasks hidden and counted.
- Container detail: node hostname, I/O labelled "since start", Restarts
hidden for swarm tasks with a note on where task failures are.
- Widget names: "Docker — Services & failures", "Docker — Busiest containers".
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The dashboard's host widgets took 18-60 s. Every "latest reading" query
(hosts overview, host-agent fleet, host page, vitals, SNMP, MCP get_host)
searched plugin_metrics for the newest row per metric. Postgres has no skip
scan, so each one read every raw sample in the 7-day window.
- plugin_metric_latest: one row per (source, resource, metric).
record_metrics upserts it in the same transaction, moving forward only and
in sorted key order. Migration 0030 seeds it by stepping the existing index.
- Retention drops a latest row once its sample leaves the raw window, so a
resource that stopped reporting disappears when its history does, as before.
Renames carry it along.
- A quiet host keeps its last values, marked stale. A time-bounded query
would have shown it as missing.
- Vitals sparklines span the 6 h they claim, averaged in SQL to ~120 points,
instead of fetching 6 h raw and drawing the last 120 samples.
- Dashboard and share-view cells: "Loading…" until the first answer; a 20 s
deadline per fetch; "Couldn't load (why)" with Retry now and an automatic
retry for cells that don't poll; "as of HH:MM" in the header, and
"couldn't refresh · as of" when a refresh fails over shown data.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- prune_by_age(column) registers a time-series column with the hourly cleanup;
Traefik (metrics, access summaries, global stats) and UniFi (WAN, client,
WLAN, DPI, events, alarms, speedtests) register theirs in setup().
- Alert events join monitor results and Ansible runs on data.retention_days,
read from Settings each run; Ansible run log files older than it are deleted.
- audit.retention_days (365) keeps the audit log longer, set on Settings > General.
- metric_series.points reads metrics.retention.raw_days, so the chart's
raw/rollup seam is where the rollup put it (RAW_DAYS removed).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
MAX_SAMPLES_PER_INGEST was 2 while agent 1.10 drains a backlog five samples a
post, so the server dropped the oldest three of every such post. Posts now
answer in 0.4-1.3 s on dev, including the boot cleanup window, so five fit
the agent's 10 s. A test holds the server cap at or above the agent's.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
