The full-metrics page rendered once server-side with the history query inline,
so the charts blocked first paint and nothing refreshed without a reload (it
reads the latest snapshot the server holds — the agent pushes ~every 30s).
Split it into a shell + two HTMX fragments:
- Shell (host_detail.html): header + shared time-range toggle + two containers;
paints instantly.
- Current state (/<id>/metrics → _host_metrics.html): identity + gauges +
per-core + filesystems + interfaces/disks + temps, from the DISTINCT ON latest
query. hx-trigger "load, every 15s" → live numbers at ~the agent cadence.
- History charts (/<id>/charts → _host_charts.html): the 3 charts + Chart.js,
from the date_bin history query. hx-trigger "load, every 60s, rangeChange" so
they lazy-load (never block paint), refresh slowly, and follow the range
selector. Leak-safe: previous Chart instances are destroyed before re-render.
- Range toggle switched from full-reload links to the shared _time_range.html
(setTimeRange + rangeChange), so only the fragments refetch.
- routes: host_detail (shell) + host_detail_metrics + host_detail_charts, with a
_split_host_metrics helper.
- tests/test_templates_parse.py now also parses plugin templates (these
fragments aren't rendered in the unit lane).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC
The previous commit's integration test failed at collection-time import:
importing plugins.host_agent.routes (which imports the host_agent ORM models at
top level) double-registers host_agent_registrations against the app-loaded
plugin's metadata ("Table already defined").
Move the two pure read helpers (_latest_metrics_for_host, _history_for_host)
plus SOURCE_MODULE / HISTORY_METRICS into plugins/host_agent/metrics_query.py,
which imports only the core PluginMetric model — no plugin models. routes.py
imports them back. The integration test now imports from metrics_query and no
longer trips the loader's registration guard. No behavior change.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC
Follow-on to the plugin_metrics indexes. plugin_metrics is already retention-
bounded (core cleanup prunes > data.retention_days, default 90d) and charts top
out at 30d, so the cost wasn't growth — it was the read path shipping raw rows
to Python.
- _history_for_host: bucket + average in SQL via date_bin (epoch-aligned, ~120
buckets) instead of fetching every raw sample (a 30d range was hundreds of
thousands of rows) and downsampling in Python. Uses the new
(source_module, resource_name, recorded_at) index.
- _latest_metrics_for_host: DISTINCT ON (resource_name, metric_name) ORDER BY
recorded_at DESC — newest row per group in one index-ordered pass, replacing
the GROUP-BY-max subquery self-joined back to the whole history.
- Integration test validates both against Postgres.
Deliberately not a materialized raw→hourly rollup: these query-side changes
deliver the speed; a rollup would additionally cut storage and remains a future
option if scale demands it.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC
Two issues on the host full-metrics view:
- fmt_bytes capped at GB, so large filesystems read "25369.0 GB" instead of
"24.8 TB". Add TB and PB tiers.
- Interfaces/Disks/Temperatures shared one 3-column grid, so all three cards
stretched to the Temperatures height — a 40-core CPU made one very tall column
and left Interfaces/Disks with large empty space. Split them: Interfaces +
Disks sit side-by-side at natural height (align-items:start); Temperatures
becomes a full-width card whose readings flow into a compact multi-column grid
(wide-and-short instead of one tall column).
UI-only.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC
docker_metrics.mem_usage_bytes and docker_containers.mem_usage_bytes /
mem_limit_bytes were int4 (max 2,147,483,647). A container using >2.1 GB of RAM
(e.g. 8.18 GB) overflowed the column, so asyncpg raised "value out of int32
range" and the entire docker ingest batch failed — no metrics stored for any
host with a large container.
The I/O counters and the milestone-77 rollup/disk tables already used
BigInteger; this trio (docker_001-era) was missed.
- models.py: DockerMetric.mem_usage_bytes, DockerContainer.mem_usage_bytes,
DockerContainer.mem_limit_bytes → BigInteger.
- migration docker_008_bigint_mem: ALTER COLUMN ... TYPE BIGINT (safe in-place
int4→int8 promotion).
- integration regression test: persist an ~8 GB container, assert the
current-state and time-series rows round-trip.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC
Plugin UIs had no nav home (only dashboard widgets / typed URLs). Add an
optional get_nav() plugin export and surface it in the sidebar.
- plugin_manager: _PLUGIN_NAV registry + get_plugin_nav() getter + a tolerant
_collect_plugin_nav() (missing hook = fine; raising/malformed = logged &
skipped; idempotent per plugin so hot-reload re-runs cleanly). Collected in
both the startup load path and the hot-reload path.
- app.py: inject plugin_nav into the template context, filtered to enabled
plugins so a hot-disabled plugin's link can't linger before restart.
- base.html: render plugin links under the Infrastructure group, with the same
request.path active-state treatment as core links.
- docker/snmp/unifi/traefik: each exports get_nav() → its /plugins/<name>/ view.
- Tests: collector behavior (collect, missing hook, reload-replace, malformed
item, raising hook, sorted output).
With docker enabled, "Docker" now appears under Infrastructure → its fleet view.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC
The SNMP plugin ships in the image but logged "pysnmp not installed — SNMP
polling disabled" on every poll, so polling never worked. Two coupled defects:
1. The Dockerfile installed only `.[ansible]`, so the `snmp` extra (pysnmp)
was never bundled even though the plugin is first-party and shipped.
2. poller.py used the synchronous pysnmp HLAPI (`next(getCmd(...))`), which
pysnmp-lextudio 6.x removed — it's asyncio-only now — so even with the dep
present, polling would have thrown and silently returned nothing. The 5.x
line that still has the sync API isn't safe on the image's Python 3.13.
Fix:
- Dockerfile: install `.[ansible,snmp]`.
- poller.py: `poll_device_sync` → `async def poll_device` on the asyncio HLAPI,
with a dual-version import (pysnmp 7.x `pysnmp.hlapi.v3arch.asyncio`/`get_cmd`
+ async `UdpTransportTarget.create`; pysnmp-lextudio 6.2.x
`pysnmp.hlapi.asyncio`/`getCmd` + direct `UdpTransportTarget`) so a dependency
bump can't silently re-break it.
- scheduler.py: await poll_device directly; drop the run_in_executor wrapper
and the now-unused asyncio import.
- Add tests/plugins/snmp/test_poller.py covering the version→mpModel mapping,
that the poller is a coroutine, and the graceful no-pysnmp path.
Note: CI confirms import/load and the no-pysnmp path, but has no SNMP target —
live polling against real devices is verified after deploy.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC
The page-top chrome was inconsistent: most nested views used the breadcrumb
kicker + page-title pattern, but plugin sub-pages used ad-hoc "← back" links,
~11 pages stacked a breadcrumb AND a redundant ancestor back button, and two
(monitors/edit, hosts/uptime) had a back button but no breadcrumb. The
settings/plugin_detail page stacked all of it at once.
Unify on the breadcrumb-led model:
- settings/_tabs.html: drop the hardcoded "Settings" h1; the breadcrumb
("Settings › …") plus the tab strip is the header.
- settings/plugin_detail: drop the "← Plugins" back button.
- docker container_detail/swarm/disk + snmp/device: replace ad-hoc back links
with the standard crumbs() breadcrumb.
- host_agent, ansible/*, alerts/maintenance: remove redundant ancestor back
buttons (the breadcrumb's parent crumbs already link there); keep lateral
shortcuts (Inventory/Schedules/Browse/Targets/Groups/New).
- monitors/edit, hosts/uptime: add the missing breadcrumb, drop the back link.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC
Container list now sub-groups each host's containers by compose project (or
swarm service) with a small subheading, preserving the running-first order;
hosts with no such labels render flat as before. The dashboard status widget
links each container to its detail page and surfaces enriched state inline —
health dot (healthy/unhealthy), restart count, and last non-zero exit code for
stopped containers. Completes the #942 UI surface.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC
Two new read-only sub-pages, linked from the Docker index header only when the
data exists (non-swarm / Docker-less installs aren't offered empty pages):
- /plugins/docker/swarm — services with replica health (running/desired,
colour-coded green/amber/red), Swarm nodes (role/availability/status, leader
badge), and task→node placement with node ids resolved to hostnames. Grouped
by reporting manager host. Empty state explains manager-only collection.
- /plugins/docker/disk — per-host reclaimable space, image/layer/container/
volume/build-cache sizes, stopped-container count, and a per-image table
(size, shared, ref count, reclaimable badge). Notes prune actions are deferred.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC
New /plugins/docker/container/<host_id>/<name> detail page (v1 quality):
status/health badge, uptime, CPU/mem, restart count, last exit code (+OOM),
net + block I/O (humanised), image/compose/swarm-service/node/ports, a
range-toggled CPU/mem history graph (HTMX fragment reusing the time-range
selector), and a lifecycle timeline rendered from docker_events (glyph+colour
per event kind). Not-found and empty-history/empty-timeline states included.
Container names in the main list and the per-host hub panel now link to it.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC
Backend for the image/disk panel (milestone 77 #942). Agent gains
collect_disk_usage() — one /system/df call (gated on containers existing, so
Docker-less hosts pay nothing), surfacing reclaimable bytes (image size held by
unreferenced images), layers/containers/volumes/build-cache sizes, and the top
50 images by size. Emitted as sample["docker_disk"]; host_agent ingest tracks
the newest sample's copy and hands it to the docker capability as a 5th arg.
New current-state tables docker_disk_usage (one row/host) + docker_images
(per-host image rows), docker_007 migration; ingest upserts the summary and
replaces the image set per host (stale images pruned). Unit tests for the df
parsing/reclaimable math + build_sample gating; integration test for
persistence + image-set replacement.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC
Bounds Docker time-series growth (the main scaling concern). New
docker_metrics_hourly table + docker_006 migration; a plugin retention module
(docker.run_retention capability) rolls raw docker_metrics older than the raw
window into hourly averages (idempotent upsert), deletes the rolled raw rows,
then prunes stale rollups + lifecycle events. Core cleanup.py drives it each
hourly run via the capability (no plugin-model import), reading the three
retention windows fresh from settings so changes apply without restart (rule 25).
Settings → "Thresholds & Retention" gains a Docker retention card (raw /
rolled-up / events windows, working defaults 7/90/30 days). Unit tests cover the
hour-aligned cutoff/bucketing helpers; integration test exercises the real
rollup-average + prune across both windows.
Milestone 77 task #941.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC
Wires the agent's enriched + swarm payloads through the docker.persist_host_
samples capability:
* Swarm topology — persist sample["swarm"] into docker_swarm_services /
docker_swarm_nodes (upsert + prune stale, host-scoped so two managers don't
clobber). Migration docker_005 adds services.placement_json for the
task→node placement the agent now reports.
* Lifecycle events — _derive_events (pure, unit-tested) diffs the newest
snapshot against stored per-container state: start / stop / die (non-zero
exit) / oom / health_change → docker_events rows. Skipped on a host's first
snapshot so the baseline doesn't emit a start per existing container.
* Alerts — record restart_count (always) and is_healthy (1.0/0.0, only when a
HEALTHCHECK exists) alongside cpu/mem, under host-scoped resource names;
METRIC_CATALOG[docker] gains restart_count + is_healthy so they're alertable.
host_agent ingest captures the newest sample's swarm object and threads it to
the capability (now persist_host_docker(session, host, snapshots, swarm=None));
invoked when containers OR swarm are present, under the same SAVEPOINT. Unit
tests cover the event-diff matrix; integration tests cover event derivation
across two snapshots and swarm topology round-trip (incl. placement).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC
Adds collect_swarm(socket_path) to the host agent. Self-detects a Swarm
manager via /info (Swarm.ControlAvailable) — workers and non-swarm daemons
return None and never touch the manager-only endpoints (one cheap /info call,
no 503s). On a manager it queries /services, /tasks, /nodes and emits
sample["swarm"] = {services, nodes}:
* services roll desired-vs-running replicas up from the task list (replica
health isn't on the service object), handle replicated + global mode, strip
the @sha256 image digest, and carry cross-node task→node placement.
* nodes normalise role / availability / status + the manager leader flag.
build_sample omits the swarm key entirely off managers, same silent contract
as collect_docker. Unit tests cover manager detection, replica roll-up +
placement (replicated & global), node normalisation, worker silent-skip, and
build_sample wiring.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC
Adds the milestone-77 storage that doesn't fit on the per-container row:
* docker_events — lifecycle (start/stop/die/oom/health_change), to be
derived by diffing consecutive host snapshots; host-scoped, indexed for
timeline lookups (host_id, container_name, at) and retention pruning (at).
* docker_swarm_services / docker_swarm_nodes — manager-reported Swarm
topology (desired-vs-running replicas, node role/availability/status).
Migration docker_004 extends the docker branch (down_revision docker_003);
purely additive, no DROP+recreate. event/mode/role are plain strings (no
CHECK whitelist), matching how docker_containers models status. Integration
guard asserts the three new host-scoped tables exist.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC
First slice of milestone 77 (Docker monitoring depth). Surfaces real per-container
stats beyond basic state, all read-only on the existing push model.
- agent (→1.4.0): collect_docker now inspects each container (health, restart
count, exit code, OOM) and reads net + block I/O from the stats payload; pulls
compose project + swarm service/task/node from container labels. Per-container
inspect+stats calls run over a small bounded ThreadPool so the ~1s-per-stats
blocking doesn't stretch the sample on a busy host.
- schema (docker_003): additive columns on docker_containers — health, exit_code,
oom_killed, compose_project, service_name, task_id, node_id, and BigInteger
net/blk byte counters.
- ingest: persists the enrichment + restart_count (.get keeps older agents working).
- ui: Docker page rows now show health badge, uptime ("up 3d 4h"), restart count,
exit code (+OOM) for stopped containers, and compose/service grouping label.
- tests: agent helpers (grouping, inspect fields, net/IO sum) + collect_docker
assembly incl. inspect; integration asserts enrichment round-trips.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC
Docker collection moves off the central single-socket scrape onto the host
agent, giving Docker a real per-host dimension. The Steward host now reports
its own containers like any other host, and same-named containers on different
hosts no longer collide.
- agent: stdlib UDS Docker client (AF_UNIX HTTP/1.1, Connection: close,
chunked-aware), collect_docker() ports the cpu%/mem math; sample["docker"]
added best-effort (silent-skip on absent/unreadable socket). AGENT_VERSION
1.2.0 → 1.3.0; optional docker_socket config key.
- ingest: host_agent ingest hands per-host container snapshots to the docker
plugin via a new "docker.persist_host_samples" capability (no hard import,
no-op when docker disabled), inside a SAVEPOINT so a docker failure never
sinks the host metrics. Resource names are host-scoped ("<host>/<name>").
- schema: docker_containers re-keyed (host_id, name); docker_metrics gains
host_id; docker_002 migration DROP+recreates (dev-only, rule 122).
- ui: Docker page + widgets grouped by host with host links; new per-host
Docker panel embedded on the Hosts hub (gated on docker enabled via a new
enabled_plugins template context). Replaces the SQLite-only strftime
bucketing with DB-agnostic Python bucketing.
- provisioning: install/provision playbooks add steward-agent to the docker
group (best-effort) so the agent can read the socket.
- removed central scrape: docker scheduler.py + scraper.py deleted; plugin.yaml
socket_path/scrape_interval_seconds/include_stopped dropped (plugin 2.0.0).
- tests: agent docker collector units (math, chunked decode, silent-skip,
sample shape, config) + integration (host-scoped schema + persistence).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC
Collapse the three former check types into a single core `Monitor` entity
with one management surface (/monitors), one result table (monitor_results),
and a single scheduled task. Every type can now watch a free-standing custom
destination (optional host_id) — not just a registered Host.
- models: Monitor + MonitorResult replace PingResult/DnsResult; Host loses its
ping/dns facet columns (now Monitor rows linked by host_id).
- checks: monitors/{ping,dns,http}.py pure probes + runner.run_monitor
dispatcher; one monitor_check scheduler with a per-monitor due-filter.
- status: single monitor_status_source replaces the three sources.
- UI: /monitors blueprint (type-aware add/edit/list/widget); host hub shows a
host's linked monitors + "add monitor for this host"; nav + widget registry
+ alert metric catalog rewired. http plugin folded into core and removed.
- migration 0022 merges the http branch, data-migrates host facets +
http_monitors + all three result histories, drops the old tables/columns.
Resolves the per-host ping/dns auto-attach issue (#275): monitors are now
explicit, never auto-added to every host.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC
Move the hardcoded warn/crit cutoffs into Settings -> Thresholds (DB-backed,
live, no restart). New thresholds.* keys + to_thresholds_cfg() + a
threshold_style(value, kind) jinja global that reads them; latency reuses the
existing ping good/warn keys, uptime is direction-aware (floors).
Replace the _macros metric_style/uptime_style macros (now removed) with the
global across Hosts-Overview, host_agent fleet + panel, Uptime/SLA widget, and
the ping page uptime column — all now honor the configured cutoffs. Uptime keeps
its green 'good' look when not degraded.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add a live range switch to the history widget that re-requests the fragment
without entering edit mode. It rewrites the parent cell's hx-get so the choice
survives polling (htmx re-reads the attribute each poll), then fetches at once.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The agent already collects + ingests net throughput, disk I/O, and temps;
add them to the fleet-glance rows (fmt_bps + two-line io cells + temp with
70/85C threshold color), each shown only when the host reports it so VMs/
containers without sensors don't show blank cells.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Hosts-Overview: move the CPU sparkline inline next to the cpu % value it
represents (was floated far right on the name line, reading as unrelated).
- Host Agent history widget: caption the chart with the host it represents
(the panel title is generic) — links to the host hub.
- History widget: snap the y-axis to a stable 0..next-10%-band ceiling instead
of auto-scaling to the exact peak, so the 'zoom' no longer jumps each poll.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Restore the host_agent fleet-glance widget to one horizontal row per host
(name left, metric cells + sparklines right) instead of the multi-column
block grid, and zebra-stripe odd rows so adjacent hosts read as distinct.
Names still wrap rather than hard-truncate.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The agent reports every few seconds, so multi-hour history series were hundreds–
thousands of points — a dense, noisy line (esp. CPU). Bucket-average server-side
to ~120 points (keeps the shape, drops the noise) for both the history-graph
widget and the host-detail charts. Also fix the chart tooltip title showing the
raw epoch-ms (e.g. 1,781,720,471,459) — format it as HH:MM.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Address graphical issues raised from the dashboard screenshot:
- No more truncated host names. Hosts-Overview and Host-Agent-Resources put the
host name on its own line (full, wraps if needed) with its data grouped beneath
it; ping/dns (.ping-name) and uptime widget names wrap instead of ellipsis.
Prefer vertical overflow over cramming/truncating a row.
- History graph fills the panel: drop the fixed 0–100 y-axis ceiling (beginAtZero
+ 8% grace) so the lines use the vertical space instead of hugging the bottom;
axis labels still show real %.
- Container-query breakpoints: the widget body is now a query container, so
fragments restyle to their OWN panel width. Host-list widgets flow into 2 cols
≥520px and 3 cols ≥900px (.host-blocks) — use the width, remove vertical
deadspace — and collapse to one column when narrow. Mirrored into the share view.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Operator ask: show the graphs next to their fields in the fleet widget, like the
host page's AGENT panel. Each row now renders cpu/mem/disk/load as a value with a
trend sparkline beneath it (1h window), instead of bare numbers.
- _fleet_rows fetches a per-host recent series (cpu/mem/load host-level, disk from
the root mount) in one 1h query and attaches a sparkline per metric to each row.
- widget_table.html lays out a metric cell (label + threshold-coloured value +
sparkline) per field, mirroring panel.html. Threshold colour is computed in the
loop and passed into the cell macro (keeps Jinja macro scope clean).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Milestone 72 phase C — bring the host-view graphs onto the dashboard:
- host_resource_history widget reworked into a real host-view chart: epoch-ms
linear axis (no Chart.js date adapter), themed like the host-detail charts,
maintainAspectRatio:false so it fills the resized panel, unique canvas per
widget instance (wid), and empty states ("pick a host" / "no metrics yet").
Was previously unusable — it had a broken time axis and no way to choose a host.
- Add a "host" param type: the edit form renders a live dropdown of hosts
(dashboard routes now pass the host list to the editor); the chosen host_id is
stored in config and fed to the widget.
- Hosts-overview widget gains a per-row CPU sparkline (last hour) via the shared
sparkline_svg helper — the host-view at-a-glance trend, on the main widget.
Charts/sparklines render only in the browser, so CI can't exercise them — needs
an operator visual check.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Milestone 72 phase A (clarity wins, no schema change):
- Add shared metric_style() macro in _macros.html: colours a numeric metric
amber (>=warn) / red (>=crit) on the value ITSELF, not just the status dot.
Defaults 80/90 for percentage gauges; load /core uses warn 80 / crit 100.
Applied to CPU/mem/disk across host_agent panel + fleet widget + the unified
hosts-overview widget.
- Link every host reference to the host hub (/hosts/<id>) in ping, dns,
hosts-overview, host_agent fleet, and uptime widgets; status widget entries
link to their own detail_url. All guarded on session.user_id so the public
share view degrades to plain text (the bare href carries no share token).
- Fix truncation: title= tooltips on all names that ellipsis, plus a hover
underline affordance on the now-clickable ping-name links.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Host agent panel (reporting state) gains a "Re-provision" collapsible (admin +
linked target + managed key): bootstrap user/password → provision.yml, which
reinstalls the steward account + managed key + agent. This is the missing path
after regenerating the managed key — Update alone can't fix a broken key.
- Remove the Ansible sections (run-playbook + target link) from the host EDIT
page — they were the stale free-text version and the wrong place. They live on
the host detail hub (dropdown + discovered variables). Edit page now links to
the host page; edit_host route simplified (no ansible fetch).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Run UX: when start_run throws before/around launch (e.g. ENOSPC creating the
temp dir — the box is out of disk), the run was marked "failed" with empty
output. Now the exception is broadcast + written to the run output/results so
the run view shows e.g. "[run error] OSError: [Errno 28] No space left on
device" instead of a blank failure.
Widget audit follow-ups (no broken links were found; these are consistency):
- host_resource_history widget now charts root (/) disk, consistent with the
host panel (was the opaque "disk worst").
- host_resources widget: tooltip on the health dot explaining it warns on the
worst mount while the number shows root.
- status_overview widget detail_url /status → /status/ (avoid redirect).
- Normalize ad-hoc widget empty-states to the shared .empty style (wording,
which distinguishes "configured" vs "data yet", preserved).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Bring the dashboard in line with the unified host IA + improved displays.
- New core "Hosts — Overview" widget (/hosts/overview/widget): one row per host
combining monitor status (ping dot + latency, uptime 24h) with the agent
glance (CPU / memory / disk root + stale flag), each row linking to the host
hub. Reads agent data from the generic PluginMetric table via a core-safe
_agent_overview_by_host helper (no host_agent import); freshness vs the
plugin's stale window. The granular Ping/DNS/Uptime/Agent widgets stay.
- Group the add-widget picker into Core monitors / Monitoring capabilities /
Integrations (a `group` field on every WIDGET_REGISTRY entry + section
headings in _edit_panels.html), matching the Settings → Plugins taxonomy.
- Fix the agent fleet widget rows to link to the host hub (/hosts/<id>) instead
of the old /plugins/host_agent/<id>/ page.
Scribe #903.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Each at-a-glance metric (CPU, Memory, Disk /, Load) now shows a 6h sparkline
(reused core.status.sparkline_svg + a recent-series query; disk uses the root
mount sub-resource) so trend/consistency is visible, not just the instant.
- Load is now normalized: "Load /core" = 1m load ÷ CPU cores as % (100% = run
queue matches capacity), comparable across different hardware. Cores derived
from the per-core CPU metrics already collected — no agent change. Raw load
in the tooltip.
- Added a "Pressure 10s" line: PSI cpu/mem/io (some, avg10), the
hardware-independent saturation signal already collected by the agent.
Scribe #898.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The "Disk (worst)" number was opaque at a glance. Show the root filesystem (/)
usage instead — what people actually care about — on the host panel and the
dashboard widget. "Worst" is kept only for the widget's health dot (so a full
/var or /data still warns) and on the full-metrics page (per-mount + worst
trend). Added hover tooltips defining CPU / Memory / Disk / Load.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The assertion failed ("Pass steward_url/token/pubkey") because those were
injected as inventory HOST vars, but the playbooks declared them in the play
`vars:` block — and play vars OUTRANK inventory host vars, so the empty
defaults won and the injected values never reached the play.
- Pass globals (steward_url, steward_pubkey, steward_user, agent_interval) as
extra-vars via the JSON -e @file (highest precedence, space-safe). Keep only
the per-host steward_token as an inventory host var.
- provision.yml / install.yml: drop steward_token from `vars:` so the host var
isn't shadowed; assertions use `| default('')` for the un-defaulted token.
This was the next layer under the inventory-format fix — first the inventory
wouldn't parse, now the injected vars actually apply.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A failed provision looked successful in two ways:
1. The host panel showed the agent as deployed (metrics + Update/Rotate/Remove)
because provision/deploy mint the registration row BEFORE the playbook runs
and the panel keyed "installed" on that row. Now gated on the agent actually
checking in (reg.last_seen_at). Three states: reporting (metrics + lifecycle),
pending (token minted but no check-in → "no metrics yet, deploy may be
running/failed" banner + retry + Clear pending registration), and none
(install path).
2. The run reported success though nothing ran — ansible-playbook exits 0 on
"no hosts matched"/empty inventory. The executor now treats an empty PLAY
RECAP (returncode 0 but no hosts executed) as failed, with a clear failure
note. Non-zero exits and recap failed/unreachable were already caught.
Scribe issue #887.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
generate_inventory() emits Ansible's dynamic --list shape (all.hosts is a
LIST, vars under _meta) — valid only as an executable inventory script's
stdout. We were writing it to a static file and passing -i, so Ansible's yaml
plugin rejected it ("Invalid 'hosts' entry for 'all' group, requires a
dictionary, found ...list...") and fell back to implicit localhost → "no hosts
matched". Affected every steward:* scope run; surfaced on the first real
provision.
- New inventory_to_yaml(inv): convert the --list dict → a valid static YAML
inventory (all.hosts dict keyed by host, groups under all.children, group
vars preserved, injected per-host vars like steward_token retained).
- Wire it into runner.trigger_run, host_agent deploy + provision.
- executor writes the file as inventory.yml so the yaml plugin's extension
check reliably claims it.
- generate_inventory unchanged (still the --list dict); conversion happens at
write time. Unit tests added.
Scribe issue #885.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
fetch_scope_targets() runs a SELECT that autobegins the session transaction
(SQLAlchemy 2.0), so the following `async with db.begin()` raised
"A transaction is already begun on this Session" → 500 on both
/plugins/host_agent/deploy and /provision.
Mint registrations directly into the autobegun transaction, build the
inventory while the ORM objects are still live, then await db.commit().
Escaped CI because unit tests use a mock db_sessionmaker; no integration
test exercises these routes (follow-up noted in Scribe issue #884).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Dev-only instance, no bookmarks — per family rule 22, fully remove old paths
instead of shimming them.
- Delete the /plugins/host_agent/ (index) and /plugins/host_agent/settings/
redirect routes; delete the now-dead host_list.html fleet template.
- Move the remaining management POST routes off /settings/ to /fleet/
(add-host, rotate-token, delete) — single canonical prefix.
- Repoint real callers to canonical URLs: dashboard widgets (host resources →
/hosts/, history → /plugins/host_agent/fleet/), the full-metrics page
breadcrumb + back link (→ the host hub), Settings→Ansible link, and the
agent panel's curl-install link.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Hosts list: new Agent column (latest CPU/mem read from the generic
PluginMetric table — no host_agent import) + an admin "Agent fleet" button.
- /plugins/host_agent/ (old fleet page) now redirects to /hosts/ (folded into
the hub; kept as a redirect so widgets/links don't 404).
- Agent management moved off the "settings" URL: the management page is now
/plugins/host_agent/fleet/ ("Agent fleet" — bulk provision/install/update +
registrations + curl install), reachable from the Hosts list. Old
/plugins/host_agent/settings/ redirects there. Per-host management lives on
the host detail page; this page is now explicitly the bulk/fleet view.
Milestone 70 phases 2-3. Phase 4 (plugins capability/integration split + nav
cleanup) next.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Make Hosts the front-and-center hub. A host now has a real detail page at
/hosts/<id> that pulls its facets into one view, instead of management being
scattered across a nav-less Host-Agents area and the edit form.
- hosts: new GET /hosts/<id> detail route + hosts/detail.html. Shows the
monitors summary (ping/DNS status + latency + uptime 24h/7d/30d), an Ansible
section (linked target, link/create, run-playbook), and an embedded Agent
panel. Hosts list name links here; ansible-link redirects here.
- host_agent: GET /plugins/host_agent/panel/<host_id> — a self-contained HTMX
fragment embedded into the core hub across the plugin boundary (core never
imports plugin models). Shows live agent metrics + Update/Rotate/Remove when
installed, or the provisioning path when not: inline "generate managed key"
warning, a prompt to link an Ansible target first, then Provision (bootstrap
password) / Install (managed key) tied to the host's target scope.
Part of milestone 70 (Hosts hub). Phase 2+ will enrich the list, redirect the
old fleet/settings pages, and re-taxonomize plugins into capabilities vs
integrations.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Provisioning review corrections + the matching frontend, plus breadcrumb
header integration.
- provision.yml: authorize the managed pubkey with a regexp match on the
' steward-managed' comment so rotating the key REPLACES the host's steward
key in place instead of stacking a second authorized entry. Hand-added keys
(other comments) are untouched.
- update.yml (new): refresh agent.py + restart only. Does NOT rotate the token
or rewrite /etc/steward-agent.conf — the host keeps its identity. Asserts the
agent is already installed and fails clearly otherwise.
- host_agent /update route: runs update.yml as the managed steward user (no
token minting). Token rotation stays a deliberate action.
- settings/ansible/generate-key honors a safe relative `next` redirect, so an
inline trigger elsewhere returns to its page.
- Host Agents settings: reworked into a clear lifecycle — an intro card that
explains it runs Ansible to deploy the agent (+ inline "generate managed key"
warning/trigger when none exists), then three labelled cards: 1 Provision,
2 Install/enroll, 3 Update. Each explains what it does.
- base.html: breadcrumb now renders as a kicker line directly above the page
title (moved below alerts, tightened margin) so nested and top-level views
share one consistent header.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Turn the agent-install playbook into a full provisioning + maintenance
path. Solves the bootstrap chicken-and-egg: first contact uses an
operator-supplied password (one run, never stored), which creates a
dedicated `steward` login account with NOPASSWD sudo + Steward's managed
public key. Every run thereafter connects as `steward` with the managed
key — fully unattended (scheduled prune, agent updates).
- core/crypto: generate_ssh_keypair() — ed25519, OpenSSH formats.
- settings: ansible.ssh_public_key (non-secret, displayed) + ansible.ssh_user
(default steward); to_ansible_cfg extended.
- settings UI + route: "Generate managed key" (private encrypted, public
shown to copy) + SSH-user field.
- executor: build_bootstrap() writes a 0600 vars file (-e @file) for the
per-run user/password — never argv, never DB, never logged; drops the
managed key when a bootstrap password is given; --user floor from the
global ssh_user when no override.
- runner.trigger_run: pass-through `connection` kwarg, deliberately NOT
persisted on AnsibleRun.params (password stays out of the DB).
- bundled/host_agent/provision.yml: create steward user + authorized_keys
+ /etc/sudoers.d/steward (visudo-validated) + agent install.
- host_agent: /provision route + "Provision a fresh host" card (bootstrap
user/password; injects pubkey + user + token as space-safe JSON hostvars).
- Dockerfile: add sshpass (Ansible shells out to it for password SSH).
- tests: keypair generation + build_bootstrap (secret stays off argv).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds a breadcrumb trail to nested views for orientation. Mechanism: a
{% block breadcrumb %} slot in base.html (+ styling) and a shared crumbs()
macro in templates/_macros.html; each nested page fills the block with its
trail (root→current, last item is the current page). Pages without the block
render no bar, so top-level nav roots stay clean.
Applied to: Ansible (browse, schedules, playbook editor, run detail) +
inventory (targets/groups + detail), host_agent (fleet, host detail,
settings), hosts form, settings tabs (ansible/auth/notifications/plugins/
reports + plugin detail), dashboard (list, edit), and alerts (rule form,
maintenance + new). Dynamic labels (host/target/run/dashboard names) come
from the page context — no route changes. All 60 templates Jinja-compile.
Task #873.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Implements #253's framework: a small core capability registry
(steward/core/capabilities.py) where a module/plugin publishes a named,
role-gated action and a consumer discovers it via has_capability() and runs it
via invoke_capability() — no hard import, graceful degradation, permission
propagation (actor role checked against the capability's required_role).
Core publishes "ansible.run_playbook" (operator) wrapping ansible.runner.
trigger_run (extended to accept a caller-built inventory). First consumer: the
host_agent plugin gains "Deploy via Ansible" on its settings page — pick an
inventory target/group and it installs/updates the agent via the bundled
host_agent/install.yml, minting a fresh token per host and injecting it as an
inventory hostvar (turning per-host curl|sh into one run). Exposed role
ordering as middleware.role_meets. Unit tests for the registry + role checks.
Task #253 (milestone #37).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Surfaces the metrics the agent now collects (task #868). Adds a viewer-facing
fleet page (/plugins/host_agent/) with per-host cards (CPU/mem/disk/load/temp,
stale flag) and a per-host detail page (/plugins/host_agent/<id>/) — the link
the fleet widget already pointed at but had no route for.
Detail page shows current gauges (CPU, memory incl. swap/cache, load, network
rx/tx, disk I/O, temp max, memory/cpu/io PSI), per-core CPU bars, per-mount
filesystem bars, and per-interface/disk/sensor breakdowns, plus history charts
(utilization %, throughput B/s, load & pressure) over a selectable range.
Charts use a linear epoch-ms x-axis so no Chart.js date adapter is needed.
Stale state uses the plugin's stale_after_seconds threshold. Repoints the
host_resources widget detail link from admin settings to the new fleet page.
Completes milestone #68 (task #867).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Extends the push agent (v1.2.0) with the Netdata-style signals the operator
wants: per-core CPU, per-interface network throughput and per-disk I/O
(rates derived in-agent from monotonic /proc counter deltas, with
counter-reset clamping), hardware temperatures (/sys/class/hwmon),
memory-pressure PSI (/proc/pressure), and cached/buffers memory breakdown.
All stdlib-only.
Server side needs no migration — these land as additional rows in the shared
PluginMetric table via _expand_sample_to_metrics. Per-resource series use a
':' in resource_name (host:net:eth0, host:core0, host:temp:Package) so the
existing fleet widget's ':' filter ignores them; host-level totals/max are
emitted at the bare host resource. New sample keys are optional, so older
agents keep ingesting unchanged.
Unit tests for the new parsers, rate/reset logic, and the expander contract.
Data foundation for the Netdata-style host view (task #867).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds a Kuma-style "is everything up?" surface that aggregates heterogeneous
monitor types via a status-source registry (steward/core/status.py): each
type registers an async source(db) -> [StatusEntry]. Core registers ping/DNS;
the http plugin registers its own from setup() so core never imports plugin
tables. Per entry: current up/down, last-30 heartbeat bar, uptime %
(24h/7d/30d), latest latency + response sparkline, and TLS expiry countdown
(HTTP). New /status page (live htmx refresh) + a status_overview dashboard
widget + nav link. Pure-function unit tests for registry + sparkline.
First deliverable of milestone #68 (task #866).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The agent now detects its own primary non-loopback IP (UDP-socket trick, no
packets sent) and sends it in the metadata bag; bumps AGENT_VERSION 1.0.0->1.1.0
and re-detects on SIGHUP. The server persists it on HostAgentRegistration.host_ip
and mirrors it into Host.address ONLY when that field is blank — an admin-typed
address is never overwritten. The reported IP shows in the settings table so
admins can see drift regardless.
- agent.py: detect_primary_ip(); host_ip in collect_metadata(); refresh on reload
- routes.py: pure pick_host_address() helper; ingest persists host_ip + mirrors
- models.py + migration host_agent_002_host_ip: new String(45) column
- settings_list.html: Reported IP column
- plugin.yaml: 1.0.0 -> 1.1.0
- tests: pick_host_address (fill-when-blank / never-clobber / no-op),
detect_primary_ip (never raises, non-loopback), metadata carries host_ip
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The fabledscryer->steward rename had only ever reached host_agent. The other
five bundled plugins (http, snmp, traefik, unifi, docker) still imported
`from fabledscryer.*` (package no longer exists) and read FABLEDSCRYER_* env
vars — so every one of them was broken at import since the original rebrand.
CI stayed green only because none are enabled by default and migrations don't
import plugin modules. Now that they version in-tree, complete the rename:
- fabledscryer.* -> steward.* imports across all five plugins
- FABLEDSCRYER_* -> STEWARD_* in plugin migration env.py files
- author/repository/homepage + user-facing 'Fabled Scryer' strings -> Steward
- snmp/scheduler.py: also drop dead `now`/datetime; record_metric from steward
Adds tests/test_no_legacy_names.py — fails if 'scryer'/'roundtable' ever
reappear in shipped code (the drift bit twice; this stops a third time).
Also clears pre-existing ruff lint debt (unused imports, semicolon statements,
mid-file import) surfaced by the new lint lane.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
bvandeusen