8af297670eb611fc245ecf733ea9ae2e41979101
226 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 bvandeusen
|
8af297670e |
feat(metrics): roll plugin_metrics up to hourly to bound storage
plugin_metrics grows by (sources × resources × ~30s cadence); keeping 90d of raw
is a large table. Add a raw→hourly rollup (mirroring the Docker plugin) so only a
short raw window is kept at full resolution, with hourly averages archived longer.
- PluginMetricHourly model + core migration 0024 (plugin_metrics_hourly: avg/max/
count per source/resource/metric/hour, unique bucket constraint + lookup index).
- steward/core/metrics_retention.rollup_plugin_metrics: date_trunc('hour') agg of
raw older than the hour-aligned raw window, idempotent pg upsert into hourly,
delete the rolled raw, prune hourly beyond the rollup window.
- cleanup.py: plugin_metrics is no longer blanket-deleted at data.retention_days;
_run_metrics_retention drives the rollup with windows read live from settings.
- Settings: metrics.retention.raw_days (7) + rollup_days (90), tunable on the
Thresholds & Retention page (new "Host metrics retention" card).
- Chart read: _history_for_host merges the hourly rollup (older part of the range)
with raw date_bin (recent part, capped ≤1h), so 30d charts keep working —
recent at full resolution, older at hourly. Route passes raw_days from settings.
- Tests: unit (cutoff helpers) + integration (rollup aggregates/prunes; history
merges hourly + raw) against Postgres.
Speed was already handled by the indexes + SQL aggregation; this is the storage
lever (raw window ~10x smaller).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC
|
||
|
bvandeusen
|
b0d3e83bdd |
feat(hosts): true live vitals strip; agent panel becomes management-only
Realizes the chosen host-summary layout: a thin live vitals bar at the very top, separate from the agent management panel (no more duplicated CPU/MEM/DISK/LOAD). - New fragment _host_vitals.html + route /plugins/host_agent/vitals/<id>: compact CPU / Memory / Disk(/) / Load-per-core (threshold-coloured + sparkline) + Pressure + live/stale·version·last-seen. Polled every 15s; renders nothing until the agent reports. - The metric computation (latest snapshot + 6h sparkline query + load/core + PSI) moves from host_panel into host_vitals. host_panel slims to management only (reg/target/reporting/stale/ansible) and no longer queries metrics; panel.html drops the gauge row + pressure block, keeping status + lifecycle actions. - hosts/detail.html: vitals strip on top (full width, live), then a 2-col [Monitors | Agent] grid, Ansible full width below, docker fragment last. UI only. Templates parse; plugin-template parse test covers the new fragment. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC |
||
|
bvandeusen
|
a35c369dd4 |
polish(ui): sidebar nav a11y + scroll/focus refinements (nav slice 3)
- Only the nav list scrolls when long (min-height:0 + overflow on .side-nav); the brand and user/logout stay pinned top/bottom. - Keyboard focus rings (:focus-visible) on every interactive sidebar element (brand, nav links, user/logout) and the mobile ☰ toggle. - Mobile toggle is now a real toggle: aria-controls/aria-expanded kept in sync, Escape closes the off-canvas sidebar, scrim close stays in sync too. - prefers-reduced-motion disables the sidebar slide + link transitions. - Confirmed no dead top-nav CSS/markup remnants from the old top bar. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC |
||
|
bvandeusen
|
28c9a4dd2f |
fix(docker): collapse Swarm view per-cluster so multi-manager doesn't duplicate
Swarm services/nodes are cluster-global (every manager's API returns the same
list), but each manager reports them independently (rows keyed by host_id) and
the Swarm page grouped by reporting manager — so two managers in one cluster
listed every service and node twice.
Group the reporting managers into swarms in the swarm() view (managers sharing
any node_id are the same cluster, via union-find over node-set intersection),
then dedup within each: one service per name and one node per node_id, keeping
the freshest. Render one section per swarm ("reported by N managers · names").
node_id is globally unique so node dedup is always safe; grouping by node
overlap also keeps it correct if two separate swarms are ever monitored.
View-layer only — no schema/agent change. The main per-host container page is
unaffected (those are genuinely distinct per-node tasks).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC
|
||
|
bvandeusen
|
10dfd8ffd2 |
fix(host_agent): update full-metrics charts in place to kill refresh flicker
The live refresh re-created the Chart.js charts each poll, which blanked the
canvases and re-ran the grow-in animation — a jarring flash/jump every cycle.
Make the canvases persistent in the page shell and poll only the data:
- The 3 chart canvases + their Chart instances are created once in the shell,
with animation disabled.
- /charts is now a data-only fragment swapped into a hidden div; it calls
window.applyHostSeries(series, range), which sets each dataset's data and
calls chart.update("none") — in-place, no re-create, no animation, fixed
height. Range labels update via .hm-chart-range spans.
- Current-state fragment keeps its atomic innerHTML poll into fixed-height
cards, so numbers update without a layout jump.
Net: live updates morph smoothly with no flicker or layout shift.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC
|
||
|
bvandeusen
|
9e4f1983f8 |
feat(host_agent): lazy-load full-metrics charts + live-poll current state
The full-metrics page rendered once server-side with the history query inline, so the charts blocked first paint and nothing refreshed without a reload (it reads the latest snapshot the server holds — the agent pushes ~every 30s). Split it into a shell + two HTMX fragments: - Shell (host_detail.html): header + shared time-range toggle + two containers; paints instantly. - Current state (/<id>/metrics → _host_metrics.html): identity + gauges + per-core + filesystems + interfaces/disks + temps, from the DISTINCT ON latest query. hx-trigger "load, every 15s" → live numbers at ~the agent cadence. - History charts (/<id>/charts → _host_charts.html): the 3 charts + Chart.js, from the date_bin history query. hx-trigger "load, every 60s, rangeChange" so they lazy-load (never block paint), refresh slowly, and follow the range selector. Leak-safe: previous Chart instances are destroyed before re-render. - Range toggle switched from full-reload links to the shared _time_range.html (setTimeRange + rangeChange), so only the fragments refetch. - routes: host_detail (shell) + host_detail_metrics + host_detail_charts, with a _split_host_metrics helper. - tests/test_templates_parse.py now also parses plugin templates (these fragments aren't rendered in the unit lane). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC |
||
|
bvandeusen
|
a78af23793 |
refactor(host_agent): extract metric-query helpers to a model-free module
The previous commit's integration test failed at collection-time import:
importing plugins.host_agent.routes (which imports the host_agent ORM models at
top level) double-registers host_agent_registrations against the app-loaded
plugin's metadata ("Table already defined").
Move the two pure read helpers (_latest_metrics_for_host, _history_for_host)
plus SOURCE_MODULE / HISTORY_METRICS into plugins/host_agent/metrics_query.py,
which imports only the core PluginMetric model — no plugin models. routes.py
imports them back. The integration test now imports from metrics_query and no
longer trips the loader's registration guard. No behavior change.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC
|
||
|
bvandeusen
|
aff0c36d37 |
perf(host_agent): aggregate metric history in SQL; DISTINCT ON for latest
Follow-on to the plugin_metrics indexes. plugin_metrics is already retention- bounded (core cleanup prunes > data.retention_days, default 90d) and charts top out at 30d, so the cost wasn't growth — it was the read path shipping raw rows to Python. - _history_for_host: bucket + average in SQL via date_bin (epoch-aligned, ~120 buckets) instead of fetching every raw sample (a 30d range was hundreds of thousands of rows) and downsampling in Python. Uses the new (source_module, resource_name, recorded_at) index. - _latest_metrics_for_host: DISTINCT ON (resource_name, metric_name) ORDER BY recorded_at DESC — newest row per group in one index-ordered pass, replacing the GROUP-BY-max subquery self-joined back to the whole history. - Integration test validates both against Postgres. Deliberately not a materialized raw→hourly rollup: these query-side changes deliver the speed; a rollup would additionally cut storage and remains a future option if scale demands it. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC |
||
|
bvandeusen
|
6d08db0d89 |
feat(hosts): rework host summary layout to use horizontal space
The host detail page stacked three full-width cards (Monitors, Agent, Ansible) left-aligned, using only ~40% of the width with lots of vertical whitespace. Rework toward the operator-chosen "vitals on top + 2-column" layout: - The agent panel (CPU/MEM/DISK/LOAD vitals + sparklines + lifecycle) moves to the top, full width, so a host's vitals lead the page. - Monitors and Ansible sit side by side in a responsive 2-column grid (auto-fit, stacks under ~420px), filling the width. - The Docker per-host fragment moves below, full width. Note: the vitals live in the host_agent fragment (loaded across the plugin boundary) and uptime in the host-detail context, so a single literal top "strip" mixing both isn't clean — the agent card on top is the faithful, contained realization. UI-only. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC |
||
|
bvandeusen
|
3a54d6d71d |
perf(metrics): index plugin_metrics for host/dashboard reads
plugin_metrics had only a PK on id, so every host-detail, full-metrics, and dashboard-widget load sequentially scanned the entire time-series table — which grows by (sources × resources × sample cadence), so the host views got slower over time (operator-reported "blocked/slow" loads). monitor_results was already indexed, so the uptime aggregation wasn't the bottleneck. Add two composite indexes matching the hot query shapes: - (source_module, resource_name, recorded_at) — history range scans, fleet/ widget queries, and the 'host:%' sub-resource prefix. - (source_module, resource_name, metric_name, recorded_at) — the latest-value- per-metric group-by/self-join. Model __table_args__ + core migration 0023. Integration lane validates the migration via `alembic upgrade head`. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC |
||
|
bvandeusen
|
7d144780df |
fix(host_agent): TB/PB byte scaling + rebalance the full-metrics layout
Two issues on the host full-metrics view: - fmt_bytes capped at GB, so large filesystems read "25369.0 GB" instead of "24.8 TB". Add TB and PB tiers. - Interfaces/Disks/Temperatures shared one 3-column grid, so all three cards stretched to the Temperatures height — a 40-core CPU made one very tall column and left Interfaces/Disks with large empty space. Split them: Interfaces + Disks sit side-by-side at natural height (align-items:start); Temperatures becomes a full-width card whose readings flow into a compact multi-column grid (wide-and-short instead of one tall column). UI-only. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC |
||
|
bvandeusen
|
2545e8c6ce |
fix(docker): widen memory byte columns to BIGINT (int32 overflow on ingest)
docker_metrics.mem_usage_bytes and docker_containers.mem_usage_bytes / mem_limit_bytes were int4 (max 2,147,483,647). A container using >2.1 GB of RAM (e.g. 8.18 GB) overflowed the column, so asyncpg raised "value out of int32 range" and the entire docker ingest batch failed — no metrics stored for any host with a large container. The I/O counters and the milestone-77 rollup/disk tables already used BigInteger; this trio (docker_001-era) was missed. - models.py: DockerMetric.mem_usage_bytes, DockerContainer.mem_usage_bytes, DockerContainer.mem_limit_bytes → BigInteger. - migration docker_008_bigint_mem: ALTER COLUMN ... TYPE BIGINT (safe in-place int4→int8 promotion). - integration regression test: persist an ~8 GB container, assert the current-state and time-series rows round-trip. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC |
||
|
bvandeusen
|
c8b6719b37 |
feat(ui): plugin get_nav() hook → sidebar Infrastructure links (slice 2)
Plugin UIs had no nav home (only dashboard widgets / typed URLs). Add an optional get_nav() plugin export and surface it in the sidebar. - plugin_manager: _PLUGIN_NAV registry + get_plugin_nav() getter + a tolerant _collect_plugin_nav() (missing hook = fine; raising/malformed = logged & skipped; idempotent per plugin so hot-reload re-runs cleanly). Collected in both the startup load path and the hot-reload path. - app.py: inject plugin_nav into the template context, filtered to enabled plugins so a hot-disabled plugin's link can't linger before restart. - base.html: render plugin links under the Infrastructure group, with the same request.path active-state treatment as core links. - docker/snmp/unifi/traefik: each exports get_nav() → its /plugins/<name>/ view. - Tests: collector behavior (collect, missing hook, reload-replace, malformed item, raising hook, sorted output). With docker enabled, "Docker" now appears under Infrastructure → its fleet view. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC |
||
|
bvandeusen
|
95ebdf7045 |
feat(ui): replace flat top nav with a grouped left sidebar (slice 1)
The flat top bar was a set of ungrouped peers and gave plugin data no home. Move to a persistent left sidebar with grouped sections, per the navigation redesign (operator-chosen shell). - base.html: top <nav> → left <aside class="sidebar"> + content column. Groups: Overview (Dashboard, Status), Infrastructure (Hosts; plugin links arrive in slice 2), Monitoring (Monitors, Alerts), Automation (Ansible), Admin (Settings, Audit; admin-only). Brand on top, user/logout at the bottom. - Active link highlighted by request.path prefix (aria-current). - Responsive: sidebar slides off-canvas under 900px via a ☰ toggle + scrim (inline class toggle, no new JS deps). Candle-glow preserved. - Logged-out pages (login/setup) render without the sidebar (gated on session.user_id), content area full-width and centered as before. - Add tests/test_templates_parse.py: syntax-parses every steward template so a broken tag fails the unit lane (only the login page renders there today). Plugin nav links (Docker/SNMP/UniFi/Traefik) come next in slice 2 via a get_nav hook. UI-only; no behavior/route changes. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC |
||
|
bvandeusen
|
0940dc6972 |
feat(crypto): fail loudly on unpersistable secret key; flag undecryptable secrets
Follow-up to the incident where an unwritable /data made Steward silently mint a fresh ephemeral secret key on every boot, orphaning all encrypted secrets — only discovered when an Ansible run failed. Make both failure modes loud and visible. - config._resolve_secret_key: if a new key must be generated but can't be persisted (no STEWARD_SECRET_KEY, unwritable /data), raise RuntimeError and refuse to start, with an actionable fix (set STEWARD_SECRET_KEY, or make /data writable by uid 1000). Also raises a clear error if an existing key file can't be read. First-run on a writable volume still generates + persists normally. - core.settings: detect secrets stored as ciphertext that won't decrypt with the current key (scan_undecryptable_secrets at startup; _is_undecryptable helper). Cached in memory; set_setting discards a key on a fresh write so the banner clears without a restart. - app.py: run the scan after migrate_plaintext_secrets, log a warning, and inject undecryptable_secrets into the template context. - base.html: admin banner naming which secrets to re-enter, each linked to its settings tab. - compose.deploy.yml: document the uid-1000 /data write requirement and the STEWARD_SECRET_KEY option for multi-node Swarm. - Tests: secret-key resolver (reuse existing file, generate when writable, raise when unpersistable) and the undecryptable-secret detection helper. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC |
||
|
bvandeusen
|
4fc8c96c41 |
fix(snmp): bundle pysnmp in image and port poller to the asyncio HLAPI
The SNMP plugin ships in the image but logged "pysnmp not installed — SNMP polling disabled" on every poll, so polling never worked. Two coupled defects: 1. The Dockerfile installed only `.[ansible]`, so the `snmp` extra (pysnmp) was never bundled even though the plugin is first-party and shipped. 2. poller.py used the synchronous pysnmp HLAPI (`next(getCmd(...))`), which pysnmp-lextudio 6.x removed — it's asyncio-only now — so even with the dep present, polling would have thrown and silently returned nothing. The 5.x line that still has the sync API isn't safe on the image's Python 3.13. Fix: - Dockerfile: install `.[ansible,snmp]`. - poller.py: `poll_device_sync` → `async def poll_device` on the asyncio HLAPI, with a dual-version import (pysnmp 7.x `pysnmp.hlapi.v3arch.asyncio`/`get_cmd` + async `UdpTransportTarget.create`; pysnmp-lextudio 6.2.x `pysnmp.hlapi.asyncio`/`getCmd` + direct `UdpTransportTarget`) so a dependency bump can't silently re-break it. - scheduler.py: await poll_device directly; drop the run_in_executor wrapper and the now-unused asyncio import. - Add tests/plugins/snmp/test_poller.py covering the version→mpModel mapping, that the poller is a coroutine, and the graceful no-pysnmp path. Note: CI confirms import/load and the no-pysnmp path, but has no SNMP target — live polling against real devices is verified after deploy. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC |
||
|
bvandeusen
|
88091936c5 |
fix(ui): unify header/breadcrumb treatment, drop redundant back buttons
The page-top chrome was inconsistent: most nested views used the breadcrumb
kicker + page-title pattern, but plugin sub-pages used ad-hoc "← back" links,
~11 pages stacked a breadcrumb AND a redundant ancestor back button, and two
(monitors/edit, hosts/uptime) had a back button but no breadcrumb. The
settings/plugin_detail page stacked all of it at once.
Unify on the breadcrumb-led model:
- settings/_tabs.html: drop the hardcoded "Settings" h1; the breadcrumb
("Settings › …") plus the tab strip is the header.
- settings/plugin_detail: drop the "← Plugins" back button.
- docker container_detail/swarm/disk + snmp/device: replace ad-hoc back links
with the standard crumbs() breadcrumb.
- host_agent, ansible/*, alerts/maintenance: remove redundant ancestor back
buttons (the breadcrumb's parent crumbs already link there); keep lateral
shortcuts (Inventory/Schedules/Browse/Targets/Groups/New).
- monitors/edit, hosts/uptime: add the missing breadcrumb, drop the back link.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC
|
||
|
bvandeusen
|
626ba69934 |
test(docker): parse-check templates + smoke routes/_human_bytes (milestone 77 #943)
CI never renders docker templates through the app (the unit-lane app uses testing=True, which skips plugin loading), so a Jinja syntax error could ship green. Add unit tests that parse every docker template, smoke-import the routes module + confirm the #942 view functions exist, and cover the _human_bytes / _human_uptime presentation helpers. Closes the render-regression gap the new UI pages introduced. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC |
||
|
bvandeusen
|
9615f9abcd |
feat(docker): group containers by compose/swarm + enrich widget (milestone 77 #942)
Container list now sub-groups each host's containers by compose project (or swarm service) with a small subheading, preserving the running-first order; hosts with no such labels render flat as before. The dashboard status widget links each container to its detail page and surfaces enriched state inline — health dot (healthy/unhealthy), restart count, and last non-zero exit code for stopped containers. Completes the #942 UI surface. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC |
||
|
bvandeusen
|
114262dbf9 |
feat(docker): swarm topology view + image/disk usage page (milestone 77 #942)
Two new read-only sub-pages, linked from the Docker index header only when the data exists (non-swarm / Docker-less installs aren't offered empty pages): - /plugins/docker/swarm — services with replica health (running/desired, colour-coded green/amber/red), Swarm nodes (role/availability/status, leader badge), and task→node placement with node ids resolved to hostnames. Grouped by reporting manager host. Empty state explains manager-only collection. - /plugins/docker/disk — per-host reclaimable space, image/layer/container/ volume/build-cache sizes, stopped-container count, and a per-image table (size, shared, ref count, reclaimable badge). Notes prune actions are deferred. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC |
||
|
bvandeusen
|
3e4e35de96 |
feat(docker): container detail page + lifecycle timeline (milestone 77 #942)
New /plugins/docker/container/<host_id>/<name> detail page (v1 quality): status/health badge, uptime, CPU/mem, restart count, last exit code (+OOM), net + block I/O (humanised), image/compose/swarm-service/node/ports, a range-toggled CPU/mem history graph (HTMX fragment reusing the time-range selector), and a lifecycle timeline rendered from docker_events (glyph+colour per event kind). Not-found and empty-history/empty-timeline states included. Container names in the main list and the per-host hub panel now link to it. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC |
||
|
bvandeusen
|
277eb40165 |
fix(docker): close read txn before second begin in disk-usage test
test_disk_usage_persisted opened a second session.begin() after interleaved
SELECTs, which autobegin a transaction → "A transaction is already begun".
Roll back the read transaction before the re-report write. Restores the
integration lane green for the /system/df slice (
|
||
|
bvandeusen
|
a840d6f823 |
feat(docker): collect + persist /system/df image/disk usage (agent 1.6.0)
Backend for the image/disk panel (milestone 77 #942). Agent gains collect_disk_usage() — one /system/df call (gated on containers existing, so Docker-less hosts pay nothing), surfacing reclaimable bytes (image size held by unreferenced images), layers/containers/volumes/build-cache sizes, and the top 50 images by size. Emitted as sample["docker_disk"]; host_agent ingest tracks the newest sample's copy and hands it to the docker capability as a 5th arg. New current-state tables docker_disk_usage (one row/host) + docker_images (per-host image rows), docker_007 migration; ingest upserts the summary and replaces the image set per host (stale images pruned). Unit tests for the df parsing/reclaimable math + build_sample gating; integration test for persistence + image-set replacement. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC |
||
|
bvandeusen
|
faecac3ec6 |
feat(docker): retention + hourly rollup for metrics/events with Settings windows
Bounds Docker time-series growth (the main scaling concern). New docker_metrics_hourly table + docker_006 migration; a plugin retention module (docker.run_retention capability) rolls raw docker_metrics older than the raw window into hourly averages (idempotent upsert), deletes the rolled raw rows, then prunes stale rollups + lifecycle events. Core cleanup.py drives it each hourly run via the capability (no plugin-model import), reading the three retention windows fresh from settings so changes apply without restart (rule 25). Settings → "Thresholds & Retention" gains a Docker retention card (raw / rolled-up / events windows, working defaults 7/90/30 days). Unit tests cover the hour-aligned cutoff/bucketing helpers; integration test exercises the real rollup-average + prune across both windows. Milestone 77 task #941. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC |
||
|
bvandeusen
|
578cc33cc0 |
feat(docker): ingest swarm topology + lifecycle events + health/restart alerts
Wires the agent's enriched + swarm payloads through the docker.persist_host_
samples capability:
* Swarm topology — persist sample["swarm"] into docker_swarm_services /
docker_swarm_nodes (upsert + prune stale, host-scoped so two managers don't
clobber). Migration docker_005 adds services.placement_json for the
task→node placement the agent now reports.
* Lifecycle events — _derive_events (pure, unit-tested) diffs the newest
snapshot against stored per-container state: start / stop / die (non-zero
exit) / oom / health_change → docker_events rows. Skipped on a host's first
snapshot so the baseline doesn't emit a start per existing container.
* Alerts — record restart_count (always) and is_healthy (1.0/0.0, only when a
HEALTHCHECK exists) alongside cpu/mem, under host-scoped resource names;
METRIC_CATALOG[docker] gains restart_count + is_healthy so they're alertable.
host_agent ingest captures the newest sample's swarm object and threads it to
the capability (now persist_host_docker(session, host, snapshots, swarm=None));
invoked when containers OR swarm are present, under the same SAVEPOINT. Unit
tests cover the event-diff matrix; integration tests cover event derivation
across two snapshots and swarm topology round-trip (incl. placement).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC
|
||
|
bvandeusen
|
448258c5b4 |
feat(docker): agent manager-only swarm collector (AGENT_VERSION 1.5.0)
Adds collect_swarm(socket_path) to the host agent. Self-detects a Swarm
manager via /info (Swarm.ControlAvailable) — workers and non-swarm daemons
return None and never touch the manager-only endpoints (one cheap /info call,
no 503s). On a manager it queries /services, /tasks, /nodes and emits
sample["swarm"] = {services, nodes}:
* services roll desired-vs-running replicas up from the task list (replica
health isn't on the service object), handle replicated + global mode, strip
the @sha256 image digest, and carry cross-node task→node placement.
* nodes normalise role / availability / status + the manager leader flag.
build_sample omits the swarm key entirely off managers, same silent contract
as collect_docker. Unit tests cover manager detection, replica roll-up +
placement (replicated & global), node normalisation, worker silent-skip, and
build_sample wiring.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC
|
||
|
bvandeusen
|
fee654b53a |
feat(docker): schema for lifecycle events + swarm topology
Adds the milestone-77 storage that doesn't fit on the per-container row:
* docker_events — lifecycle (start/stop/die/oom/health_change), to be
derived by diffing consecutive host snapshots; host-scoped, indexed for
timeline lookups (host_id, container_name, at) and retention pruning (at).
* docker_swarm_services / docker_swarm_nodes — manager-reported Swarm
topology (desired-vs-running replicas, node role/availability/status).
Migration docker_004 extends the docker branch (down_revision docker_003);
purely additive, no DROP+recreate. event/mode/role are plain strings (no
CHECK whitelist), matching how docker_containers models status. Integration
guard asserts the three new host-scoped tables exist.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC
|
||
|
bvandeusen
|
82c3d2cf36 |
feat(docker): per-container enrichment — health, restarts, exit code, I/O, grouping
First slice of milestone 77 (Docker monitoring depth). Surfaces real per-container
stats beyond basic state, all read-only on the existing push model.
- agent (→1.4.0): collect_docker now inspects each container (health, restart
count, exit code, OOM) and reads net + block I/O from the stats payload; pulls
compose project + swarm service/task/node from container labels. Per-container
inspect+stats calls run over a small bounded ThreadPool so the ~1s-per-stats
blocking doesn't stretch the sample on a busy host.
- schema (docker_003): additive columns on docker_containers — health, exit_code,
oom_killed, compose_project, service_name, task_id, node_id, and BigInteger
net/blk byte counters.
- ingest: persists the enrichment + restart_count (.get keeps older agents working).
- ui: Docker page rows now show health badge, uptime ("up 3d 4h"), restart count,
exit code (+OOM) for stopped containers, and compose/service grouping label.
- tests: agent helpers (grouping, inspect fields, net/IO sum) + collect_docker
assembly incl. inspect; integration asserts enrichment round-trips.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC
|
||
|
bvandeusen
|
7b80552a7d |
feat(docker): per-host collection via the host agent; drop central scrape
Docker collection moves off the central single-socket scrape onto the host
agent, giving Docker a real per-host dimension. The Steward host now reports
its own containers like any other host, and same-named containers on different
hosts no longer collide.
- agent: stdlib UDS Docker client (AF_UNIX HTTP/1.1, Connection: close,
chunked-aware), collect_docker() ports the cpu%/mem math; sample["docker"]
added best-effort (silent-skip on absent/unreadable socket). AGENT_VERSION
1.2.0 → 1.3.0; optional docker_socket config key.
- ingest: host_agent ingest hands per-host container snapshots to the docker
plugin via a new "docker.persist_host_samples" capability (no hard import,
no-op when docker disabled), inside a SAVEPOINT so a docker failure never
sinks the host metrics. Resource names are host-scoped ("<host>/<name>").
- schema: docker_containers re-keyed (host_id, name); docker_metrics gains
host_id; docker_002 migration DROP+recreates (dev-only, rule 122).
- ui: Docker page + widgets grouped by host with host links; new per-host
Docker panel embedded on the Hosts hub (gated on docker enabled via a new
enabled_plugins template context). Replaces the SQLite-only strftime
bucketing with DB-agnostic Python bucketing.
- provisioning: install/provision playbooks add steward-agent to the docker
group (best-effort) so the agent can read the socket.
- removed central scrape: docker scheduler.py + scraper.py deleted; plugin.yaml
socket_path/scrape_interval_seconds/include_stopped dropped (plugin 2.0.0).
- tests: agent docker collector units (math, chunked decode, silent-skip,
sample shape, config) + integration (host-scoped schema + persistence).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC
|
||
|
bvandeusen
|
35f658b573 |
feat(monitors): unify ping/dns/http into one Monitor entity + custom targets
Collapse the three former check types into a single core `Monitor` entity
with one management surface (/monitors), one result table (monitor_results),
and a single scheduled task. Every type can now watch a free-standing custom
destination (optional host_id) — not just a registered Host.
- models: Monitor + MonitorResult replace PingResult/DnsResult; Host loses its
ping/dns facet columns (now Monitor rows linked by host_id).
- checks: monitors/{ping,dns,http}.py pure probes + runner.run_monitor
dispatcher; one monitor_check scheduler with a per-monitor due-filter.
- status: single monitor_status_source replaces the three sources.
- UI: /monitors blueprint (type-aware add/edit/list/widget); host hub shows a
host's linked monitors + "add monitor for this host"; nav + widget registry
+ alert metric catalog rewired. http plugin folded into core and removed.
- migration 0022 merges the http branch, data-migrates host facets +
http_monitors + all three result histories, drops the old tables/columns.
Resolves the per-host ping/dns auto-attach issue (#275): monitors are now
explicit, never auto-added to every host.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC
|
||
|
bvandeusen
|
591706bd39 |
feat(settings): configurable monitoring thresholds
Move the hardcoded warn/crit cutoffs into Settings -> Thresholds (DB-backed, live, no restart). New thresholds.* keys + to_thresholds_cfg() + a threshold_style(value, kind) jinja global that reads them; latency reuses the existing ping good/warn keys, uptime is direction-aware (floors). Replace the _macros metric_style/uptime_style macros (now removed) with the global across Hosts-Overview, host_agent fleet + panel, Uptime/SLA widget, and the ping page uptime column — all now honor the configured cutoffs. Uptime keeps its green 'good' look when not degraded. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
eefa38cc75 |
feat(host_agent): in-widget 1h/6h/24h time-range toggle on history graph
Add a live range switch to the history widget that re-requests the fragment without entering edit mode. It rewrites the parent cell's hx-get so the choice survives polling (htmx re-reads the attribute each poll), then fetches at once. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
e7b96fbfa7 |
feat(host_agent): surface network, disk I/O, and temperature in fleet widget
The agent already collects + ingests net throughput, disk I/O, and temps; add them to the fleet-glance rows (fmt_bps + two-line io cells + temp with 70/85C threshold color), each shown only when the host reports it so VMs/ containers without sensors don't show blank cells. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
e446b7099e |
feat(dashboard): threshold colors on Hosts-Overview ping + 24h uptime
Add an inverted uptime_style macro (low-is-bad mirror of metric_style) and color the inline ping latency (warn 100ms / crit 250ms) and 24h uptime values in the Hosts-Overview widget, which were the remaining uncolored metrics. (Uptime/SLA + Ping widgets already colored degraded values.) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
ae03f09234 |
feat(dashboard): merge top summary strip + Status widget into one Overview
Operator chose a single canonical summary. Repurpose the status_overview widget into 'Overview' (host count + monitor up/down/pending + Alerts link; key kept so existing dashboards upgrade in place) and remove the redundant fixed top strip from the dashboard view. Drops _get_summary_stats and its now-unused PingResult/DnsResult imports (rule 22). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
10808c1c5d |
fix(dashboard): inline cpu sparkline, history graph host label + stable y-axis
- Hosts-Overview: move the CPU sparkline inline next to the cpu % value it represents (was floated far right on the name line, reading as unrelated). - Host Agent history widget: caption the chart with the host it represents (the panel title is generic) — links to the host hub. - History widget: snap the y-axis to a stable 0..next-10%-band ceiling instead of auto-scaling to the exact peak, so the 'zoom' no longer jumps each poll. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
ebc67723d2 |
feat(host_agent): horizontal zebra-striped fleet widget rows
Restore the host_agent fleet-glance widget to one horizontal row per host (name left, metric cells + sparklines right) instead of the multi-column block grid, and zebra-stripe odd rows so adjacent hosts read as distinct. Names still wrap rather than hard-truncate. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
fed9973899 |
fix(dashboard): downsample history graphs + readable tooltip time
The agent reports every few seconds, so multi-hour history series were hundreds– thousands of points — a dense, noisy line (esp. CPU). Bucket-average server-side to ~120 points (keeps the shape, drops the noise) for both the history-graph widget and the host-detail charts. Also fix the chart tooltip title showing the raw epoch-ms (e.g. 1,781,720,471,459) — format it as HH:MM. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
e58c86cf01 |
feat(dashboard): widget readability — name-per-line, fill graphs, container breakpoints
Address graphical issues raised from the dashboard screenshot: - No more truncated host names. Hosts-Overview and Host-Agent-Resources put the host name on its own line (full, wraps if needed) with its data grouped beneath it; ping/dns (.ping-name) and uptime widget names wrap instead of ellipsis. Prefer vertical overflow over cramming/truncating a row. - History graph fills the panel: drop the fixed 0–100 y-axis ceiling (beginAtZero + 8% grace) so the lines use the vertical space instead of hugging the bottom; axis labels still show real %. - Container-query breakpoints: the widget body is now a query container, so fragments restyle to their OWN panel width. Host-list widgets flow into 2 cols ≥520px and 3 cols ≥900px (.host-blocks) — use the width, remove vertical deadspace — and collapse to one column when narrow. Mirrored into the share view. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
cb47b5e977 |
feat(dashboard): per-metric sparklines in the Host Agent — Resources widget
Operator ask: show the graphs next to their fields in the fleet widget, like the host page's AGENT panel. Each row now renders cpu/mem/disk/load as a value with a trend sparkline beneath it (1h window), instead of bare numbers. - _fleet_rows fetches a per-host recent series (cpu/mem/load host-level, disk from the root mount) in one 1h query and attaches a sparkline per metric to each row. - widget_table.html lays out a metric cell (label + threshold-coloured value + sparkline) per field, mirroring panel.html. Threshold colour is computed in the loop and passed into the cell macro (keeps Jinja macro scope clean). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
88dca32d3c |
feat(dashboard): edit in place over the live dashboard with a bottom widget drawer
Milestone 72 phase D — the dashboard view IS the edit surface (operator ask): - /d/<id> renders the grid via Gridstack in STATIC mode — positioned exactly like before, live HTMX widget bodies keep polling. An "Edit" button flips the same grid interactive (grid.setStatic(false)) in place, so you drag/resize over real data. "Done" flips it back. Layout autosaves on change. - The widget picker is now a bottom drawer (position:fixed overlay) revealed by body.dash-editing — so the dashboard width is identical entering/leaving edit. - Add: POST returns a single grid item; JS inserts it + grid.makeWidget + htmx.process so it loads live data. Remove: POST 204 + grid.removeWidget. Per-panel drag handle + remove ✕ are in the DOM for editors, shown only while editing. - Gridstack loads for everyone (viewers get the static grid); edit wiring is gated on can_edit. Mobile collapses to one column. - Removed the separate /d/<id>/edit route + edit.html + _edit_panels.html (rule 22). Dashboard-list Edit and new-dashboard create now deep-link /d/<id>?edit=1 which opens edit mode on load. Browser-only behaviour — CI can't exercise it; needs an operator visual check. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
988c13d51f |
feat(dashboard): phase C richer panels — host time-series graph + cpu sparklines
Milestone 72 phase C — bring the host-view graphs onto the dashboard:
- host_resource_history widget reworked into a real host-view chart: epoch-ms
linear axis (no Chart.js date adapter), themed like the host-detail charts,
maintainAspectRatio:false so it fills the resized panel, unique canvas per
widget instance (wid), and empty states ("pick a host" / "no metrics yet").
Was previously unusable — it had a broken time axis and no way to choose a host.
- Add a "host" param type: the edit form renders a live dropdown of hosts
(dashboard routes now pass the host list to the editor); the chosen host_id is
stored in config and fed to the widget.
- Hosts-overview widget gains a per-row CPU sparkline (last hour) via the shared
sparkline_svg helper — the host-view at-a-glance trend, on the main widget.
Charts/sparklines render only in the browser, so CI can't exercise them — needs
an operator visual check.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
||
|
bvandeusen
|
525f6eedbd |
feat(dashboard): phase B drag-resize grid (Gridstack) replacing masonry
Milestone 72 phase B — Grafana-style drag-resize grid for dashboard widgets: - DashboardWidget: replace `position` with a 12-col grid placement (grid_x/grid_y/grid_w/grid_h). Migration 0021 backfills the old position order into a 3-up grid (4 cols x 4 cells each) and drops position. - View + share render a static CSS grid from x/y/w/h: fixed cell height (h * 70px) with the body scrolling, so the arranged layout is what's shown; collapses to a single column under 820px. - Edit view: Gridstack.js 12.6.0 (vanilla, CDN, pinned) — drag the title bar to move, drag a corner/edge to resize; every change autosaves to a new /d/<id>/edit/layout endpoint. Replaces the SortableJS position reorder. - add_widget appends at the bottom-left; remove no longer renumbers. _get_widgets now orders by grid position (drives DOM + mobile fallback order). Note: Gridstack drag-resize is browser-only, so CI can't exercise it — needs an operator visual check of the edit experience. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
5f92340c0c |
feat(dashboard): phase A widget clarity — threshold colours, host links, tooltips
Milestone 72 phase A (clarity wins, no schema change): - Add shared metric_style() macro in _macros.html: colours a numeric metric amber (>=warn) / red (>=crit) on the value ITSELF, not just the status dot. Defaults 80/90 for percentage gauges; load /core uses warn 80 / crit 100. Applied to CPU/mem/disk across host_agent panel + fleet widget + the unified hosts-overview widget. - Link every host reference to the host hub (/hosts/<id>) in ping, dns, hosts-overview, host_agent fleet, and uptime widgets; status widget entries link to their own detail_url. All guarded on session.user_id so the public share view degrades to plain text (the bare href carries no share token). - Fix truncation: title= tooltips on all names that ellipsis, plus a hover underline affordance on the now-clickable ping-name links. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
2ea8c2f9af |
feat(ansible): bundled system_update maintenance playbook
New maintenance/system_update.yml: cross-distro OS package upgrade (apt +
dnf/yum) with two run-form flags:
- restart_services: restart every service that needs it after the upgrade
(Debian via needrestart -r a, installed if missing; RHEL via
needs-restarting -s → systemctl try-restart).
- reboot_if_required: reboot the host only when a reboot is actually pending
(Debian /var/run/reboot-required; RHEL needs-restarting -r). Never reboots
otherwise.
Tagged steward:category=maintenance and steward:confirm=true (significant /
reboot-capable), so it shows in the run UI with the confirm gate and both flags
as fill-in fields. No code changes — auto-discovered from the builtin source.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
||
|
bvandeusen
|
88afad9de4 |
feat(hosts): re-provision action on reporting hosts; Ansible off the edit page
- Host agent panel (reporting state) gains a "Re-provision" collapsible (admin + linked target + managed key): bootstrap user/password → provision.yml, which reinstalls the steward account + managed key + agent. This is the missing path after regenerating the managed key — Update alone can't fix a broken key. - Remove the Ansible sections (run-playbook + target link) from the host EDIT page — they were the stale free-text version and the wrong place. They live on the host detail hub (dropdown + discovered variables). Edit page now links to the host page; edit_host route simplified (no ansible fetch). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
71715c38d8 |
feat(auth): return to the original view after an auth-expiry redirect
When a session has expired, require_role now bounces to /login?next=<path> and sends the user back there after re-login. - middleware: safe_next_url() (same-site relative path/query only; rejects off-site, protocol-relative, javascript:, and the auth pages — no open redirect). _login_redirect() builds the next param; for HTMX requests it uses HX-Current-URL (the page, not the fragment) and HX-Redirect so the whole browser navigates instead of swapping the login page into a fragment. - login GET/POST carry `next` (hidden field), validated, used on success for local + LDAP; OIDC stashes it in session across the IdP round-trip. - login.html: hidden next field + next on the SSO link. - tests for safe_next_url. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
2594ca517d |
fix(ansible): clear error when the managed SSH key can't be decrypted
When ansible.ssh_private_key can't be decrypted (app secret key changed), decrypt_secret returns the ciphertext unchanged; the executor was writing that enc:v1: blob as the SSH key file → cryptic "Load key ...: error in libcrypto" → Permission denied. Now: - build_credentials skips a still-encrypted key value (never writes ciphertext as a key file). - start_run broadcasts a plain-language run error: "The managed SSH key could not be decrypted (the app secret key changed). Regenerate it in Settings → Ansible and re-provision the host(s)." The run still fails (no usable key), but the reason is now obvious instead of a libcrypto error. Operator remedy: regenerate the managed key + re-provision. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
e0253fba48 |
fix(ansible): surface run failure reason; widget audit cleanup
Run UX: when start_run throws before/around launch (e.g. ENOSPC creating the temp dir — the box is out of disk), the run was marked "failed" with empty output. Now the exception is broadcast + written to the run output/results so the run view shows e.g. "[run error] OSError: [Errno 28] No space left on device" instead of a blank failure. Widget audit follow-ups (no broken links were found; these are consistency): - host_resource_history widget now charts root (/) disk, consistent with the host panel (was the opaque "disk worst"). - host_resources widget: tooltip on the health dot explaining it warns on the worst mount while the number shows root. - status_overview widget detail_url /status → /status/ (avoid redirect). - Normalize ad-hoc widget empty-states to the shared .empty style (wording, which distinguishes "configured" vs "data yet", preserved). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
bvandeusen
|
609bd78af2 |
feat(dashboard): unified Hosts widget + grouped widget picker
Bring the dashboard in line with the unified host IA + improved displays. - New core "Hosts — Overview" widget (/hosts/overview/widget): one row per host combining monitor status (ping dot + latency, uptime 24h) with the agent glance (CPU / memory / disk root + stale flag), each row linking to the host hub. Reads agent data from the generic PluginMetric table via a core-safe _agent_overview_by_host helper (no host_agent import); freshness vs the plugin's stale window. The granular Ping/DNS/Uptime/Agent widgets stay. - Group the add-widget picker into Core monitors / Monitoring capabilities / Integrations (a `group` field on every WIDGET_REGISTRY entry + section headings in _edit_panels.html), matching the Settings → Plugins taxonomy. - Fix the agent fleet widget rows to link to the host hub (/hosts/<id>) instead of the old /plugins/host_agent/<id>/ page. Scribe #903. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |