Ansible schedule form: structured playbook-variable fields + first-item dropdown defaults #2

Merged

bvandeusen merged 126 commits from dev into main 2026-06-30 23:44:04 -04:00

Conversation 0 Commits 126 Files Changed 290

+26118 -1482

bvandeusen commented 2026-06-30 23:43:52 -04:00

Owner

Copy Link

Copy Source

Two related Ansible run-form improvements from dev.

Structured playbook-variable fields on the schedule form (issue #1217)

The Schedules form only had a free-form extra-vars textarea, unlike the browse/host run forms which auto-populate a field per declared playbook variable (vars:/vars_prompt:) plus the playbook description.

• Playbook <select> loads /ansible/playbook-vars?schedule=1 into a #sch-playbook-vars container (mirrors the host detail form).
• Edit mode pre-renders the saved playbook's variables server-side with their stored values; the extra-vars textarea keeps only leftover (non-declared) vars.
• Shared _playbook_vars.html gains optional values (prefill) + schedule flags: in the schedule context secret vars render disabled ("secrets can't be scheduled" — they're dropped downstream) and the destructive-confirm gate is shown but not required.

Dropdowns default to the first item (task #1219)

Source/Playbook <select>s opened on a "— choose … —" placeholder, forcing an extra click.

• Removed the placeholders (schedule + host source/playbook, shared _playbook_options).
• schedules() defaults the source to the first (or edited) source and pre-renders that source's playbooks (create mode no longer shows the all-sources union).
• Cascade: source change repopulates the playbook list, then hx-on::after-settle re-triggers the playbook select to refresh its variable fields.

Backend already parsed var__* fields for schedules, so this is primarily a front-end/partial parity fix. CI green on both commits (runs 1805, 1807 — lint/unit/integration).

Note: the alert-rule form's opt-in Ansible-action source dropdown intentionally keeps its placeholder (different pattern — free-text playbook, checkbox-gated).

🤖 Generated with Claude Code

Two related Ansible run-form improvements from `dev`. ## Structured playbook-variable fields on the schedule form (issue #1217) The Schedules form only had a free-form extra-vars textarea, unlike the browse/host run forms which auto-populate a field per declared playbook variable (`vars:`/`vars_prompt:`) plus the playbook description. - Playbook `<select>` loads `/ansible/playbook-vars?schedule=1` into a `#sch-playbook-vars` container (mirrors the host detail form). - Edit mode pre-renders the saved playbook's variables server-side with their stored values; the extra-vars textarea keeps only leftover (non-declared) vars. - Shared `_playbook_vars.html` gains optional `values` (prefill) + `schedule` flags: in the schedule context secret vars render disabled ("secrets can't be scheduled" — they're dropped downstream) and the destructive-confirm gate is shown but not `required`. ## Dropdowns default to the first item (task #1219) Source/Playbook `<select>`s opened on a "— choose … —" placeholder, forcing an extra click. - Removed the placeholders (schedule + host source/playbook, shared `_playbook_options`). - `schedules()` defaults the source to the first (or edited) source and pre-renders that source's playbooks (create mode no longer shows the all-sources union). - Cascade: source change repopulates the playbook list, then `hx-on::after-settle` re-triggers the playbook select to refresh its variable fields. Backend already parsed `var__*` fields for schedules, so this is primarily a front-end/partial parity fix. CI green on both commits (runs 1805, 1807 — lint/unit/integration). Note: the alert-rule form's opt-in Ansible-action source dropdown intentionally keeps its placeholder (different pattern — free-text playbook, checkbox-gated). 🤖 Generated with [Claude Code](https://claude.com/claude-code)

bvandeusen added 126 commits 2026-06-30 23:43:52 -04:00

feat(plugins): fold first-party plugins in-tree; bundled + external roots ... a7a281cb11

First-party plugins (host_agent, http, snmp, traefik, unifi, docker) are now
tracked under plugins/ and baked into the image, so they version atomically
with core — ending the cross-repo import drift the roundtable->steward rename
exposed. History for these files is preserved in the archived Roundtable-plugins
repo.

Plugin discovery becomes multi-root: PLUGIN_DIR (single) -> PLUGIN_DIRS
(bundled first, then external) + PLUGIN_INSTALL_DIR. Bundled ships in the image;
third-party plugins still mount at runtime into the external root
(STEWARD_PLUGIN_DIR, default /data/plugins) and downloads/installs land there.
Bundled shadows external on a name collision.

- config.py: load_bootstrap returns plugin_dirs + plugin_install_dir
- app.py: iterate PLUGIN_DIRS at the migration + load sites
- migration_runner.py: discover_all_in() unions every plugin root
- plugin_manager.py: resolve_plugin_path() (pure, first-root-wins); load /
  install / hot-reload span all roots; installs target the external root
- settings/routes.py: _discover_plugins scans all roots, dedup bundled-first
- Dockerfile: COPY plugins/ ; docker-compose: drop host bind, document external
- tests/test_plugin_dirs.py: resolution, multi-root discovery, bootstrap split

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

ci: add Forgejo CI (lint + unit + Postgres integration) + ci-requirements.md ... d925709c77

Single-repo CI now that plugins are bundled in-tree. Three lanes on push to
dev/main, modeled on FabledCurator's canonical ci.yml (minus frontend/Redis):
- lint: ruff check steward/ plugins/ tests/ (no dep install)
- unit: pytest -m 'not integration' — whole current suite (testing=True mocks DB)
- integration: postgres:16-alpine service via socket-discovered bridge IP; a
  boot-and-migrate test creates the real app, running core + all bundled-plugin
  migrations, then round-trips the DB — guards the folded-in migration graph.

Registers the 'integration' pytest marker; adds tests/integration/test_boot.py.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

fix(plugins): complete steward rename across bundled plugins; clear lint debt ... af60ca446d

The fabledscryer->steward rename had only ever reached host_agent. The other
five bundled plugins (http, snmp, traefik, unifi, docker) still imported
`from fabledscryer.*` (package no longer exists) and read FABLEDSCRYER_* env
vars — so every one of them was broken at import since the original rebrand.
CI stayed green only because none are enabled by default and migrations don't
import plugin modules. Now that they version in-tree, complete the rename:
- fabledscryer.* -> steward.* imports across all five plugins
- FABLEDSCRYER_* -> STEWARD_* in plugin migration env.py files
- author/repository/homepage + user-facing 'Fabled Scryer' strings -> Steward
- snmp/scheduler.py: also drop dead `now`/datetime; record_metric from steward

Adds tests/test_no_legacy_names.py — fails if 'scryer'/'roundtable' ever
reappear in shipped code (the drift bit twice; this stops a third time).

Also clears pre-existing ruff lint debt (unused imports, semicolon statements,
mid-file import) surfaced by the new lint lane.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

feat(host_agent): agent reports its primary IP; mirror into Host.address (task 274) ... 3b2146bc7a

The agent now detects its own primary non-loopback IP (UDP-socket trick, no
packets sent) and sends it in the metadata bag; bumps AGENT_VERSION 1.0.0->1.1.0
and re-detects on SIGHUP. The server persists it on HostAgentRegistration.host_ip
and mirrors it into Host.address ONLY when that field is blank — an admin-typed
address is never overwritten. The reported IP shows in the settings table so
admins can see drift regardless.

- agent.py: detect_primary_ip(); host_ip in collect_metadata(); refresh on reload
- routes.py: pure pick_host_address() helper; ingest persists host_ip + mirrors
- models.py + migration host_agent_002_host_ip: new String(45) column
- settings_list.html: Reported IP column
- plugin.yaml: 1.0.0 -> 1.1.0
- tests: pick_host_address (fill-when-blank / never-clobber / no-op),
  detect_primary_ip (never raises, non-loopback), metadata carries host_ip

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

test(host_agent): assert parsed agent version matches AGENT_VERSION constant ... 712aec60c1

The on-disk-version test hardcoded '1.0.0' and broke on the 1.1.0 bump. Compare
against agent.AGENT_VERSION so future bumps don't require touching the test.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

fix(docker): stop ignoring plugins/ in image build; dev-friendly compose ... ccc7601182

.dockerignore excluded plugins/ (leftover from the separate-repo era), so the
Dockerfile's COPY plugins/ found nothing and the build failed. Track it now that
first-party plugins ship in the image.

docker-compose.yml reworked for local dev: live-mounted ./steward + ./plugins
with PYTHONPATH=/app, --debug auto-reload, fixed dev SECRET_KEY, Postgres port
exposed, and the host-specific traefik/docker.sock mounts commented out.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

feat(ansible): ship ansible in image + allow system-triggered runs (task 545) ... fb579bcf97

Foundation for the Ansible automation milestone (#37) — makes the existing
manual playbook runner actually executable and the schema automation-ready.

- pyproject: [ansible] extra (full ansible package, batteries-included, pinned)
- Dockerfile: pip install .[ansible]; add openssh-client for remote runs
- models/ansible.py + migration 0012: AnsibleRun.triggered_by now nullable so
  automated (alert/schedule) runs need no human actor
- ansible/routes.py + run_detail.html: show 'Triggered by' (username or 'system')
- CI: integration lane installs .[dev,ansible]; new tests/integration/
  test_ansible_foundation.py runs a real connection:local playbook end-to-end,
  asserts success+output, and round-trips a NULL-triggered run

No extra-vars/limit/credentials/scheduling here — those are their own #37 tasks.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

fix(ansible): shorten migration 0012 revision id to fit alembic_version(32) ... 5af7312a72

The id 0012_ansible_run_triggered_by_nullable was 38 chars; alembic_version.
version_num is VARCHAR(32), so stamping it raised StringDataRightTruncationError
and the boot/migrate integration lane failed. Renamed to 0012_ansible_run_nullable.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

feat(ansible): parameterized runs — extra-vars, limit, tags, dry-run (task 546) ... 8b62eb2ca3

Run form + executor now support optional params, passed as argv (never
shell-interpolated):
- extra_vars: one key=value per line -> repeated -e
- limit -> --limit; tags -> --tags
- dry-run checkbox -> --check --diff

- executor.py: pure build_ansible_command(playbook, inventory, params); start_run
  gains an optional params arg (backward-compatible)
- models/ansible.py + migration 0013: nullable params JSON column
- routes.py: create_run parses + validates (extra-var lines need '='), stores
  params on the run, passes to the executor
- browse.html run form: Extra vars / Limit / Tags / Dry-run fields
- run_detail.html: shows the params used
- tests/test_ansible_command.py: unit coverage of the builder; integration test
  runs a real playbook with extra-var + limit + check and asserts substitution

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

feat(alerts): run an Ansible playbook on alert firing (task 250) ... 4771d17f6d

Rebuilds the deleted NUT/UPS automation as a general alert action: any metric
can drive a playbook run on transition-to-firing.

- models/alerts.py + migration 0014: AlertRule.ansible_action (JSON, admin-only,
  reuses the #546 param shape); AlertEvent.ansible_run_id links a firing event to
  the run it triggered
- core/alerts.py: pure alert_extra_vars() injects steward_alert_* context; on
  ('firing', event) with an action set, schedule _run_ansible_action (deferred
  after commit, same pattern as notifications) — fires once per transition,
  consecutive_failures_required is the debounce; system-triggered AnsibleRun
- alerts/routes.py: admin-only parse/validate of the action (source must be
  configured, playbook must exist); operators keep editing rules, action preserved
- rules_form.html: admin-only 'On firing -> run a playbook' section
- tests: unit for alert_extra_vars; integration drives record_metric to firing
  and asserts a system AnsibleRun ran the playbook with the injected var and the
  event linked to it

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

fix(alerts): persist AlertOperator by value to match the DB enum ... 43a5325e69

The alertoperator Postgres enum (migration 0002) was created with the operator
VALUES as labels ('>', '<', …), but the model's Enum(AlertOperator) defaulted to
persisting enum-member NAMES ('gt', 'lt', …) — so inserting any AlertRule against
real Postgres raised 'invalid input value for enum alertoperator: "gt"'. Never
caught because unit tests mock the DB and no integration test inserted a rule
until task 250's. Add values_callable so the ORM round-trips the values.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

test(alerts): commit the test user before the rule's created_by FK ... 534ed030b8

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

chore(dev): mount ./ansible-dev as a local Ansible source at /srv/ansible ... 0a36a57901

Dev convenience so you can author playbooks on the host and have Steward's
local-source discovery pick them up. ./ansible-dev/ is gitignored (operator
content); a safe demo playbook lives there locally.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

feat(ansible): credentials — SSH key, become, vault, host-key checking (task 548) ... 0bf007173b

Global Ansible credentials, applied to every run (manual + alert-triggered):
- core/settings.py: ansible.ssh_private_key / become_password / vault_password
  (plaintext at rest, masked in UI — encryption tracked in #580) + host_key_checking
  (default off); surfaced via to_ansible_cfg into app.config[ANSIBLE]
- executor.py: pure build_credentials() materializes creds into a 0600 temp dir
  (--private-key, --vault-password-file, become via -e @vars-file so the password
  never hits argv) cleaned up in finally; pure ansible_env() sets
  ANSIBLE_HOST_KEY_CHECKING. build_ansible_command stays param-only
- settings/routes.py + ansible.html: admin-only Credentials section, masked-update
  (blank keeps current, explicit Clear checkbox), reload app config on save
- tests: unit (build_credentials per cred + none; ansible_env toggle); integration
  vault round-trip (ansible-vault encrypt a vars file, run via executor with the
  vault password, assert it decrypted)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

feat(ansible): run a playbook against a single Steward host (task 547) ... 38f61b71c1

From a host's edit page, an operator can run a playbook against just that host
via an ephemeral one-host inventory — no need for the host to exist in a source
inventory.

- ansible/sources.py: pure host_inventory_content(host) -> '<name> ansible_host=<addr>'
- executor.start_run: optional inventory_content written to the temp dir and used
  as -i (overrides inventory_path); cleaned up with the creds dir
- hosts/routes.py: POST /hosts/<id>/run-playbook (operator) — validates source +
  playbook, builds the ephemeral inventory, starts a user-triggered run, redirects
  to the live run detail; edit page gets the ansible source list
- hosts/form.html: 'Run Ansible playbook against this host' panel (shown when
  editing an existing host and sources exist) — source + playbook + extra-vars/
  tags/dry-run (no --limit; single host)
- tests: unit host_inventory_content; integration runs a hosts:all/connection:local
  playbook against the ephemeral inventory and asserts inventory_hostname

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

feat(ci): publish steward image to the registry + remote-deploy compose ... fe57dde57b

Add the rule-46 image-publish lane: a `publish` job gated on
[lint, unit, integration] that builds the runtime Dockerfile in ci-builder
and pushes git.fabledsword.com/bvandeusen/steward:<sha> always, :dev on dev
and :latest on main. Authenticates with the REGISTRY_TOKEN repo secret
(the injected GITHUB_TOKEN lacks write:package). Steward is the first family
app to implement the publish lane; mechanics mirror CI-runner's
build-ci-python.yml.

Add compose.deploy.yml for a persistent remote instance that pulls :dev
(no source bind-mounts, persistent app_data+pgdata volumes, bundled Postgres),
plus a README "Remote dev instance" runbook and the POSTGRES_PASSWORD env key.
Fix .env.example's database URL var to the canonical STEWARD_DATABASE_URL
(single underscore) and document the publish lane in ci-requirements.md.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

fix(deploy): move POSTGRES_PASSWORD default into .env.example ... 42d9a7ba07

Plain ${POSTGRES_PASSWORD} in compose; default 'steward' lives in .env.example
where it's visible and editable rather than buried in compose YAML.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Revert "fix(deploy): move POSTGRES_PASSWORD default into .env.example" ... ef6c90c022

This reverts commit 42d9a7ba07.

fix(deploy): let app own secret key — remove STEWARD_SECRET_KEY from deploy compose ... 98edb12abb

The app auto-generates a key on first boot and persists it to /data/secret.key,
which the app_data volume keeps across image updates. No reason to surface this
as a user-facing env var in the deploy compose. Removed from .env.example too.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

fix(deploy): explicit backend network for steward↔db traffic ... 06dc5073ca

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

docs: Ansible inventory infrastructure implementation plan eb319d715e

feat(ansible): AnsibleTarget model + 0015_ansible_inventory_targets migration c11b3f01ed

feat(ansible): AnsibleGroup + join table — 0016_ansible_inventory_groups migration fa32488fdf

feat(ansible): add inventory_scope to AnsibleRun — 0017_ansible_run_scope migration 10df05c13e

feat(ansible): generate_inventory() + fetch_scope_targets() + unit tests ea47169049

feat(ansible): GIT_ASKPASS per-source HTTP token auth + settings UI d906232eb2

feat(ansible): scope picker in run form + create_run() DB inventory generation 2c991eabd2

feat(ansible): inventory groups CRUD routes + templates + blueprint registration 935cbc105c

feat(ansible): inventory targets CRUD templates 37d9ca8a8a

feat(ansible): host edit page Ansible target link/create/unlink section dd0acaf623

test(ansible): integration tests for inventory tables + fetch_scope_targets 923905e72d

chore(lint): remove unused imports flagged by ruff ... 80613d6310

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

feat(plugins): default-enable generic bundled plugins on fresh install ... b49496b57b

A fresh install enabled zero plugins — settings.py DEFAULTS had no plugin.*
keys, so to_plugins_cfg returned {} and every plugin had to be flipped on
by hand. Seed docker, host_agent, http, snmp as default-on (generic, non-
vendor-specific); traefik and unifi stay opt-in. Stored choices override
the default, so disabling persists.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

fix(test): call sync create_app directly in ansible inventory integration test ... 67a1bc740d

create_app is synchronous (returns Quart) and runs its own internal
asyncio.run for settings/migrations. _make_app wrapped it in asyncio.run,
which nested event loops and raised "a coroutine is required" — the other
integration tests already call create_app() directly. This test was added
in an unpushed commit, so CI never exercised it until now.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

feat(status): unified Status page + widget across ping/DNS/HTTP monitors ... 3b6e005ed8

Adds a Kuma-style "is everything up?" surface that aggregates heterogeneous
monitor types via a status-source registry (steward/core/status.py): each
type registers an async source(db) -> [StatusEntry]. Core registers ping/DNS;
the http plugin registers its own from setup() so core never imports plugin
tables. Per entry: current up/down, last-30 heartbeat bar, uptime %
(24h/7d/30d), latest latency + response sparkline, and TLS expiry countdown
(HTTP). New /status page (live htmx refresh) + a status_overview dashboard
widget + nav link. Pure-function unit tests for registry + sparkline.

First deliverable of milestone #68 (task #866).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

feat(host_agent): collect network, disk I/O, per-core CPU, temps, memory PSI ... a9e7baee6a

Extends the push agent (v1.2.0) with the Netdata-style signals the operator
wants: per-core CPU, per-interface network throughput and per-disk I/O
(rates derived in-agent from monotonic /proc counter deltas, with
counter-reset clamping), hardware temperatures (/sys/class/hwmon),
memory-pressure PSI (/proc/pressure), and cached/buffers memory breakdown.
All stdlib-only.

Server side needs no migration — these land as additional rows in the shared
PluginMetric table via _expand_sample_to_metrics. Per-resource series use a
':' in resource_name (host:net:eth0, host:core0, host:temp:Package) so the
existing fleet widget's ':' filter ignores them; host-level totals/max are
emitted at the bare host resource. New sample keys are optional, so older
agents keep ingesting unchanged.

Unit tests for the new parsers, rate/reset logic, and the expander contract.
Data foundation for the Netdata-style host view (task #867).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

test(host_agent): update existing collector tests for new agent API ... f037b69c58

collect_cpu now returns (aggregate, [per_core]) and build_sample takes a
rate-state dict, so the pre-existing tests that asserted the old float
return / no-state signature needed updating. Mocks the new collectors
(temps/psi/net/diskio) for determinism.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

feat(host_agent): Netdata-style fleet overview + per-host detail view ... bca1b92cc6

Surfaces the metrics the agent now collects (task #868). Adds a viewer-facing
fleet page (/plugins/host_agent/) with per-host cards (CPU/mem/disk/load/temp,
stale flag) and a per-host detail page (/plugins/host_agent/<id>/) — the link
the fleet widget already pointed at but had no route for.

Detail page shows current gauges (CPU, memory incl. swap/cache, load, network
rx/tx, disk I/O, temp max, memory/cpu/io PSI), per-core CPU bars, per-mount
filesystem bars, and per-interface/disk/sensor breakdowns, plus history charts
(utilization %, throughput B/s, load & pressure) over a selectable range.
Charts use a linear epoch-ms x-axis so no Chart.js date adapter is needed.
Stale state uses the plugin's stale_after_seconds threshold. Repoints the
host_resources widget detail link from admin settings to the new fleet page.

Completes milestone #68 (task #867).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

feat(ansible): scheduled recurring playbook runs ... 4a0a3ee46e

Adds cron-like recurring runs (the engine the maintenance-automation work
needs). New AnsibleSchedule model + migration 0018; a core ScheduledTask
(ansible_scheduled_runs, 60s) fires due schedules, each creating a
system-triggered AnsibleRun (triggered_by=None). Centralises the
resolve-inventory → create-run → launch flow in ansible/runner.trigger_run,
shared by the manual route (refactored to use it) and the scheduler.

Schedules UI under /ansible/schedules: create/edit/pause/delete/run-now,
with interval presets, scope targeting (all / group / target), extra-vars /
limit / tags / dry-run, and last-run status (resolved via last_run_id) +
next-run. Unit test for the due-check.

Task #549 (milestone #37).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

feat(ansible): bundled first-party playbooks + built-in source ... 656bda2e3d

Ships a read-only "steward-builtin" source (steward/ansible/bundled/) that
always appears alongside operator-configured sources, with two playbooks:
- maintenance/docker_prune.yml — docker system prune for swarm/standalone
  nodes (safe by default; prune_all_images / prune_volumes extra-vars to
  widen). Schedule it against a swarm-node group for recurring cleanup (#869).
- host_agent/install.yml — installs/updates the host agent (mirrors
  install.sh: user, agent.py, config, hardened systemd unit), parameterised
  with steward_url + steward_token extra-vars.

get_sources() now prepends the builtin source. Tests updated to find the
configured git source by name; added coverage that the bundled playbooks are
discoverable.

Tasks #869 + agent-install (milestone #37).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

feat(plugins): plugin capability registry + host_agent→Ansible deploy synergy ... f3e919892d

Implements #253's framework: a small core capability registry
(steward/core/capabilities.py) where a module/plugin publishes a named,
role-gated action and a consumer discovers it via has_capability() and runs it
via invoke_capability() — no hard import, graceful degradation, permission
propagation (actor role checked against the capability's required_role).

Core publishes "ansible.run_playbook" (operator) wrapping ansible.runner.
trigger_run (extended to accept a caller-built inventory). First consumer: the
host_agent plugin gains "Deploy via Ansible" on its settings page — pick an
inventory target/group and it installs/updates the agent via the bundled
host_agent/install.yml, minting a fresh token per host and injecting it as an
inventory hostvar (turning per-host curl|sh into one run). Exposed role
ordering as middleware.role_meets. Unit tests for the registry + role checks.

Task #253 (milestone #37).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

feat(ansible): in-app playbook authoring/editor ... c10eae1c74

Adds create/edit/delete of playbooks from the UI (admin only), so a homelab
user without a git workflow can author automation in-app. A new always-present
writable local source "steward-local" (/data/ansible/playbooks, env-overridable,
created on first save) is editable alongside operator local-dir sources; the
bundled and git sources stay read-only (git is GitOps, clobbered on pull).

sources.py: write_playbook / delete_playbook (traversal-guarded, .yml/.yaml
only) + validate_playbook_yaml (YAML + play-list check) + is_editable_source.
routes.py: /playbooks/new, /edit, /save, /delete (admin). Browse gains a
"New playbook" button and per-playbook + view-page Edit/Delete for editable
sources. Plain textarea editor with save-time YAML validation. Unit tests for
write/delete/guard/validate.

Task #579 — completes milestone #37 (Ansible automation).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

feat(ansible): cross-link inventory/schedules/browse for discoverability ... 71e4724286

The inventory CRUD UI (/ansible/inventory/targets + /groups) existed but was
unreachable from the main Ansible pages — only a buried text hint pointed to
it. Add an "Inventory" button to the Runs, Browse, and Schedules headers, and
"← Ansible" back-links on the inventory target/group pages, so the targeting
features (manual runs, schedules, Deploy-via-Ansible) are findable.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

feat(ui): breadcrumb navigation across nested pages ... 389002fc6f

Adds a breadcrumb trail to nested views for orientation. Mechanism: a
{% block breadcrumb %} slot in base.html (+ styling) and a shared crumbs()
macro in templates/_macros.html; each nested page fills the block with its
trail (root→current, last item is the current page). Pages without the block
render no bar, so top-level nav roots stay clean.

Applied to: Ansible (browse, schedules, playbook editor, run detail) +
inventory (targets/groups + detail), host_agent (fleet, host detail,
settings), hosts form, settings tabs (ansible/auth/notifications/plugins/
reports + plugin detail), dashboard (list, edit), and alerts (rule form,
maintenance + new). Dynamic labels (host/target/run/dashboard names) come
from the page context — no route changes. All 60 templates Jinja-compile.

Task #873.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

feat(ansible): runner robustness — cancel, concurrency, structured results, retention ... 88857be24e

Closes #550 (all four):

- Cancellation: track live subprocesses; POST /ansible/runs/<id>/cancel
  (operator) SIGTERMs then SIGKILLs after a grace; new 'cancelled' status
  (+ migration 0019, ALTER TYPE in autocommit). Queued runs cancel cleanly
  before launch. Cancel button on run detail.
- Concurrency: global semaphore (ansible.max_concurrent_runs, default 3,
  Settings→Ansible) caps simultaneous runs; excess show 'queued' (new status)
  until a slot frees. Semaphore bound lazily per running loop.
- Structured results: parse PLAY RECAP into per-host ok/changed/unreachable/
  failed/skipped + capture failed-task lines, stored in new results JSON
  column (migration 0020); rendered as a host-summary table on run detail.
  Keeps live streaming (no json-callback swap).
- Retention: full output written to a persistent log artifact
  (/data/ansible/runs/<id>.log, env-overridable) beyond the 1 MB DB cap and
  across restarts; in-memory replay buffer bounded + GC'd after completion;
  Download-log route. Boot reconciliation now also sweeps stale 'queued'.

Unit tests for recap parsing + cancel flagging. Status colors updated across
run list / detail / schedules.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

feat(security): encrypt sensitive settings at rest (Fernet) ... 6e91bdc82b

Secrets (smtp.password, oidc.client_secret, ldap.bind_password, ansible
ssh_private_key/become_password/vault_password) were stored plaintext in
app_settings. Add transparent encryption-at-rest:

- steward/core/crypto.py: Fernet keyed off the app secret (/data/secret.key),
  enc:v1: prefix marks ciphertext; passthrough for plaintext/empty/no-key,
  never reveals plaintext on a wrong key.
- settings.py: SECRET_KEYS registry; set_setting encrypts on write; all read
  paths (get_setting / get_all_settings / load_settings_sync) decrypt
  transparently; migrate_plaintext_secrets() converts legacy rows in place.
- app.py startup: init_crypto(SECRET_KEY) + one-time legacy-secret migration
  before settings load.
- Add cryptography dependency.

UI masking is unchanged (it checks decrypted truthiness). Key-loss caveat
documented: secrets are unrecoverable if the app secret key is lost. Unit
tests cover round-trip, empty/plaintext passthrough, and wrong-key safety.

Task #580.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

feat(ansible): host provisioning via steward managed SSH identity ... 0318f6423f

Turn the agent-install playbook into a full provisioning + maintenance
path. Solves the bootstrap chicken-and-egg: first contact uses an
operator-supplied password (one run, never stored), which creates a
dedicated `steward` login account with NOPASSWD sudo + Steward's managed
public key. Every run thereafter connects as `steward` with the managed
key — fully unattended (scheduled prune, agent updates).

- core/crypto: generate_ssh_keypair() — ed25519, OpenSSH formats.
- settings: ansible.ssh_public_key (non-secret, displayed) + ansible.ssh_user
  (default steward); to_ansible_cfg extended.
- settings UI + route: "Generate managed key" (private encrypted, public
  shown to copy) + SSH-user field.
- executor: build_bootstrap() writes a 0600 vars file (-e @file) for the
  per-run user/password — never argv, never DB, never logged; drops the
  managed key when a bootstrap password is given; --user floor from the
  global ssh_user when no override.
- runner.trigger_run: pass-through `connection` kwarg, deliberately NOT
  persisted on AnsibleRun.params (password stays out of the DB).
- bundled/host_agent/provision.yml: create steward user + authorized_keys
  + /etc/sudoers.d/steward (visudo-validated) + agent install.
- host_agent: /provision route + "Provision a fresh host" card (bootstrap
  user/password; injects pubkey + user + token as space-safe JSON hostvars).
- Dockerfile: add sshpass (Ansible shells out to it for password SSH).
- tests: keypair generation + build_bootstrap (secret stays off argv).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

feat(ansible): per-variable fields in the playbook run form ... a996cc6908

When you click Run, Steward now parses the playbook and renders a field
for each declared variable instead of a blank extra-vars textarea. The
form loads on demand via HTMX (/ansible/run-form/<source>/<playbook>).

- sources.discover_playbook_variables: parse vars: defaults + vars_prompt:
  (vars_prompt wins on name collision; non-scalar vars skipped; role/include
  vars not traversed). Flags secret-looking names + vars_prompt private.
- Run-time values flow through a JSON extra-vars file (-e @file), which is
  space/quote-safe — fixes a latent shlex-split bug in the old -e key=value
  textarea path. executor.build_extra_vars_file (pure) + start_run merge.
- Secret-flagged fields are masked AND routed through an unpersisted
  secret_vars channel (runner.trigger_run → start_run), so passwords entered
  at run time never land in the DB / run history.
- Defaults shown as placeholders (not prefilled): an untouched field falls
  through to the inventory/play default instead of overriding it.
- routes: run_form HTMX endpoint; _parse_run_params now returns
  (params, secret_vars, err) and reads var__/secret__ fields. Schedules drop
  secret vars (can't prompt unattended).
- templates: ansible/_run_form.html fragment; browse.html rewired to HTMX,
  static JS run-form removed. Advanced section keeps limit/tags/check + a
  free-form extra-vars escape hatch.
- tests: test_playbook_variables.py (discovery + extra-vars file).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

fix(ansible): replace steward key on reprovision + distinct agent update path ... 6a8146b544

Provisioning review corrections + the matching frontend, plus breadcrumb
header integration.

- provision.yml: authorize the managed pubkey with a regexp match on the
  ' steward-managed' comment so rotating the key REPLACES the host's steward
  key in place instead of stacking a second authorized entry. Hand-added keys
  (other comments) are untouched.
- update.yml (new): refresh agent.py + restart only. Does NOT rotate the token
  or rewrite /etc/steward-agent.conf — the host keeps its identity. Asserts the
  agent is already installed and fails clearly otherwise.
- host_agent /update route: runs update.yml as the managed steward user (no
  token minting). Token rotation stays a deliberate action.
- settings/ansible/generate-key honors a safe relative `next` redirect, so an
  inline trigger elsewhere returns to its page.
- Host Agents settings: reworked into a clear lifecycle — an intro card that
  explains it runs Ansible to deploy the agent (+ inline "generate managed key"
  warning/trigger when none exists), then three labelled cards: 1 Provision,
  2 Install/enroll, 3 Update. Each explains what it does.
- base.html: breadcrumb now renders as a kicker line directly above the page
  title (moved below alerts, tightened margin) so nested and top-level views
  share one consistent header.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

feat(hosts): Phase 1 — host detail hub page (unify host IA) ... 8bdf07f709

Make Hosts the front-and-center hub. A host now has a real detail page at
/hosts/<id> that pulls its facets into one view, instead of management being
scattered across a nav-less Host-Agents area and the edit form.

- hosts: new GET /hosts/<id> detail route + hosts/detail.html. Shows the
  monitors summary (ping/DNS status + latency + uptime 24h/7d/30d), an Ansible
  section (linked target, link/create, run-playbook), and an embedded Agent
  panel. Hosts list name links here; ansible-link redirects here.
- host_agent: GET /plugins/host_agent/panel/<host_id> — a self-contained HTMX
  fragment embedded into the core hub across the plugin boundary (core never
  imports plugin models). Shows live agent metrics + Update/Rotate/Remove when
  installed, or the provisioning path when not: inline "generate managed key"
  warning, a prompt to link an Ansible target first, then Provision (bootstrap
  password) / Install (managed key) tied to the host's target scope.

Part of milestone 70 (Hosts hub). Phase 2+ will enrich the list, redirect the
old fleet/settings pages, and re-taxonomize plugins into capabilities vs
integrations.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

feat(hosts): Phase 2+3 — agent column, fold fleet + management into Hosts ... f29255039d

- Hosts list: new Agent column (latest CPU/mem read from the generic
  PluginMetric table — no host_agent import) + an admin "Agent fleet" button.
- /plugins/host_agent/ (old fleet page) now redirects to /hosts/ (folded into
  the hub; kept as a redirect so widgets/links don't 404).
- Agent management moved off the "settings" URL: the management page is now
  /plugins/host_agent/fleet/ ("Agent fleet" — bulk provision/install/update +
  registrations + curl install), reachable from the Hosts list. Old
  /plugins/host_agent/settings/ redirects there. Per-host management lives on
  the host detail page; this page is now explicitly the bulk/fleet view.

Milestone 70 phases 2-3. Phase 4 (plugins capability/integration split + nav
cleanup) next.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

feat(settings): Phase 4 — capabilities vs integrations + nav cleanup ... 7ef1af2184

Final phase of the host-IA unification (milestone 70).

- Settings → Plugins split into two tiers: "Monitoring capabilities"
  (host_agent, http, snmp, docker — built-in host facets, surfaced via
  Hosts/Status, on by default) and "Integrations" (traefik, unifi — external
  systems, off until configured). Presentation only: a CAPABILITY_PLUGINS set
  (overridable by plugin.yaml `kind:`) tags each plugin; module loading,
  optional deps, and migrations are untouched.
- Drop the "default-enable a plugin" framing in the UI copy — capabilities are
  described as built-in, not optional add-ons.
- Nav: remove the standalone "Uptime" item (folded into Hosts; still reachable
  via the SLA button on the Hosts list).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

refactor(hosts): remove legacy host_agent redirects/paths (no back-compat) ... 17c9c875e4

Dev-only instance, no bookmarks — per family rule 22, fully remove old paths
instead of shimming them.

- Delete the /plugins/host_agent/ (index) and /plugins/host_agent/settings/
  redirect routes; delete the now-dead host_list.html fleet template.
- Move the remaining management POST routes off /settings/ to /fleet/
  (add-host, rotate-token, delete) — single canonical prefix.
- Repoint real callers to canonical URLs: dashboard widgets (host resources →
  /hosts/, history → /plugins/host_agent/fleet/), the full-metrics page
  breadcrumb + back link (→ the host hub), Settings→Ansible link, and the
  agent panel's curl-install link.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

fix(host_agent): deploy/provision 500 — double-begin on session ... cfe6b4c25f

fetch_scope_targets() runs a SELECT that autobegins the session transaction
(SQLAlchemy 2.0), so the following `async with db.begin()` raised
"A transaction is already begun on this Session" → 500 on both
/plugins/host_agent/deploy and /provision.

Mint registrations directly into the autobegun transaction, build the
inventory while the ORM objects are still live, then await db.commit().
Escaped CI because unit tests use a mock db_sessionmaker; no integration
test exercises these routes (follow-up noted in Scribe issue #884).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

feat(crypto): name the failing setting in wrong-key decrypt log ... 9ce4cce5c5

The "could not decrypt a stored secret" warning was generic, so an operator
couldn't tell which of the six secret settings was encrypted under an old key.
Thread the setting key through _decode → decrypt_secret(context=...) so the log
now reads e.g. "Could not decrypt stored secret smtp.password (wrong/rotated
key — re-enter it)". Pure diagnostic; decrypt behaviour unchanged.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

fix(ansible): write DB inventory as static YAML, not dynamic --list JSON ... a92d1995d5

generate_inventory() emits Ansible's dynamic --list shape (all.hosts is a
LIST, vars under _meta) — valid only as an executable inventory script's
stdout. We were writing it to a static file and passing -i, so Ansible's yaml
plugin rejected it ("Invalid 'hosts' entry for 'all' group, requires a
dictionary, found ...list...") and fell back to implicit localhost → "no hosts
matched". Affected every steward:* scope run; surfaced on the first real
provision.

- New inventory_to_yaml(inv): convert the --list dict → a valid static YAML
  inventory (all.hosts dict keyed by host, groups under all.children, group
  vars preserved, injected per-host vars like steward_token retained).
- Wire it into runner.trigger_run, host_agent deploy + provision.
- executor writes the file as inventory.yml so the yaml plugin's extension
  check reliably claims it.
- generate_inventory unchanged (still the --list dict); conversion happens at
  write time. Unit tests added.

Scribe issue #885.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

fix(host_agent): gate "deployed" on agent check-in; fail no-op runs ... bb90411f00

A failed provision looked successful in two ways:

1. The host panel showed the agent as deployed (metrics + Update/Rotate/Remove)
   because provision/deploy mint the registration row BEFORE the playbook runs
   and the panel keyed "installed" on that row. Now gated on the agent actually
   checking in (reg.last_seen_at). Three states: reporting (metrics + lifecycle),
   pending (token minted but no check-in → "no metrics yet, deploy may be
   running/failed" banner + retry + Clear pending registration), and none
   (install path).

2. The run reported success though nothing ran — ansible-playbook exits 0 on
   "no hosts matched"/empty inventory. The executor now treats an empty PLAY
   RECAP (returncode 0 but no hosts executed) as failed, with a clear failure
   note. Non-zero exits and recap failed/unreachable were already caught.

Scribe issue #887.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

feat(ansible): dropdown playbook selection + auto-populated variable fields ... ad726e65f3

Stop making operators type playbook paths and guess extra-var names.

- Reusable infra: shared ansible/_playbook_vars.html (the discovered-variable
  fields) + ansible/_playbook_options.html; two HTMX endpoints —
  /ansible/playbook-options (a source's playbooks, optional ?selected for edit)
  and /ansible/playbook-vars (a playbook's vars:/vars_prompt: as fill-in
  fields). browse _run_form.html refactored to include the shared partial.
- Host "Run a playbook against this host": source dropdown → playbook dropdown
  → variable fields, all chained via HTMX. Handler reuses _parse_run_params so
  var__/secret__ fields flow through extra_vars_map + the unpersisted
  secret_vars channel.
- Schedules: playbook free-text+datalist → source-dependent dropdown; fixed the
  extra-vars edit pre-fill to read extra_vars_map (stale list key after the
  earlier JSON-extra-vars change).

Scribe #895.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

fix(host_agent): provision/deploy vars shadowed by play-var precedence ... 7e6e63521b

The assertion failed ("Pass steward_url/token/pubkey") because those were
injected as inventory HOST vars, but the playbooks declared them in the play
`vars:` block — and play vars OUTRANK inventory host vars, so the empty
defaults won and the injected values never reached the play.

- Pass globals (steward_url, steward_pubkey, steward_user, agent_interval) as
  extra-vars via the JSON -e @file (highest precedence, space-safe). Keep only
  the per-host steward_token as an inventory host var.
- provision.yml / install.yml: drop steward_token from `vars:` so the host var
  isn't shadowed; assertions use `| default('')` for the un-defaulted token.

This was the next layer under the inventory-format fix — first the inventory
wouldn't parse, now the injected vars actually apply.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

feat(auth): capture Steward URL during first-run admin setup (OOBE) ... f80f6c87e8

A fresh install had no prompt for general.public_base_url, so first-run agent
installs/share links silently used the request Host header. Add a "Steward URL"
field to the first-run /setup page (create-admin), pre-filled with the current
address, with help text. setup_post saves general.public_base_url and applies
it to app.config immediately (no restart). Editable later in Settings → General.

Scribe #896.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

feat(host_agent): at-a-glance disk = root (/) not "worst"; metric tooltips ... 36212dc58b

The "Disk (worst)" number was opaque at a glance. Show the root filesystem (/)
usage instead — what people actually care about — on the host panel and the
dashboard widget. "Worst" is kept only for the widget's health dot (so a full
/var or /data still warns) and on the full-metrics page (per-mount + worst
trend). Added hover tooltips defining CPU / Memory / Disk / Load.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

feat(host_agent): sparklines + load/core + PSI on the host panel ... a0d1c5f07c

- Each at-a-glance metric (CPU, Memory, Disk /, Load) now shows a 6h sparkline
  (reused core.status.sparkline_svg + a recent-series query; disk uses the root
  mount sub-resource) so trend/consistency is visible, not just the instant.
- Load is now normalized: "Load /core" = 1m load ÷ CPU cores as % (100% = run
  queue matches capacity), comparable across different hardware. Cores derived
  from the per-core CPU metrics already collected — no agent change. Raw load
  in the tooltip.
- Added a "Pressure 10s" line: PSI cpu/mem/io (some, avg10), the
  hardware-independent saturation signal already collected by the agent.

Scribe #898.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

feat(ansible): playbooks self-describe via "# description:" comment ... e5f6a11f94

Playbooks can ship a human description Steward reads and shows when one is
selected. Convention: a `# description: <text>` magic comment (Ansible rejects
unknown play keys, so a comment is the portable place — works for third-party
playbooks too); falls back to the first play's name:. sources
.discover_playbook_description().

Surfaced at the top of the shared _playbook_vars.html partial, which loads on
playbook selection in the host run form, schedules form, and browse run form.
All four bundled playbooks (provision/install/update/docker_prune) now carry a
description line. Unit tests added.

Scribe #900.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

docs: Steward playbook-authoring conventions reference ... b32fce1d74

A self-contained guide (docs/reference/playbook-authoring.md) to the contract
between a playbook and Steward's run UI — the `# description:` comment,
vars/vars_prompt → fill-in fields, secret naming, hosts: all targeting,
managed credentials, and idempotency. Includes a "what Steward reads" summary
table, the metadata/extensibility note (only `# description:` today; reserved
`# steward:<key>:` namespace for future keys), and an annotated example.
Meant to be fed to another session authoring Steward-friendly playbooks.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

feat(ansible): steward:category + steward:confirm playbook metadata ... 42f7840c26

Extend the playbook metadata convention with a namespaced `# steward:<key>:`
comment block:

- steward:category — free-text grouping label, shown as a badge in the browse
  list and on the run form.
- steward:confirm — true/yes/1/on marks a playbook destructive; the run form
  then requires a confirmation tick (required checkbox in the shared vars
  fragment) before it can launch.

sources.discover_playbook_meta() parses description + category + confirm (first
match per key; `# description:` still primary, `# steward:description:` alias).
discover_playbook_description() now delegates to it. The browse list reads
per-playbook meta to show category badges + descriptions; the run-form and
playbook-vars fragments render the badge + confirm gate.

Bundled playbooks tagged: docker_prune → category maintenance + confirm true;
provision/install/update → category host-agent.

Docs: docs/reference/playbook-authoring.md updated (keys now implemented) and a
quick reference added next to the code at steward/ansible/PLAYBOOK_CONVENTIONS.md.
Tests added for category/confirm/alias parsing.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

feat(dashboard): unified Hosts widget + grouped widget picker ... 609bd78af2

Bring the dashboard in line with the unified host IA + improved displays.

- New core "Hosts — Overview" widget (/hosts/overview/widget): one row per host
  combining monitor status (ping dot + latency, uptime 24h) with the agent
  glance (CPU / memory / disk root + stale flag), each row linking to the host
  hub. Reads agent data from the generic PluginMetric table via a core-safe
  _agent_overview_by_host helper (no host_agent import); freshness vs the
  plugin's stale window. The granular Ping/DNS/Uptime/Agent widgets stay.
- Group the add-widget picker into Core monitors / Monitoring capabilities /
  Integrations (a `group` field on every WIDGET_REGISTRY entry + section
  headings in _edit_panels.html), matching the Settings → Plugins taxonomy.
- Fix the agent fleet widget rows to link to the host hub (/hosts/<id>) instead
  of the old /plugins/host_agent/<id>/ page.

Scribe #903.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

fix(ansible): surface run failure reason; widget audit cleanup ... e0253fba48

Run UX: when start_run throws before/around launch (e.g. ENOSPC creating the
temp dir — the box is out of disk), the run was marked "failed" with empty
output. Now the exception is broadcast + written to the run output/results so
the run view shows e.g. "[run error] OSError: [Errno 28] No space left on
device" instead of a blank failure.

Widget audit follow-ups (no broken links were found; these are consistency):
- host_resource_history widget now charts root (/) disk, consistent with the
  host panel (was the opaque "disk worst").
- host_resources widget: tooltip on the health dot explaining it warns on the
  worst mount while the number shows root.
- status_overview widget detail_url /status → /status/ (avoid redirect).
- Normalize ad-hoc widget empty-states to the shared .empty style (wording,
  which distinguishes "configured" vs "data yet", preserved).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

fix(ansible): clear error when the managed SSH key can't be decrypted ... 2594ca517d

When ansible.ssh_private_key can't be decrypted (app secret key changed),
decrypt_secret returns the ciphertext unchanged; the executor was writing that
enc:v1: blob as the SSH key file → cryptic "Load key ...: error in libcrypto"
→ Permission denied. Now:
- build_credentials skips a still-encrypted key value (never writes ciphertext
  as a key file).
- start_run broadcasts a plain-language run error: "The managed SSH key could
  not be decrypted (the app secret key changed). Regenerate it in Settings →
  Ansible and re-provision the host(s)."

The run still fails (no usable key), but the reason is now obvious instead of a
libcrypto error. Operator remedy: regenerate the managed key + re-provision.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

feat(auth): return to the original view after an auth-expiry redirect ... 71715c38d8

When a session has expired, require_role now bounces to /login?next=<path> and
sends the user back there after re-login.

- middleware: safe_next_url() (same-site relative path/query only; rejects
  off-site, protocol-relative, javascript:, and the auth pages — no open
  redirect). _login_redirect() builds the next param; for HTMX requests it uses
  HX-Current-URL (the page, not the fragment) and HX-Redirect so the whole
  browser navigates instead of swapping the login page into a fragment.
- login GET/POST carry `next` (hidden field), validated, used on success for
  local + LDAP; OIDC stashes it in session across the IdP round-trip.
- login.html: hidden next field + next on the SSO link.
- tests for safe_next_url.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

feat(hosts): re-provision action on reporting hosts; Ansible off the edit page ... 88afad9de4

- Host agent panel (reporting state) gains a "Re-provision" collapsible (admin +
  linked target + managed key): bootstrap user/password → provision.yml, which
  reinstalls the steward account + managed key + agent. This is the missing path
  after regenerating the managed key — Update alone can't fix a broken key.
- Remove the Ansible sections (run-playbook + target link) from the host EDIT
  page — they were the stale free-text version and the wrong place. They live on
  the host detail hub (dropdown + discovered variables). Edit page now links to
  the host page; edit_host route simplified (no ansible fetch).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

feat(ansible): bundled system_update maintenance playbook ... 2ea8c2f9af

New maintenance/system_update.yml: cross-distro OS package upgrade (apt +
dnf/yum) with two run-form flags:
  - restart_services: restart every service that needs it after the upgrade
    (Debian via needrestart -r a, installed if missing; RHEL via
    needs-restarting -s → systemctl try-restart).
  - reboot_if_required: reboot the host only when a reboot is actually pending
    (Debian /var/run/reboot-required; RHEL needs-restarting -r). Never reboots
    otherwise.

Tagged steward:category=maintenance and steward:confirm=true (significant /
reboot-capable), so it shows in the run UI with the confirm gate and both flags
as fill-in fields. No code changes — auto-discovered from the builtin source.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

feat(dashboard): phase A widget clarity — threshold colours, host links, tooltips ... 5f92340c0c

Milestone 72 phase A (clarity wins, no schema change):
- Add shared metric_style() macro in _macros.html: colours a numeric metric
  amber (>=warn) / red (>=crit) on the value ITSELF, not just the status dot.
  Defaults 80/90 for percentage gauges; load /core uses warn 80 / crit 100.
  Applied to CPU/mem/disk across host_agent panel + fleet widget + the unified
  hosts-overview widget.
- Link every host reference to the host hub (/hosts/<id>) in ping, dns,
  hosts-overview, host_agent fleet, and uptime widgets; status widget entries
  link to their own detail_url. All guarded on session.user_id so the public
  share view degrades to plain text (the bare href carries no share token).
- Fix truncation: title= tooltips on all names that ellipsis, plus a hover
  underline affordance on the now-clickable ping-name links.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

feat(dashboard): phase B drag-resize grid (Gridstack) replacing masonry ... 525f6eedbd

Milestone 72 phase B — Grafana-style drag-resize grid for dashboard widgets:
- DashboardWidget: replace `position` with a 12-col grid placement
  (grid_x/grid_y/grid_w/grid_h). Migration 0021 backfills the old position
  order into a 3-up grid (4 cols x 4 cells each) and drops position.
- View + share render a static CSS grid from x/y/w/h: fixed cell height
  (h * 70px) with the body scrolling, so the arranged layout is what's shown;
  collapses to a single column under 820px.
- Edit view: Gridstack.js 12.6.0 (vanilla, CDN, pinned) — drag the title bar
  to move, drag a corner/edge to resize; every change autosaves to a new
  /d/<id>/edit/layout endpoint. Replaces the SortableJS position reorder.
- add_widget appends at the bottom-left; remove no longer renumbers.
  _get_widgets now orders by grid position (drives DOM + mobile fallback order).

Note: Gridstack drag-resize is browser-only, so CI can't exercise it — needs an
operator visual check of the edit experience.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

feat(dashboard): phase C richer panels — host time-series graph + cpu sparklines ... 988c13d51f

Milestone 72 phase C — bring the host-view graphs onto the dashboard:
- host_resource_history widget reworked into a real host-view chart: epoch-ms
  linear axis (no Chart.js date adapter), themed like the host-detail charts,
  maintainAspectRatio:false so it fills the resized panel, unique canvas per
  widget instance (wid), and empty states ("pick a host" / "no metrics yet").
  Was previously unusable — it had a broken time axis and no way to choose a host.
- Add a "host" param type: the edit form renders a live dropdown of hosts
  (dashboard routes now pass the host list to the editor); the chosen host_id is
  stored in config and fed to the widget.
- Hosts-overview widget gains a per-row CPU sparkline (last hour) via the shared
  sparkline_svg helper — the host-view at-a-glance trend, on the main widget.

Charts/sparklines render only in the browser, so CI can't exercise them — needs
an operator visual check.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

feat(dashboard): edit in place over the live dashboard with a bottom widget drawer ... 88dca32d3c

Milestone 72 phase D — the dashboard view IS the edit surface (operator ask):
- /d/<id> renders the grid via Gridstack in STATIC mode — positioned exactly like
  before, live HTMX widget bodies keep polling. An "Edit" button flips the same
  grid interactive (grid.setStatic(false)) in place, so you drag/resize over real
  data. "Done" flips it back. Layout autosaves on change.
- The widget picker is now a bottom drawer (position:fixed overlay) revealed by
  body.dash-editing — so the dashboard width is identical entering/leaving edit.
- Add: POST returns a single grid item; JS inserts it + grid.makeWidget +
  htmx.process so it loads live data. Remove: POST 204 + grid.removeWidget.
  Per-panel drag handle + remove ✕ are in the DOM for editors, shown only while
  editing.
- Gridstack loads for everyone (viewers get the static grid); edit wiring is
  gated on can_edit. Mobile collapses to one column.
- Removed the separate /d/<id>/edit route + edit.html + _edit_panels.html
  (rule 22). Dashboard-list Edit and new-dashboard create now deep-link
  /d/<id>?edit=1 which opens edit mode on load.

Browser-only behaviour — CI can't exercise it; needs an operator visual check.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

feat(dashboard): per-metric sparklines in the Host Agent — Resources widget ... cb47b5e977

Operator ask: show the graphs next to their fields in the fleet widget, like the
host page's AGENT panel. Each row now renders cpu/mem/disk/load as a value with a
trend sparkline beneath it (1h window), instead of bare numbers.

- _fleet_rows fetches a per-host recent series (cpu/mem/load host-level, disk from
  the root mount) in one 1h query and attaches a sparkline per metric to each row.
- widget_table.html lays out a metric cell (label + threshold-coloured value +
  sparkline) per field, mirroring panel.html. Threshold colour is computed in the
  loop and passed into the cell macro (keeps Jinja macro scope clean).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

feat(dashboard): widget readability — name-per-line, fill graphs, container breakpoints ... e58c86cf01

Address graphical issues raised from the dashboard screenshot:
- No more truncated host names. Hosts-Overview and Host-Agent-Resources put the
  host name on its own line (full, wraps if needed) with its data grouped beneath
  it; ping/dns (.ping-name) and uptime widget names wrap instead of ellipsis.
  Prefer vertical overflow over cramming/truncating a row.
- History graph fills the panel: drop the fixed 0–100 y-axis ceiling (beginAtZero
  + 8% grace) so the lines use the vertical space instead of hugging the bottom;
  axis labels still show real %.
- Container-query breakpoints: the widget body is now a query container, so
  fragments restyle to their OWN panel width. Host-list widgets flow into 2 cols
  ≥520px and 3 cols ≥900px (.host-blocks) — use the width, remove vertical
  deadspace — and collapse to one column when narrow. Mirrored into the share view.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

fix(dashboard): downsample history graphs + readable tooltip time ... fed9973899

The agent reports every few seconds, so multi-hour history series were hundreds–
thousands of points — a dense, noisy line (esp. CPU). Bucket-average server-side
to ~120 points (keeps the shape, drops the noise) for both the history-graph
widget and the host-detail charts. Also fix the chart tooltip title showing the
raw epoch-ms (e.g. 1,781,720,471,459) — format it as HH:MM.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

feat(host_agent): horizontal zebra-striped fleet widget rows ... ebc67723d2

Restore the host_agent fleet-glance widget to one horizontal row per host
(name left, metric cells + sparklines right) instead of the multi-column
block grid, and zebra-stripe odd rows so adjacent hosts read as distinct.
Names still wrap rather than hard-truncate.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

fix(dashboard): inline cpu sparkline, history graph host label + stable y-axis ... 10808c1c5d

- Hosts-Overview: move the CPU sparkline inline next to the cpu % value it
  represents (was floated far right on the name line, reading as unrelated).
- Host Agent history widget: caption the chart with the host it represents
  (the panel title is generic) — links to the host hub.
- History widget: snap the y-axis to a stable 0..next-10%-band ceiling instead
  of auto-scaling to the exact peak, so the 'zoom' no longer jumps each poll.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

feat(dashboard): merge top summary strip + Status widget into one Overview ... ae03f09234

Operator chose a single canonical summary. Repurpose the status_overview
widget into 'Overview' (host count + monitor up/down/pending + Alerts link;
key kept so existing dashboards upgrade in place) and remove the redundant
fixed top strip from the dashboard view. Drops _get_summary_stats and its
now-unused PingResult/DnsResult imports (rule 22).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

feat(dashboard): threshold colors on Hosts-Overview ping + 24h uptime ... e446b7099e

Add an inverted uptime_style macro (low-is-bad mirror of metric_style) and
color the inline ping latency (warn 100ms / crit 250ms) and 24h uptime values
in the Hosts-Overview widget, which were the remaining uncolored metrics.
(Uptime/SLA + Ping widgets already colored degraded values.)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

feat(host_agent): surface network, disk I/O, and temperature in fleet widget ... e7b96fbfa7

The agent already collects + ingests net throughput, disk I/O, and temps;
add them to the fleet-glance rows (fmt_bps + two-line io cells + temp with
70/85C threshold color), each shown only when the host reports it so VMs/
containers without sensors don't show blank cells.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

feat(host_agent): in-widget 1h/6h/24h time-range toggle on history graph ... eefa38cc75

Add a live range switch to the history widget that re-requests the fragment
without entering edit mode. It rewrites the parent cell's hx-get so the choice
survives polling (htmx re-reads the attribute each poll), then fetches at once.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

feat(settings): configurable monitoring thresholds ... 591706bd39

Move the hardcoded warn/crit cutoffs into Settings -> Thresholds (DB-backed,
live, no restart). New thresholds.* keys + to_thresholds_cfg() + a
threshold_style(value, kind) jinja global that reads them; latency reuses the
existing ping good/warn keys, uptime is direction-aware (floors).

Replace the _macros metric_style/uptime_style macros (now removed) with the
global across Hosts-Overview, host_agent fleet + panel, Uptime/SLA widget, and
the ping page uptime column — all now honor the configured cutoffs. Uptime keeps
its green 'good' look when not degraded.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

feat(monitors): unify ping/dns/http into one Monitor entity + custom targets ... 35f658b573

Collapse the three former check types into a single core `Monitor` entity
with one management surface (/monitors), one result table (monitor_results),
and a single scheduled task. Every type can now watch a free-standing custom
destination (optional host_id) — not just a registered Host.

- models: Monitor + MonitorResult replace PingResult/DnsResult; Host loses its
  ping/dns facet columns (now Monitor rows linked by host_id).
- checks: monitors/{ping,dns,http}.py pure probes + runner.run_monitor
  dispatcher; one monitor_check scheduler with a per-monitor due-filter.
- status: single monitor_status_source replaces the three sources.
- UI: /monitors blueprint (type-aware add/edit/list/widget); host hub shows a
  host's linked monitors + "add monitor for this host"; nav + widget registry
  + alert metric catalog rewired. http plugin folded into core and removed.
- migration 0022 merges the http branch, data-migrates host facets +
  http_monitors + all three result histories, drops the old tables/columns.

Resolves the per-host ping/dns auto-attach issue (#275): monitors are now
explicit, never auto-added to every host.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC

feat(docker): per-host collection via the host agent; drop central scrape ... 7b80552a7d

Docker collection moves off the central single-socket scrape onto the host
agent, giving Docker a real per-host dimension. The Steward host now reports
its own containers like any other host, and same-named containers on different
hosts no longer collide.

- agent: stdlib UDS Docker client (AF_UNIX HTTP/1.1, Connection: close,
  chunked-aware), collect_docker() ports the cpu%/mem math; sample["docker"]
  added best-effort (silent-skip on absent/unreadable socket). AGENT_VERSION
  1.2.0 → 1.3.0; optional docker_socket config key.
- ingest: host_agent ingest hands per-host container snapshots to the docker
  plugin via a new "docker.persist_host_samples" capability (no hard import,
  no-op when docker disabled), inside a SAVEPOINT so a docker failure never
  sinks the host metrics. Resource names are host-scoped ("<host>/<name>").
- schema: docker_containers re-keyed (host_id, name); docker_metrics gains
  host_id; docker_002 migration DROP+recreates (dev-only, rule 122).
- ui: Docker page + widgets grouped by host with host links; new per-host
  Docker panel embedded on the Hosts hub (gated on docker enabled via a new
  enabled_plugins template context). Replaces the SQLite-only strftime
  bucketing with DB-agnostic Python bucketing.
- provisioning: install/provision playbooks add steward-agent to the docker
  group (best-effort) so the agent can read the socket.
- removed central scrape: docker scheduler.py + scraper.py deleted; plugin.yaml
  socket_path/scrape_interval_seconds/include_stopped dropped (plugin 2.0.0).
- tests: agent docker collector units (math, chunked decode, silent-skip,
  sample shape, config) + integration (host-scoped schema + persistence).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC

feat(docker): per-container enrichment — health, restarts, exit code, I/O, grouping ... 82c3d2cf36

First slice of milestone 77 (Docker monitoring depth). Surfaces real per-container
stats beyond basic state, all read-only on the existing push model.

- agent (→1.4.0): collect_docker now inspects each container (health, restart
  count, exit code, OOM) and reads net + block I/O from the stats payload; pulls
  compose project + swarm service/task/node from container labels. Per-container
  inspect+stats calls run over a small bounded ThreadPool so the ~1s-per-stats
  blocking doesn't stretch the sample on a busy host.
- schema (docker_003): additive columns on docker_containers — health, exit_code,
  oom_killed, compose_project, service_name, task_id, node_id, and BigInteger
  net/blk byte counters.
- ingest: persists the enrichment + restart_count (.get keeps older agents working).
- ui: Docker page rows now show health badge, uptime ("up 3d 4h"), restart count,
  exit code (+OOM) for stopped containers, and compose/service grouping label.
- tests: agent helpers (grouping, inspect fields, net/IO sum) + collect_docker
  assembly incl. inspect; integration asserts enrichment round-trips.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC

feat(docker): schema for lifecycle events + swarm topology ... fee654b53a

Adds the milestone-77 storage that doesn't fit on the per-container row:

  * docker_events — lifecycle (start/stop/die/oom/health_change), to be
    derived by diffing consecutive host snapshots; host-scoped, indexed for
    timeline lookups (host_id, container_name, at) and retention pruning (at).
  * docker_swarm_services / docker_swarm_nodes — manager-reported Swarm
    topology (desired-vs-running replicas, node role/availability/status).

Migration docker_004 extends the docker branch (down_revision docker_003);
purely additive, no DROP+recreate. event/mode/role are plain strings (no
CHECK whitelist), matching how docker_containers models status. Integration
guard asserts the three new host-scoped tables exist.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC

feat(docker): agent manager-only swarm collector (AGENT_VERSION 1.5.0) ... 448258c5b4

Adds collect_swarm(socket_path) to the host agent. Self-detects a Swarm
manager via /info (Swarm.ControlAvailable) — workers and non-swarm daemons
return None and never touch the manager-only endpoints (one cheap /info call,
no 503s). On a manager it queries /services, /tasks, /nodes and emits
sample["swarm"] = {services, nodes}:

  * services roll desired-vs-running replicas up from the task list (replica
    health isn't on the service object), handle replicated + global mode, strip
    the @sha256 image digest, and carry cross-node task→node placement.
  * nodes normalise role / availability / status + the manager leader flag.

build_sample omits the swarm key entirely off managers, same silent contract
as collect_docker. Unit tests cover manager detection, replica roll-up +
placement (replicated & global), node normalisation, worker silent-skip, and
build_sample wiring.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC

feat(docker): ingest swarm topology + lifecycle events + health/restart alerts ... 578cc33cc0

Wires the agent's enriched + swarm payloads through the docker.persist_host_
samples capability:

  * Swarm topology — persist sample["swarm"] into docker_swarm_services /
    docker_swarm_nodes (upsert + prune stale, host-scoped so two managers don't
    clobber). Migration docker_005 adds services.placement_json for the
    task→node placement the agent now reports.
  * Lifecycle events — _derive_events (pure, unit-tested) diffs the newest
    snapshot against stored per-container state: start / stop / die (non-zero
    exit) / oom / health_change → docker_events rows. Skipped on a host's first
    snapshot so the baseline doesn't emit a start per existing container.
  * Alerts — record restart_count (always) and is_healthy (1.0/0.0, only when a
    HEALTHCHECK exists) alongside cpu/mem, under host-scoped resource names;
    METRIC_CATALOG[docker] gains restart_count + is_healthy so they're alertable.

host_agent ingest captures the newest sample's swarm object and threads it to
the capability (now persist_host_docker(session, host, snapshots, swarm=None));
invoked when containers OR swarm are present, under the same SAVEPOINT. Unit
tests cover the event-diff matrix; integration tests cover event derivation
across two snapshots and swarm topology round-trip (incl. placement).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC

feat(docker): retention + hourly rollup for metrics/events with Settings windows ... faecac3ec6

Bounds Docker time-series growth (the main scaling concern). New
docker_metrics_hourly table + docker_006 migration; a plugin retention module
(docker.run_retention capability) rolls raw docker_metrics older than the raw
window into hourly averages (idempotent upsert), deletes the rolled raw rows,
then prunes stale rollups + lifecycle events. Core cleanup.py drives it each
hourly run via the capability (no plugin-model import), reading the three
retention windows fresh from settings so changes apply without restart (rule 25).

Settings → "Thresholds & Retention" gains a Docker retention card (raw /
rolled-up / events windows, working defaults 7/90/30 days). Unit tests cover the
hour-aligned cutoff/bucketing helpers; integration test exercises the real
rollup-average + prune across both windows.

Milestone 77 task #941.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC

feat(docker): collect + persist /system/df image/disk usage (agent 1.6.0) ... a840d6f823

Backend for the image/disk panel (milestone 77 #942). Agent gains
collect_disk_usage() — one /system/df call (gated on containers existing, so
Docker-less hosts pay nothing), surfacing reclaimable bytes (image size held by
unreferenced images), layers/containers/volumes/build-cache sizes, and the top
50 images by size. Emitted as sample["docker_disk"]; host_agent ingest tracks
the newest sample's copy and hands it to the docker capability as a 5th arg.

New current-state tables docker_disk_usage (one row/host) + docker_images
(per-host image rows), docker_007 migration; ingest upserts the summary and
replaces the image set per host (stale images pruned). Unit tests for the df
parsing/reclaimable math + build_sample gating; integration test for
persistence + image-set replacement.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC

fix(docker): close read txn before second begin in disk-usage test ... 277eb40165

test_disk_usage_persisted opened a second session.begin() after interleaved
SELECTs, which autobegin a transaction → "A transaction is already begun".
Roll back the read transaction before the re-report write. Restores the
integration lane green for the /system/df slice (a840d6f).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC

feat(docker): container detail page + lifecycle timeline (milestone 77 #942) ... 3e4e35de96

New /plugins/docker/container/<host_id>/<name> detail page (v1 quality):
status/health badge, uptime, CPU/mem, restart count, last exit code (+OOM),
net + block I/O (humanised), image/compose/swarm-service/node/ports, a
range-toggled CPU/mem history graph (HTMX fragment reusing the time-range
selector), and a lifecycle timeline rendered from docker_events (glyph+colour
per event kind). Not-found and empty-history/empty-timeline states included.
Container names in the main list and the per-host hub panel now link to it.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC

feat(docker): swarm topology view + image/disk usage page (milestone 77 #942) ... 114262dbf9

Two new read-only sub-pages, linked from the Docker index header only when the
data exists (non-swarm / Docker-less installs aren't offered empty pages):

- /plugins/docker/swarm — services with replica health (running/desired,
  colour-coded green/amber/red), Swarm nodes (role/availability/status, leader
  badge), and task→node placement with node ids resolved to hostnames. Grouped
  by reporting manager host. Empty state explains manager-only collection.

- /plugins/docker/disk — per-host reclaimable space, image/layer/container/
  volume/build-cache sizes, stopped-container count, and a per-image table
  (size, shared, ref count, reclaimable badge). Notes prune actions are deferred.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC

feat(docker): group containers by compose/swarm + enrich widget (milestone 77 #942) ... 9615f9abcd

Container list now sub-groups each host's containers by compose project (or
swarm service) with a small subheading, preserving the running-first order;
hosts with no such labels render flat as before. The dashboard status widget
links each container to its detail page and surfaces enriched state inline —
health dot (healthy/unhealthy), restart count, and last non-zero exit code for
stopped containers. Completes the #942 UI surface.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC

test(docker): parse-check templates + smoke routes/_human_bytes (milestone 77 #943) ... 626ba69934

CI never renders docker templates through the app (the unit-lane app uses
testing=True, which skips plugin loading), so a Jinja syntax error could ship
green. Add unit tests that parse every docker template, smoke-import the routes
module + confirm the #942 view functions exist, and cover the _human_bytes /
_human_uptime presentation helpers. Closes the render-regression gap the new UI
pages introduced.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC

fix(ui): unify header/breadcrumb treatment, drop redundant back buttons ... 88091936c5

The page-top chrome was inconsistent: most nested views used the breadcrumb
kicker + page-title pattern, but plugin sub-pages used ad-hoc "← back" links,
~11 pages stacked a breadcrumb AND a redundant ancestor back button, and two
(monitors/edit, hosts/uptime) had a back button but no breadcrumb. The
settings/plugin_detail page stacked all of it at once.

Unify on the breadcrumb-led model:
- settings/_tabs.html: drop the hardcoded "Settings" h1; the breadcrumb
  ("Settings › …") plus the tab strip is the header.
- settings/plugin_detail: drop the "← Plugins" back button.
- docker container_detail/swarm/disk + snmp/device: replace ad-hoc back links
  with the standard crumbs() breadcrumb.
- host_agent, ansible/*, alerts/maintenance: remove redundant ancestor back
  buttons (the breadcrumb's parent crumbs already link there); keep lateral
  shortcuts (Inventory/Schedules/Browse/Targets/Groups/New).
- monitors/edit, hosts/uptime: add the missing breadcrumb, drop the back link.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC

fix(snmp): bundle pysnmp in image and port poller to the asyncio HLAPI ... 4fc8c96c41

The SNMP plugin ships in the image but logged "pysnmp not installed — SNMP
polling disabled" on every poll, so polling never worked. Two coupled defects:

1. The Dockerfile installed only `.[ansible]`, so the `snmp` extra (pysnmp)
   was never bundled even though the plugin is first-party and shipped.
2. poller.py used the synchronous pysnmp HLAPI (`next(getCmd(...))`), which
   pysnmp-lextudio 6.x removed — it's asyncio-only now — so even with the dep
   present, polling would have thrown and silently returned nothing. The 5.x
   line that still has the sync API isn't safe on the image's Python 3.13.

Fix:
- Dockerfile: install `.[ansible,snmp]`.
- poller.py: `poll_device_sync` → `async def poll_device` on the asyncio HLAPI,
  with a dual-version import (pysnmp 7.x `pysnmp.hlapi.v3arch.asyncio`/`get_cmd`
  + async `UdpTransportTarget.create`; pysnmp-lextudio 6.2.x
  `pysnmp.hlapi.asyncio`/`getCmd` + direct `UdpTransportTarget`) so a dependency
  bump can't silently re-break it.
- scheduler.py: await poll_device directly; drop the run_in_executor wrapper
  and the now-unused asyncio import.
- Add tests/plugins/snmp/test_poller.py covering the version→mpModel mapping,
  that the poller is a coroutine, and the graceful no-pysnmp path.

Note: CI confirms import/load and the no-pysnmp path, but has no SNMP target —
live polling against real devices is verified after deploy.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC

feat(crypto): fail loudly on unpersistable secret key; flag undecryptable secrets ... 0940dc6972

Follow-up to the incident where an unwritable /data made Steward silently mint a
fresh ephemeral secret key on every boot, orphaning all encrypted secrets — only
discovered when an Ansible run failed. Make both failure modes loud and visible.

- config._resolve_secret_key: if a new key must be generated but can't be
  persisted (no STEWARD_SECRET_KEY, unwritable /data), raise RuntimeError and
  refuse to start, with an actionable fix (set STEWARD_SECRET_KEY, or make /data
  writable by uid 1000). Also raises a clear error if an existing key file can't
  be read. First-run on a writable volume still generates + persists normally.
- core.settings: detect secrets stored as ciphertext that won't decrypt with the
  current key (scan_undecryptable_secrets at startup; _is_undecryptable helper).
  Cached in memory; set_setting discards a key on a fresh write so the banner
  clears without a restart.
- app.py: run the scan after migrate_plaintext_secrets, log a warning, and inject
  undecryptable_secrets into the template context.
- base.html: admin banner naming which secrets to re-enter, each linked to its
  settings tab.
- compose.deploy.yml: document the uid-1000 /data write requirement and the
  STEWARD_SECRET_KEY option for multi-node Swarm.
- Tests: secret-key resolver (reuse existing file, generate when writable, raise
  when unpersistable) and the undecryptable-secret detection helper.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC

feat(ui): replace flat top nav with a grouped left sidebar (slice 1) ... 95ebdf7045

The flat top bar was a set of ungrouped peers and gave plugin data no home.
Move to a persistent left sidebar with grouped sections, per the navigation
redesign (operator-chosen shell).

- base.html: top <nav> → left <aside class="sidebar"> + content column.
  Groups: Overview (Dashboard, Status), Infrastructure (Hosts; plugin links
  arrive in slice 2), Monitoring (Monitors, Alerts), Automation (Ansible),
  Admin (Settings, Audit; admin-only). Brand on top, user/logout at the bottom.
- Active link highlighted by request.path prefix (aria-current).
- Responsive: sidebar slides off-canvas under 900px via a ☰ toggle + scrim
  (inline class toggle, no new JS deps). Candle-glow preserved.
- Logged-out pages (login/setup) render without the sidebar (gated on
  session.user_id), content area full-width and centered as before.
- Add tests/test_templates_parse.py: syntax-parses every steward template so a
  broken tag fails the unit lane (only the login page renders there today).

Plugin nav links (Docker/SNMP/UniFi/Traefik) come next in slice 2 via a get_nav
hook. UI-only; no behavior/route changes.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC

feat(ui): plugin get_nav() hook → sidebar Infrastructure links (slice 2) ... c8b6719b37

Plugin UIs had no nav home (only dashboard widgets / typed URLs). Add an
optional get_nav() plugin export and surface it in the sidebar.

- plugin_manager: _PLUGIN_NAV registry + get_plugin_nav() getter + a tolerant
  _collect_plugin_nav() (missing hook = fine; raising/malformed = logged &
  skipped; idempotent per plugin so hot-reload re-runs cleanly). Collected in
  both the startup load path and the hot-reload path.
- app.py: inject plugin_nav into the template context, filtered to enabled
  plugins so a hot-disabled plugin's link can't linger before restart.
- base.html: render plugin links under the Infrastructure group, with the same
  request.path active-state treatment as core links.
- docker/snmp/unifi/traefik: each exports get_nav() → its /plugins/<name>/ view.
- Tests: collector behavior (collect, missing hook, reload-replace, malformed
  item, raising hook, sorted output).

With docker enabled, "Docker" now appears under Infrastructure → its fleet view.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC

fix(docker): widen memory byte columns to BIGINT (int32 overflow on ingest) ... 2545e8c6ce

docker_metrics.mem_usage_bytes and docker_containers.mem_usage_bytes /
mem_limit_bytes were int4 (max 2,147,483,647). A container using >2.1 GB of RAM
(e.g. 8.18 GB) overflowed the column, so asyncpg raised "value out of int32
range" and the entire docker ingest batch failed — no metrics stored for any
host with a large container.

The I/O counters and the milestone-77 rollup/disk tables already used
BigInteger; this trio (docker_001-era) was missed.

- models.py: DockerMetric.mem_usage_bytes, DockerContainer.mem_usage_bytes,
  DockerContainer.mem_limit_bytes → BigInteger.
- migration docker_008_bigint_mem: ALTER COLUMN ... TYPE BIGINT (safe in-place
  int4→int8 promotion).
- integration regression test: persist an ~8 GB container, assert the
  current-state and time-series rows round-trip.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC

fix(host_agent): TB/PB byte scaling + rebalance the full-metrics layout ... 7d144780df

Two issues on the host full-metrics view:
- fmt_bytes capped at GB, so large filesystems read "25369.0 GB" instead of
  "24.8 TB". Add TB and PB tiers.
- Interfaces/Disks/Temperatures shared one 3-column grid, so all three cards
  stretched to the Temperatures height — a 40-core CPU made one very tall column
  and left Interfaces/Disks with large empty space. Split them: Interfaces +
  Disks sit side-by-side at natural height (align-items:start); Temperatures
  becomes a full-width card whose readings flow into a compact multi-column grid
  (wide-and-short instead of one tall column).

UI-only.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC

perf(metrics): index plugin_metrics for host/dashboard reads ... 3a54d6d71d

plugin_metrics had only a PK on id, so every host-detail, full-metrics, and
dashboard-widget load sequentially scanned the entire time-series table — which
grows by (sources × resources × sample cadence), so the host views got slower
over time (operator-reported "blocked/slow" loads). monitor_results was already
indexed, so the uptime aggregation wasn't the bottleneck.

Add two composite indexes matching the hot query shapes:
- (source_module, resource_name, recorded_at) — history range scans, fleet/
  widget queries, and the 'host:%' sub-resource prefix.
- (source_module, resource_name, metric_name, recorded_at) — the latest-value-
  per-metric group-by/self-join.

Model __table_args__ + core migration 0023. Integration lane validates the
migration via `alembic upgrade head`.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC

feat(hosts): rework host summary layout to use horizontal space ... 6d08db0d89

The host detail page stacked three full-width cards (Monitors, Agent, Ansible)
left-aligned, using only ~40% of the width with lots of vertical whitespace.

Rework toward the operator-chosen "vitals on top + 2-column" layout:
- The agent panel (CPU/MEM/DISK/LOAD vitals + sparklines + lifecycle) moves to
  the top, full width, so a host's vitals lead the page.
- Monitors and Ansible sit side by side in a responsive 2-column grid
  (auto-fit, stacks under ~420px), filling the width.
- The Docker per-host fragment moves below, full width.

Note: the vitals live in the host_agent fragment (loaded across the plugin
boundary) and uptime in the host-detail context, so a single literal top "strip"
mixing both isn't clean — the agent card on top is the faithful, contained
realization. UI-only.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC

perf(host_agent): aggregate metric history in SQL; DISTINCT ON for latest ... aff0c36d37

Follow-on to the plugin_metrics indexes. plugin_metrics is already retention-
bounded (core cleanup prunes > data.retention_days, default 90d) and charts top
out at 30d, so the cost wasn't growth — it was the read path shipping raw rows
to Python.

- _history_for_host: bucket + average in SQL via date_bin (epoch-aligned, ~120
  buckets) instead of fetching every raw sample (a 30d range was hundreds of
  thousands of rows) and downsampling in Python. Uses the new
  (source_module, resource_name, recorded_at) index.
- _latest_metrics_for_host: DISTINCT ON (resource_name, metric_name) ORDER BY
  recorded_at DESC — newest row per group in one index-ordered pass, replacing
  the GROUP-BY-max subquery self-joined back to the whole history.
- Integration test validates both against Postgres.

Deliberately not a materialized raw→hourly rollup: these query-side changes
deliver the speed; a rollup would additionally cut storage and remains a future
option if scale demands it.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC

refactor(host_agent): extract metric-query helpers to a model-free module ... a78af23793

The previous commit's integration test failed at collection-time import:
importing plugins.host_agent.routes (which imports the host_agent ORM models at
top level) double-registers host_agent_registrations against the app-loaded
plugin's metadata ("Table already defined").

Move the two pure read helpers (_latest_metrics_for_host, _history_for_host)
plus SOURCE_MODULE / HISTORY_METRICS into plugins/host_agent/metrics_query.py,
which imports only the core PluginMetric model — no plugin models. routes.py
imports them back. The integration test now imports from metrics_query and no
longer trips the loader's registration guard. No behavior change.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC

feat(host_agent): lazy-load full-metrics charts + live-poll current state ... 9e4f1983f8

The full-metrics page rendered once server-side with the history query inline,
so the charts blocked first paint and nothing refreshed without a reload (it
reads the latest snapshot the server holds — the agent pushes ~every 30s).

Split it into a shell + two HTMX fragments:
- Shell (host_detail.html): header + shared time-range toggle + two containers;
  paints instantly.
- Current state (/<id>/metrics → _host_metrics.html): identity + gauges +
  per-core + filesystems + interfaces/disks + temps, from the DISTINCT ON latest
  query. hx-trigger "load, every 15s" → live numbers at ~the agent cadence.
- History charts (/<id>/charts → _host_charts.html): the 3 charts + Chart.js,
  from the date_bin history query. hx-trigger "load, every 60s, rangeChange" so
  they lazy-load (never block paint), refresh slowly, and follow the range
  selector. Leak-safe: previous Chart instances are destroyed before re-render.
- Range toggle switched from full-reload links to the shared _time_range.html
  (setTimeRange + rangeChange), so only the fragments refetch.
- routes: host_detail (shell) + host_detail_metrics + host_detail_charts, with a
  _split_host_metrics helper.
- tests/test_templates_parse.py now also parses plugin templates (these
  fragments aren't rendered in the unit lane).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC

fix(host_agent): update full-metrics charts in place to kill refresh flicker ... 10dfd8ffd2

The live refresh re-created the Chart.js charts each poll, which blanked the
canvases and re-ran the grow-in animation — a jarring flash/jump every cycle.

Make the canvases persistent in the page shell and poll only the data:
- The 3 chart canvases + their Chart instances are created once in the shell,
  with animation disabled.
- /charts is now a data-only fragment swapped into a hidden div; it calls
  window.applyHostSeries(series, range), which sets each dataset's data and
  calls chart.update("none") — in-place, no re-create, no animation, fixed
  height. Range labels update via .hm-chart-range spans.
- Current-state fragment keeps its atomic innerHTML poll into fixed-height
  cards, so numbers update without a layout jump.

Net: live updates morph smoothly with no flicker or layout shift.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC

fix(docker): collapse Swarm view per-cluster so multi-manager doesn't duplicate ... 28c9a4dd2f

Swarm services/nodes are cluster-global (every manager's API returns the same
list), but each manager reports them independently (rows keyed by host_id) and
the Swarm page grouped by reporting manager — so two managers in one cluster
listed every service and node twice.

Group the reporting managers into swarms in the swarm() view (managers sharing
any node_id are the same cluster, via union-find over node-set intersection),
then dedup within each: one service per name and one node per node_id, keeping
the freshest. Render one section per swarm ("reported by N managers · names").
node_id is globally unique so node dedup is always safe; grouping by node
overlap also keeps it correct if two separate swarms are ever monitored.

View-layer only — no schema/agent change. The main per-host container page is
unaffected (those are genuinely distinct per-node tasks).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC

polish(ui): sidebar nav a11y + scroll/focus refinements (nav slice 3) ... a35c369dd4

- Only the nav list scrolls when long (min-height:0 + overflow on .side-nav);
  the brand and user/logout stay pinned top/bottom.
- Keyboard focus rings (:focus-visible) on every interactive sidebar element
  (brand, nav links, user/logout) and the mobile ☰ toggle.
- Mobile toggle is now a real toggle: aria-controls/aria-expanded kept in sync,
  Escape closes the off-canvas sidebar, scrim close stays in sync too.
- prefers-reduced-motion disables the sidebar slide + link transitions.
- Confirmed no dead top-nav CSS/markup remnants from the old top bar.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC

feat(hosts): true live vitals strip; agent panel becomes management-only ... b0d3e83bdd

Realizes the chosen host-summary layout: a thin live vitals bar at the very top,
separate from the agent management panel (no more duplicated CPU/MEM/DISK/LOAD).

- New fragment _host_vitals.html + route /plugins/host_agent/vitals/<id>: compact
  CPU / Memory / Disk(/) / Load-per-core (threshold-coloured + sparkline) +
  Pressure + live/stale·version·last-seen. Polled every 15s; renders nothing
  until the agent reports.
- The metric computation (latest snapshot + 6h sparkline query + load/core + PSI)
  moves from host_panel into host_vitals. host_panel slims to management only
  (reg/target/reporting/stale/ansible) and no longer queries metrics; panel.html
  drops the gauge row + pressure block, keeping status + lifecycle actions.
- hosts/detail.html: vitals strip on top (full width, live), then a 2-col
  [Monitors | Agent] grid, Ansible full width below, docker fragment last.

UI only. Templates parse; plugin-template parse test covers the new fragment.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC

feat(metrics): roll plugin_metrics up to hourly to bound storage ... 8af297670e

plugin_metrics grows by (sources × resources × ~30s cadence); keeping 90d of raw
is a large table. Add a raw→hourly rollup (mirroring the Docker plugin) so only a
short raw window is kept at full resolution, with hourly averages archived longer.

- PluginMetricHourly model + core migration 0024 (plugin_metrics_hourly: avg/max/
  count per source/resource/metric/hour, unique bucket constraint + lookup index).
- steward/core/metrics_retention.rollup_plugin_metrics: date_trunc('hour') agg of
  raw older than the hour-aligned raw window, idempotent pg upsert into hourly,
  delete the rolled raw, prune hourly beyond the rollup window.
- cleanup.py: plugin_metrics is no longer blanket-deleted at data.retention_days;
  _run_metrics_retention drives the rollup with windows read live from settings.
- Settings: metrics.retention.raw_days (7) + rollup_days (90), tunable on the
  Thresholds & Retention page (new "Host metrics retention" card).
- Chart read: _history_for_host merges the hourly rollup (older part of the range)
  with raw date_bin (recent part, capped ≤1h), so 30d charts keep working —
  recent at full resolution, older at hourly. Route passes raw_days from settings.
- Tests: unit (cutoff helpers) + integration (rollup aggregates/prunes; history
  merges hourly + raw) against Postgres.

Speed was already handled by the indexes + SQL aggregation; this is the storage
lever (raw window ~10x smaller).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC

feat(snmp): surface SNMP device readings on the matching host page ... 51682f130a

SNMP devices are config-defined by IP/hostname, not Steward Host records, so
they had no presence on a host's page. Map them by address and embed a fragment
(mirrors the Docker per-host fragment).

- _devices_for_host(devices_cfg, address, name): case-insensitive match of a
  device's configured host to the Steward host's address or name (tolerates
  non-dict / host-less entries).
- Route /plugins/snmp/host/<id>: renders the matched device(s) + latest readings,
  or nothing when none map (so hosts without SNMP carry no empty card).
- snmp/host_panel.html: per-device card (name · address · reachability) with a
  readings grid (K/M scaling) and a History link to the full device page.
- hosts/detail.html embeds it after the Docker fragment, gated on snmp enabled.
- Unit test for the matching helper (by address, by name, no-match, blank).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC

fix(docker): reap vanished containers; clarify the resources widget ... 0c055ed6fa

The container persister upserted current state keyed (host_id, name) but — unlike
the image/swarm/disk persisters — never deleted containers that vanished from the
host's latest listing. Every removed one-shot container (CI job runners, buildkit
builders, codex jobs) left a permanent "stopped" row, so the dashboard counts
ballooned (e.g. 856 stopped) and read as "not dedup'd". It wasn't dedup — it was
a missing reaper.

- ingest.py: after the upsert loop, delete this host's containers whose name is
  notin the newest snapshot (the listing is authoritative — event derivation
  already treats absence as removal). Mirrors images/swarm/disk. The existing
  `if not snapshots: return` keeps swarm/disk-only samples from touching
  containers; an empty container snapshot legitimately means none exist.
- widget_resources.html: this widget shows the busiest running containers by CPU.
  Replaced the two unlabeled hairline bars with labeled CPU/MEM bars (label · bar ·
  % value, warn/crit colors) so it's self-explanatory.
- Regression test: a container present then absent across snapshots is reaped.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC

fix(docker): dedup widget counts across swarm managers; show failures not stopped ... e289b6c49f

The running count was inflated: swarm-aware agents on every manager list
cluster-wide tasks, so the same container (identical container_id) was reported
once per manager and the dashboard widgets summed them. The earlier swarm dedup
only covered the Swarm topology page — the widgets read raw rows.

- _dedup_by_container_id(): collapse rows sharing a non-empty container_id
  (globally unique, so only true duplicates merge); first occurrence wins under
  the running-first ordering. Applied in both the containers and resources
  widgets before counting/limiting. Rows without a container_id (older agents)
  are kept as-is.
- Replaced the static "stopped" tally — not actionable — with "failed (24h)":
  distinct containers with a die/oom event in the last 24h (DISTINCT so the
  managers' duplicate events don't double it), rendered red when non-zero.
- widget.html empty-state now keys off total_count.
- Regression test: same container_id from two managers counts once; id-less rows
  all kept.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC

fix(docker): move dedup helper to a model-free module (CI green) ... 431f804037

The widget-dedup test imported plugins.docker.routes, which re-imports
plugins.docker.models after the app already loaded the docker plugin —
"Table 'docker_containers' is already defined". Same plugin-loader gotcha the
host_agent query helpers avoid.

Extract dedup_by_container_id into plugins/docker/dedup.py (imports no ORM
models, so it's safe to import in either environment); routes.py imports it
from there and the test targets the model-free module.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC

feat(docker): swarm-aware container views — collapse replicas, show host, surface agent-less tasks ... 24df8458ea

The container list and dashboard widget listed each swarm task as its own cryptic
row (svc.1.<taskid>) and could only show tasks on nodes that run a Steward agent.
Make both views service-centric and cluster-complete.

- swarm_view.build_swarm_services(): model-free builder that merges real container
  rows (collected local-per-node, so host_id = the node a task runs on) with the
  managers' placement (DockerSwarmService.placement_json). Where placement counts
  exceed the local rows on a node — i.e. nodes with no agent — it synthesizes
  "ghost" replicas so the replica list matches the cluster. Non-swarm containers
  pass through grouped by host.
- /rows + rows.html: a Swarm-services panel collapses each service to one block,
  every replica a host chip (status + cpu); ghosts render dashed "N · no agent".
  Non-swarm/compose containers keep the per-host table below. New Services stat.
- /widget + widget.html: same service collapse with host chips; standalone
  containers stay grouped by host.
- Unit tests for the builder (real+ghost merge, full coverage, partial agent-less,
  synthesized service, down state).

No migration — node_id, placement_json and swarm-node hostnames are already stored.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC

feat(snmp): explicit host binding (host_id / steward_host) with implicit fallback ... b1aabb7dfa

SNMP devices map onto a Steward host's detail page via _devices_for_host,
which previously matched the device's `host` field (the SNMP poll target)
against the Host's address/name — a coincidence of strings that breaks when
the poll target differs from how the Host is recorded.

Add two optional per-device config fields that decouple the binding from the
poll target:
  • host_id      — exact Steward Host UUID match (the explicit link)
  • steward_host — friendly bind by Host name or address (case-insensitive)

An explicit binding is exclusive: a device bound to host A never implicitly
matches host B by a coincidental address. No explicit binding → existing
implicit `host`-string match (fully backward compatible).

host_panel passes host.id and badges explicitly-bound devices. plugin.yaml
documents the new fields. Tests cover explicit id/name, exclusivity, wrong
uuid, and legacy fallback.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC

fix(snmp): close SnmpEngine after each poll to stop fd leak ... 2df5fc94a3

A fresh SnmpEngine() was created on every poll_device() call and never
closed. pysnmp opens a UDP transport socket per engine and doesn't release
it on GC, so each scheduler tick (default 60s, per device) leaked a file
descriptor. Over hours of polling the process hit its fd ceiling and the
listening socket could no longer accept connections — OSError: [Errno 24]
Too many open files on socket.accept(), locking up the app.

Wrap the engine in try/finally and release its transport socket via a new
_close_engine() helper that probes both pysnmp API shapes (6.2.x lextudio
camelCase, canonical 7.x snake_case); all close paths are best-effort so a
failed close never breaks the poll loop. Regression tests cover both shapes
and the never-raises contract.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

fix(unifi): close the cached client before discarding it (fd leak) ... 0c5a1573da

The UniFi poller caches a UnifiClient holding a long-lived httpx.AsyncClient
(a real socket pool). On a login failure or a mid-poll error it set
`_client = None` to force re-auth, but never called close() — orphaning the
connection pool until GC. Under a flapping/unreachable controller that leaks
a file descriptor every failure tick: the same class of bug as the SNMP
engine leak (OSError: [Errno 24] Too many open files).

Add _drop_client() that aclose()s the client (best-effort) before nulling
the cache, and route both discard paths through it. Regression tests cover
both failure paths plus the noop/close-error contracts.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

feat(reliability): fd-leak rails — self-watchdog, poll-overlap guard, fd tests ... f40063a74d

Guardrails so the fd-leak class of bug (Errno 24 lockup; recent SNMP + UniFi
fixes) surfaces early or can't compound, instead of silently killing the app.

- Self-fd watchdog (steward/core/self_monitor.py): records open_fds and
  open_fds_pct (% of soft RLIMIT_NOFILE) as "steward"/"process" metrics each
  minute through the normal alert pipeline, so the operator can alert on them
  via the existing alert-rules UI. Built-in WARNING floor at 80% gives a
  zero-config early signal. Stdlib-only (/proc + resource); degrades to a no-op
  off Linux. Registered as a core ScheduledTask in app.py.

- Poll-overlap guard (steward/core/scheduler.py): extract a pure _DueTracker
  that skips a tick while a task's prior run is still in flight, so a hung poll
  can't stack overlapping runs (which amplify per-poll resource/fd use). A
  skipped task isn't penalised — it retries the next tick after it completes.

- fd-stability tests (tests/core/): _DueTracker overlap policy, the watchdog
  metric/warning/degradation paths, and a real-fd canary that hammers tcp_check
  and asserts /proc/self/fd doesn't grow.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

feat(ansible): structured playbook-variable fields on the schedule form ... efc11c1837

The Ansible Schedules form only offered a free-form extra-vars textarea,
unlike the browse/host run forms which auto-populate a field per declared
playbook variable (vars:/vars_prompt:) plus the playbook description. Wire
the schedule form to the same shared `_playbook_vars.html` partial:

- Playbook <select> now loads /ansible/playbook-vars?schedule=1 on change
  into a #sch-playbook-vars container (mirrors the host detail form).
- Edit mode pre-renders the saved playbook's variables server-side with
  their stored values; the extra-vars textarea keeps only leftover
  (non-declared) vars.
- Partial gains optional `values` (prefill) and `schedule` flags. In the
  schedule context secret vars render disabled ("secrets can't be
  scheduled") since they're dropped downstream, and the destructive-confirm
  gate is shown but not required.

Backend already parsed var__* fields for schedules, so this is the
front-end/partial parity fix.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

feat(ansible): default source/playbook dropdowns to the first item ... 524fd1f509

The schedule and host run forms opened their Source/Playbook <select>s on a
"— choose … —" placeholder, forcing an extra click. Default them to the first
item and cascade the dependent fields so the initial state is real:

- Remove the placeholder options (schedules source+playbook, host source+
  playbook, and the shared _playbook_options fragment).
- schedules(): pass sel_source (first source, or the edited schedule's) and
  that source's pre-rendered playbooks so the playbook dropdown starts populated
  (create mode too, no longer the all-sources union).
- Cascade on load/change: the source <select> fires playbook-options then
  (hx-on::after-settle) re-triggers the playbook <select>, which loads that
  playbook's variable fields. Host form drives the whole chain from a load
  trigger; schedule create mode loads vars via the playbook <select>'s own load
  trigger, while edit mode keeps its server-rendered prefill untouched.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

bvandeusen merged commit 001cbafe56 into main 2026-06-30 23:44:04 -04:00

Sign in to join this conversation.

Reviewers

No Reviewers

Labels

No items

No Label

Milestone

No items

No Milestone

Projects

Clear projects

No project

Assignees

Clear assignees

bvandeusen

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: bvandeusen/FabledSteward#2