Guardrails so the fd-leak class of bug (Errno 24 lockup; recent SNMP + UniFi
fixes) surfaces early or can't compound, instead of silently killing the app.
- Self-fd watchdog (steward/core/self_monitor.py): records open_fds and
open_fds_pct (% of soft RLIMIT_NOFILE) as "steward"/"process" metrics each
minute through the normal alert pipeline, so the operator can alert on them
via the existing alert-rules UI. Built-in WARNING floor at 80% gives a
zero-config early signal. Stdlib-only (/proc + resource); degrades to a no-op
off Linux. Registered as a core ScheduledTask in app.py.
- Poll-overlap guard (steward/core/scheduler.py): extract a pure _DueTracker
that skips a tick while a task's prior run is still in flight, so a hung poll
can't stack overlapping runs (which amplify per-poll resource/fd use). A
skipped task isn't penalised — it retries the next tick after it completes.
- fd-stability tests (tests/core/): _DueTracker overlap policy, the watchdog
metric/warning/degradation paths, and a real-fd canary that hammers tcp_check
and asserts /proc/self/fd doesn't grow.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
plugin_metrics grows by (sources × resources × ~30s cadence); keeping 90d of raw
is a large table. Add a raw→hourly rollup (mirroring the Docker plugin) so only a
short raw window is kept at full resolution, with hourly averages archived longer.
- PluginMetricHourly model + core migration 0024 (plugin_metrics_hourly: avg/max/
count per source/resource/metric/hour, unique bucket constraint + lookup index).
- steward/core/metrics_retention.rollup_plugin_metrics: date_trunc('hour') agg of
raw older than the hour-aligned raw window, idempotent pg upsert into hourly,
delete the rolled raw, prune hourly beyond the rollup window.
- cleanup.py: plugin_metrics is no longer blanket-deleted at data.retention_days;
_run_metrics_retention drives the rollup with windows read live from settings.
- Settings: metrics.retention.raw_days (7) + rollup_days (90), tunable on the
Thresholds & Retention page (new "Host metrics retention" card).
- Chart read: _history_for_host merges the hourly rollup (older part of the range)
with raw date_bin (recent part, capped ≤1h), so 30d charts keep working —
recent at full resolution, older at hourly. Route passes raw_days from settings.
- Tests: unit (cutoff helpers) + integration (rollup aggregates/prunes; history
merges hourly + raw) against Postgres.
Speed was already handled by the indexes + SQL aggregation; this is the storage
lever (raw window ~10x smaller).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC
Plugin UIs had no nav home (only dashboard widgets / typed URLs). Add an
optional get_nav() plugin export and surface it in the sidebar.
- plugin_manager: _PLUGIN_NAV registry + get_plugin_nav() getter + a tolerant
_collect_plugin_nav() (missing hook = fine; raising/malformed = logged &
skipped; idempotent per plugin so hot-reload re-runs cleanly). Collected in
both the startup load path and the hot-reload path.
- app.py: inject plugin_nav into the template context, filtered to enabled
plugins so a hot-disabled plugin's link can't linger before restart.
- base.html: render plugin links under the Infrastructure group, with the same
request.path active-state treatment as core links.
- docker/snmp/unifi/traefik: each exports get_nav() → its /plugins/<name>/ view.
- Tests: collector behavior (collect, missing hook, reload-replace, malformed
item, raising hook, sorted output).
With docker enabled, "Docker" now appears under Infrastructure → its fleet view.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC
Follow-up to the incident where an unwritable /data made Steward silently mint a
fresh ephemeral secret key on every boot, orphaning all encrypted secrets — only
discovered when an Ansible run failed. Make both failure modes loud and visible.
- config._resolve_secret_key: if a new key must be generated but can't be
persisted (no STEWARD_SECRET_KEY, unwritable /data), raise RuntimeError and
refuse to start, with an actionable fix (set STEWARD_SECRET_KEY, or make /data
writable by uid 1000). Also raises a clear error if an existing key file can't
be read. First-run on a writable volume still generates + persists normally.
- core.settings: detect secrets stored as ciphertext that won't decrypt with the
current key (scan_undecryptable_secrets at startup; _is_undecryptable helper).
Cached in memory; set_setting discards a key on a fresh write so the banner
clears without a restart.
- app.py: run the scan after migrate_plaintext_secrets, log a warning, and inject
undecryptable_secrets into the template context.
- base.html: admin banner naming which secrets to re-enter, each linked to its
settings tab.
- compose.deploy.yml: document the uid-1000 /data write requirement and the
STEWARD_SECRET_KEY option for multi-node Swarm.
- Tests: secret-key resolver (reuse existing file, generate when writable, raise
when unpersistable) and the undecryptable-secret detection helper.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC
Collapse the three former check types into a single core `Monitor` entity
with one management surface (/monitors), one result table (monitor_results),
and a single scheduled task. Every type can now watch a free-standing custom
destination (optional host_id) — not just a registered Host.
- models: Monitor + MonitorResult replace PingResult/DnsResult; Host loses its
ping/dns facet columns (now Monitor rows linked by host_id).
- checks: monitors/{ping,dns,http}.py pure probes + runner.run_monitor
dispatcher; one monitor_check scheduler with a per-monitor due-filter.
- status: single monitor_status_source replaces the three sources.
- UI: /monitors blueprint (type-aware add/edit/list/widget); host hub shows a
host's linked monitors + "add monitor for this host"; nav + widget registry
+ alert metric catalog rewired. http plugin folded into core and removed.
- migration 0022 merges the http branch, data-migrates host facets +
http_monitors + all three result histories, drops the old tables/columns.
Resolves the per-host ping/dns auto-attach issue (#275): monitors are now
explicit, never auto-added to every host.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC
generate_inventory() emits Ansible's dynamic --list shape (all.hosts is a
LIST, vars under _meta) — valid only as an executable inventory script's
stdout. We were writing it to a static file and passing -i, so Ansible's yaml
plugin rejected it ("Invalid 'hosts' entry for 'all' group, requires a
dictionary, found ...list...") and fell back to implicit localhost → "no hosts
matched". Affected every steward:* scope run; surfaced on the first real
provision.
- New inventory_to_yaml(inv): convert the --list dict → a valid static YAML
inventory (all.hosts dict keyed by host, groups under all.children, group
vars preserved, injected per-host vars like steward_token retained).
- Wire it into runner.trigger_run, host_agent deploy + provision.
- executor writes the file as inventory.yml so the yaml plugin's extension
check reliably claims it.
- generate_inventory unchanged (still the --list dict); conversion happens at
write time. Unit tests added.
Scribe issue #885.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Turn the agent-install playbook into a full provisioning + maintenance
path. Solves the bootstrap chicken-and-egg: first contact uses an
operator-supplied password (one run, never stored), which creates a
dedicated `steward` login account with NOPASSWD sudo + Steward's managed
public key. Every run thereafter connects as `steward` with the managed
key — fully unattended (scheduled prune, agent updates).
- core/crypto: generate_ssh_keypair() — ed25519, OpenSSH formats.
- settings: ansible.ssh_public_key (non-secret, displayed) + ansible.ssh_user
(default steward); to_ansible_cfg extended.
- settings UI + route: "Generate managed key" (private encrypted, public
shown to copy) + SSH-user field.
- executor: build_bootstrap() writes a 0600 vars file (-e @file) for the
per-run user/password — never argv, never DB, never logged; drops the
managed key when a bootstrap password is given; --user floor from the
global ssh_user when no override.
- runner.trigger_run: pass-through `connection` kwarg, deliberately NOT
persisted on AnsibleRun.params (password stays out of the DB).
- bundled/host_agent/provision.yml: create steward user + authorized_keys
+ /etc/sudoers.d/steward (visudo-validated) + agent install.
- host_agent: /provision route + "Provision a fresh host" card (bootstrap
user/password; injects pubkey + user + token as space-safe JSON hostvars).
- Dockerfile: add sshpass (Ansible shells out to it for password SSH).
- tests: keypair generation + build_bootstrap (secret stays off argv).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Closes#550 (all four):
- Cancellation: track live subprocesses; POST /ansible/runs/<id>/cancel
(operator) SIGTERMs then SIGKILLs after a grace; new 'cancelled' status
(+ migration 0019, ALTER TYPE in autocommit). Queued runs cancel cleanly
before launch. Cancel button on run detail.
- Concurrency: global semaphore (ansible.max_concurrent_runs, default 3,
Settings→Ansible) caps simultaneous runs; excess show 'queued' (new status)
until a slot frees. Semaphore bound lazily per running loop.
- Structured results: parse PLAY RECAP into per-host ok/changed/unreachable/
failed/skipped + capture failed-task lines, stored in new results JSON
column (migration 0020); rendered as a host-summary table on run detail.
Keeps live streaming (no json-callback swap).
- Retention: full output written to a persistent log artifact
(/data/ansible/runs/<id>.log, env-overridable) beyond the 1 MB DB cap and
across restarts; in-memory replay buffer bounded + GC'd after completion;
Download-log route. Boot reconciliation now also sweeps stale 'queued'.
Unit tests for recap parsing + cancel flagging. Status colors updated across
run list / detail / schedules.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds create/edit/delete of playbooks from the UI (admin only), so a homelab
user without a git workflow can author automation in-app. A new always-present
writable local source "steward-local" (/data/ansible/playbooks, env-overridable,
created on first save) is editable alongside operator local-dir sources; the
bundled and git sources stay read-only (git is GitOps, clobbered on pull).
sources.py: write_playbook / delete_playbook (traversal-guarded, .yml/.yaml
only) + validate_playbook_yaml (YAML + play-list check) + is_editable_source.
routes.py: /playbooks/new, /edit, /save, /delete (admin). Browse gains a
"New playbook" button and per-playbook + view-page Edit/Delete for editable
sources. Plain textarea editor with save-time YAML validation. Unit tests for
write/delete/guard/validate.
Task #579 — completes milestone #37 (Ansible automation).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Implements #253's framework: a small core capability registry
(steward/core/capabilities.py) where a module/plugin publishes a named,
role-gated action and a consumer discovers it via has_capability() and runs it
via invoke_capability() — no hard import, graceful degradation, permission
propagation (actor role checked against the capability's required_role).
Core publishes "ansible.run_playbook" (operator) wrapping ansible.runner.
trigger_run (extended to accept a caller-built inventory). First consumer: the
host_agent plugin gains "Deploy via Ansible" on its settings page — pick an
inventory target/group and it installs/updates the agent via the bundled
host_agent/install.yml, minting a fresh token per host and injecting it as an
inventory hostvar (turning per-host curl|sh into one run). Exposed role
ordering as middleware.role_meets. Unit tests for the registry + role checks.
Task #253 (milestone #37).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Ships a read-only "steward-builtin" source (steward/ansible/bundled/) that
always appears alongside operator-configured sources, with two playbooks:
- maintenance/docker_prune.yml — docker system prune for swarm/standalone
nodes (safe by default; prune_all_images / prune_volumes extra-vars to
widen). Schedule it against a swarm-node group for recurring cleanup (#869).
- host_agent/install.yml — installs/updates the host agent (mirrors
install.sh: user, agent.py, config, hardened systemd unit), parameterised
with steward_url + steward_token extra-vars.
get_sources() now prepends the builtin source. Tests updated to find the
configured git source by name; added coverage that the bundled playbooks are
discoverable.
Tasks #869 + agent-install (milestone #37).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds a Kuma-style "is everything up?" surface that aggregates heterogeneous
monitor types via a status-source registry (steward/core/status.py): each
type registers an async source(db) -> [StatusEntry]. Core registers ping/DNS;
the http plugin registers its own from setup() so core never imports plugin
tables. Per entry: current up/down, last-30 heartbeat bar, uptime %
(24h/7d/30d), latest latency + response sparkline, and TLS expiry countdown
(HTTP). New /status page (live htmx refresh) + a status_overview dashboard
widget + nav link. Pure-function unit tests for registry + sparkline.
First deliverable of milestone #68 (task #866).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A fresh install enabled zero plugins — settings.py DEFAULTS had no plugin.*
keys, so to_plugins_cfg returned {} and every plugin had to be flipped on
by hand. Seed docker, host_agent, http, snmp as default-on (generic, non-
vendor-specific); traefik and unifi stay opt-in. Stored choices override
the default, so disabling persists.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Renames the Python package directory, CLI command, env var prefix,
docker-compose service/container/image, Postgres role/db, and all
visible branding. Marketing form is "Fabled Steward".
Clean break from the previous rebrand: drops the fabledscryer→roundtable
import shim in __init__.py and the FABLEDSCRYER_* env var fallback in
config.py and migrations/env.py. Env vars are now STEWARD_* only.
Heads-up for existing deployments:
- Postgres user/db renamed fabledscryer → steward in docker-compose.yml.
Existing volumes need the role/db renamed inside Postgres, or override
POSTGRES_USER/POSTGRES_DB to keep the old names.
- Host-agent systemd unit is now steward-agent.service. Existing agents
keep running under the old name; reinstall to switch.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
bvandeusen