Operator saw the `http` plugin -- deleted when ping/dns/http were unified into
the Monitor entity -- still reported as enabled, with no way to clear it.
It was never an orphaned row. `plugin.http` was hardcoded in core DEFAULTS, so
deleting the app_settings row did nothing: the default reasserted it on the
next settings load. That is precisely why no cleanup UI could have fixed it.
Rule 22 says the removed subsystem should have taken its setting with it.
Purged the surviving references -- the DEFAULTS key, "http" in
CAPABILITY_PLUGINS, the stale plugin_manager docstring example, and the
capabilities blurb still advertising HTTP/uptime as a bundled capability --
plus a migration dropping any stored plugin.http row. Untouched: `http` as a
MONITOR TYPE (icmp/tcp/dns/http) everywhere it appears, and http_001_initial,
which is kept deliberately so existing DBs resolve the revision graph.
For the general case, cleanup splits by provenance rather than being uniformly
automatic or uniformly manual:
Bundled plugins ship in the image and version atomically with core, so they
cannot be transiently missing -- a plugin.* default with no bundled directory
is unambiguously a bug. Guarded by a unit test that fails CI, which is the
only part safe to automate.
External plugins live in operator-mounted /data/plugins, where absence is
ambiguous: unmounted volume, failed install, mid-upgrade. Auto-deleting their
config would silently destroy unrecoverable credentials on a transient
condition, so Settings now lists them under "Configured but not installed"
with an explicit, confirmed, audit-logged Remove. The section hides entirely
when empty, and the remove route refuses if the plugin is actually installed.
Orphan detection reads STORED rows, never the DEFAULTS-merged view -- offering
to remove a default-only key would be a lie -- and returns names only, since
plugin config can hold credentials and listing orphans never needs their values.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Bound docker_logs growth in the periodic cleanup task (same architecture as the
metrics/events retention). run_docker_retention gains logs_retention_days +
logs_max_bytes_per_container: it prunes lines past the age window, then keeps
only the newest ~cap bytes per (host, container) via a window-function ring
(exclusive-prefix sum, so the newest line always survives even if it alone
exceeds the cap). Containers rotate independently.
- settings DEFAULTS: docker.logs.enabled/exclude/retention_days(3)/
max_bytes_per_container(5MB) — operator preference: ~3 days / ~5 MB
- cleanup.py reads the two windows fresh each run (rule 25, no restart)
- integration rotate test: age prune + per-container byte cap + isolation
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CAGR73DUowdVFVvYzLXC5C
plugin_metrics grows by (sources × resources × ~30s cadence); keeping 90d of raw
is a large table. Add a raw→hourly rollup (mirroring the Docker plugin) so only a
short raw window is kept at full resolution, with hourly averages archived longer.
- PluginMetricHourly model + core migration 0024 (plugin_metrics_hourly: avg/max/
count per source/resource/metric/hour, unique bucket constraint + lookup index).
- steward/core/metrics_retention.rollup_plugin_metrics: date_trunc('hour') agg of
raw older than the hour-aligned raw window, idempotent pg upsert into hourly,
delete the rolled raw, prune hourly beyond the rollup window.
- cleanup.py: plugin_metrics is no longer blanket-deleted at data.retention_days;
_run_metrics_retention drives the rollup with windows read live from settings.
- Settings: metrics.retention.raw_days (7) + rollup_days (90), tunable on the
Thresholds & Retention page (new "Host metrics retention" card).
- Chart read: _history_for_host merges the hourly rollup (older part of the range)
with raw date_bin (recent part, capped ≤1h), so 30d charts keep working —
recent at full resolution, older at hourly. Route passes raw_days from settings.
- Tests: unit (cutoff helpers) + integration (rollup aggregates/prunes; history
merges hourly + raw) against Postgres.
Speed was already handled by the indexes + SQL aggregation; this is the storage
lever (raw window ~10x smaller).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC
Follow-up to the incident where an unwritable /data made Steward silently mint a
fresh ephemeral secret key on every boot, orphaning all encrypted secrets — only
discovered when an Ansible run failed. Make both failure modes loud and visible.
- config._resolve_secret_key: if a new key must be generated but can't be
persisted (no STEWARD_SECRET_KEY, unwritable /data), raise RuntimeError and
refuse to start, with an actionable fix (set STEWARD_SECRET_KEY, or make /data
writable by uid 1000). Also raises a clear error if an existing key file can't
be read. First-run on a writable volume still generates + persists normally.
- core.settings: detect secrets stored as ciphertext that won't decrypt with the
current key (scan_undecryptable_secrets at startup; _is_undecryptable helper).
Cached in memory; set_setting discards a key on a fresh write so the banner
clears without a restart.
- app.py: run the scan after migrate_plaintext_secrets, log a warning, and inject
undecryptable_secrets into the template context.
- base.html: admin banner naming which secrets to re-enter, each linked to its
settings tab.
- compose.deploy.yml: document the uid-1000 /data write requirement and the
STEWARD_SECRET_KEY option for multi-node Swarm.
- Tests: secret-key resolver (reuse existing file, generate when writable, raise
when unpersistable) and the undecryptable-secret detection helper.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC
Bounds Docker time-series growth (the main scaling concern). New
docker_metrics_hourly table + docker_006 migration; a plugin retention module
(docker.run_retention capability) rolls raw docker_metrics older than the raw
window into hourly averages (idempotent upsert), deletes the rolled raw rows,
then prunes stale rollups + lifecycle events. Core cleanup.py drives it each
hourly run via the capability (no plugin-model import), reading the three
retention windows fresh from settings so changes apply without restart (rule 25).
Settings → "Thresholds & Retention" gains a Docker retention card (raw /
rolled-up / events windows, working defaults 7/90/30 days). Unit tests cover the
hour-aligned cutoff/bucketing helpers; integration test exercises the real
rollup-average + prune across both windows.
Milestone 77 task #941.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Jg27rgypiW2efULXJDtMC
Move the hardcoded warn/crit cutoffs into Settings -> Thresholds (DB-backed,
live, no restart). New thresholds.* keys + to_thresholds_cfg() + a
threshold_style(value, kind) jinja global that reads them; latency reuses the
existing ping good/warn keys, uptime is direction-aware (floors).
Replace the _macros metric_style/uptime_style macros (now removed) with the
global across Hosts-Overview, host_agent fleet + panel, Uptime/SLA widget, and
the ping page uptime column — all now honor the configured cutoffs. Uptime keeps
its green 'good' look when not degraded.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The "could not decrypt a stored secret" warning was generic, so an operator
couldn't tell which of the six secret settings was encrypted under an old key.
Thread the setting key through _decode → decrypt_secret(context=...) so the log
now reads e.g. "Could not decrypt stored secret smtp.password (wrong/rotated
key — re-enter it)". Pure diagnostic; decrypt behaviour unchanged.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Turn the agent-install playbook into a full provisioning + maintenance
path. Solves the bootstrap chicken-and-egg: first contact uses an
operator-supplied password (one run, never stored), which creates a
dedicated `steward` login account with NOPASSWD sudo + Steward's managed
public key. Every run thereafter connects as `steward` with the managed
key — fully unattended (scheduled prune, agent updates).
- core/crypto: generate_ssh_keypair() — ed25519, OpenSSH formats.
- settings: ansible.ssh_public_key (non-secret, displayed) + ansible.ssh_user
(default steward); to_ansible_cfg extended.
- settings UI + route: "Generate managed key" (private encrypted, public
shown to copy) + SSH-user field.
- executor: build_bootstrap() writes a 0600 vars file (-e @file) for the
per-run user/password — never argv, never DB, never logged; drops the
managed key when a bootstrap password is given; --user floor from the
global ssh_user when no override.
- runner.trigger_run: pass-through `connection` kwarg, deliberately NOT
persisted on AnsibleRun.params (password stays out of the DB).
- bundled/host_agent/provision.yml: create steward user + authorized_keys
+ /etc/sudoers.d/steward (visudo-validated) + agent install.
- host_agent: /provision route + "Provision a fresh host" card (bootstrap
user/password; injects pubkey + user + token as space-safe JSON hostvars).
- Dockerfile: add sshpass (Ansible shells out to it for password SSH).
- tests: keypair generation + build_bootstrap (secret stays off argv).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Closes#550 (all four):
- Cancellation: track live subprocesses; POST /ansible/runs/<id>/cancel
(operator) SIGTERMs then SIGKILLs after a grace; new 'cancelled' status
(+ migration 0019, ALTER TYPE in autocommit). Queued runs cancel cleanly
before launch. Cancel button on run detail.
- Concurrency: global semaphore (ansible.max_concurrent_runs, default 3,
Settings→Ansible) caps simultaneous runs; excess show 'queued' (new status)
until a slot frees. Semaphore bound lazily per running loop.
- Structured results: parse PLAY RECAP into per-host ok/changed/unreachable/
failed/skipped + capture failed-task lines, stored in new results JSON
column (migration 0020); rendered as a host-summary table on run detail.
Keeps live streaming (no json-callback swap).
- Retention: full output written to a persistent log artifact
(/data/ansible/runs/<id>.log, env-overridable) beyond the 1 MB DB cap and
across restarts; in-memory replay buffer bounded + GC'd after completion;
Download-log route. Boot reconciliation now also sweeps stale 'queued'.
Unit tests for recap parsing + cancel flagging. Status colors updated across
run list / detail / schedules.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A fresh install enabled zero plugins — settings.py DEFAULTS had no plugin.*
keys, so to_plugins_cfg returned {} and every plugin had to be flipped on
by hand. Seed docker, host_agent, http, snmp as default-on (generic, non-
vendor-specific); traefik and unifi stay opt-in. Stored choices override
the default, so disabling persists.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Global Ansible credentials, applied to every run (manual + alert-triggered):
- core/settings.py: ansible.ssh_private_key / become_password / vault_password
(plaintext at rest, masked in UI — encryption tracked in #580) + host_key_checking
(default off); surfaced via to_ansible_cfg into app.config[ANSIBLE]
- executor.py: pure build_credentials() materializes creds into a 0600 temp dir
(--private-key, --vault-password-file, become via -e @vars-file so the password
never hits argv) cleaned up in finally; pure ansible_env() sets
ANSIBLE_HOST_KEY_CHECKING. build_ansible_command stays param-only
- settings/routes.py + ansible.html: admin-only Credentials section, masked-update
(blank keeps current, explicit Clear checkbox), reload app config on save
- tests: unit (build_credentials per cred + none; ansible_env toggle); integration
vault round-trip (ansible-vault encrypt a vars file, run via executor with the
vault password, assert it decrypted)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Renames the Python package directory, CLI command, env var prefix,
docker-compose service/container/image, Postgres role/db, and all
visible branding. Marketing form is "Fabled Steward".
Clean break from the previous rebrand: drops the fabledscryer→roundtable
import shim in __init__.py and the FABLEDSCRYER_* env var fallback in
config.py and migrations/env.py. Env vars are now STEWARD_* only.
Heads-up for existing deployments:
- Postgres user/db renamed fabledscryer → steward in docker-compose.yml.
Existing volumes need the role/db renamed inside Postgres, or override
POSTGRES_USER/POSTGRES_DB to keep the old names.
- Host-agent systemd unit is now steward-agent.service. Existing agents
keep running under the old name; reinstall to switch.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
bvandeusenandClaude Opus 5