#5212 "Plain labels: no raw ids, scopes, metric keys or unexplained jargon",
batch 2:
- A run records where it came from: AnsibleRun.origin ("schedule", "alert"
or "mcp") and origin_id, set by the scheduler, Run now, alert actions and
the MCP tools. Migration 0031 backfills past runs from alert events,
schedules' last runs, and MCP audit entries.
- The runs list has a Started by column: "schedule nightly", "alert rule
nas down", "alice via MCP" or "alice", linked where there is a page.
The raw scope tooltip is gone.
- Run detail: no run id in the subtitle or body, Started by in words, and
the host summary lists failed and unreachable hosts first. The live
output only follows new lines while the reader is at the bottom, and
when the run finishes the page reloads once so the badge and summary
are final.
- list_ansible_runs and get_ansible_run report origin and origin_id.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The dashboard's host widgets took 18-60 s. Every "latest reading" query
(hosts overview, host-agent fleet, host page, vitals, SNMP, MCP get_host)
searched plugin_metrics for the newest row per metric. Postgres has no skip
scan, so each one read every raw sample in the 7-day window.
- plugin_metric_latest: one row per (source, resource, metric).
record_metrics upserts it in the same transaction, moving forward only and
in sorted key order. Migration 0030 seeds it by stepping the existing index.
- Retention drops a latest row once its sample leaves the raw window, so a
resource that stopped reporting disappears when its history does, as before.
Renames carry it along.
- A quiet host keeps its last values, marked stale. A time-bounded query
would have shown it as missing.
- Vitals sparklines span the 6 h they claim, averaged in SQL to ~120 points,
instead of fetching 6 h raw and drawing the last 120 samples.
- Dashboard and share-view cells: "Loading…" until the first answer; a 20 s
deadline per fetch; "Couldn't load (why)" with Retry now and an automatic
retry for cells that don't poll; "as of HH:MM" in the header, and
"couldn't refresh · as of" when a refresh fails over shown data.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- One run per host: start_run claims every host it touches (addresses read
from the inventory Steward wrote; a source's own inventory file is claimed
as that file) before it takes a run slot. The claim is all-or-nothing. A run
waiting on a host shows queued and names the run it waits for, honours a
cancel, and fails after HOST_WAIT_SECONDS (1 h) with which host and run.
- A due schedule whose previous run is still queued or running skips the
firing: last_run_at moves on, last_run_id still names the run going, and
last_error says "Skipped: the previous run was still going". The Schedules
page shows that beside the run (muted), and a launch error in red.
- An alert rule that fires while the action run from an earlier firing is
still going starts no new run; the event says so and names the run. An
in-process set closes the gap between the check and the run being linked.
- docs/core/ansible.md: queued/cancelled statuses and the overlap rules.
Tests: tests/core/test_ansible_overlap.py (claim keys, waiting, all-or-nothing,
cancel, deadline) and tests/integration/test_ansible_overlap.py (two real
runs on one host take turns; schedule skip then fire; alert action skip then
start).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- git ships in the image; checkouts live under /data/ansible/sources
(STEWARD_ANSIBLE_SOURCE_DIR), on the persistent volume.
- One task, ansible_git_pulls, reads the sources every 30 s and pulls each
as its interval passes: a source added in Settings is pulled without a
restart, a removed one is forgotten.
- git_pull: a 120 s timeout ending the process group, GIT_TERMINAL_PROMPT=0,
pull --ff-only, a failed clone removed; it raises GitPullError with git's
message, so the Sync button stops saying Synced after a failure (and the
message is now escaped).
- An alert action's vars and the firing's steward_alert_* vars reach
ansible-playbook as one JSON extra-vars file, not -e key=value, which
Ansible split on whitespace (a rule named 'High CPU on web').
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The first instrumented drop on dev read "dropped after 10.0s … (completed:
parse=0.1s)": the whole stall is inside record_metrics. It now marks its
three steps (flush of the rows, the rule and state reads, the per-sample
evaluation) on the caller's phases, and the drop line carries the request's
samples and metric row count.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5093 "Shared format filters and status_class driven by Settings → Thresholds".
- steward/core/formatting.py: human_bytes (binary units), human_rate,
human_duration (two largest units), ago (a <time> titled with the absolute
UTC time) and status_class(value, kind, thresholds) -> ok/warn/crit/unknown,
which raises on a kind the thresholds don't name.
- app.py registers them as the bytes, rate, duration and ago filters and the
status_class global; threshold_style is now graded by status_class (and caps
its weight at 500).
- Replaces the copies: docker's _human_bytes/_human_uptime, core alerts'
_duration, the host-agent fmt_bps/fmt_bytes macros, the Traefik and UniFi
byte-rate arithmetic, the swarm-failures ago macro and two uptime spans.
- Tests: a table per helper, and every template compiled against the app's
filters, so an unknown filter fails CI rather than a page.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Notifications and alert actions now start only once the evaluating
transaction commits, and never for one that rolls back. record_metrics
queues them on the session (_after_commit), and after_commit/after_rollback
listeners on the sync session spawn or drop them. Spawned tasks are held in a
module set so they cannot be garbage-collected before they run.
A maintenance window no longer skips evaluation. Alerts in its scope still
change state, but their notifications and actions are held, and each event
says so ("notification muted: maintenance window <name>"; also on no-data
and resume events). notify_after_maintenance, run every minute with the
silence check as the new core task alert_reconcile, sends one notification
for an alert still firing after its window ends and runs its action. The
event's note records the release.
Alert actions go through runner.trigger_run like every other run path, so
the playbook and the inventory must be ones the source lists (#5034 guards).
The firing event links to the run, or its note says why the action did not
start. The maintenance page and form wording now say what a window does.
Tests:
- integration: a rolled-back evaluation notifies nobody, and dispatch waits
for the commit; maintenance mutes while the state moves and notifies once
when the window ends; an action whose playbook is outside its source is
refused, with the note set;
- the alert-action test uses a listed inventory name and waits, with a
deadline, for the run the action starts;
- unit: _muted_by.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent offline is an ordinary rule now. The host_agent scheduler records
host_agent/<host>/seconds_since_seen every minute (from last_seen_at, or
registration time for an agent that never reported), replacing the task
that only logged stale agents. The catalog offers the metric, and picking
it on a new rule fills "> 180".
No data: each rule's state tracks its last sample and its usual gap,
smoothed so that an outage counts as at most three gaps. A core task,
alert_silence_check, runs every 60s and marks a rule no_data_since once its
series has had no sample for three usual gaps (at least 5 minutes; an hour
while the cadence is unknown). A never-sampled rule counts from when it
started watching. The mark sits beside the state, so a firing alert that
goes silent stays firing. Each silence records one event and sends one
notification (rule option notify_no_data, default on; muted in maintenance
windows). The next sample clears it with a "data resumed" event and
notification. Creating, changing, disabling or re-enabling a rule restarts
its watch.
Migration 0029 adds the columns and seeds each existing rule's last sample
and gap from its last day of data, so an upgrade does not flag every rule.
Shown on the Alerts page (a NO DATA badge with its start time), in MCP
list_alerts, list_alert_rules and get_host (state "no_data", no_data_since),
in get_overview findings (alert_no_data, or a firing alert marked silent)
and in the weekly report. The rule form has the notify option, and its
metric reference now matches what traefik records and lists host_agent.
Emails say "no data" and carry the event's note.
The metric-catalog guard also reads positional MetricBatch.add arguments.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5060 "One evaluation path: record_metrics batch API; host-agent ingest
evaluates rules".
- record_metrics(session, rows) writes metric rows and evaluates the rules
that watch them, reading rules, states and maintenance windows once per
batch. Each rule sees its key's samples oldest first, so an agent catching
up after an outage counts consecutive checks as they happened.
- Host-agent ingest now goes through it. Before, it wrote plugin_metrics
directly and no host_agent rule could ever fire. The catalog gains the
per-mount disk_used_pct ("<host>:<mount>").
- MetricBatch gathers a transaction's samples; the docker ingest, traefik,
unifi, snmp, monitor and self-monitor writers record one batch each instead
of a rule query per sample.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5059 "Alert state machine: acknowledgement holds until resolve; N
consecutive OK to resolve; editing, disabling or deleting a rule resets it".
- The state machine is now next_step(), a pure function of one sample. An
acknowledged alert stays acknowledged while breached; before, the next
breaching sample re-fired it. Firing and acknowledged alerts resolve after
consecutive_ok_required OK checks in a row (new rule column, default 1;
rule form, rules list and MCP create/update_alert_rule clear_after_checks).
- Changing what a rule watches (source, resource, metric, operator,
threshold) or disabling it resets its alert to inactive; a firing or
acknowledged one records a resolved event whose new note says why, and
nobody is notified. Web and MCP edits share alerts.service.apply_rule_change.
- Migration 0028: consecutive_ok_required, consecutive_ok_count,
alert_events.note, and metric_value nullable (a reset has no sample).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Rebuilds the deleted NUT/UPS automation as a general alert action: any metric
can drive a playbook run on transition-to-firing.
- models/alerts.py + migration 0014: AlertRule.ansible_action (JSON, admin-only,
reuses the #546 param shape); AlertEvent.ansible_run_id links a firing event to
the run it triggered
- core/alerts.py: pure alert_extra_vars() injects steward_alert_* context; on
('firing', event) with an action set, schedule _run_ansible_action (deferred
after commit, same pattern as notifications) — fires once per transition,
consecutive_failures_required is the debounce; system-triggered AnsibleRun
- alerts/routes.py: admin-only parse/validate of the action (source must be
configured, playbook must exist); operators keep editing rules, action preserved
- rules_form.html: admin-only 'On firing -> run a playbook' section
- tests: unit for alert_extra_vars; integration drives record_metric to firing
and asserts a system AnsibleRun ran the playbook with the injected var and the
event linked to it
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Renames the Python package directory, CLI command, env var prefix,
docker-compose service/container/image, Postgres role/db, and all
visible branding. Marketing form is "Fabled Steward".
Clean break from the previous rebrand: drops the fabledscryer→roundtable
import shim in __init__.py and the FABLEDSCRYER_* env var fallback in
config.py and migrations/env.py. Env vars are now STEWARD_* only.
Heads-up for existing deployments:
- Postgres user/db renamed fabledscryer → steward in docker-compose.yml.
Existing volumes need the role/db renamed inside Postgres, or override
POSTGRES_USER/POSTGRES_DB to keep the old names.
- Host-agent systemd unit is now steward-agent.service. Existing agents
keep running under the old name; reinstall to switch.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
bvandeusenandClaude Opus 5.5