Commit Graph
42 Commits
Author SHA1 Message Date
bvandeusenandClaude Opus 5.5 5ca13f9278 feat(prometheus): the Prometheus dashboard widget, target + template (#5691)
CI / lint (push) Successful in 3s
CI / integration (push) Successful in 1m14s
CI / unit (push) Successful in 1m29s
CI / publish (push) Successful in 1m12s
#5691 "Generic Prometheus dashboard widget: pick a target and a template",
step 3 of milestone 508 "Prometheus widget templates".

- /plugins/prometheus/widget draws one target with one template in widget
  shape: the verdict (led by "isn't answering" when the target is down),
  then each block with problems first, figures beside a share bar or the
  last 24h of bars, and the healthy rest folding to "+N more healthy" past
  six. A missing target or template, or one that no longer checks, says so.
- library.py: bundled templates from plugins/prometheus/widgets/*.yaml,
  the picker's target and template lists, and the default title.
- widgets/fabled_archives.yaml: the first bundled template, against the
  series Archives reports (refined in #5693).
- Dashboard: a widget setting may take `options_from` (read when the picker
  opens) with an `empty` sentence; `default_title`; a Details link may name
  settings ("/plugins/prometheus/targets/{target}"). Prometheus is grouped
  with the integrations.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 07:05:15 -04:00
bvandeusenandClaude Opus 5.5 ec3e5e7e0d feat(prometheus): fold state sets on the target page; forget series an endpoint drops (#5689)
CI / lint (push) Successful in 2s
CI / integration (push) Successful in 1m20s
CI / unit (push) Successful in 1m28s
CI / publish (push) Successful in 20s
#5689 "Prometheus target page: fold state sets, show epoch times, size
bytes", step 1 of milestone 508 "Prometheus widget templates".

- A state set (name ending in _state, a state label, every series 0 or 1;
  or the OpenMetrics form, the label named after the metric) reads as one
  row per thing it describes, naming the current state, the others on
  hover. fabledarchives_jobs{state} is a count and stays as it is.
- label_pairs / state_label / fold_states in service.py, for the widget
  templates to reuse.
- After a good scrape, series it no longer has lose their newest value
  (forget_gone): Archives' old build_info row read as current beside the
  new one. History stays; a failed scrape forgets nothing; alert rules keep
  their own last-sample time.
- Epoch times and _bytes sizes were already read by shown_value /
  metric_value.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 15:11:24 -04:00
bvandeusenandClaude Opus 5.5 49799bd0b3 feat(backup): daily backups to an S3 bucket, Settings → Backups, and the guide (#5673, #5675)
CI / lint (push) Successful in 2s
CI / integration (push) Failing after 1m15s
CI / unit (push) Failing after 1m28s
CI / publish (push) Skipped
#5673 "S3 client and the backup job: settings, schedule, history, metrics":
- steward/backup/s3.py: SigV4 over httpx, path-style; put and list only
  (an append-only key is enough; retention is the bucket's). Signing is held
  to AWS's published examples.
- steward/backup/job.py: a 15-minute tick. The daily backup goes at the hour
  chosen on the reader's clock; a missed day is made up, and a failure is
  retried an hour later. History of the last 30 runs, audit
  (backup.made / backup.failed), and backup_age_secs / backup_bytes under
  "Steward itself", resource backup, for alert rules.
- Settings → Backups: verdict first, then the bucket (secret sealed; a
  blank box keeps it), the schedule, Test the bucket, Back up now, and the
  recent runs.

#5675 "Docs: backing up to Archives, and restoring": docs/core/backups.md.

Also: the audit page now words the Prometheus target actions from #5668,
which showed as codes, and a guard fails on any logged action without
wording.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 07:23:58 -04:00
bvandeusenandClaude Opus 5.5 002181253e fix(prometheus): the list page is a section top; the scrape test builds its text without format(); docs (#5668, #5670)
- tests/test_breadcrumbs.py: the Prometheus target list is its section's
  own top page, like SNMP's, so it carries no trail.
- The integration test's exposition has label braces, which str.format read
  as fields: the fake scrape raised, the target was recorded down and no
  series landed. It substitutes the counter with replace() now.
- docs/plugins/prometheus.md: adding a target, filtering, how series are
  named and stored, alerting, and the Fabled Archives recipe.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 07:06:31 -04:00
bvandeusenandClaude Opus 5.5 400e91a0d8 fix(host_agent): a post that reached Steward counts as the agent seen, even if storing it failed (#5514)
CI / lint (push) Successful in 2s
CI / integration (push) Successful in 1m14s
CI / unit (push) Successful in 1m28s
CI / publish (push) Successful in 1m16s
At 20:25-20:39Z the NFS storage under steward_db slowed badly (checkpoint
sync 4.6 s, then 70.7 s with one file at 21.9 s, against the usual 0.002 s).
Ingest commits stalled past the agent's 10 s timeout and were dropped, and
last_seen_at, written in that same transaction, went with them. Agents
posting every few seconds read as silent for 3 minutes, and "Agent offline"
fired and emailed for all three hosts.

Ingest now notes each agent's arrival in memory right after the (read-only)
registration lookup, before any write; seconds_since_seen counts from the
later of last_seen_at and that arrival.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 16:48:19 -04:00
bvandeusenandClaude Opus 5.5 57e1960dc1 feat(dashboard): an Alerts widget beside the Overview; the Overview's overflow links to Alerts and Monitors (#5510)
CI / lint (push) Successful in 3s
CI / integration (push) Successful in 1m16s
CI / unit (push) Failing after 1m28s
CI / publish (push) Skipped
The Alerts widget (/alerts/widget) is shaped like every widget: a verdict
("2 firing, 1 silent." / "Nothing firing: all 12 rules are quiet."), then
what is firing or silent with its reading now, what the rule fires on and
since when, then what someone acknowledged and who; a line when alert
email failed in the last 7 days or is retrying (outbox.alert_trouble),
linking admins to Settings -> System; the quiet rules counted, a covering
rule once. It sits beside the Overview in the recommended layout (6x4
each).

The Overview's "+N more" line pointed at the removed /status/; alerts now
link to /alerts/ and monitors to /monitors/.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 16:27:17 -04:00
bvandeusenandClaude Opus 5.5 defad2e98b feat(email): delivery you can see: each alert's email in history, outgoing email on Settings -> System (#5509)
CI / lint (push) Successful in 2s
CI / integration (push) Successful in 1m10s
CI / unit (push) Successful in 1m25s
CI / publish (push) Successful in 1m1s
#5509 "Delivery you can see: each alert's email state in history, and
outgoing email on Settings -> System" (milestone 501).

- Alerts -> History: an alert's email reads from the outbox (#5506):
  "gathering" with when it sends, "sending", "retrying" with the attempt,
  next try and the server's last answer, "sent", or "failed" naming each
  reader it didn't reach and why (outbox.delivery / email_states). Events
  from before the outbox keep what they recorded. The Notified column
  shows once an alert has a queued email.
- Settings -> System gets "Outgoing email": every email of the last 7
  days not yet sent, failures first, then the longest retrying, with
  recipient, kind, state and last error, and a count of what was sent.
  Failures and anything still retrying after 5 minutes lead the page's
  verdict (outbox.outgoing).
- get_alert_history (MCP) adds `email`: the same delivery state.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 16:12:40 -04:00
bvandeusenandClaude Opus 5.5 9a3e7ddba2 feat(email): the report, invites, resets and test sends go through the outbox; only it talks SMTP (#5508)
CI / lint (push) Successful in 2s
CI / integration (push) Failing after 1m11s
CI / unit (push) Successful in 1m26s
CI / publish (push) Skipped
#5508 "Every other email through the outbox: weekly report, invites,
password resets, and the test buttons" (milestone 501).

- outbox.send_each queues one email per reader (claimed, so the sender
  doesn't take it too) and tries them at once, for a page that answers
  with what happened; outbox.said words the result. attempt takes a cfg
  for a test of the form's unsaved settings and returns the settled rows.
- Weekly report (scheduled and Send now) and invites: queued, tried at
  once; one the server couldn't take yet is retried for an hour. An invite
  that didn't go shows its link with the reason, as before.
- Password reset: queued in the reset link's own transaction; the sender
  delivers it on its next tick.
- Settings -> Notifications test email and a rule's Send test: queued and
  recorded with retry=False, tried once (System 156 ruling).
- Removed mail.send, notifications._send_email and _smtp_send_sync;
  dispatch_notifications is webhook only. A guard asserts only the outbox
  (and smtp_open itself) opens an SMTP connection, with a self-test.

Also fixes run 8941 (#5507's integration test): its Also-send-to
addresses were "@test", which is_real_address rightly drops; they are
example.com now.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 16:05:01 -04:00
bvandeusenandClaude Opus 5.5 5237ffc663 feat(alerts): alert email waits 2 minutes in the outbox and each alert records its outcome (#5507)
CI / lint (push) Successful in 3s
CI / integration (push) Failing after 1m13s
CI / unit (push) Successful in 1m25s
CI / publish (push) Skipped
#5507 "Alert emails through the outbox, gathered for 2 minutes"
(milestone 501). Replaces the in-memory core/alert_burst.py.

- outbox.hold_alert: an alert joins each recipient's held alert email, or
  opens one due 2 minutes later (System 156 ruling). A Postgres advisory
  lock keeps two alerts at once from opening two windows for one reader.
- email_outbox_events.alert_json (migration 0043) keeps the alert as it
  fired, so the email written when the window closes says what it said.
- write_held writes it then, in the reader's zone: the usual alert email
  for one alert, the list for several (outbox.alert_words).
- after_settle records on each AlertEvent, once every email carrying it
  has settled: email_sent, or email_error naming each reader it didn't
  reach and the server's reason.
- A held email survives a restart and goes out when its window closes.
- A refusal of a message's one recipient no longer repeats the address.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 15:59:17 -04:00
bvandeusenandClaude Opus 5.5 c8dbe811f8 feat(health): Steward records its own health and Settings → System shows it (#5467)
CI / lint (push) Successful in 2s
CI / integration (push) Successful in 1m15s
CI / unit (push) Failing after 1m29s
CI / publish (push) Skipped
#5467 "Steward's own health: metrics, rules can watch them, a page that shows them".

The self-monitor tick now records, under source "steward", resource "process":
ingest_posts / ingest_failed / ingest_slowest_ms (the host-agent posts since the
last tick, tallied in steward/core/health.py), db_roundtrip_ms (a timed SELECT 1),
and unwatched_secs (the gap before this tick past watch.GAP_S, from Steward's own
last sample). The hourly cleanup records cleanup_secs. The open-file pair is still
recorded, and is now only skipped on its own where /proc is absent. All of these
are in CORE_METRICS, so an alert rule can watch them under "Steward itself".

Settings → System (/settings/system/, admin) shows a verdict first (failed posts,
a gap in watching, open files near the limit), then each measure now and over the
last day, with a sparkline scaled to its own peak.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 14:02:50 -04:00
bvandeusenandClaude Opus 5.5 656e127fd8 perf(metrics): a day of one metric reads from the index alone, built after boot (#5490)
CI / lint (push) Successful in 2s
CI / integration (push) Successful in 1m15s
CI / unit (push) Successful in 1m25s
CI / publish (push) Successful in 1m3s
#5490 "A day of one metric reads fast even cold". A cold 24 h series read
measured 18.3 s against 63 ms warm (#5485): every sample's row sat on its own
heap page. ix_plugin_metrics_series_value carries `value` (INCLUDE), so the
read is an index-only scan; it replaces the plain series index from 0023.

It is built once, with CREATE INDEX CONCURRENTLY from a task started after the
app is serving, never in a migration, so the boot isn't held while it builds
over the live table (#5220) and ingest and pages carry on. A build cut short
by a restart leaves an INVALID index; the next boot drops and rebuilds it.
Later boots find it present and only check. Postgres keeps it current on each
insert, as it did the index it replaces.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 13:23:46 -04:00
bvandeusenandClaude Opus 5.5 57e040f1d0 feat(mcp): update_group_members puts hosts into an inventory group or takes them out (#5487)
CI / lint (push) Successful in 2s
CI / integration (push) Successful in 1m9s
CI / unit (push) Successful in 1m29s
CI / publish (push) Successful in 1m19s
#5487 "MCP: edit inventory group membership". Group membership decides which
hosts a group alert rule covers, and the MCP could only read it. The new
operator tool takes a group and lists to add and remove, each a target's
name or id or the Steward host it is linked to; an unknown name refuses the
whole change. It answers with the group's targets after and the alert rules
covering the group (whose per-host rules follow within a minute), and audits
as ansible.group.updated by "<user> via MCP", as the Inventory page does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 11:48:08 -04:00
bvandeusenandClaude Opus 5.5 73cfbda1da fix(alerts): a rule's page doesn't wait for its metric preview (#5485); Send test up top (#5486)
CI / lint (push) Successful in 2s
CI / integration (push) Successful in 1m10s
CI / unit (push) Successful in 1m26s
CI / publish (push) Successful in 1m5s
#5485 "A rule's page waits up to 25 s for its metric preview: a cold 24 h
series read from plugin_metrics". Opening "Agent offline" took 25.6 s; a
cold day of one series measured 18.3 s over MCP, a warm one 63 ms. The
pickers still come with the page from plugin_metric_latest; the preview is
its own request (hx-trigger=load), showing "Reading the last day…" until it
lands. Picker swaps no longer read it either.

#5486 "Send test where it can be found": the button moves from inside "What
happens" to the page header beside History, its result under the verdict.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 11:44:19 -04:00
bvandeusenandClaude Opus 5.5 d19df51ffb feat(alerts): pick the host, then its part, and see what the metric reads (#5475)
CI / lint (push) Successful in 2s
CI / integration (push) Successful in 1m8s
CI / unit (push) Failing after 1m29s
CI / publish (push) Skipped
#5475 "Rule form: pick the host, then its part; metrics that part reports;
a link to groups; a preview of the metric", from the operator's review of
the New rule page: hosts and their parts shared one long picker.

- Host agent rules pick a Host, then a Part: "The whole host", then its
  mounts, CPU cores, disks, interfaces and sensors under a heading each,
  cores in number order. A part that reports nothing a rule can watch
  (an interface's traffic) isn't offered.
- The Metric list holds what the picked resource reports, in catalog order.
- Under it, a preview: the reading now, and the last 24 h low to high with
  an hourly sparkline drawn from zero (UI ruling 8), read from one series.
- The group picker links to Ansible > Inventory to make a group, in a new
  tab, and re-reads the groups when the tab comes back into view.
- Picking a host or part swaps the pickers alone (/alerts/api/rule_target);
  picking a metric swaps the preview (/alerts/api/metric_preview).
- A rule's own resource and metric stay on offer after they go quiet.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 09:54:08 -04:00
bvandeusenandClaude Opus 5.5 5ef18c992c feat(alerts): Send test on a rule checks that its alerts reach someone (#5466)
CI / lint (push) Successful in 2s
CI / integration (push) Successful in 1m9s
CI / unit (push) Successful in 1m36s
CI / publish (push) Successful in 1m6s
#5466 "Send test: check a rule's delivery from the rule". A rule's page gets
a Send test button (operator role), and MCP gets send_test_alert. Each sends
a TEST alert for that rule now, by email to everyone who gets alerts and by
webhook. The alert carries the rule's name and condition, at the latest
reading of what it watches; a rule covering hosts uses its first host. The
result for each channel shows under the button, or "Nothing is set up to
send to". Nothing fires, no event is written, and the test is audited as
alert_rule.tested. An alert with no reading now says "has no reading yet"
instead of "is None".

Docs: alerting.md also gains the burst email from #5465 "One email per burst
of alerts".

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 08:33:32 -04:00
bvandeusenandClaude Opus 5.5 48a0de5600 feat(alerts): one rule can cover every host, or every host in a group (#5464)
CI / lint (push) Successful in 3s
CI / integration (push) Successful in 1m7s
CI / unit (push) Successful in 1m25s
CI / publish (push) Successful in 1m12s
#5464 "Group rules: one rule covers every host, or every host in a group",
step 2 of milestone 499 "Alerts you can trust". Each rule named one host, so
the live instance carried six rules for three hosts, and a new host had no
agent-offline or disk alert until someone copied them.

- alert_rules.covers / cover_group / parent_id (migration 0041). A rule on
  host_agent may cover "all_hosts" or a "group" (an inventory group: the
  Hosts ruling, #5294). A covering rule has no state; Steward keeps one
  ordinary child rule per covered host, named "<rule>: <host>", so each
  host's alert keeps its own state, history and acknowledgement.
- alerts.service.reconcile_children: a covered host that has reported in
  the last 7 days gets a child; a host no longer covered loses its child;
  children take the parent's condition, checks, notify setting, action and
  on/off. Run on save and each minute (alert_reconcile).
- A child is changed through its parent: edit, toggle and delete of a child
  are refused (409 / PartOfCoveringRule); its edit page opens the parent.
- Rule form: a Covers choice for host sources, with the group picker; fixed
  once the rule exists. Rules list: a covering rule reads "every host in
  swarm · 3 hosts" with its hosts' rules nested under it, each opening its
  history. The history picker and the rule count leave covering rules out.
- MCP: create_alert_rule takes covers/group; update_alert_rule takes group
  and passes changes down; list_alert_rules shows covers, hosts and part_of.
- docs/core/alerting.md: "Rules that cover hosts".

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 08:25:51 -04:00
bvandeusenandClaude Opus 5.5 734d313a54 feat(alerts): an acknowledgement is on the record, with who made it (#5463)
CI / lint (push) Successful in 3s
CI / integration (push) Successful in 1m7s
CI / unit (push) Successful in 1m27s
CI / publish (push) Successful in 1m8s
#5463 "Acknowledgements on record: an event and who did it", step 1 of
milestone 499 "Alerts you can trust". The live walk in #5461 found that
acknowledging changed only the alert's current state: once it resolved,
nothing said who had acknowledged it or when.

- alert_events.actor_id (migration 0040): the person behind a transition.
- Acknowledging writes a firing → acknowledged event by the acknowledger.
- A rule edit, disable or enable that resets an alert carries its actor
  (web and MCP alike).
- The history page notes "acknowledged by X" and "rule disabled by X"; MCP
  get_alert_history returns `by`.
- One usernames() helper in alerts.service, used by the active list, the
  history page and the MCP tools.
- docs/core/alerting.md: the state diagram no longer shows an acknowledged
  alert re-firing, and says what an event records.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 08:19:48 -04:00
bvandeusenandClaude Opus 5.5 f67678998c refactor(plugins): no plugin catalog; plugins ship in the image or are mounted (#5374)
CI / lint (push) Successful in 3s
CI / integration (push) Failing after 1m12s
CI / unit (push) Successful in 1m25s
CI / publish (push) Skipped
First-party plugins have shipped inside the image since a7a281c, and the
catalog's only repository (Steward-plugins) answers 404, so Steward polled a
dead URL every minute for nothing. The operator chose to remove the catalog
entirely (#5214 "Review: readable and fast on the live instance").

Removed:
- core/plugin_index.py and the plugin_catalog_refresh startup task;
- the plugins.index_url default, and migration 0035 deletes the stored
  plugins.index_url / plugins.repositories rows;
- the Settings -> Plugins repository editor, catalog, refresh, install and
  reload routes and their templates;
- the download/checksum/extract install path in plugin_manager, and the
  PLUGIN_INSTALL_DIR config it wrote to;
- plugins/index.yaml, docs/plugins/index.yaml.example and the guide's
  "Publishing to the Catalog" section (now "Shipping a Plugin").

Kept: enabling and configuring plugins, hot-reload on enable, restart, orphan
cleanup, and the external root (/data/plugins) for operator-mounted plugins.
tests/test_no_plugin_catalog.py guards the routes, default and task.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-08 13:15:55 -04:00
bvandeusenandClaude Opus 5.5 9ca2e38fa1 feat(snmp): devices you add, test and read on the SNMP page (milestone 491)
CI / lint (push) Successful in 2s
CI / integration (push) Failing after 55s
CI / unit (push) Successful in 1m24s
CI / publish (push) Skipped
The operator, reviewing SNMP on #5214: "this system looks incomplete like
there's something missing from it". The one device shown was plugin.yaml's
example (core-switch 192.168.1.1, community public), polled every minute;
devices could only be defined by editing YAML in the image; a failed poll was
logged at DEBUG and stored nowhere, so the page said "waiting for the next
poll" forever; v3 ran as v2c; counters showed as lifetime totals.

- Devices are rows (snmp_devices, migration snmp_001): address, port,
  v1/v2c community or a v3 user with auth/priv protocols and keys (secrets
  encrypted), presets plus custom readings, an optional host binding, and
  the newest poll's outcome. The YAML device list and its example are gone.
- The SNMP page adds, edits and deletes them (admin, audited, with
  get_audit_entities so the audit log names and links a device). Readings
  come from presets (system uptime, interface traffic by number, UPS-MIB,
  Linux net-snmp) and custom "OID name [unit]" lines.
- One GET per device with every OID, devices polled at once. The outcome is
  written to the row in words ("No reply from … within 5 s. Check the address
  and port, that SNMP is on, and the community"), with readings it lacks.
  v1's whole-request error drops that OID and asks again. SNMPv3 signs in
  with USM (MD5/SHA-1/SHA-2, DES/3DES/AES).
- Test asks the unsaved form's device for its name, description, uptime and
  interfaces by number. It is an admin's explicit action; every page reads
  only stored rows (System 178's ruling).
- Each reading has a unit; a counter is stored as a per-second rate (bytes as
  bit/s), a reset or wrap skipped. The page, widget and host panel say values
  in units and why a device is silent; the device page draws a chart per
  reading on a dated axis.

Tests: readings and presets, the form and its secrets, v3 checks, rates,
the poller against a stand-in pysnmp (one GET, v1 retry, timeout wording,
USM), the pages, and an integration test of the poll writing outcomes and
rates on Postgres.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-08 10:54:11 -04:00
bvandeusenandClaude Opus 5.5 5492336f1e feat(dashboard): a curated default layout, merged duplicate widgets, a labelled Overview (#5213)
CI / lint (push) Successful in 2s
CI / integration (push) Successful in 50s
CI / unit (push) Failing after 1m16s
CI / publish (push) Skipped
- New installs start from widgets.RECOMMENDED, packed row by row for the
  enabled plugins; the edit drawer can reset a dashboard to it.
- Host Agent — Resources folds into Hosts — Overview (which gains load 1m),
  UniFi — Devices into UniFi — Overview; migration 0032 moves existing
  dashboards onto the survivors.
- The Overview labels what it counts (hosts, monitors up/down/pending,
  alerts firing) and lists firing alerts and down monitors with since-times.
- Uptime polls every 5 minutes (poll_seconds) instead of every cycle.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-07 00:42:04 -04:00
bvandeusenandClaude Opus 5.5 b8f73270e7 perf(metrics): a latest-value table for "what is it now"; dashboard cells say loading, failed and as-of (#5208)
CI / lint (push) Successful in 3s
CI / integration (push) Failing after 1m14s
CI / unit (push) Successful in 1m33s
CI / publish (push) Skipped
The dashboard's host widgets took 18-60 s. Every "latest reading" query
(hosts overview, host-agent fleet, host page, vitals, SNMP, MCP get_host)
searched plugin_metrics for the newest row per metric. Postgres has no skip
scan, so each one read every raw sample in the 7-day window.

- plugin_metric_latest: one row per (source, resource, metric).
  record_metrics upserts it in the same transaction, moving forward only and
  in sorted key order. Migration 0030 seeds it by stepping the existing index.
- Retention drops a latest row once its sample leaves the raw window, so a
  resource that stopped reporting disappears when its history does, as before.
  Renames carry it along.
- A quiet host keeps its last values, marked stale. A time-bounded query
  would have shown it as missing.
- Vitals sparklines span the 6 h they claim, averaged in SQL to ~120 points,
  instead of fetching 6 h raw and drawing the last 120 samples.
- Dashboard and share-view cells: "Loading…" until the first answer; a 20 s
  deadline per fetch; "Couldn't load (why)" with Retry now and an automatic
  retry for cells that don't poll; "as of HH:MM" in the header, and
  "couldn't refresh · as of" when a refresh fails over shown data.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-06 23:06:34 -04:00
bvandeusenandClaude Opus 5.5 3c24c633df fix(ansible): overlapping runs: one per host, one per schedule, one action run per alert rule (#5158)
CI / lint (push) Successful in 3s
CI / unit (push) Successful in 1m27s
CI / integration (push) Failing after 1m1s
CI / publish (push) Skipped
- One run per host: start_run claims every host it touches (addresses read
  from the inventory Steward wrote; a source's own inventory file is claimed
  as that file) before it takes a run slot. The claim is all-or-nothing. A run
  waiting on a host shows queued and names the run it waits for, honours a
  cancel, and fails after HOST_WAIT_SECONDS (1 h) with which host and run.
- A due schedule whose previous run is still queued or running skips the
  firing: last_run_at moves on, last_run_id still names the run going, and
  last_error says "Skipped: the previous run was still going". The Schedules
  page shows that beside the run (muted), and a launch error in red.
- An alert rule that fires while the action run from an earlier firing is
  still going starts no new run; the event says so and names the run. An
  in-process set closes the gap between the check and the run being linked.
- docs/core/ansible.md: queued/cancelled statuses and the overlap rules.

Tests: tests/core/test_ansible_overlap.py (claim keys, waiting, all-or-nothing,
cancel, deadline) and tests/integration/test_ansible_overlap.py (two real
runs on one host take turns; schedule skip then fire; alert action skip then
start).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-06 21:24:51 -04:00
bvandeusenandClaude Opus 5.5 edcfa4d3db fix(ansible): git sources work in the image and survive a redeploy; pulls time out; alert vars go in a JSON file (#5157)
CI / lint (push) Successful in 2s
CI / integration (push) Successful in 49s
CI / unit (push) Successful in 1m11s
CI / publish (push) Successful in 1m3s
- git ships in the image; checkouts live under /data/ansible/sources
  (STEWARD_ANSIBLE_SOURCE_DIR), on the persistent volume.
- One task, ansible_git_pulls, reads the sources every 30 s and pulls each
  as its interval passes: a source added in Settings is pulled without a
  restart, a removed one is forgotten.
- git_pull: a 120 s timeout ending the process group, GIT_TERMINAL_PROMPT=0,
  pull --ff-only, a failed clone removed; it raises GitPullError with git's
  message, so the Sync button stops saying Synced after a failure (and the
  message is now escaped).
- An alert action's vars and the firing's steward_alert_* vars reach
  ansible-playbook as one JSON extra-vars file, not -e key=value, which
  Ansible split on whitespace (a rule named 'High CPU on web').

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-06 21:13:14 -04:00
bvandeusenandClaude Opus 5.5 3474cdff46 feat(mcp): connect card, MCP docs, and an end-to-end roles/audit test over HTTP
CI / lint (push) Successful in 1s
CI / integration (push) Successful in 44s
CI / unit (push) Successful in 1m5s
CI / publish (push) Successful in 1m15s
MCP-8 #4972 "Connect card, docs, and end-to-end integration test".

- AI access page: a "Connect a client" card with the /mcp URL (from
  Settings → General → Public base URL, else the request host) and recipes
  for Claude Code, Claude Desktop (mcp-remote) and other Streamable HTTP
  clients, plus the reverse-proxy note. The token-created page shows the
  same recipes with the new token filled in, ready to paste.
- docs/core/mcp.md: connecting, the reverse-proxy requirements, the
  security model (token ceiling, default deny, check-mode default, secrets,
  audit), tools by role, response conventions, adding tools.
  docs/architecture.md describes the /mcp dispatch outside the blueprint
  pipeline; docs/reference/code-map.md gains an MCP section.
- tests/integration/test_mcp.py: over HTTP, a viewer token lists no write
  tools and is refused one by name; an operator token reads get_overview
  and its maintenance-window create/end land in the audit log as
  "<user> via MCP" with the token id. (Revoked → 401 was already covered.)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-06 08:25:41 -04:00
bvandeusenandClaude Opus 5.5 41c7509366 feat(mcp): diagnostic tools — overview, hosts, monitors, alerts, metrics, audit
CI / lint (push) Successful in 2s
CI / integration (push) Successful in 36s
CI / unit (push) Successful in 58s
CI / publish (push) Successful in 1m2s
MCP-4 of milestone 461 "MCP server: Steward as an LLM-inspectable,
LLM-operable hub". Twelve read-only viewer tools:

- get_overview: what is wrong now, ranked critical-then-oldest. Failing
  monitors with their error and when the run of failures began, firing,
  pending and still-breaching acknowledged alerts, TLS certificates expiring
  within 14 days, agents that stopped reporting, hosts over the configured
  CPU/memory/disk thresholds (Settings > Thresholds), failed Ansible runs in
  the last 24h, and findings from enabled plugins through a new optional
  `mcp_overview(db)` plugin hook. Each finding names the call that shows
  its evidence.
- list_hosts, get_host, get_host_metrics
- list_monitors, get_monitor (HTTP header values and URL passwords are
  redacted)
- list_alerts, get_alert_history, list_alert_rules, list_maintenance_windows
- query_metrics: the whole metric store, narrowed one argument at a time
  (sources > resources > latest values > a series summary)
- list_audit_events

Shared shaping in steward/mcp/tools/_shape.py: UTC ISO times with age_s,
bounded lists that say when and how they were cut, series as
min/avg/max/p95/last plus a ≤60-point series, lookup by id or
case-insensitive name with a did-you-mean, and a range argument that
refuses an unknown key instead of reading it as 24h.

New core reads in steward/core/metric_series.py (sources and resources
via loose index scans, latest values via DISTINCT ON, series across the
raw/hourly-rollup seam). monitors/service.py gains monitor_states,
down_since and recent_failures; agent_overview_by_host now carries the
fullest disk and the last sample time.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-06 07:39:41 -04:00
bvandeusenandClaude Opus 5.5 3984f639e8 feat(mcp): /mcp endpoint with bearer auth and role-gated tools
CI / lint (push) Successful in 2s
CI / integration (push) Failing after 35s
CI / unit (push) Failing after 1m0s
CI / publish (push) Skipped
Second step of milestone 461. An MCP client (Claude Code, Claude Desktop
via mcp-remote, Open WebUI) connects to /mcp with a personal API token.

- mcp>=2.2 (SDK v2 MCPServer). Mounted by wrapping app.asgi_app, so /mcp
  never enters Quart's pipeline; stateless streamable HTTP so a redeploy
  never strands a client on a session id the new process didn't issue.
- Auth: Authorization: Bearer stw_… resolves to a Caller (user +
  effective role) in a ContextVar; 401 with WWW-Authenticate otherwise.
  The web pipeline's database gate runs first, so a DB restart answers
  503 instead of failing inside a tool.
- Registry, default-deny: register_tool(fn, role=, read_only=) is the only
  door; role and read_only are required. Tools above the caller's role,
  or from a disabled plugin, are neither listed nor callable. Undeclared
  arguments are refused (not silently dropped); a tool's ValueError
  reaches the agent as its reason; a DB failure mid-tool says so.
- Plugins contribute tools via an optional register_mcp_tools(register)
  export, all-or-nothing, replaced on hot-reload. Documented in
  docs/plugins/writing-a-plugin.md.
- First tool: whoami.
- Tests drive app.asgi_app directly (Quart's test client never reaches
  /mcp); the integration test mints a real token against Postgres and
  runs initialize → tools/list → whoami → revoke → 401.

Task #4966 "MCP-2 — /mcp transport".

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-06 07:24:00 -04:00
bvandeusenandClaude Opus 4.8 8c2d994e4c docs(host-agent): document container-log collection [M79 step 6]
CI / lint (push) Successful in 2s
CI / unit (push) Successful in 42s
CI / integration (push) Successful in 2m21s
CI / publish (push) Successful in 5s
Design-spec updates for m79: the docker_logs_enabled / docker_log_exclude config
keys, the collect_docker_logs collector + the drop-logs-on-backoff note in the
agent internals, and a wire-format design point on docker_logs riding in the same
push (with server-side toggle/exclude enforcement + the per-container ring).

The end-to-end chain (store → rotate → query + ingest enforcement) is already
covered by the integration tests added across steps 2–5, so no capstone test is
added. AGENT_VERSION bump (1.7.0) landed in step 1.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CAGR73DUowdVFVvYzLXC5C
2026-07-19 19:07:53 -04:00
bvandeusenandClaude Opus 4.8 42f7840c26 feat(ansible): steward:category + steward:confirm playbook metadata
CI / lint (push) Successful in 2s
CI / unit (push) Successful in 8s
CI / integration (push) Successful in 2m14s
CI / publish (push) Successful in 59s
Extend the playbook metadata convention with a namespaced `# steward:<key>:`
comment block:

- steward:category — free-text grouping label, shown as a badge in the browse
  list and on the run form.
- steward:confirm — true/yes/1/on marks a playbook destructive; the run form
  then requires a confirmation tick (required checkbox in the shared vars
  fragment) before it can launch.

sources.discover_playbook_meta() parses description + category + confirm (first
match per key; `# description:` still primary, `# steward:description:` alias).
discover_playbook_description() now delegates to it. The browse list reads
per-playbook meta to show category badges + descriptions; the run-form and
playbook-vars fragments render the badge + confirm gate.

Bundled playbooks tagged: docker_prune → category maintenance + confirm true;
provision/install/update → category host-agent.

Docs: docs/reference/playbook-authoring.md updated (keys now implemented) and a
quick reference added next to the code at steward/ansible/PLAYBOOK_CONVENTIONS.md.
Tests added for category/confirm/alias parsing.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-17 11:35:35 -04:00
bvandeusenandClaude Opus 4.8 b32fce1d74 docs: Steward playbook-authoring conventions reference
CI / lint (push) Successful in 3s
CI / unit (push) Successful in 8s
CI / integration (push) Successful in 2m22s
CI / publish (push) Successful in 4s
A self-contained guide (docs/reference/playbook-authoring.md) to the contract
between a playbook and Steward's run UI — the `# description:` comment,
vars/vars_prompt → fill-in fields, secret naming, hosts: all targeting,
managed credentials, and idempotency. Includes a "what Steward reads" summary
table, the metadata/extensibility note (only `# description:` today; reserved
`# steward:<key>:` namespace for future keys), and an annotated example.
Meant to be fed to another session authoring Steward-friendly playbooks.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-17 11:26:56 -04:00
bvandeusen eb319d715e docs: Ansible inventory infrastructure implementation plan 2026-06-05 18:19:17 -04:00
bvandeusenandClaude Opus 4.7 88ab5b917e chore: rename project Roundtable → Steward
Renames the Python package directory, CLI command, env var prefix,
docker-compose service/container/image, Postgres role/db, and all
visible branding. Marketing form is "Fabled Steward".

Clean break from the previous rebrand: drops the fabledscryer→roundtable
import shim in __init__.py and the FABLEDSCRYER_* env var fallback in
config.py and migrations/env.py. Env vars are now STEWARD_* only.

Heads-up for existing deployments:
- Postgres user/db renamed fabledscryer → steward in docker-compose.yml.
  Existing volumes need the role/db renamed inside Postgres, or override
  POSTGRES_USER/POSTGRES_DB to keep the old names.
- Host-agent systemd unit is now steward-agent.service. Existing agents
  keep running under the old name; reinstall to switch.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-04-25 16:20:14 -04:00
bvandeusenandClaude Opus 4.6 44cc2f2fce docs(plugins): catalog entry for host_agent
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-04-15 00:31:27 -04:00
bvandeusenandClaude Opus 4.6 b1cfc377da docs(plugins): host_agent plan — path 1 note (no DB tests)
Records the mid-execution pivot: test harness uses testing=True which
mocks the DB, so plugin tests stick to pure-Python unit tests (agent +
metric expander). Server routes are verified manually.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-04-14 18:13:41 -04:00
bvandeusenandClaude Opus 4.6 98003f3458 docs(plugins): host_agent implementation plan
15-task TDD plan covering plugin scaffolding, agent collectors, ingest
route, install script, settings UI, widgets, detail page, scheduler,
and integration test.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-04-13 23:46:37 -04:00
bvandeusenandClaude Opus 4.6 e243e0e0e1 docs(plugins): host-agent plugin design spec
Design for a push-based Python agent that collects CPU/memory/storage/
load/uptime from remote Linux hosts and POSTs to a new host_agent
plugin. Uses per-host bearer tokens, systemd install via curl|sh,
in-memory ring buffer with exponential backoff, and writes to the
existing PluginMetric bus plus a plugin-private registrations table.

No core schema changes — plugin stays strictly isolated.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-04-13 23:35:12 -04:00
bvandeusenandClaude Opus 4.6 d6e094bbb5 refactor: remove broken NUT/UPS plugin and managed-NUT startup
The UPS plugin's default driver was snmp-ups, so NUT was just translating
network SNMP back into its own protocol — no value over polling the UPS
directly through the existing snmp plugin. The managed-NUT entrypoint path
also broke container startup when config was incomplete.

Deletes nut_setup.py, the NUT block from entrypoint.sh, NUT packages from
the Dockerfile, the ups entry from the plugin catalog example, and the
ups wiring in alerts METRIC_CATALOG, widgets, settings routes, and the
rules form. The plugins/ups source tree (untracked) is also removed from
the working copy. Existing ups_* DB tables are orphaned but harmless in
dev; a drop migration isn't needed.

Follow-up: rebuild "on battery → Ansible shutdown" automation on top of
the alerts system as a new action type that runs a playbook.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-04-13 22:41:08 -04:00
bvandeusen 0596f1d9c1 docs: roundtable rename sweep
Sweeps README, docs/, example configs, code docstrings, report and
notification subject lines, plugin catalog URL default, and stray
comments in Dockerfile/entrypoint.sh/.gitignore. Adds a legacy note
to docs/core/configuration.md explaining the FABLEDSCRYER_* fallback.
docker-compose DB credentials intentionally left as fabledscryer to
preserve the existing pgdata volume.
2026-04-13 20:16:25 -04:00
bvandeusenandClaude Opus 4.6 1f6404f0fd docs: rebrand implementation plan
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-04-13 14:40:26 -04:00
bvandeusenandClaude Opus 4.6 39074252d1 docs: rebrand design spec (FabledScryer → Roundtable)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-04-13 14:30:40 -04:00
bvandeusenandClaude Sonnet 4.6 edc9b752d2 feat: drag-and-drop dashboard editing, alert dynamic fields
- Replace up/down arrow reorder buttons on dashboard edit page with
  SortableJS drag handles; add POST /d/<id>/edit/reorder endpoint
  accepting ordered widget ID array
- Add {% block extra_scripts %} hook to base.html for page-specific JS
- Add dynamic alert rule field loading via HTMX partial (_rule_fields)
  so metric/threshold fields update when source is changed
- Add docs/plugins/index.yaml.example showing catalog index format

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-23 08:16:04 -04:00
bvandeusenandClaude Sonnet 4.6 3f437141fb chore: update plugin catalog URL to git.fabledsword.com
Default plugins.index_url and all documentation URLs now point to
https://git.fabledsword.com/bvandeusen/FabledScryer-plugins.
Anchor /plugins/ in .gitignore so it only matches the root directory
(not docs/plugins/).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-22 18:36:09 -04:00
bvandeusenandClaude Sonnet 4.6 230b542015 feat: rename to FabledScryer, multi-dashboard system, plugin management, branding
- Rename package fablednetmon → fabledscryer throughout
- Multi-dashboard: ownership, per-user defaults, HTMX edit (add/remove/reorder)
- Read-only share tokens scoped to individual dashboards
- Dashboard edit is HTMX-driven (no page reloads)
- Plugin management system: remote catalog, download/install, hot-reload, in-app restart
- plugin_index.py: fetch/cache remote index.yaml; default URL → bvandeusen/fabledscryer-plugins
- plugin_manager.py: download_and_install_plugin, hot_reload_plugin, restart_app
  - ZIP extraction handles GitHub archive formats (name-v1.0.0/, name-main/)
- Settings split into tabbed sections: General, Notifications, Ansible, Plugins
- Plugins tab: catalog browser (HTMX), install/activate/update/restart actions
- UI/branding: dark palette (#07071a), crystal ball SVG logo, animated star field,
  Libertinus Serif applied to headings, nav, labels, and section titles
- Widget registry (core/widgets.py) for dashboard plugin integration
- UPS widget.html (dashboard card) and settings/_tabs.html include
- Migrations 0005–0008: dashboards, is_default, ownership, share tokens
- docs/plugins/: writing-a-plugin.md updated with publishing guide,
  index.yaml.example template for fabledscryer-plugins repo

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-22 18:27:56 -04:00