Commit Graph
12 Commits
Author SHA1 Message Date
bvandeusenandClaude Opus 5.5 3a772716ae feat(ui): audit, alert rules and the rule form say things in words (#5212)
CI / lint (push) Successful in 2s
CI / integration (push) Successful in 51s
CI / unit (push) Failing after 1m14s
CI / publish (push) Skipped
#5212 "Plain labels: no raw ids, scopes, metric keys or unexplained jargon",
batch 1:

- Audit log: each action reads as a sentence ("paused monitor"), with the
  code as its tooltip. Detail shows flags as yes/no, lists joined, a run's
  scope by name, spans as spans, and no ids the entity column already gives.
  The filter is now a picker of areas.
- Alert rules, history and Status say the condition in words: "is down",
  "Fullest disk >= 85%". Readings use metric_value, and history times use
  |ago.
- Rule form: sources and metrics by name, the threshold's unit, "fires
  when it is down/up" for an up/down metric instead of 0.5, and a metric
  reference built from the catalog.
- Schedules and Maintenance: viewers no longer see Run now, Pause, Edit,
  Delete or New. Intervals and next runs read as "every 2h" and "in 3h";
  a scope reads "Every alert on Host agent".

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-07 00:23:02 -04:00
bvandeusenandClaude Opus 5.5 7275fad767 feat(ui): audit log, alert history and Ansible runs load as you scroll (#5211)
CI / lint (push) Successful in 2s
CI / integration (push) Successful in 51s
CI / unit (push) Successful in 1m13s
CI / publish (push) Successful in 1m7s
#5211 "Lists load as you scroll: audit, alert history, Ansible runs"
(milestone 477), per preference 172.

- core/paging.py: keyset paging over (time, id), newest first, with an
  opaque "<iso>|<id>" cursor; a page reads one row past itself to know
  whether another follows. No OFFSET, so rows added while someone scrolls
  never shift the pages after them.
- _macros.html scroll_more / scroll_end and static/scroll-more.js, written
  once: an IntersectionObserver with a 600px margin fetches the next page
  before the reader reaches the end; "Loading more…" while in flight,
  "End of the list." at the end, and a failed page says so with a Retry
  button instead of re-firing on every scroll.
- Audit log: no "Load more" link and no 500-row cap.
- Alert history: no 200-row stop with "pick a shorter range";
  alert_svc.history takes the cursor, so the MCP tool and the page share
  one query.
- Ansible runs: no silent cap of 50.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-07 00:10:53 -04:00
bvandeusenandClaude Opus 5.5 8e890c6914 feat(ui): host page leads with its alerts; old readings say how old (#5210)
CI / integration (push) Successful in 50s
CI / lint (push) Successful in 2s
CI / unit (push) Successful in 1m20s
CI / publish (push) Successful in 1m13s
#5210 "Problems first, with since-time, across every list" (milestone 477),
second batch.

- Host page: an "Alerts on this host" strip first (rules on its agent
  series, its containers or its monitors: alert_svc.watches_host); its
  monitors problems first with "Down 12m · reason"; Docker above Ansible.
- Uptime page and widget: worst 24h uptime first, then 7d and 30d; hosts
  with no uptime yet last.
- Agent panel and fleet widget: last seen as "3m ago", not a clock time.
- SNMP: a device is "Reachable" only while it answered within the last 3
  polls; older readings say "No answer" with their age, and every device
  says when it was last read (was: reachable whenever any reading existed).
- Traefik: a router missing from 3 scrapes in a row is listed apart as no
  longer reported, by when it was last seen, instead of keeping its last
  state (a removed router stayed Failing).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-07 00:02:20 -04:00
bvandeusenandClaude Opus 5.5 5a14e7c693 feat(ui): alerts, monitors, hosts, schedules and maintenance lists put problems first, with since-times (#5210)
CI / lint (push) Successful in 3s
CI / integration (push) Successful in 1m8s
CI / unit (push) Successful in 1m25s
CI / publish (push) Successful in 1m17s
#5210 "Problems first, with since-time, across every list" (milestone 477),
first batch.

- Alerts: the Active list reads firing, no data, acknowledged, pending, the
  longest-running first, and re-polls itself. An acknowledged alert says how
  long it has been breaching (from the alert's events; acknowledging moves
  last_transition_at) and, apart from that, who acknowledged it and when.
  Rules show what each is doing now; disabled rules sort last. The current
  value reads the latest-value table (#5208) instead of plugin_metrics.
- Monitors: a down monitor says "Down 12m · connection refused" in the list.
- Hosts: sorted down, agent gone quiet, pending, healthy. The agent column
  shows "—" for a missing reading (was 0%) and a Stale badge with its age.
  agent_metrics_by_host is gone; the list reads agent_overview_by_host.
- Schedules: failing first (failed or interrupted run, or a firing that could
  not start one), then running, then the rest, paused last; status badge
  from the shared states.
- Maintenance windows: active (ending soonest), then upcoming (starting
  soonest), with expired folded away; each says when it ends/starts/ended.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-06 23:58:56 -04:00
bvandeusenandClaude Opus 5.5 4fbe93bdf7 feat(alerts): a sentence per alert, a link to what it watches, a count in the sidebar (#5124)
CI / lint (push) Successful in 3s
CI / integration (push) Successful in 45s
CI / unit (push) Failing after 1m7s
CI / publish (push) Skipped
The Alerts page's active list now says each alert in one line around the
resource it watches, e.g. "cpu_pct on web-1 is 94% (> 90%), firing for 12m",
with web-1 linking to its host, monitor or container page (or the plugin's
page for Traefik, UniFi and SNMP). Values take their unit from the metric
name through formatting.metric_value. A silent series reads "No cpu_pct from
web-1 for 20m"; an up/down metric says the word.

Actions follow the routes' roles: acknowledge, edit, toggle and New rule for
operators and admins, Delete for admins only, nothing for viewers.

The sidebar shows how many alerts need attention beside Alerts, from an
async context processor that skips htmx fragments, logged-out pages and the
database-down page, and renders without the count if the query fails.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-06 18:12:50 -04:00
bvandeusenandClaude Opus 5.5 dc22a35e9d feat(alerts): alert history page — transitions, deliveries, mutes and actions (#5066)
CI / lint (push) Successful in 3s
CI / integration (push) Successful in 50s
CI / unit (push) Successful in 1m7s
CI / publish (push) Successful in 1m4s
#5066 "Alert history page". /alerts/history lists every alert transition
in a range (1h–30d, the shared range picker), newest first, all rules or
one (?rule=). Each row: from → to, the value against the rule's condition
("no data" for a silent one), email and webhook sent or failed with the
error, "muted · <window>" (and "notified when the window ended") taken out
of the event note, other notes, and a link to the Ansible run an action
started. Range and rule changes swap only the rows (HTMX). Viewer role.

The Alerts page links each rule's history and the whole history. The query
moves to alert_svc.history, which MCP get_alert_history now uses too.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-06 14:06:04 -04:00
bvandeusenandClaude Opus 5.5 c368e93e2c feat(alerts): silence is alertable — agent offline and no-data (#5062)
CI / lint (push) Successful in 2s
CI / unit (push) Failing after 1m14s
CI / integration (push) Successful in 1m14s
CI / publish (push) Skipped
Agent offline is an ordinary rule now. The host_agent scheduler records
host_agent/<host>/seconds_since_seen every minute (from last_seen_at, or
registration time for an agent that never reported), replacing the task
that only logged stale agents. The catalog offers the metric, and picking
it on a new rule fills "> 180".

No data: each rule's state tracks its last sample and its usual gap,
smoothed so that an outage counts as at most three gaps. A core task,
alert_silence_check, runs every 60s and marks a rule no_data_since once its
series has had no sample for three usual gaps (at least 5 minutes; an hour
while the cadence is unknown). A never-sampled rule counts from when it
started watching. The mark sits beside the state, so a firing alert that
goes silent stays firing. Each silence records one event and sends one
notification (rule option notify_no_data, default on; muted in maintenance
windows). The next sample clears it with a "data resumed" event and
notification. Creating, changing, disabling or re-enabling a rule restarts
its watch.

Migration 0029 adds the columns and seeds each existing rule's last sample
and gap from its last day of data, so an upgrade does not flag every rule.

Shown on the Alerts page (a NO DATA badge with its start time), in MCP
list_alerts, list_alert_rules and get_host (state "no_data", no_data_since),
in get_overview findings (alert_no_data, or a firing alert marked silent)
and in the weekly report. The rule form has the notify option, and its
metric reference now matches what traefik records and lists host_agent.
Emails say "no data" and carry the event's note.

The metric-catalog guard also reads positional MetricBatch.add arguments.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-06 13:06:54 -04:00
bvandeusenandClaude Opus 5.5 0742b7a2b7 feat(alerts): the metric catalog is written by what emits it (#5061)
#5061 "Metric catalog belongs to what emits: plugins register their
metrics; names match what is recorded".

- The static METRIC_CATALOG is gone. metric_catalog() is core's own sources
  (the four monitor types and "steward", the app's open_fds/open_fds_pct,
  which were recorded but never offered) plus each loaded plugin's
  get_alert_metrics() export. A disabled plugin's sources leave the picker.
- traefik offers what it records: error_rate_4xx_pct and error_rate_5xx_pct.
  The old "error_rate" was never recorded, so a rule on it could not fire.
- tests/core/test_metric_catalog.py walks every MetricBatch.add /
  record_metric call (traefik's tuple loop included) and fails on a recorded
  metric the catalog lacks; it shows on a synthetic writer that it can fail.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-06 12:55:55 -04:00
bvandeusenandClaude Opus 5.5 58bfc7c361 feat(alerts): one evaluation path; host-agent ingest fires rules (#5060)
CI / lint (push) Successful in 2s
CI / integration (push) Failing after 45s
CI / unit (push) Successful in 1m7s
CI / publish (push) Skipped
#5060 "One evaluation path: record_metrics batch API; host-agent ingest
evaluates rules".

- record_metrics(session, rows) writes metric rows and evaluates the rules
  that watch them, reading rules, states and maintenance windows once per
  batch. Each rule sees its key's samples oldest first, so an agent catching
  up after an outage counts consecutive checks as they happened.
- Host-agent ingest now goes through it. Before, it wrote plugin_metrics
  directly and no host_agent rule could ever fire. The catalog gains the
  per-mount disk_used_pct ("<host>:<mount>").
- MetricBatch gathers a transaction's samples; the docker ingest, traefik,
  unifi, snmp, monitor and self-monitor writers record one batch each instead
  of a rule query per sample.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-06 12:53:40 -04:00
bvandeusenandClaude Opus 5.5 a6c24299a2 fix(alerts): acknowledgements hold until resolve; clear after N OK; rule changes reset (#5059)
CI / lint (push) Successful in 2s
CI / integration (push) Successful in 45s
CI / unit (push) Successful in 1m8s
CI / publish (push) Successful in 1m1s
#5059 "Alert state machine: acknowledgement holds until resolve; N
consecutive OK to resolve; editing, disabling or deleting a rule resets it".

- The state machine is now next_step(), a pure function of one sample. An
  acknowledged alert stays acknowledged while breached; before, the next
  breaching sample re-fired it. Firing and acknowledged alerts resolve after
  consecutive_ok_required OK checks in a row (new rule column, default 1;
  rule form, rules list and MCP create/update_alert_rule clear_after_checks).
- Changing what a rule watches (source, resource, metric, operator,
  threshold) or disabling it resets its alert to inactive; a firing or
  acknowledged one records a resolved event whose new note says why, and
  nobody is notified. Web and MCP edits share alerts.service.apply_rule_change.
- Migration 0028: consecutive_ok_required, consecutive_ok_count,
  alert_events.note, and metric_value nullable (a reset has no sample).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-06 11:32:17 -04:00
bvandeusenandClaude Opus 5.5 9abe0beecc feat(mcp): monitoring-config write tools — monitors, alert rules, acks, maintenance
CI / lint (push) Successful in 2s
CI / integration (push) Successful in 41s
CI / unit (push) Successful in 1m0s
CI / publish (push) Successful in 58s
MCP-7 #4971 "Monitoring-config write tools: monitors, alert rules, acks,
maintenance".

The validation the forms did inline (or not at all) moves to the services,
and both the routes and the tools call it, so bad input is refused the same
way with the same reason:
- monitors/service: monitor_fields, build_config, normalise_target. Port
  1-65535, status 100-599, timeout 1-60, method GET/HEAD/POST, headers an
  object of strings, a key the type does not take is named. The forms now
  answer 400 with the reason instead of redirecting silently or a 500.
- alerts/service: METRIC_CATALOG and SOURCE_MODULES, resources_for,
  metrics_for, rule_fields (source, resource and metric from the same
  choices the rule form offers), add_rule, acknowledge, window_fields.

Tools: create_monitor, update_monitor (config merged over the stored one;
a rename warns about rules still watching the old name),
set_monitor_enabled, create_alert_rule, update_alert_rule,
acknowledge_alert, create_maintenance_window (duration or end),
end_maintenance_window (ends one in progress, cancels an upcoming one) —
operator. delete_monitor, delete_alert_rule — admin. get_alert_rule_options
— viewer. Every write is audited as "<user> via MCP".

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-06 08:19:54 -04:00
bvandeusenandClaude Opus 5.5 312ea95bcf refactor: move host, docker, alert and monitor queries into service modules
CI / lint (push) Successful in 2s
CI / integration (push) Successful in 35s
CI / unit (push) Successful in 59s
CI / publish (push) Successful in 56s
MCP-3 of milestone 461 "MCP server: Steward as an LLM-inspectable,
LLM-operable hub". The /mcp tools run outside Quart's request pipeline, so
the queries they share with the pages now live in plain functions that take
an open session:

- steward/hosts/service.py: uptime, status, monitors, agent metrics,
  agent overview (stale threshold passed in, not read from current_app),
  CPU sparks.
- plugins/docker/service.py: bucket, container_history, host_map,
  list_containers, container_events, query_logs.
- steward/alerts/service.py: active_alerts (also used by core/reports),
  list_rules, list_maintenance_windows.
- steward/monitors/service.py: monitor_rows (was routes._rows_data).

Routes are repointed and render the same data. One behavior fix rides
along: the container-log search now escapes % and _ so they match
literally instead of acting as LIKE wildcards.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-06 07:29:21 -04:00