bvandeusenandClaude Opus 5.5
48a0de5600
#5464 "Group rules: one rule covers every host, or every host in a group", step 2 of milestone 499 "Alerts you can trust". Each rule named one host, so the live instance carried six rules for three hosts, and a new host had no agent-offline or disk alert until someone copied them. - alert_rules.covers / cover_group / parent_id (migration 0041). A rule on host_agent may cover "all_hosts" or a "group" (an inventory group: the Hosts ruling, #5294). A covering rule has no state; Steward keeps one ordinary child rule per covered host, named "<rule>: <host>", so each host's alert keeps its own state, history and acknowledgement. - alerts.service.reconcile_children: a covered host that has reported in the last 7 days gets a child; a host no longer covered loses its child; children take the parent's condition, checks, notify setting, action and on/off. Run on save and each minute (alert_reconcile). - A child is changed through its parent: edit, toggle and delete of a child are refused (409 / PartOfCoveringRule); its edit page opens the parent. - Rule form: a Covers choice for host sources, with the group picker; fixed once the rule exists. Rules list: a covering rule reads "every host in swarm · 3 hosts" with its hosts' rules nested under it, each opening its history. The history picker and the rule count leave covering rules out. - MCP: create_alert_rule takes covers/group; update_alert_rule takes group and passes changes down; list_alert_rules shows covers, hosts and part_of. - docs/core/alerting.md: "Rules that cover hosts". Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
170 lines
7.3 KiB
Markdown
170 lines
7.3 KiB
Markdown
# Alerting
|
|
|
|
Alert rules evaluate every metric that flows through `record_metric()`. There is no separate polling process — evaluation is inline with every metric write.
|
|
|
|
---
|
|
|
|
## How It Works
|
|
|
|
When any monitor or plugin calls `record_metric()`:
|
|
|
|
1. The metric value is written to the `plugin_metrics` table, and `plugin_metric_latest` moves to it if it is the newest sample of that metric
|
|
2. All enabled `AlertRule` rows matching `(source_module, resource_name, metric_name)` are loaded
|
|
3. Each matching rule is evaluated against the new value
|
|
4. If a state transition occurs, an `AlertEvent` row is written
|
|
5. Notification I/O is deferred outside the transaction via `asyncio.create_task()`
|
|
|
|
**Key function:** `steward/core/alerts.py` → `record_metric(session, source_module, resource_name, metric_name, value)`
|
|
|
|
`record_metric()` must always be called inside an active transaction:
|
|
|
|
```python
|
|
async with session.begin():
|
|
await record_metric(session, "ping", "my-server", "response_time_ms", 42.3)
|
|
```
|
|
|
|
Notifications are sent via `asyncio.create_task()` after the function returns, so no network I/O blocks the DB transaction.
|
|
|
|
---
|
|
|
|
## Alert State Machine
|
|
|
|
Each `AlertRule` has one associated `AlertState` row. The state transitions are:
|
|
|
|
```
|
|
inactive ──(breached)──► pending ──(N breaching checks in a row)──► FIRING
|
|
▲ │ │
|
|
└──(an OK check)───────┘ │
|
|
FIRING ──(acknowledged)──► ACKNOWLEDGED
|
|
FIRING / ACKNOWLEDGED ──(M OK checks in a row)──► RESOLVED ──► inactive
|
|
```
|
|
|
|
N is the rule's consecutive checks, M its checks to clear. An acknowledgement
|
|
holds for as long as the breach lasts: only a resolve clears it, and a breaching
|
|
check never sends an acknowledged alert back to firing.
|
|
|
|
`RESOLVED` is transient — the evaluator writes the event and immediately sets state back to `inactive` within the same transaction. `RESOLVED` never persists as a final state in the DB.
|
|
|
|
Every transition is an `AlertEvent`, the history the Alerts → History page and
|
|
MCP `get_alert_history` read. A transition a person made carries them in
|
|
`actor_id`: an acknowledgement (firing → acknowledged), or a rule edit, disable
|
|
or enable that reset the alert (→ resolved, with a note such as "rule
|
|
disabled"). Transitions caused by readings, or by Steward itself, have none.
|
|
|
|
### State Definitions
|
|
|
|
| State | Meaning |
|
|
|---|---|
|
|
| `inactive` | Threshold not breached |
|
|
| `pending` | Threshold breached but consecutive count not yet met; no notification sent |
|
|
| `firing` | Consecutive count met; FIRING notification sent |
|
|
| `acknowledged` | Someone has seen it; no further notifications; clears when the readings recover |
|
|
| `resolved` | Transient — notification sent, immediately transitions to `inactive` |
|
|
|
|
---
|
|
|
|
## Creating Alert Rules
|
|
|
|
Alert rules are created in the UI at `/alerts/`. Each rule requires:
|
|
|
|
- **Name** — human-readable label
|
|
- **Source module** — `ping`, `dns`, `traefik`, or any plugin name
|
|
- **Resource name** — host name, router name, etc. (must match exactly what the monitor writes)
|
|
- **Metric name** — the metric key (e.g. `response_time_ms`, `up`, `error_rate_5xx_pct`)
|
|
- **Operator** — `>`, `<`, `>=`, `<=`, `==`, `!=`
|
|
- **Threshold** — numeric value
|
|
- **Consecutive failures required** — how many consecutive breaches before FIRING (default 1)
|
|
|
|
### Rules that cover hosts
|
|
|
|
A rule on `host_agent` can cover **every host**, or **every host in an
|
|
inventory group**, instead of naming one resource (#5464). Inventory groups
|
|
are Steward's grouping system: a host is in a group through its linked
|
|
Ansible target.
|
|
|
|
A covering rule has no state of its own. Steward keeps one ordinary rule per
|
|
covered host under it, named `<rule>: <host>`, so each host's alert keeps its
|
|
own state, history and acknowledgement. Each minute (the `alert_reconcile`
|
|
task), and whenever the rule is saved:
|
|
|
|
- a host that has reported in the last 7 days, and is covered, gets a rule;
|
|
- a host that is no longer covered loses its rule, and that rule's history;
|
|
- the hosts' rules take the covering rule's condition, checks, notification
|
|
setting, action and on/off.
|
|
|
|
A host's rule is changed through its covering rule; editing, disabling or
|
|
deleting it alone is refused. What a rule covers is fixed once it exists, but
|
|
a group rule can move to another group. Over MCP, `create_alert_rule` takes
|
|
`covers` ("all_hosts" or "group") and `group`, and `list_alert_rules` shows
|
|
each covering rule's `hosts` and each host's rule's `part_of`.
|
|
|
|
### Available Metrics by Module
|
|
|
|
| `source_module` | `metric_name` | Description |
|
|
|---|---|---|
|
|
| `ping` | `response_time_ms` | Probe latency (0.0 if down) |
|
|
| `ping` | `up` | 1.0 = up, 0.0 = down |
|
|
| `dns` | `resolved` | 1.0 = resolved, 0.0 = failed |
|
|
| `dns` | `ip_changed` | 1.0 = IP changed from previous result |
|
|
| `traefik` | `request_rate` | Requests per second |
|
|
| `traefik` | `error_rate_4xx_pct` | 4xx errors as % of requests |
|
|
| `traefik` | `error_rate_5xx_pct` | 5xx errors as % of requests |
|
|
| `traefik` | `latency_p50_ms` | Approximate p50 latency (ms) |
|
|
| `traefik` | `latency_p95_ms` | Approximate p95 latency (ms) |
|
|
| `traefik` | `latency_p99_ms` | Approximate p99 latency (ms) |
|
|
|
|
---
|
|
|
|
## Notifications
|
|
|
|
Notifications fire on `FIRING` and `RESOLVED` transitions. All configured channels receive every notification; there is no per-rule channel routing.
|
|
|
|
### Email
|
|
|
|
Configured at `/settings/` under SMTP. Settings:
|
|
|
|
| Setting key | Description |
|
|
|---|---|
|
|
| `smtp.host` | SMTP server |
|
|
| `smtp.port` | Port (default 587) |
|
|
| `smtp.tls` | STARTTLS (default true) |
|
|
| `smtp.username` | Login |
|
|
| `smtp.password` | Password |
|
|
| `smtp.recipients` | List of email addresses |
|
|
|
|
Email is skipped if `smtp.host` is empty.
|
|
|
|
### Webhook
|
|
|
|
Configured at `/settings/` under Webhook. The body is a Jinja2 template rendered to JSON. Content-Type is always `application/json`. If the rendered template is not valid JSON, the delivery is logged as failed and no request is sent.
|
|
|
|
| Template variable | Type | Description |
|
|
|---|---|---|
|
|
| `alert.rule_name` | str | Alert rule name |
|
|
| `alert.state` | str | `FIRING` or `RESOLVED` |
|
|
| `alert.metric` | str | Metric name |
|
|
| `alert.value` | float | Current value |
|
|
| `alert.threshold` | float | Configured threshold |
|
|
| `alert.resource` | str | Resource name |
|
|
| `alert.source_module` | str | `ping`, `dns`, `traefik`, etc. |
|
|
| `alert.timestamp` | str | ISO 8601 UTC |
|
|
|
|
Default template (Discord-compatible):
|
|
```json
|
|
{"content": "**{{ alert.state }}** — {{ alert.resource }} — {{ alert.rule_name }} ({{ alert.metric }} = {{ alert.value }})"}
|
|
```
|
|
|
|
Webhook is skipped if `webhook.url` is empty.
|
|
|
|
---
|
|
|
|
## Data Models
|
|
|
|
Defined in `steward/models/alerts.py`:
|
|
|
|
- **`alert_rules`** — one row per configured rule
|
|
- **`alert_states`** — one row per rule, tracks current state and consecutive failure count
|
|
- **`alert_events`** — append-only log of all state transitions and notification outcomes
|
|
- **`plugin_metrics`** — all metric values written by any monitor or plugin
|
|
- **`plugin_metric_latest`** — the newest sample of each (source, resource, metric). Pages that show "what it is now" read this instead of searching history; retention drops a row once its sample leaves the raw window
|