FabledSteward/docs/core/alerting.md

# Alerting

Alert rules evaluate every metric that flows through `record_metric()`. There is no separate polling process — evaluation is inline with every metric write.

---

## How It Works

When any monitor or plugin calls `record_metric()`:

1. The metric value is written to the `plugin_metrics` table, and `plugin_metric_latest` moves to it if it is the newest sample of that metric
2. All enabled `AlertRule` rows matching `(source_module, resource_name, metric_name)` are loaded
3. Each matching rule is evaluated against the new value
4. If a state transition occurs, an `AlertEvent` row is written
5. Notification I/O is deferred outside the transaction via `asyncio.create_task()`

**Key function:** `steward/core/alerts.py` → `record_metric(session, source_module, resource_name, metric_name, value)`

`record_metric()` must always be called inside an active transaction:

```python
async with session.begin():
    await record_metric(session, "ping", "my-server", "response_time_ms", 42.3)
```

Notifications are sent via `asyncio.create_task()` after the function returns, so no network I/O blocks the DB transaction.

---

## Alert State Machine

Each `AlertRule` has one associated `AlertState` row. The state transitions are:

```
inactive ──(breached)──► pending ──(N breaching checks in a row)──► FIRING
    ▲                      │                                          │
    └──(an OK check)───────┘                                          │
                                 FIRING ──(acknowledged)──► ACKNOWLEDGED
                                 FIRING / ACKNOWLEDGED ──(M OK checks in a row)──► RESOLVED ──► inactive
```

N is the rule's consecutive checks, M its checks to clear. An acknowledgement
holds for as long as the breach lasts: only a resolve clears it, and a breaching
check never sends an acknowledged alert back to firing.

`RESOLVED` is transient — the evaluator writes the event and immediately sets state back to `inactive` within the same transaction. `RESOLVED` never persists as a final state in the DB.

Every transition is an `AlertEvent`, the history the Alerts → History page and
MCP `get_alert_history` read. A transition a person made carries them in
`actor_id`: an acknowledgement (firing → acknowledged), or a rule edit, disable
or enable that reset the alert (→ resolved, with a note such as "rule
disabled"). Transitions caused by readings, or by Steward itself, have none.

### State Definitions

| State | Meaning |
|---|---|
| `inactive` | Threshold not breached |
| `pending` | Threshold breached but consecutive count not yet met; no notification sent |
| `firing` | Consecutive count met; FIRING notification sent |
| `acknowledged` | Someone has seen it; no further notifications; clears when the readings recover |
| `resolved` | Transient — notification sent, immediately transitions to `inactive` |

---

## Creating Alert Rules

Alert rules are created in the UI at `/alerts/`. Each rule requires:

- **Name** — human-readable label
- **Source module** — `ping`, `dns`, `traefik`, or any plugin name
- **Resource name** — host name, router name, etc. (must match exactly what the monitor writes)
- **Metric name** — the metric key (e.g. `response_time_ms`, `up`, `error_rate_5xx_pct`)
- **Operator** — `>`, `<`, `>=`, `<=`, `==`, `!=`
- **Threshold** — numeric value
- **Consecutive failures required** — how many consecutive breaches before FIRING (default 1)

The form offers only what has reported, read from `plugin_metric_latest`.
For the host agent, a **Host** is picked first, then a **Part** of it: "The
whole host", then its mounts, CPU cores, disks and so on, under a heading for
each kind. A part that reports nothing a rule can watch, such as an interface's
traffic, isn't listed. The **Metric** list holds only what the picked resource
reports. Under it, a preview shows the metric's current reading and its last
24 hours (lowest to highest, with an hourly sparkline; it loads just after the
page, since a day of samples can take a while to read), so the threshold can be
set against real numbers.

### Rules that cover hosts

A rule on `host_agent` can cover **every host**, or **every host in an
inventory group**, instead of naming one resource (#5464). Inventory groups
are Steward's grouping system: a host is in a group through its linked
Ansible target.

A covering rule has no state of its own. Steward keeps one ordinary rule per
covered host under it, named `<rule>: <host>`, so each host's alert keeps its
own state, history and acknowledgement. Each minute (the `alert_reconcile`
task), and whenever the rule is saved:

- a host that has reported in the last 7 days, and is covered, gets a rule;
- a host that is no longer covered loses its rule, and that rule's history;
- the hosts' rules take the covering rule's condition, checks, notification
  setting, action and on/off.

A host's rule is changed through its covering rule; editing, disabling or
deleting it alone is refused. What a rule covers is fixed once it exists, but
a group rule can move to another group. Over MCP, `create_alert_rule` takes
`covers` ("all_hosts" or "group") and `group`, and `list_alert_rules` shows
each covering rule's `hosts` and each host's rule's `part_of`.

### Available Metrics by Module

| `source_module` | `metric_name` | Description |
|---|---|---|
| `ping` | `response_time_ms` | Probe latency (0.0 if down) |
| `ping` | `up` | 1.0 = up, 0.0 = down |
| `dns` | `resolved` | 1.0 = resolved, 0.0 = failed |
| `dns` | `ip_changed` | 1.0 = IP changed from previous result |
| `traefik` | `request_rate` | Requests per second |
| `traefik` | `error_rate_4xx_pct` | 4xx errors as % of requests |
| `traefik` | `error_rate_5xx_pct` | 5xx errors as % of requests |
| `traefik` | `latency_p50_ms` | Approximate p50 latency (ms) |
| `traefik` | `latency_p95_ms` | Approximate p95 latency (ms) |
| `traefik` | `latency_p99_ms` | Approximate p99 latency (ms) |

#### Steward itself

Source `steward`, resource `process`, recorded once a minute by the
self-monitor tick (`steward/core/self_monitor.py`; the ingest tally is in
`steward/core/health.py`). Settings → System (`/settings/system/`, admin)
shows each one now and over the last day, with a verdict above that names
failed agent posts, a gap in watching, or open files near the limit.

| `metric_name` | Description |
|---|---|
| `ingest_posts` | Host-agent posts Steward took in the last minute |
| `ingest_failed` | Posts in the last minute that didn't land (handler raised, or the agent gave up) |
| `ingest_slowest_ms` | The slowest post in the last minute; agents give up at 10 s |
| `db_roundtrip_ms` | A `SELECT 1`, connection checkout included |
| `unwatched_secs` | The gap before this tick when it exceeds `watch.GAP_S` (a deploy or an outage); 0 when unbroken |
| `restart_gap_secs` | Recorded once, on a new process's first tick: the time since the last process's last sample, however short |
| `cleanup_secs` | How long the last hourly retention and rollup run took |
| `open_fds` / `open_fds_pct` | Steward's open files, and as a share of its limit (absent without `/proc`) |

---

## Notifications

Notifications fire on `FIRING` and `RESOLVED` transitions. All configured channels receive every notification; there is no per-rule channel routing.

### Email

Configured at `/settings/` under SMTP. Settings:

| Setting key | Description |
|---|---|
| `smtp.host` | SMTP server |
| `smtp.port` | Port (default 587) |
| `smtp.tls` | STARTTLS (default true) |
| `smtp.username` | Login |
| `smtp.password` | Password |
| `smtp.recipients` | List of email addresses |

Email is skipped if `smtp.host` is empty.

Alerts that arrive together go out as one email. The first alert opens a
2-minute window. When it closes, each recipient gets one message listing every
alert in the window, oldest first, in their own zone, with a subject like
"[Steward] 3 alerts: 2 firing, 1 resolved". One alert on its own keeps its
usual email.

Every email Steward sends (alerts, the weekly report, invites, password
resets and the test buttons) waits in the database (`email_outbox`,
`core/outbox.py`) until the mail server accepts it, so nothing waiting is lost
when Steward restarts: the sender picks it up at boot. The sender runs every 15
seconds and settles each email on the server's answer:

- **Accepted:** sent.
- **Refused** (an SMTP 5xx, such as "no such mailbox"): failed at once, with
  the server's words. Trying again would only repeat the refusal.
- **Temporary** (no connection, a timeout, a 4xx): tried again after 30 s,
  1, 2, 5, 10 and then every 15 minutes. An email the server still hasn't
  taken an hour after its first try is failed with the last error.

Once every email carrying an alert has settled, that alert's event records
whether the email went, and to whom it didn't.

Where to see it: each alert's row in **Alerts → History** says what its email
is doing (gathering, with when it sends; retrying, with the attempt, the next
try and the server's last answer; sent; failed, with who wasn't reached and
why), and `get_alert_history` over MCP reports the same. **Settings → System →
Outgoing email** lists every email of the last 7 days not yet sent, failures
first, and the page's verdict names failures and any email still retrying
after 5 minutes.

An email a person is waiting on is tried at once instead of on the next tick:
the weekly report's **Send now**, an invite, and the test buttons, so the page
can say what the server answered. A report or invite the server couldn't take
yet stays queued and is retried as above. A test (the Settings → Notifications
test email, a rule's **Send test**) is recorded but never retried: its one
answer is the result.

### Webhook

Configured at `/settings/` under Webhook. The body is a Jinja2 template rendered to JSON. Content-Type is always `application/json`. If the rendered template is not valid JSON, the delivery is logged as failed and no request is sent.

| Template variable | Type | Description |
|---|---|---|
| `alert.rule_name` | str | Alert rule name |
| `alert.state` | str | `FIRING` or `RESOLVED` |
| `alert.metric` | str | Metric name |
| `alert.value` | float | Current value |
| `alert.threshold` | float | Configured threshold |
| `alert.resource` | str | Resource name |
| `alert.source_module` | str | `ping`, `dns`, `traefik`, etc. |
| `alert.timestamp` | str | ISO 8601 UTC |

Default template (Discord-compatible):
```json
{"content": "**{{ alert.state }}** — {{ alert.resource }} — {{ alert.rule_name }} ({{ alert.metric }} = {{ alert.value }})"}
```

Webhook is skipped if `webhook.url` is empty. Webhooks are not gathered into
bursts: each alert posts on its own, at once.

### Send test

**Send test** at the top of a rule's page (operator role), or `send_test_alert` over MCP,
sends a `TEST` alert for that rule now, by email and webhook, the same way a
real alert would go. It uses the rule's name and condition and the latest
reading of what the rule watches. A rule that covers hosts uses its first
host. Each channel's result shows under the button. Nothing fires, the rule's
history doesn't change, and the test is recorded in the audit log as
`alert_rule.tested`.

### On the dashboard

The **Alerts** widget (`/alerts/widget`) sits beside the Overview in the
recommended layout. It opens with a verdict ("2 firing, 1 silent." or
"Nothing firing: all 12 rules are quiet."). Below that it lists what is
firing or silent: each rule, what it watches, its reading now against what
it fires on, and since when. Acknowledged alerts follow, with who
acknowledged them and when. A line appears when alert email has failed in
the last 7 days or is still retrying, linking admins to Settings → System.
The quiet rules are counted at the end.

---

## Data Models

Defined in `steward/models/alerts.py`:

- **`alert_rules`** — one row per configured rule
- **`alert_states`** — one row per rule, tracks current state and consecutive failure count
- **`alert_events`** — append-only log of all state transitions and notification outcomes
- **`plugin_metrics`** — all metric values written by any monitor or plugin. Its series index carries the value (`ix_plugin_metrics_series_value`), so a day of one metric is read from the index alone even when nothing is cached; Steward builds it in the background after startup (`core/metric_index.py`), never during a migration
- **`plugin_metric_latest`** — the newest sample of each (source, resource, metric). Pages that show "what it is now" read this instead of searching history; retention drops a row once its sample leaves the raw window