FabledSteward/docs/core/alerting.md

Alerting

Alert rules evaluate every metric that flows through record_metric(). There is no separate polling process — evaluation is inline with every metric write.

---

How It Works

When any monitor or plugin calls record_metric():

1. The metric value is written to the plugin_metrics table, and plugin_metric_latest moves to it if it is the newest sample of that metric
2. All enabled AlertRule rows matching (source_module, resource_name, metric_name) are loaded
3. Each matching rule is evaluated against the new value
4. If a state transition occurs, an AlertEvent row is written
5. Notification I/O is deferred outside the transaction via asyncio.create_task()

Key function: steward/core/alerts.py → record_metric(session, source_module, resource_name, metric_name, value)

record_metric() must always be called inside an active transaction:

async with session.begin():
    await record_metric(session, "ping", "my-server", "response_time_ms", 42.3)

Notifications are sent via asyncio.create_task() after the function returns, so no network I/O blocks the DB transaction.

---

Alert State Machine

Each AlertRule has one associated AlertState row. The state transitions are:

inactive ──(breached)──► pending ──(N breaching checks in a row)──► FIRING
    ▲                      │                                          │
    └──(an OK check)───────┘                                          │
                                 FIRING ──(acknowledged)──► ACKNOWLEDGED
                                 FIRING / ACKNOWLEDGED ──(M OK checks in a row)──► RESOLVED ──► inactive

N is the rule's consecutive checks, M its checks to clear. An acknowledgement holds for as long as the breach lasts: only a resolve clears it, and a breaching check never sends an acknowledged alert back to firing.

RESOLVED is transient — the evaluator writes the event and immediately sets state back to inactive within the same transaction. RESOLVED never persists as a final state in the DB.

Every transition is an AlertEvent, the history the Alerts → History page and MCP get_alert_history read. A transition a person made carries them in actor_id: an acknowledgement (firing → acknowledged), or a rule edit, disable or enable that reset the alert (→ resolved, with a note such as "rule disabled"). Transitions caused by readings, or by Steward itself, have none.

State Definitions

State	Meaning
inactive	Threshold not breached
pending	Threshold breached but consecutive count not yet met; no notification sent
firing	Consecutive count met; FIRING notification sent
acknowledged	Someone has seen it; no further notifications; clears when the readings recover
resolved	Transient — notification sent, immediately transitions to inactive

---

Creating Alert Rules

Alert rules are created in the UI at /alerts/. Each rule requires:

• Name — human-readable label
• Source module — ping, dns, traefik, or any plugin name
• Resource name — host name, router name, etc. (must match exactly what the monitor writes)
• Metric name — the metric key (e.g. response_time_ms, up, error_rate_5xx_pct)
• Operator — >, <, >=, <=, ==, !=
• Threshold — numeric value
• Consecutive failures required — how many consecutive breaches before FIRING (default 1)

Rules that cover hosts

A rule on host_agent can cover every host, or every host in an inventory group, instead of naming one resource (#5464). Inventory groups are Steward's grouping system: a host is in a group through its linked Ansible target.

A covering rule has no state of its own. Steward keeps one ordinary rule per covered host under it, named <rule>: <host>, so each host's alert keeps its own state, history and acknowledgement. Each minute (the alert_reconcile task), and whenever the rule is saved:

• a host that has reported in the last 7 days, and is covered, gets a rule;
• a host that is no longer covered loses its rule, and that rule's history;
• the hosts' rules take the covering rule's condition, checks, notification setting, action and on/off.

A host's rule is changed through its covering rule; editing, disabling or deleting it alone is refused. What a rule covers is fixed once it exists, but a group rule can move to another group. Over MCP, create_alert_rule takes covers ("all_hosts" or "group") and group, and list_alert_rules shows each covering rule's hosts and each host's rule's part_of.

Available Metrics by Module

source_module	metric_name	Description
ping	response_time_ms	Probe latency (0.0 if down)
ping	up	1.0 = up, 0.0 = down
dns	resolved	1.0 = resolved, 0.0 = failed
dns	ip_changed	1.0 = IP changed from previous result
traefik	request_rate	Requests per second
traefik	error_rate_4xx_pct	4xx errors as % of requests
traefik	error_rate_5xx_pct	5xx errors as % of requests
traefik	latency_p50_ms	Approximate p50 latency (ms)
traefik	latency_p95_ms	Approximate p95 latency (ms)
traefik	latency_p99_ms	Approximate p99 latency (ms)

---

Notifications

Notifications fire on FIRING and RESOLVED transitions. All configured channels receive every notification; there is no per-rule channel routing.

Email

Configured at /settings/ under SMTP. Settings:

Setting key	Description
smtp.host	SMTP server
smtp.port	Port (default 587)
smtp.tls	STARTTLS (default true)
smtp.username	Login
smtp.password	Password
smtp.recipients	List of email addresses

Email is skipped if smtp.host is empty.

Alerts that arrive together go out as one email (core/alert_burst.py). The first alert opens a 30-second window. When it closes, each recipient gets one message listing every alert in the window, oldest first, in their own zone, with a subject like "[Steward] 3 alerts: 2 firing, 1 resolved". One alert on its own keeps its usual email. Every alert's event records whether the email went. A burst still waiting when Steward stops is not sent; its alerts stay on the Alerts page and in the history.

Webhook

Configured at /settings/ under Webhook. The body is a Jinja2 template rendered to JSON. Content-Type is always application/json. If the rendered template is not valid JSON, the delivery is logged as failed and no request is sent.

Template variable	Type	Description
alert.rule_name	str	Alert rule name
alert.state	str	FIRING or RESOLVED
alert.metric	str	Metric name
alert.value	float	Current value
alert.threshold	float	Configured threshold
alert.resource	str	Resource name
alert.source_module	str	ping, dns, traefik, etc.
alert.timestamp	str	ISO 8601 UTC

Default template (Discord-compatible):

{"content": "**{{ alert.state }}** — {{ alert.resource }} — {{ alert.rule_name }} ({{ alert.metric }} = {{ alert.value }})"}

Webhook is skipped if webhook.url is empty. Webhooks are not gathered into bursts: each alert posts on its own, at once.

Send test

Send test on a rule's page (operator role), or send_test_alert over MCP, sends a TEST alert for that rule now, by email and webhook, the same way a real alert would go. It uses the rule's name and condition and the latest reading of what the rule watches. A rule that covers hosts uses its first host. Each channel's result shows under the button. Nothing fires, the rule's history doesn't change, and the test is recorded in the audit log as alert_rule.tested.

---

Data Models

Defined in steward/models/alerts.py:

• alert_rules — one row per configured rule
• alert_states — one row per rule, tracks current state and consecutive failure count
• alert_events — append-only log of all state transitions and notification outcomes
• plugin_metrics — all metric values written by any monitor or plugin
• plugin_metric_latest — the newest sample of each (source, resource, metric). Pages that show "what it is now" read this instead of searching history; retention drops a row once its sample leaves the raw window