FabledSteward/docs/core/alerting.md

Alerting

Alert rules evaluate every metric that flows through record_metric(). There is no separate polling process — evaluation is inline with every metric write.

---

How It Works

When any monitor or plugin calls record_metric():

1. The metric value is written to the plugin_metrics table, and plugin_metric_latest moves to it if it is the newest sample of that metric
2. All enabled AlertRule rows matching (source_module, resource_name, metric_name) are loaded
3. Each matching rule is evaluated against the new value
4. If a state transition occurs, an AlertEvent row is written
5. Notification I/O is deferred outside the transaction via asyncio.create_task()

Key function: steward/core/alerts.py → record_metric(session, source_module, resource_name, metric_name, value)

record_metric() must always be called inside an active transaction:

async with session.begin():
    await record_metric(session, "ping", "my-server", "response_time_ms", 42.3)

Notifications are sent via asyncio.create_task() after the function returns, so no network I/O blocks the DB transaction.

---

Alert State Machine

Each AlertRule has one associated AlertState row. The state transitions are:

inactive ──(breached)──► pending ──(N breaching checks in a row)──► FIRING
    ▲                      │                                          │
    └──(an OK check)───────┘                                          │
                                 FIRING ──(acknowledged)──► ACKNOWLEDGED
                                 FIRING / ACKNOWLEDGED ──(M OK checks in a row)──► RESOLVED ──► inactive

N is the rule's consecutive checks, M its checks to clear. An acknowledgement holds for as long as the breach lasts: only a resolve clears it, and a breaching check never sends an acknowledged alert back to firing.

RESOLVED is transient — the evaluator writes the event and immediately sets state back to inactive within the same transaction. RESOLVED never persists as a final state in the DB.

Every transition is an AlertEvent, the history the Alerts → History page and MCP get_alert_history read. A transition a person made carries them in actor_id: an acknowledgement (firing → acknowledged), or a rule edit, disable or enable that reset the alert (→ resolved, with a note such as "rule disabled"). Transitions caused by readings, or by Steward itself, have none.

State Definitions

State	Meaning
inactive	Threshold not breached
pending	Threshold breached but consecutive count not yet met; no notification sent
firing	Consecutive count met; FIRING notification sent
acknowledged	Someone has seen it; no further notifications; clears when the readings recover
resolved	Transient — notification sent, immediately transitions to inactive

---

Creating Alert Rules

Alert rules are created in the UI at /alerts/. Each rule requires:

• Name — human-readable label
• Source module — ping, dns, traefik, or any plugin name
• Resource name — host name, router name, etc. (must match exactly what the monitor writes)
• Metric name — the metric key (e.g. response_time_ms, up, error_rate_5xx_pct)
• Operator — >, <, >=, <=, ==, !=
• Threshold — numeric value
• Consecutive failures required — how many consecutive breaches before FIRING (default 1)

The form offers only what has reported, read from plugin_metric_latest. For the host agent, a Host is picked first, then a Part of it: "The whole host", then its mounts, CPU cores, disks and so on, under a heading for each kind. A part that reports nothing a rule can watch, such as an interface's traffic, isn't listed. The Metric list holds only what the picked resource reports. Under it, a preview shows the metric's current reading and its last 24 hours (lowest to highest, with an hourly sparkline; it loads just after the page, since a day of samples can take a while to read), so the threshold can be set against real numbers.

Rules that cover hosts

A rule on host_agent can cover every host, or every host in an inventory group, instead of naming one resource (#5464). Inventory groups are Steward's grouping system: a host is in a group through its linked Ansible target.

A covering rule has no state of its own. Steward keeps one ordinary rule per covered host under it, named <rule>: <host>, so each host's alert keeps its own state, history and acknowledgement. Each minute (the alert_reconcile task), and whenever the rule is saved:

• a host that has reported in the last 7 days, and is covered, gets a rule;
• a host that is no longer covered loses its rule, and that rule's history;
• the hosts' rules take the covering rule's condition, checks, notification setting, action and on/off.

A host's rule is changed through its covering rule; editing, disabling or deleting it alone is refused. What a rule covers is fixed once it exists, but a group rule can move to another group. Over MCP, create_alert_rule takes covers ("all_hosts" or "group") and group, and list_alert_rules shows each covering rule's hosts and each host's rule's part_of.

Available Metrics by Module

source_module	metric_name	Description
ping	response_time_ms	Probe latency (0.0 if down)
ping	up	1.0 = up, 0.0 = down
dns	resolved	1.0 = resolved, 0.0 = failed
dns	ip_changed	1.0 = IP changed from previous result
traefik	request_rate	Requests per second
traefik	error_rate_4xx_pct	4xx errors as % of requests
traefik	error_rate_5xx_pct	5xx errors as % of requests
traefik	latency_p50_ms	Approximate p50 latency (ms)
traefik	latency_p95_ms	Approximate p95 latency (ms)
traefik	latency_p99_ms	Approximate p99 latency (ms)

Steward itself

Source steward, resource process, recorded once a minute by the self-monitor tick (steward/core/self_monitor.py; the ingest tally is in steward/core/health.py). Settings → System (/settings/system/, admin) shows each one now and over the last day, with a verdict above that names failed agent posts, a gap in watching, or open files near the limit.

metric_name	Description
ingest_posts	Host-agent posts Steward took in the last minute
ingest_failed	Posts in the last minute that didn't land (handler raised, or the agent gave up)
ingest_slowest_ms	The slowest post in the last minute; agents give up at 10 s
db_roundtrip_ms	A SELECT 1, connection checkout included
unwatched_secs	The gap before this tick when it exceeds watch.GAP_S (a deploy or an outage); 0 when unbroken
cleanup_secs	How long the last hourly retention and rollup run took
open_fds / open_fds_pct	Steward's open files, and as a share of its limit (absent without /proc)

---

Notifications

Notifications fire on FIRING and RESOLVED transitions. All configured channels receive every notification; there is no per-rule channel routing.

Email

Configured at /settings/ under SMTP. Settings:

Setting key	Description
smtp.host	SMTP server
smtp.port	Port (default 587)
smtp.tls	STARTTLS (default true)
smtp.username	Login
smtp.password	Password
smtp.recipients	List of email addresses

Email is skipped if smtp.host is empty.

Alerts that arrive together go out as one email. The first alert opens a 2-minute window. When it closes, each recipient gets one message listing every alert in the window, oldest first, in their own zone, with a subject like "[Steward] 3 alerts: 2 firing, 1 resolved". One alert on its own keeps its usual email.

Every email Steward sends (alerts, the weekly report, invites, password resets and the test buttons) waits in the database (email_outbox, core/outbox.py) until the mail server accepts it, so nothing waiting is lost when Steward restarts: the sender picks it up at boot. The sender runs every 15 seconds and settles each email on the server's answer:

• Accepted: sent.
• Refused (an SMTP 5xx, such as "no such mailbox"): failed at once, with the server's words. Trying again would only repeat the refusal.
• Temporary (no connection, a timeout, a 4xx): tried again after 30 s, 1, 2, 5, 10 and then every 15 minutes. An email the server still hasn't taken an hour after its first try is failed with the last error.

Once every email carrying an alert has settled, that alert's event records whether the email went, and to whom it didn't.

Where to see it: each alert's row in Alerts → History says what its email is doing (gathering, with when it sends; retrying, with the attempt, the next try and the server's last answer; sent; failed, with who wasn't reached and why), and get_alert_history over MCP reports the same. Settings → System → Outgoing email lists every email of the last 7 days not yet sent, failures first, and the page's verdict names failures and any email still retrying after 5 minutes.

An email a person is waiting on is tried at once instead of on the next tick: the weekly report's Send now, an invite, and the test buttons, so the page can say what the server answered. A report or invite the server couldn't take yet stays queued and is retried as above. A test (the Settings → Notifications test email, a rule's Send test) is recorded but never retried: its one answer is the result.

Webhook

Configured at /settings/ under Webhook. The body is a Jinja2 template rendered to JSON. Content-Type is always application/json. If the rendered template is not valid JSON, the delivery is logged as failed and no request is sent.

Template variable	Type	Description
alert.rule_name	str	Alert rule name
alert.state	str	FIRING or RESOLVED
alert.metric	str	Metric name
alert.value	float	Current value
alert.threshold	float	Configured threshold
alert.resource	str	Resource name
alert.source_module	str	ping, dns, traefik, etc.
alert.timestamp	str	ISO 8601 UTC

Default template (Discord-compatible):

{"content": "**{{ alert.state }}** — {{ alert.resource }} — {{ alert.rule_name }} ({{ alert.metric }} = {{ alert.value }})"}

Webhook is skipped if webhook.url is empty. Webhooks are not gathered into bursts: each alert posts on its own, at once.

Send test

Send test at the top of a rule's page (operator role), or send_test_alert over MCP, sends a TEST alert for that rule now, by email and webhook, the same way a real alert would go. It uses the rule's name and condition and the latest reading of what the rule watches. A rule that covers hosts uses its first host. Each channel's result shows under the button. Nothing fires, the rule's history doesn't change, and the test is recorded in the audit log as alert_rule.tested.

On the dashboard

The Alerts widget (/alerts/widget) sits beside the Overview in the recommended layout. It opens with a verdict ("2 firing, 1 silent." or "Nothing firing: all 12 rules are quiet."). Below that it lists what is firing or silent: each rule, what it watches, its reading now against what it fires on, and since when. Acknowledged alerts follow, with who acknowledged them and when. A line appears when alert email has failed in the last 7 days or is still retrying, linking admins to Settings → System. The quiet rules are counted at the end.

---

Data Models

Defined in steward/models/alerts.py:

• alert_rules — one row per configured rule
• alert_states — one row per rule, tracks current state and consecutive failure count
• alert_events — append-only log of all state transitions and notification outcomes
• plugin_metrics — all metric values written by any monitor or plugin. Its series index carries the value (ix_plugin_metrics_series_value), so a day of one metric is read from the index alone even when nothing is cached; Steward builds it in the background after startup (core/metric_index.py), never during a migration
• plugin_metric_latest — the newest sample of each (source, resource, metric). Pages that show "what it is now" read this instead of searching history; retention drops a row once its sample leaves the raw window