feat(alerts): silence is alertable — agent offline and no-data (#5062)

Agent offline is an ordinary rule now. The host_agent scheduler records
host_agent/<host>/seconds_since_seen every minute (from last_seen_at, or
registration time for an agent that never reported), replacing the task
that only logged stale agents. The catalog offers the metric, and picking
it on a new rule fills "> 180".

No data: each rule's state tracks its last sample and its usual gap,
smoothed so that an outage counts as at most three gaps. A core task,
alert_silence_check, runs every 60s and marks a rule no_data_since once its
series has had no sample for three usual gaps (at least 5 minutes; an hour
while the cadence is unknown). A never-sampled rule counts from when it
started watching. The mark sits beside the state, so a firing alert that
goes silent stays firing. Each silence records one event and sends one
notification (rule option notify_no_data, default on; muted in maintenance
windows). The next sample clears it with a "data resumed" event and
notification. Creating, changing, disabling or re-enabling a rule restarts
its watch.

Migration 0029 adds the columns and seeds each existing rule's last sample
and gap from its last day of data, so an upgrade does not flag every rule.

Shown on the Alerts page (a NO DATA badge with its start time), in MCP
list_alerts, list_alert_rules and get_host (state "no_data", no_data_since),
in get_overview findings (alert_no_data, or a firing alert marked silent)
and in the weekly report. The rule form has the notify option, and its
metric reference now matches what traefik records and lists host_agent.
Emails say "no data" and carry the event's note.

The metric-catalog guard also reads positional MetricBatch.add arguments.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

steward/templates/alerts/list.html

@@ -24,9 +24,16 @@
                     <span class="badge badge-red">FIRING</span>
                     {% elif state.state.value == 'acknowledged' %}
                     <span class="badge badge-yellow">ACK</span>
-                    {% else %}
+                    {% elif state.state.value == 'pending' %}
                     <span class="badge badge-dim">PENDING</span>
                     {% endif %}
+                    {% if state.no_data_since %}
+                    <span class="badge badge-yellow"
+                          title="No sample since {{ state.last_sample_at.strftime('%Y-%m-%d %H:%M UTC') if state.last_sample_at else 'the rule started watching' }}">NO DATA</span>
+                    <div style="font-size:0.72rem;color:var(--text-dim);margin-top:0.2rem;">
+                      since {{ state.no_data_since.strftime("%Y-%m-%d %H:%M") }} UTC
+                    </div>
+                    {% endif %}
                 </td>
                 <td>{{ rule.name }}</td>
                 <td style="color:var(--text-muted);">{{ rule.resource_name }}</td>

steward/templates/alerts/rules_form.html

@@ -99,6 +99,18 @@
           within the threshold. An acknowledgement holds until then.</p>
       </div>
 
+      <div class="form-group">
+        <label style="display:flex;align-items:center;gap:0.5rem;font-weight:normal;">
+          <input type="checkbox" name="notify_no_data"
+                 {% if not rule or rule.notify_no_data %}checked{% endif %}>
+          Notify when this metric stops reporting
+        </label>
+        <p style="font-size:0.75rem;color:var(--text-muted);margin-top:0.2rem;">
+          Silence is always shown as NO DATA on the Alerts page: no sample for three of the
+          metric's usual gaps (at least 5 minutes). This also sends a notification when it starts
+          and when data resumes. A firing alert that goes silent stays firing.</p>
+      </div>
+
       {# ── Metric reference ────────────────────────────────────────────────── #}
       <details style="margin-bottom:1rem;">
         <summary style="cursor:pointer;font-size:0.82rem;color:var(--text-muted);user-select:none;">
@@ -113,7 +125,8 @@
             "dns":     [("is_up", "1.0 = resolved (and matches expected IP), 0.0 = failed"),
                         ("response_ms", "resolution time in milliseconds")],
             "traefik": [("request_rate", "requests/sec for this router"),
-                        ("error_rate", "fraction of 5xx responses (0–1)"),
+                        ("error_rate_4xx_pct", "% of responses that are 4xx"),
+                        ("error_rate_5xx_pct", "% of responses that are 5xx"),
                         ("latency_p50_ms", "median response time ms"),
                         ("latency_p95_ms", "p95 response time ms"),
                         ("latency_p99_ms", "p99 response time ms"),
@@ -126,6 +139,11 @@
                         ("mem_pct", "container memory usage %")],
             "http":    [("is_up", "1.0 = check passed, 0.0 = failed"),
                         ("response_ms", "HTTP response time in ms")],
+            "host_agent": [("seconds_since_seen", "seconds since the agent last reported; > 180 = offline"),
+                           ("cpu_pct", "host CPU usage %"),
+                           ("mem_used_pct", "host memory usage %"),
+                           ("disk_used_pct_worst", "fullest disk, % used"),
+                           ("disk_used_pct", "one mount, % used (resource host:mount)")],
           } %}
           {% for src, pairs in descs.items() %}
           <div>
@@ -192,6 +210,20 @@
       </div>
       {% endif %}
 
+      {% if not rule %}
+      <script>
+        // "Agent offline" is seconds_since_seen > 180 (the agent's stale window).
+        document.addEventListener("change", function (e) {
+          if (e.target.name !== "metric_name" || e.target.value !== "seconds_since_seen") return;
+          var form = e.target.form, threshold = form.elements["threshold"];
+          if (threshold.value === "") {
+            form.elements["operator"].value = ">";
+            threshold.value = "180";
+          }
+        });
+      </script>
+      {% endif %}
+
       <div style="display:flex;gap:1rem;margin-top:1.5rem;">
         <button type="submit" class="btn">{% if rule %}Save Changes{% else %}Create Rule{% endif %}</button>
         <a href="/alerts/" class="btn btn-ghost">Cancel</a>