bvandeusenandClaude Opus 5.5
c368e93e2c
Agent offline is an ordinary rule now. The host_agent scheduler records host_agent/<host>/seconds_since_seen every minute (from last_seen_at, or registration time for an agent that never reported), replacing the task that only logged stale agents. The catalog offers the metric, and picking it on a new rule fills "> 180". No data: each rule's state tracks its last sample and its usual gap, smoothed so that an outage counts as at most three gaps. A core task, alert_silence_check, runs every 60s and marks a rule no_data_since once its series has had no sample for three usual gaps (at least 5 minutes; an hour while the cadence is unknown). A never-sampled rule counts from when it started watching. The mark sits beside the state, so a firing alert that goes silent stays firing. Each silence records one event and sends one notification (rule option notify_no_data, default on; muted in maintenance windows). The next sample clears it with a "data resumed" event and notification. Creating, changing, disabling or re-enabling a rule restarts its watch. Migration 0029 adds the columns and seeds each existing rule's last sample and gap from its last day of data, so an upgrade does not flag every rule. Shown on the Alerts page (a NO DATA badge with its start time), in MCP list_alerts, list_alert_rules and get_host (state "no_data", no_data_since), in get_overview findings (alert_no_data, or a firing alert marked silent) and in the weekly report. The rule form has the notify option, and its metric reference now matches what traefik records and lists host_agent. Emails say "no data" and carry the event's note. The metric-catalog guard also reads positional MetricBatch.add arguments. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
235 lines
12 KiB
HTML
235 lines
12 KiB
HTML
{% extends "base.html" %}
|
|
{% from "_macros.html" import crumbs %}
|
|
{% block title %}{% if rule %}Edit Rule{% else %}New Alert Rule{% endif %} — Steward{% endblock %}
|
|
{% block breadcrumb %}{{ crumbs([("Alerts", "/alerts/"), ("Edit rule" if rule else "New rule", "")]) }}{% endblock %}
|
|
{% block content %}
|
|
<div style="max-width:600px;margin:2rem auto;">
|
|
<h1 class="page-title">{% if rule %}Edit Rule{% else %}New Alert Rule{% endif %}</h1>
|
|
<div class="card">
|
|
<form method="post" action="{% if rule %}/alerts/rules/{{ rule.id }}{% else %}/alerts/rules{% endif %}">{{ csrf_field() }}
|
|
|
|
<div class="form-group">
|
|
<label>Rule Name</label>
|
|
<input type="text" name="name" required autofocus
|
|
value="{{ rule.name if rule else '' }}"
|
|
placeholder="e.g. Home server down">
|
|
</div>
|
|
|
|
{# ── Source module ───────────────────────────────────────────────────── #}
|
|
<div class="form-group">
|
|
<label>Source</label>
|
|
<select name="source_module" required
|
|
hx-get="/alerts/api/rule_fields"
|
|
hx-trigger="change"
|
|
hx-target="#rule-fields"
|
|
hx-include="[name='source_module']">
|
|
{% for src in source_modules %}
|
|
<option value="{{ src }}" {% if src == initial_source %}selected{% endif %}>
|
|
{{ src }}
|
|
</option>
|
|
{% endfor %}
|
|
</select>
|
|
</div>
|
|
|
|
{# ── Dynamic resource + metric ────────────────────────────────────────── #}
|
|
<div id="rule-fields">
|
|
{# Inline the same partial logic so the page is fully usable without JS #}
|
|
<div class="form-group">
|
|
<label>Resource Name</label>
|
|
{% if resources %}
|
|
<select name="resource_name" required>
|
|
{% if not (rule and rule.resource_name) %}<option value="" disabled selected>— pick a resource —</option>{% endif %}
|
|
{% for r in resources %}
|
|
<option value="{{ r }}" {% if rule and r == rule.resource_name %}selected{% endif %}>{{ r }}</option>
|
|
{% endfor %}
|
|
</select>
|
|
{% else %}
|
|
<select name="resource_name" required disabled>
|
|
<option value="">No data yet — run monitors first</option>
|
|
</select>
|
|
<p style="font-size:0.78rem;color:var(--text-muted);margin-top:0.25rem;">
|
|
Resource names populate from live metric data. Start the scheduler and try again.
|
|
</p>
|
|
{% endif %}
|
|
</div>
|
|
|
|
<div class="form-group">
|
|
<label>Metric Name</label>
|
|
<select name="metric_name" required>
|
|
{% if not (rule and rule.metric_name) %}<option value="" disabled selected>— pick a metric —</option>{% endif %}
|
|
{% for m in metrics %}
|
|
<option value="{{ m }}" {% if rule and m == rule.metric_name %}selected{% endif %}>{{ m }}</option>
|
|
{% endfor %}
|
|
</select>
|
|
</div>
|
|
</div>
|
|
|
|
{# ── Condition ───────────────────────────────────────────────────────── #}
|
|
<div style="display:grid;grid-template-columns:1fr 1fr;gap:1rem;">
|
|
<div class="form-group">
|
|
<label>Operator</label>
|
|
<select name="operator">
|
|
{% for val, label in operators %}
|
|
<option value="{{ val }}" {% if rule and rule.operator.value == val %}selected{% endif %}>
|
|
{{ label }}
|
|
</option>
|
|
{% endfor %}
|
|
</select>
|
|
</div>
|
|
<div class="form-group">
|
|
<label>Threshold</label>
|
|
<input type="number" name="threshold" step="any" required
|
|
value="{{ rule.threshold if rule else '' }}"
|
|
placeholder="0.5">
|
|
</div>
|
|
</div>
|
|
|
|
<div class="form-group">
|
|
<label>Consecutive failures before FIRING</label>
|
|
<input type="number" name="consecutive_failures_required" min="1"
|
|
value="{{ rule.consecutive_failures_required if rule else 1 }}">
|
|
</div>
|
|
|
|
<div class="form-group">
|
|
<label>Consecutive OK checks before it clears</label>
|
|
<input type="number" name="consecutive_ok_required" min="1"
|
|
value="{{ rule.consecutive_ok_required if rule else 1 }}">
|
|
<p style="font-size:0.75rem;color:var(--text-muted);margin-top:0.2rem;">
|
|
A firing or acknowledged alert resolves only after this many checks in a row are back
|
|
within the threshold. An acknowledgement holds until then.</p>
|
|
</div>
|
|
|
|
<div class="form-group">
|
|
<label style="display:flex;align-items:center;gap:0.5rem;font-weight:normal;">
|
|
<input type="checkbox" name="notify_no_data"
|
|
{% if not rule or rule.notify_no_data %}checked{% endif %}>
|
|
Notify when this metric stops reporting
|
|
</label>
|
|
<p style="font-size:0.75rem;color:var(--text-muted);margin-top:0.2rem;">
|
|
Silence is always shown as NO DATA on the Alerts page: no sample for three of the
|
|
metric's usual gaps (at least 5 minutes). This also sends a notification when it starts
|
|
and when data resumes. A firing alert that goes silent stays firing.</p>
|
|
</div>
|
|
|
|
{# ── Metric reference ────────────────────────────────────────────────── #}
|
|
<details style="margin-bottom:1rem;">
|
|
<summary style="cursor:pointer;font-size:0.82rem;color:var(--text-muted);user-select:none;">
|
|
Metric reference
|
|
</summary>
|
|
<div style="margin-top:0.75rem;display:grid;gap:0.5rem;">
|
|
{% set descs = {
|
|
"icmp": [("is_up", "1.0 = reachable, 0.0 = unreachable"),
|
|
("response_ms", "round-trip latency in milliseconds")],
|
|
"tcp": [("is_up", "1.0 = port open, 0.0 = unreachable"),
|
|
("response_ms", "TCP connect time in milliseconds")],
|
|
"dns": [("is_up", "1.0 = resolved (and matches expected IP), 0.0 = failed"),
|
|
("response_ms", "resolution time in milliseconds")],
|
|
"traefik": [("request_rate", "requests/sec for this router"),
|
|
("error_rate_4xx_pct", "% of responses that are 4xx"),
|
|
("error_rate_5xx_pct", "% of responses that are 5xx"),
|
|
("latency_p50_ms", "median response time ms"),
|
|
("latency_p95_ms", "p95 response time ms"),
|
|
("latency_p99_ms", "p99 response time ms"),
|
|
("response_bytes_rate", "bytes/sec of responses"),
|
|
("cert_expiry_days", "days until TLS cert expires")],
|
|
"unifi": [("is_up", "1.0 = WAN up, 0.0 = down"),
|
|
("latency_ms", "WAN latency in ms"),
|
|
("total_clients", "number of connected clients")],
|
|
"docker": [("cpu_pct", "container CPU usage %"),
|
|
("mem_pct", "container memory usage %")],
|
|
"http": [("is_up", "1.0 = check passed, 0.0 = failed"),
|
|
("response_ms", "HTTP response time in ms")],
|
|
"host_agent": [("seconds_since_seen", "seconds since the agent last reported; > 180 = offline"),
|
|
("cpu_pct", "host CPU usage %"),
|
|
("mem_used_pct", "host memory usage %"),
|
|
("disk_used_pct_worst", "fullest disk, % used"),
|
|
("disk_used_pct", "one mount, % used (resource host:mount)")],
|
|
} %}
|
|
{% for src, pairs in descs.items() %}
|
|
<div>
|
|
<div style="font-size:0.78rem;font-weight:600;color:var(--text-dim);text-transform:uppercase;letter-spacing:0.05em;margin-bottom:0.2rem;">{{ src }}</div>
|
|
{% for metric, desc in pairs %}
|
|
<div style="display:grid;grid-template-columns:160px 1fr;gap:0.5rem;font-size:0.78rem;padding:0.15rem 0;">
|
|
<code>{{ metric }}</code>
|
|
<span style="color:var(--text-muted);">{{ desc }}</span>
|
|
</div>
|
|
{% endfor %}
|
|
</div>
|
|
{% endfor %}
|
|
</div>
|
|
</details>
|
|
|
|
{# ── Ansible action (admin only) ─────────────────────────────────────── #}
|
|
{% if session.get("user_role") == "admin" %}
|
|
{% set act = rule.ansible_action if rule and rule.ansible_action else None %}
|
|
<div style="margin:1.25rem 0;padding:1rem;border:1px solid var(--border-mid);border-radius:6px;background:var(--bg-elevated);">
|
|
<label style="display:flex;align-items:center;gap:0.5rem;font-weight:600;">
|
|
<input type="checkbox" name="ansible_enabled" {% if act %}checked{% endif %}>
|
|
On firing → run an Ansible playbook
|
|
</label>
|
|
<p style="font-size:0.78rem;color:var(--text-muted);margin:0.4rem 0 0.75rem;">
|
|
Admin only. Runs once when the alert transitions into FIRING.
|
|
<code>steward_alert_*</code> context vars (resource, metric, value…) are injected automatically.
|
|
</p>
|
|
<div class="form-group">
|
|
<label>Source</label>
|
|
<select name="ansible_source">
|
|
<option value="">— select source —</option>
|
|
{% for s in ansible_sources %}
|
|
<option value="{{ s }}" {% if act and act.source == s %}selected{% endif %}>{{ s }}</option>
|
|
{% endfor %}
|
|
</select>
|
|
</div>
|
|
<div class="form-group">
|
|
<label>Playbook <span style="color:var(--text-muted);font-weight:normal;">(path within source)</span></label>
|
|
<input type="text" name="ansible_playbook" placeholder="playbooks/shutdown.yml"
|
|
value="{{ act.playbook if act else '' }}">
|
|
</div>
|
|
<div class="form-group">
|
|
<label>Inventory <span style="color:var(--text-muted);font-weight:normal;">(path within source)</span></label>
|
|
<input type="text" name="ansible_inventory" placeholder="inventory/hosts"
|
|
value="{{ act.inventory if act else '' }}">
|
|
</div>
|
|
<div class="form-group">
|
|
<label>Extra vars <span style="color:var(--text-muted);font-weight:normal;">(optional, one key=value per line)</span></label>
|
|
<textarea name="ansible_extra_vars" rows="2"
|
|
style="width:100%;font-family:ui-monospace,monospace;font-size:0.85rem;">{{ act.extra_vars | join("\n") if act and act.extra_vars else "" }}</textarea>
|
|
</div>
|
|
<div class="form-group">
|
|
<label>Limit to host(s) <span style="color:var(--text-muted);font-weight:normal;">(optional)</span></label>
|
|
<input type="text" name="ansible_limit" value="{{ act.limit if act and act.limit else '' }}">
|
|
</div>
|
|
<div class="form-group">
|
|
<label>Tags <span style="color:var(--text-muted);font-weight:normal;">(optional)</span></label>
|
|
<input type="text" name="ansible_tags" value="{{ act.tags if act and act.tags else '' }}">
|
|
</div>
|
|
<label style="display:flex;align-items:center;gap:0.5rem;font-weight:normal;">
|
|
<input type="checkbox" name="ansible_check" {% if act and act.check %}checked{% endif %}>
|
|
Dry-run (--check --diff)
|
|
</label>
|
|
</div>
|
|
{% endif %}
|
|
|
|
{% if not rule %}
|
|
<script>
|
|
// "Agent offline" is seconds_since_seen > 180 (the agent's stale window).
|
|
document.addEventListener("change", function (e) {
|
|
if (e.target.name !== "metric_name" || e.target.value !== "seconds_since_seen") return;
|
|
var form = e.target.form, threshold = form.elements["threshold"];
|
|
if (threshold.value === "") {
|
|
form.elements["operator"].value = ">";
|
|
threshold.value = "180";
|
|
}
|
|
});
|
|
</script>
|
|
{% endif %}
|
|
|
|
<div style="display:flex;gap:1rem;margin-top:1.5rem;">
|
|
<button type="submit" class="btn">{% if rule %}Save Changes{% else %}Create Rule{% endif %}</button>
|
|
<a href="/alerts/" class="btn btn-ghost">Cancel</a>
|
|
</div>
|
|
</form>
|
|
</div>
|
|
</div>
|
|
{% endblock %}
|