feat(anomalies): a metric out of its normal is an anomaly to review (#5740)
CI / lint (push) Successful in 2s
CI / integration (push) Failing after 1m24s
CI / unit (push) Successful in 1m37s
CI / publish (push) Skipped

#5740 "A metric that departs from its normal is recorded as an anomaly to
review, and is findable after it ends", milestone 513.

- Every minute each learned key's last 15 minutes is judged against its
  hour's normal: above the usual hold x1.25 and by the metric's floor opens
  an occurrence, which ends when the level comes back.
- metric_anomalies holds one row per item to review; a repeat while it
  still waits joins it as an occurrence (metric_anomaly_spans).
- An ended row nobody reviews within anomalies.review_days (14, Settings ->
  Thresholds) lapses; an ongoing one never does.
- Learning skips every hour an occurrence covers unless its anomaly was
  marked normal: no normal is learned without review.
- Alerts -> Anomalies lists those to review first, then the reviewed and
  lapsed under day dividers; marking is operator/admin and audited.
- The Overview widget and get_overview warn of open ones and count those
  to review; the host page shows its own; the weekly report lists the
  week's and the lapsed; MCP list_anomalies and review_anomaly.
- Reviewed and lapsed rows are pruned at data.retention_days, and kept
  in backups like alert history.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-10-14 13:41:57 -04:00
co-authored by Claude Opus 5.5
parent cddf7fc6c4
commit 5569f9934e
34 changed files with 1343 additions and 25 deletions
+20 -1
View File
@@ -3,6 +3,7 @@
widget is the full list of checks; this is the summary. #}
{% from "_macros.html" import status_badge %}
{% set firing = alerts | selectattr("label", "in", ["firing", "no_data"]) | list %}
{% set anomalies = anomalies or [] %}
<div style="display:flex;flex-wrap:wrap;gap:0.4rem 1.5rem;align-items:baseline;margin-bottom:0.6rem;">
<span>
{% if session.user_id %}<a href="/hosts/" style="color:inherit;text-decoration:none;">{% endif %}
@@ -24,7 +25,7 @@
</div>
{% set problems = entries | rejectattr("status", "equalto", "up") | list %}
{% if not problems and not alerts %}
{% if not problems and not alerts and not anomalies %}
<div class="tone-ok" style="font-size:0.85rem;padding:0.3rem 0;">
{% if entries %}✓ All {{ up }} monitors are up and no alert is firing.{% else %}No monitors configured, and no alert is firing.{% endif %}
</div>
@@ -54,6 +55,18 @@
{% if e.since %}<span style="color:var(--text-dim);font-size:0.75rem;white-space:nowrap;">{% if e.since_at_least %}over {% endif %}{{ e.since|ago }}</span>{% endif %}
</div>
{% endfor %}
{# Metric anomalies going on now (#5740): a warning each. #}
{% for v in anomalies[:4] %}
<div style="display:flex;align-items:center;gap:0.5rem;padding:0.2rem 0;font-size:0.82rem;">
{{ status_badge("anomaly") }}
{% set nm_style = "flex:1;min-width:0;white-space:nowrap;overflow:hidden;text-overflow:ellipsis;" %}
{% set nm = v.anomaly.resource_name ~ " · " ~ metric_label(v.anomaly.metric_name) %}
{% if session.user_id %}
<a href="/alerts/anomalies" title="{{ nm }}: {{ metric_value(v.anomaly.metric_name, v.anomaly.peak_level) }}, usual {{ metric_value(v.anomaly.metric_name, v.anomaly.usual_hold) }}" style="{{ nm_style }}color:inherit;text-decoration:none;">{{ nm }}</a>
{% else %}<span title="{{ nm }}" style="{{ nm_style }}">{{ nm }}</span>{% endif %}
<span style="color:var(--text-dim);font-size:0.75rem;white-space:nowrap;">{{ v.anomaly.last_started_at|ago }}</span>
</div>
{% endfor %}
{# What didn't fit, each linking to its own list (#5510). #}
{% set more_alerts, more_monitors = alerts|length - 6, problems|length - 8 %}
{% if more_alerts > 0 or more_monitors > 0 %}
@@ -65,3 +78,9 @@
{% endif %}
</div>
{% endif %}
{% if to_review %}
<div style="font-size:0.78rem;color:var(--text-muted);padding-top:0.3rem;">
{% if session.user_id %}<a href="/alerts/anomalies">{{ to_review }} metric anomal{{ "y" if to_review == 1 else "ies" }} to review</a>
{% else %}{{ to_review }} metric anomal{{ "y" if to_review == 1 else "ies" }} to review{% endif %}
</div>
{% endif %}