feat(anomalies): a metric out of its normal is an anomaly to review (#5740)
#5740 "A metric that departs from its normal is recorded as an anomaly to review, and is findable after it ends", milestone 513. - Every minute each learned key's last 15 minutes is judged against its hour's normal: above the usual hold x1.25 and by the metric's floor opens an occurrence, which ends when the level comes back. - metric_anomalies holds one row per item to review; a repeat while it still waits joins it as an occurrence (metric_anomaly_spans). - An ended row nobody reviews within anomalies.review_days (14, Settings -> Thresholds) lapses; an ongoing one never does. - Learning skips every hour an occurrence covers unless its anomaly was marked normal: no normal is learned without review. - Alerts -> Anomalies lists those to review first, then the reviewed and lapsed under day dividers; marking is operator/admin and audited. - The Overview widget and get_overview warn of open ones and count those to review; the host page shows its own; the weekly report lists the week's and the lapsed; MCP list_anomalies and review_anomaly. - Reviewed and lapsed rows are pruned at data.retention_days, and kept in backups like alert history. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
+19
-4
@@ -24,6 +24,10 @@ The slots:
|
||||
Only metrics registered with learn_normal() are learned: a host agent alone
|
||||
reports thousands of per-core, per-disk and per-interface series, and most
|
||||
have no use for a normal.
|
||||
|
||||
An hour an anomaly covers is learned only once someone marks the anomaly
|
||||
normal (steward.core.anomalies): a level that jumps and stays is never
|
||||
absorbed on its own, and a lapsed anomaly is never learned.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
@@ -80,11 +84,12 @@ def slot_of(at: datetime, kind: str) -> int:
|
||||
|
||||
|
||||
# One statement learns every normal: the hours of each learned key (rolled up,
|
||||
# plus the raw window not rolled up yet), which kind of slot its history
|
||||
# supports, then each slot's usual hold and peak across past days, every hour
|
||||
# counted in its own slot and its neighbours'.
|
||||
# plus the raw window not rolled up yet) less those an anomaly covers that
|
||||
# nobody marked normal (#5740), which kind of slot its history supports, then
|
||||
# each slot's usual hold and peak across past days, every hour counted in its
|
||||
# own slot and its neighbours'.
|
||||
_REBUILD = text("""
|
||||
WITH hours AS (
|
||||
WITH all_hours AS (
|
||||
SELECT source_module AS s, resource_name AS r, metric_name AS m, bucket AS b,
|
||||
COALESCE(value_p95, value_avg) AS hold, value_max AS peak
|
||||
FROM plugin_metrics_hourly
|
||||
@@ -97,6 +102,16 @@ WITH hours AS (
|
||||
AND source_module || '/' || metric_name = ANY(:keys)
|
||||
GROUP BY 1, 2, 3, 4
|
||||
),
|
||||
hours AS (
|
||||
SELECT * FROM all_hours h
|
||||
WHERE NOT EXISTS (
|
||||
SELECT 1 FROM metric_anomaly_spans sp
|
||||
JOIN metric_anomalies a ON a.id = sp.anomaly_id
|
||||
WHERE a.review <> 'normal'
|
||||
AND a.source_module = h.s AND a.resource_name = h.r AND a.metric_name = h.m
|
||||
AND sp.started_at < h.b + interval '1 hour'
|
||||
AND COALESCE(sp.ended_at, :now) > h.b)
|
||||
),
|
||||
keyed AS (
|
||||
SELECT s, r, m, count(DISTINCT date_trunc('day', b)) AS days, min(b) AS first
|
||||
FROM hours GROUP BY s, r, m
|
||||
|
||||
Reference in New Issue
Block a user