fix(anomalies): a level over its usual hold opens an anomaly only past its usual peak or once held (#6723)

Rows on the live instance were mostly bursts of a few minutes: 28 of 52
peaked within the metric's own usual peak, and 42 of 52 latest occurrences
lasted 30 minutes or less, which is what a burst shorter than the 15-minute
average looks like once its span is back-dated. Nothing asked how high or how
long, as the System 182 ruling says a normal records.

A departure now also has to be past the slot's usual peak, or held: the
median of the last hour over the same line. Those spans were also keeping the
burstiest hours out of learning, lowering the usual lines further.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-10-30 19:01:55 -04:00
co-authored by Claude Opus 5.5
parent 50b8922895
commit d83f8ffe9b
4 changed files with 96 additions and 15 deletions
+1 -1
View File
@@ -105,7 +105,7 @@ A two-minute spike raises the peak and leaves the p95 low; an hour spent at that
A metric a plugin registers with `learn_normal()` gets a normal per hour of the day, then per hour of the week once three weeks of history exist (`metric_norms`, relearned hourly by `metric_norms`; `steward/core/norms.py`).
Every minute, `metric_anomalies` averages each learned key's last 15 minutes and compares it with the normal for that hour. A level above the usual hold ×1.25, and above the metric's floor, opens an anomaly (`steward/core/anomalies.py`):
Every minute, `metric_anomalies` averages each learned key's last 15 minutes and compares it with the normal for that hour. A level above the usual hold ×1.25, and above the metric's floor, has departed. It opens an anomaly once it is also past the usual peak for that hour, or held: the median of the last hour is over the same line. A burst of a few minutes within the usual peak lifts the 15-minute average and opens nothing (`steward/core/anomalies.py`):
- `metric_anomalies` holds one row per item to review. A repeat while the row still waits joins it as another occurrence (`metric_anomaly_spans`).
- A row is `to_review` until someone marks it `normal` or `real` (Alerts → Anomalies, or MCP `review_anomaly`). An ended row nobody reviews within `anomalies.review_days` (14) becomes `lapsed`.
- Learning skips every hour an occurrence covers unless its anomaly was marked `normal`, so a new level is never learned without review.