fix(anomalies): a level over its usual hold opens an anomaly only past its usual peak or once held (#6723)

Rows on the live instance were mostly bursts of a few minutes: 28 of 52
peaked within the metric's own usual peak, and 42 of 52 latest occurrences
lasted 30 minutes or less, which is what a burst shorter than the 15-minute
average looks like once its span is back-dated. Nothing asked how high or how
long, as the System 182 ruling says a normal records.

A departure now also has to be past the slot's usual peak, or held: the
median of the last hour over the same line. Those spans were also keeping the
burstiest hours out of learning, lowering the usual lines further.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-10-30 19:01:55 -04:00
co-authored by Claude Opus 5.5
parent 50b8922895
commit d83f8ffe9b
4 changed files with 96 additions and 15 deletions
+9 -5
View File
@@ -52,7 +52,8 @@ def _row(v: anom.AnomalyView) -> dict:
async def list_anomalies(review: str = "to_review", limit: int = 50) -> dict:
"""Metric anomalies: a learned metric whose 15-minute level went well
above what is normal for it at that hour (its usual hold ×1.25, and by
more than the metric's floor).
more than the metric's floor), and either past how high it usually goes
(`usual_peak`) or held there for most of an hour.
review: "to_review" (default) — waiting for someone to mark them; "normal"
— accepted, their hours are learned; "real" — a real problem, not learned;
@@ -263,8 +264,9 @@ def slot_label(kind: str, slot: int) -> str:
def flag_above(norm_hold: float, floor: float) -> float:
"""The 15-minute level that would open an anomaly in a slot whose usual
hold is `norm_hold` (anomalies.departs)."""
"""The 15-minute level that has departed in a slot whose usual hold is
`norm_hold` (anomalies.departs). It opens an anomaly past the usual peak,
or once held."""
return max(norm_hold * anom.DEPART, norm_hold + floor)
@@ -287,8 +289,10 @@ async def get_metric_normal(source_module: str, resource: str, metric: str) -> d
(168 hours from Monday 00:00 UTC, once 21 days do). Each slot gives
`usual_hold` (the level its hours usually held: each hour's p95, at the
90th percentile across past days), `usual_peak` (how high they usually
went), `days_seen`, and `flag_above`: the 15-minute average that would
open an anomaly there (usual hold ×1.25, and at least `floor` above it).
went), `days_seen`, and `flag_above`: the 15-minute average that has
departed there (usual hold ×1.25, and at least `floor` above it). A
departure opens an anomaly once it passes `usual_peak`, or when the median
of the last hour is over `flag_above` too.
`floor` is the smallest departure worth calling an anomaly, and
`floor_setting` the Settings → Thresholds floor it comes from. Hours an
anomaly covers are learned only once someone marks it normal.