fix(anomalies): a level over its usual hold opens an anomaly only past its usual peak or once held (#6723)
Rows on the live instance were mostly bursts of a few minutes: 28 of 52 peaked within the metric's own usual peak, and 42 of 52 latest occurrences lasted 30 minutes or less, which is what a burst shorter than the 15-minute average looks like once its span is back-dated. Nothing asked how high or how long, as the System 182 ruling says a normal records. A departure now also has to be past the slot's usual peak, or held: the median of the last hour over the same line. Those spans were also keeping the burstiest hours out of learning, lowering the usual lines further. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -52,7 +52,8 @@ def _row(v: anom.AnomalyView) -> dict:
|
||||
async def list_anomalies(review: str = "to_review", limit: int = 50) -> dict:
|
||||
"""Metric anomalies: a learned metric whose 15-minute level went well
|
||||
above what is normal for it at that hour (its usual hold ×1.25, and by
|
||||
more than the metric's floor).
|
||||
more than the metric's floor), and either past how high it usually goes
|
||||
(`usual_peak`) or held there for most of an hour.
|
||||
|
||||
review: "to_review" (default) — waiting for someone to mark them; "normal"
|
||||
— accepted, their hours are learned; "real" — a real problem, not learned;
|
||||
@@ -263,8 +264,9 @@ def slot_label(kind: str, slot: int) -> str:
|
||||
|
||||
|
||||
def flag_above(norm_hold: float, floor: float) -> float:
|
||||
"""The 15-minute level that would open an anomaly in a slot whose usual
|
||||
hold is `norm_hold` (anomalies.departs)."""
|
||||
"""The 15-minute level that has departed in a slot whose usual hold is
|
||||
`norm_hold` (anomalies.departs). It opens an anomaly past the usual peak,
|
||||
or once held."""
|
||||
return max(norm_hold * anom.DEPART, norm_hold + floor)
|
||||
|
||||
|
||||
@@ -287,8 +289,10 @@ async def get_metric_normal(source_module: str, resource: str, metric: str) -> d
|
||||
(168 hours from Monday 00:00 UTC, once 21 days do). Each slot gives
|
||||
`usual_hold` (the level its hours usually held: each hour's p95, at the
|
||||
90th percentile across past days), `usual_peak` (how high they usually
|
||||
went), `days_seen`, and `flag_above`: the 15-minute average that would
|
||||
open an anomaly there (usual hold ×1.25, and at least `floor` above it).
|
||||
went), `days_seen`, and `flag_above`: the 15-minute average that has
|
||||
departed there (usual hold ×1.25, and at least `floor` above it). A
|
||||
departure opens an anomaly once it passes `usual_peak`, or when the median
|
||||
of the last hour is over `flag_above` too.
|
||||
`floor` is the smallest departure worth calling an anomaly, and
|
||||
`floor_setting` the Settings → Thresholds floor it comes from. Hours an
|
||||
anomaly covers are learned only once someone marks it normal.
|
||||
|
||||
Reference in New Issue
Block a user