fix(anomalies): a burst within what a metric reaches in most hours opens nothing (#6727)
#6727 "A metric that bursts now and then opens anomalies at hours whose usual peak reads 0". usual_peak is the p90 of a slot's hourly max, so a metric that bursts in under about 1 hour in 10 reads 0 there however often it bursts across the day. #6723 "An anomaly opens on a burst of minutes inside the metric's usual peak, so most rows aren't outside the norm" therefore still opened on those. usual_anywhere is the median, across the hours of the day, of each hour's highest peak. Height is now judged against the greater of it and the slot's usual peak. Bursts at any hour lift the median; a job at a time of its own raises a few hours and leaves it low, so the same level at another time of day is still an anomaly, as System 182 rules. A held level still opens. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -4,10 +4,10 @@ Every minute the last WINDOW of each learned key is averaged and compared
|
||||
with the normal for the slot it falls in (steward.core.norms). A level above
|
||||
the slot's usual hold by DEPART times, and by at least the metric's floor, has
|
||||
departed. That opens an occurrence once it is also outside what the normal
|
||||
records in height or in time: higher than the slot's usual peak, or held, with
|
||||
most of the last HELD above the line. A burst of a few minutes within the
|
||||
usual peak lifts the average for a while and opens nothing. The occurrence ends
|
||||
when the level comes back under the line.
|
||||
records in height or in time: higher than the metric usually goes (Norm.
|
||||
usual_height), or held, with most of the last HELD above the line. A burst of
|
||||
a few minutes within that lifts the average for a while and opens nothing. The
|
||||
occurrence ends when the level comes back under the line.
|
||||
|
||||
What the operator ruled (System 182 "Metric Norms and Anomalies"):
|
||||
- A departure is a warning, marked as a metric anomaly, and stays "to review"
|
||||
@@ -32,7 +32,7 @@ from steward.timeline.record import anomaly_changed
|
||||
|
||||
WINDOW = timedelta(minutes=15) # the level judged is this window's average
|
||||
DEPART = 1.25 # times the slot's usual hold that counts as a departure
|
||||
HELD = timedelta(hours=1) # a level within the usual peak opens once the median of this is out
|
||||
HELD = timedelta(hours=1) # a level within its usual height opens once the median of this is out
|
||||
# An occurrence that comes back within one window of ending continues the
|
||||
# same occurrence: a level wobbling around the line is one event, not ten.
|
||||
REJOIN = WINDOW
|
||||
@@ -169,7 +169,7 @@ async def detect(session, now: datetime | None = None) -> dict:
|
||||
out_now = {key for key, level in levels.items()
|
||||
if key in norms and departs(level, norms[key], LEARNED[(key[0], key[2])].least_departure)}
|
||||
held = await held_levels(session, [key for key in out_now
|
||||
if key not in open_now and levels[key] <= norms[key].usual_peak], now)
|
||||
if key not in open_now and levels[key] <= norms[key].usual_height], now)
|
||||
for key, level in levels.items():
|
||||
norm = norms.get(key)
|
||||
out = key in out_now
|
||||
@@ -182,7 +182,7 @@ async def detect(session, now: datetime | None = None) -> dict:
|
||||
_close(anomaly, span, now)
|
||||
changed.append(anomaly)
|
||||
closed += 1
|
||||
elif out and level > norm.usual_peak:
|
||||
elif out and level > norm.usual_height:
|
||||
changed.append(await _occur(session, key, level, norm, now, WINDOW))
|
||||
opened += 1
|
||||
elif out and departs(held.get(key, 0.0), norm, LEARNED[(key[0], key[2])].least_departure):
|
||||
|
||||
Reference in New Issue
Block a user