fix(anomalies): one slow reading no longer opens an anomaly (#6793)
A single tall reading lifted a 15-minute average past the usual height on its own, so one slow request opened a row that stayed open until the reading rolled out of the window. Four of the seven rows that occurred after the 10-30 detection change were this: HA web (one 6.7 s check), error-pages, codex and mailcow-svc latency. detect() now opens by height only when the window is still past the usual height, and still departs, with its highest reading left out. A window that fails that falls through to the held test, which now needs at least HELD_FEWEST (3) readings behind its median, so a lone reading on a quiet router opens nothing either way. The median and the topless average share one per-key read (_per_key). The architecture doc and get_metric_normal's description state the rule. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -105,7 +105,7 @@ A two-minute spike raises the peak and leaves the p95 low; an hour spent at that
|
||||
|
||||
A metric a plugin registers with `learn_normal()` gets a normal per hour of the day, then per hour of the week once three weeks of history exist (`metric_norms`, relearned hourly by `metric_norms`; `steward/core/norms.py`).
|
||||
|
||||
Every minute, `metric_anomalies` averages each learned key's last 15 minutes and compares it with the normal for that hour. A level above the usual hold ×1.25, and above the metric's floor, has departed. It opens an anomaly once it is also higher than the metric usually goes, or held: the median of the last hour is over the same line. How high it usually goes is the greater of the usual peak for that hour and how high it has gone in most hours of the day, so bursts that land at any hour are told from a job with a time of its own. A burst of a few minutes within that lifts the 15-minute average and opens nothing (`steward/core/anomalies.py`):
|
||||
Every minute, `metric_anomalies` averages each learned key's last 15 minutes and compares it with the normal for that hour. A level above the usual hold ×1.25, and above the metric's floor, has departed. It opens an anomaly once it is also higher than the metric usually goes, or held: the median of the last hour is over the same line. How high it usually goes is the greater of the usual peak for that hour and how high it has gone in most hours of the day, so bursts that land at any hour are told from a job with a time of its own. A burst of a few minutes within that lifts the 15-minute average and opens nothing. Height is judged with the window's highest reading left out, so one slow request can't carry a window past it, and a held level needs at least three readings behind its median, so a lone reading never opens an anomaly (`steward/core/anomalies.py`):
|
||||
- `metric_anomalies` holds one row per item to review. A repeat while the row still waits joins it as another occurrence (`metric_anomaly_spans`).
|
||||
- A row is `to_review` until someone marks it `normal` or `real` (Alerts → Anomalies, or MCP `review_anomaly`). An ended row nobody reviews within `anomalies.review_days` (14) becomes `lapsed`.
|
||||
- Learning skips every hour an occurrence covers unless its anomaly was marked `normal`, so a new level is never learned without review.
|
||||
|
||||
Reference in New Issue
Block a user