fix(anomalies): a burst within what a metric reaches in most hours opens nothing (#6727)
#6727 "A metric that bursts now and then opens anomalies at hours whose usual peak reads 0". usual_peak is the p90 of a slot's hourly max, so a metric that bursts in under about 1 hour in 10 reads 0 there however often it bursts across the day. #6723 "An anomaly opens on a burst of minutes inside the metric's usual peak, so most rows aren't outside the norm" therefore still opened on those. usual_anywhere is the median, across the hours of the day, of each hour's highest peak. Height is now judged against the greater of it and the slot's usual peak. Bursts at any hour lift the median; a job at a time of its own raises a few hours and leaves it low, so the same level at another time of day is still an anomaly, as System 182 rules. A held level still opens. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -105,7 +105,7 @@ A two-minute spike raises the peak and leaves the p95 low; an hour spent at that
|
||||
|
||||
A metric a plugin registers with `learn_normal()` gets a normal per hour of the day, then per hour of the week once three weeks of history exist (`metric_norms`, relearned hourly by `metric_norms`; `steward/core/norms.py`).
|
||||
|
||||
Every minute, `metric_anomalies` averages each learned key's last 15 minutes and compares it with the normal for that hour. A level above the usual hold ×1.25, and above the metric's floor, has departed. It opens an anomaly once it is also past the usual peak for that hour, or held: the median of the last hour is over the same line. A burst of a few minutes within the usual peak lifts the 15-minute average and opens nothing (`steward/core/anomalies.py`):
|
||||
Every minute, `metric_anomalies` averages each learned key's last 15 minutes and compares it with the normal for that hour. A level above the usual hold ×1.25, and above the metric's floor, has departed. It opens an anomaly once it is also higher than the metric usually goes, or held: the median of the last hour is over the same line. How high it usually goes is the greater of the usual peak for that hour and how high it has gone in most hours of the day, so bursts that land at any hour are told from a job with a time of its own. A burst of a few minutes within that lifts the 15-minute average and opens nothing (`steward/core/anomalies.py`):
|
||||
- `metric_anomalies` holds one row per item to review. A repeat while the row still waits joins it as another occurrence (`metric_anomaly_spans`).
|
||||
- A row is `to_review` until someone marks it `normal` or `real` (Alerts → Anomalies, or MCP `review_anomaly`). An ended row nobody reviews within `anomalies.review_days` (14) becomes `lapsed`.
|
||||
- Learning skips every hour an occurrence covers unless its anomaly was marked `normal`, so a new level is never learned without review.
|
||||
@@ -118,7 +118,7 @@ What learns a normal, and the floor each departure must clear (Settings → Thre
|
||||
| Host CPU | `host_agent` / `cpu_pct` | 10 points |
|
||||
| Host memory | `host_agent` / `mem_used_pct` | 5 points |
|
||||
| Host load per core | `host_agent` / `load_pct` (load_1m ÷ cores × 100, recorded at ingest) | 10 points |
|
||||
| Container CPU | `docker` / `cpu_pct` (100% is one core) | 10 points of one core |
|
||||
| Container CPU | `docker` / `cpu_pct` (100% is one core; a swarm service's under `swarm/<service>`, shared by its tasks) | 10 points of one core |
|
||||
| Container memory | `docker` / `mem_used_pct` (its processes' own memory, file cache left out, of its limit or of host RAM when it has none) | 5 points |
|
||||
| Monitor response time | `<type>` / `response_ms` | 100 ms |
|
||||
| Router 4xx share, p95 | `traefik` / `error_rate_4xx_pct`, `latency_p95_ms` | 2 points, 50 ms |
|
||||
|
||||
Reference in New Issue
Block a user