fix(anomalies): one slow reading no longer opens an anomaly (#6793)

A single tall reading lifted a 15-minute average past the usual height
on its own, so one slow request opened a row that stayed open until the
reading rolled out of the window. Four of the seven rows that occurred
after the 10-30 detection change were this: HA web (one 6.7 s check),
error-pages, codex and mailcow-svc latency.

detect() now opens by height only when the window is still past the
usual height, and still departs, with its highest reading left out. A
window that fails that falls through to the held test, which now needs
at least HELD_FEWEST (3) readings behind its median, so a lone reading
on a quiet router opens nothing either way.

The median and the topless average share one per-key read (_per_key).
The architecture doc and get_metric_normal's description state the
rule.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

docs/architecture.md

@@ -105,7 +105,7 @@ A two-minute spike raises the peak and leaves the p95 low; an hour spent at that
 
 A metric a plugin registers with `learn_normal()` gets a normal per hour of the day, then per hour of the week once three weeks of history exist (`metric_norms`, relearned hourly by `metric_norms`; `steward/core/norms.py`).
 
-Every minute, `metric_anomalies` averages each learned key's last 15 minutes and compares it with the normal for that hour. A level above the usual hold ×1.25, and above the metric's floor, has departed. It opens an anomaly once it is also higher than the metric usually goes, or held: the median of the last hour is over the same line. How high it usually goes is the greater of the usual peak for that hour and how high it has gone in most hours of the day, so bursts that land at any hour are told from a job with a time of its own. A burst of a few minutes within that lifts the 15-minute average and opens nothing (`steward/core/anomalies.py`):
+Every minute, `metric_anomalies` averages each learned key's last 15 minutes and compares it with the normal for that hour. A level above the usual hold ×1.25, and above the metric's floor, has departed. It opens an anomaly once it is also higher than the metric usually goes, or held: the median of the last hour is over the same line. How high it usually goes is the greater of the usual peak for that hour and how high it has gone in most hours of the day, so bursts that land at any hour are told from a job with a time of its own. A burst of a few minutes within that lifts the 15-minute average and opens nothing. Height is judged with the window's highest reading left out, so one slow request can't carry a window past it, and a held level needs at least three readings behind its median, so a lone reading never opens an anomaly (`steward/core/anomalies.py`):
 - `metric_anomalies` holds one row per item to review. A repeat while the row still waits joins it as another occurrence (`metric_anomaly_spans`).
 - A row is `to_review` until someone marks it `normal` or `real` (Alerts → Anomalies, or MCP `review_anomaly`). An ended row nobody reviews within `anomalies.review_days` (14) becomes `lapsed`.
 - Learning skips every hour an occurrence covers unless its anomaly was marked `normal`, so a new level is never learned without review.

steward/core/anomalies.py

@@ -7,7 +7,10 @@ departed. That opens an occurrence once it is also outside what the normal
 records in height or in time: higher than the metric usually goes (Norm.
 usual_height), or held, with most of the last HELD above the line. A burst of
 a few minutes within that lifts the average for a while and opens nothing. The
-occurrence ends when the level comes back under the line.
+height is judged with the window's highest reading left out, so one slow
+request or one busy sample can't carry a window past it, and a held level needs
+at least HELD_FEWEST readings behind its median: a lone reading never opens
+one. The occurrence ends when the level comes back under the line.
 
 What the operator ruled (System 182 "Metric Norms and Anomalies"):
 - A departure is a warning, marked as a metric anomaly, and stays "to review"
@@ -123,30 +126,58 @@ async def levels_now(session, now: datetime) -> dict[tuple[str, str, str], float
     return {(s, r, m): float(level) for s, r, m, level in rows}
 
 
-# The median, as a burst moves an average of the same stretch but not this.
-_HELD_LEVELS = text("""
-SELECT k.s, k.r, k.m, v.held
+def _per_key(aggregate: str):
+    """A read of `aggregate` over each given key's readings in [:since, :now].
+    A key whose aggregate is NULL is left out."""
+    return text(f"""
+SELECT k.s, k.r, k.m, v.x
 FROM unnest(:sources, :resources, :metrics) AS k(s, r, m)
 CROSS JOIN LATERAL (
-    SELECT percentile_cont(0.5) WITHIN GROUP (ORDER BY p.value) AS held FROM plugin_metrics p
+    SELECT {aggregate} AS x FROM plugin_metrics p
     WHERE p.source_module = k.s AND p.resource_name = k.r AND p.metric_name = k.m
       AND p.recorded_at >= :since AND p.recorded_at <= :now
 ) v
-WHERE v.held IS NOT NULL
+WHERE v.x IS NOT NULL
 """).bindparams(bindparam("sources", type_=ARRAY(String)), bindparam("resources", type_=ARRAY(String)),
                 bindparam("metrics", type_=ARRAY(String)))
 
 
-async def held_levels(session, keys: list[tuple[str, str, str]], now: datetime) -> dict[tuple[str, str, str], float]:
-    """{(source, resource, metric): the median of its last HELD}. Asked only of
-    keys that have departed and stayed within their usual peak, so the longer
-    read isn't paid for the rest."""
+# The median, as a burst moves an average of the same stretch but not this.
+# Fewer than HELD_FEWEST readings in the stretch are not a held level.
+HELD_FEWEST = 3
+_HELD_LEVELS = _per_key(
+    f"CASE WHEN count(*) >= {HELD_FEWEST} THEN percentile_cont(0.5) WITHIN GROUP (ORDER BY p.value) END")
+# The average with the single highest reading left out; NULL for one reading.
+_TOPLESS_LEVELS = _per_key("CASE WHEN count(*) > 1 THEN (sum(p.value) - max(p.value)) / (count(*) - 1) END")
+
+
+async def _read_per_key(session, query, keys: list[tuple[str, str, str]], since: datetime,
+                        now: datetime) -> dict[tuple[str, str, str], float]:
     if not keys:
         return {}
-    rows = await session.execute(_HELD_LEVELS, {
+    rows = await session.execute(query, {
         "sources": [k[0] for k in keys], "resources": [k[1] for k in keys], "metrics": [k[2] for k in keys],
-        "since": now - HELD, "now": now})
-    return {(s, r, m): float(held) for s, r, m, held in rows}
+        "since": since, "now": now})
+    return {(s, r, m): float(x) for s, r, m, x in rows}
+
+
+async def held_levels(session, keys: list[tuple[str, str, str]], now: datetime) -> dict[tuple[str, str, str], float]:
+    """{(source, resource, metric): the median of its last HELD}, for keys with
+    at least HELD_FEWEST readings in it. Asked only of keys that have departed
+    and not opened by height, so the longer read isn't paid for the rest."""
+    return await _read_per_key(session, _HELD_LEVELS, keys, now - HELD, now)
+
+
+async def topless_levels(session, keys: list[tuple[str, str, str]],
+                         now: datetime) -> dict[tuple[str, str, str], float]:
+    """{(source, resource, metric): its average over the last WINDOW without
+    the window's highest reading}. A key with one reading in the window is
+    left out: one reading is never a level."""
+    return await _read_per_key(session, _TOPLESS_LEVELS, keys, now - WINDOW, now)
+
+
+def _past_height(level: float, norm: Norm, floor: float) -> bool:
+    return level > norm.usual_height and departs(level, norm, floor)
 
 
 async def detect(session, now: datetime | None = None) -> dict:
@@ -166,10 +197,17 @@ async def detect(session, now: datetime | None = None) -> dict:
     }
     opened = closed = 0
     changed: list[MetricAnomaly] = []       # opened, joined or ended: their timeline entries follow
-    out_now = {key for key, level in levels.items()
-               if key in norms and departs(level, norms[key], LEARNED[(key[0], key[2])].least_departure)}
-    held = await held_levels(session, [key for key in out_now
-                                       if key not in open_now and levels[key] <= norms[key].usual_height], now)
+    def floor(key):
+        return LEARNED[(key[0], key[2])].least_departure
+
+    out_now = {key for key, level in levels.items() if key in norms and departs(level, norms[key], floor(key))}
+    waiting = [key for key in out_now if key not in open_now]
+    # Past the usual height only if it stays past it without its highest
+    # reading: one slow request lifts a window's average on its own.
+    tall = [key for key in waiting if levels[key] > norms[key].usual_height]
+    topless = await topless_levels(session, tall, now)
+    past_height = {key for key in tall if key in topless and _past_height(topless[key], norms[key], floor(key))}
+    held = await held_levels(session, [key for key in waiting if key not in past_height], now)
     for key, level in levels.items():
         norm = norms.get(key)
         out = key in out_now
@@ -182,10 +220,10 @@ async def detect(session, now: datetime | None = None) -> dict:
                 _close(anomaly, span, now)
                 changed.append(anomaly)
                 closed += 1
-        elif out and level > norm.usual_height:
+        elif key in past_height:
             changed.append(await _occur(session, key, level, norm, now, WINDOW))
             opened += 1
-        elif out and departs(held.get(key, 0.0), norm, LEARNED[(key[0], key[2])].least_departure):
+        elif out and departs(held.get(key, 0.0), norm, floor(key)):
             changed.append(await _occur(session, key, level, norm, now, HELD))
             opened += 1
     # An open occurrence whose key stopped reporting, lost its normal or fell

steward/mcp/tools/anomalies.py

@@ -293,8 +293,9 @@ async def get_metric_normal(source_module: str, resource: str, metric: str) -> d
     departed there (usual hold ×1.25, and at least `floor` above it). A
     departure opens an anomaly once it passes both `usual_peak` and
     `usual_anywhere` (how high the metric has gone in most hours of the day,
-    one figure for the whole key), or when the median of the last hour is over
-    `flag_above` too.
+    one figure for the whole key) even without the window's highest reading,
+    or when the median of the last hour (three readings or more) is over
+    `flag_above` too. A lone slow reading opens nothing.
     `floor` is the smallest departure worth calling an anomaly, and
     `floor_setting` the Settings → Thresholds floor it comes from. Hours an
     anomaly covers are learned only once someone marks it normal.

tests/integration/test_anomalies.py

@@ -140,6 +140,49 @@ def test_a_burst_past_its_usual_peak_is_an_anomaly(app, src):
     assert a.started_at == NOW - timedelta(minutes=15)
 
 
+def _window_of(app, src: str, res: str, readings: dict[int, float]):
+    """A detection pass at NOW over `readings` ({minutes before NOW: value}),
+    against the normal that holds 10 and peaks at 20."""
+    from sqlalchemy import select
+    from steward.core.anomalies import detect
+    from steward.models.metrics import PluginMetric, PluginMetricLatest
+    from steward.models.norms import MetricAnomaly
+
+    async def _go():
+        async with app.db_sessionmaker() as s:
+            async with s.begin():
+                s.add_all(_norms(src, res))
+                s.add_all([PluginMetric(source_module=src, resource_name=res, metric_name=METRIC,
+                                        value=v, recorded_at=NOW - timedelta(minutes=m))
+                           for m, v in readings.items()])
+                await s.merge(PluginMetricLatest(source_module=src, resource_name=res, metric_name=METRIC,
+                                                 value=readings[min(readings)], recorded_at=NOW))
+            async with s.begin():
+                await detect(s, NOW)
+            return list((await s.execute(select(MetricAnomaly).where(MetricAnomaly.resource_name == res))).scalars())
+    return asyncio.run(_go())
+
+
+def _every_3_minutes(value: float = 10.0) -> dict[int, float]:
+    """Six readings across the 15-minute window, as a router or monitor reports."""
+    return {m: value for m in range(0, 15, 3)} | {14: value}
+
+
+def test_one_slow_reading_is_no_anomaly(app, src):
+    # One 200 among five 10s averages 41.7, past the peak of 20; without it the window holds 10.
+    assert _window_of(app, src, _res("one"), _every_3_minutes() | {6: 200.0}) == []
+
+
+def test_two_slow_readings_past_the_peak_are_an_anomaly(app, src):
+    (a,) = _window_of(app, src, _res("two"), _every_3_minutes() | {3: 200.0, 6: 200.0})
+    assert a.started_at == NOW - timedelta(minutes=15)
+
+
+def test_a_lone_reading_is_no_anomaly(app, src):
+    # A quiet router's only request in the hour: neither a height nor a held level.
+    assert _window_of(app, src, _res("lone"), {2: 500.0}) == []
+
+
 def test_a_burst_past_its_slots_peak_but_within_what_most_hours_reach_is_no_anomaly(app, src):
     assert _hour_of(app, src, _res("anywhere"), level=30.0, for_minutes=15, anywhere=40.0) == []