fix(anomalies): a burst within what a metric reaches in most hours opens nothing (#6727)

#6727 "A metric that bursts now and then opens anomalies at hours whose usual
peak reads 0". usual_peak is the p90 of a slot's hourly max, so a metric that
bursts in under about 1 hour in 10 reads 0 there however often it bursts across
the day. #6723 "An anomaly opens on a burst of minutes inside the metric's usual
peak, so most rows aren't outside the norm" therefore still opened on those.

usual_anywhere is the median, across the hours of the day, of each hour's
highest peak. Height is now judged against the greater of it and the slot's
usual peak. Bursts at any hour lift the median; a job at a time of its own
raises a few hours and leaves it low, so the same level at another time of day
is still an anomaly, as System 182 rules. A held level still opens.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

docs/architecture.md

@@ -105,7 +105,7 @@ A two-minute spike raises the peak and leaves the p95 low; an hour spent at that
 
 A metric a plugin registers with `learn_normal()` gets a normal per hour of the day, then per hour of the week once three weeks of history exist (`metric_norms`, relearned hourly by `metric_norms`; `steward/core/norms.py`).
 
-Every minute, `metric_anomalies` averages each learned key's last 15 minutes and compares it with the normal for that hour. A level above the usual hold ×1.25, and above the metric's floor, has departed. It opens an anomaly once it is also past the usual peak for that hour, or held: the median of the last hour is over the same line. A burst of a few minutes within the usual peak lifts the 15-minute average and opens nothing (`steward/core/anomalies.py`):
+Every minute, `metric_anomalies` averages each learned key's last 15 minutes and compares it with the normal for that hour. A level above the usual hold ×1.25, and above the metric's floor, has departed. It opens an anomaly once it is also higher than the metric usually goes, or held: the median of the last hour is over the same line. How high it usually goes is the greater of the usual peak for that hour and how high it has gone in most hours of the day, so bursts that land at any hour are told from a job with a time of its own. A burst of a few minutes within that lifts the 15-minute average and opens nothing (`steward/core/anomalies.py`):
 - `metric_anomalies` holds one row per item to review. A repeat while the row still waits joins it as another occurrence (`metric_anomaly_spans`).
 - A row is `to_review` until someone marks it `normal` or `real` (Alerts → Anomalies, or MCP `review_anomaly`). An ended row nobody reviews within `anomalies.review_days` (14) becomes `lapsed`.
 - Learning skips every hour an occurrence covers unless its anomaly was marked `normal`, so a new level is never learned without review.
@@ -118,7 +118,7 @@ What learns a normal, and the floor each departure must clear (Settings → Thre
 | Host CPU | `host_agent` / `cpu_pct` | 10 points |
 | Host memory | `host_agent` / `mem_used_pct` | 5 points |
 | Host load per core | `host_agent` / `load_pct` (load_1m ÷ cores × 100, recorded at ingest) | 10 points |
-| Container CPU | `docker` / `cpu_pct` (100% is one core) | 10 points of one core |
+| Container CPU | `docker` / `cpu_pct` (100% is one core; a swarm service's under `swarm/<service>`, shared by its tasks) | 10 points of one core |
 | Container memory | `docker` / `mem_used_pct` (its processes' own memory, file cache left out, of its limit or of host RAM when it has none) | 5 points |
 | Monitor response time | `<type>` / `response_ms` | 100 ms |
 | Router 4xx share, p95 | `traefik` / `error_rate_4xx_pct`, `latency_p95_ms` | 2 points, 50 ms |

steward/core/anomalies.py

@@ -4,10 +4,10 @@ Every minute the last WINDOW of each learned key is averaged and compared
 with the normal for the slot it falls in (steward.core.norms). A level above
 the slot's usual hold by DEPART times, and by at least the metric's floor, has
 departed. That opens an occurrence once it is also outside what the normal
-records in height or in time: higher than the slot's usual peak, or held, with
-most of the last HELD above the line. A burst of a few minutes within the
-usual peak lifts the average for a while and opens nothing. The occurrence ends
-when the level comes back under the line.
+records in height or in time: higher than the metric usually goes (Norm.
+usual_height), or held, with most of the last HELD above the line. A burst of
+a few minutes within that lifts the average for a while and opens nothing. The
+occurrence ends when the level comes back under the line.
 
 What the operator ruled (System 182 "Metric Norms and Anomalies"):
 - A departure is a warning, marked as a metric anomaly, and stays "to review"
@@ -32,7 +32,7 @@ from steward.timeline.record import anomaly_changed
 
 WINDOW = timedelta(minutes=15)   # the level judged is this window's average
 DEPART = 1.25                    # times the slot's usual hold that counts as a departure
-HELD = timedelta(hours=1)        # a level within the usual peak opens once the median of this is out
+HELD = timedelta(hours=1)        # a level within its usual height opens once the median of this is out
 # An occurrence that comes back within one window of ending continues the
 # same occurrence: a level wobbling around the line is one event, not ten.
 REJOIN = WINDOW
@@ -169,7 +169,7 @@ async def detect(session, now: datetime | None = None) -> dict:
     out_now = {key for key, level in levels.items()
                if key in norms and departs(level, norms[key], LEARNED[(key[0], key[2])].least_departure)}
     held = await held_levels(session, [key for key in out_now
-                                       if key not in open_now and levels[key] <= norms[key].usual_peak], now)
+                                       if key not in open_now and levels[key] <= norms[key].usual_height], now)
     for key, level in levels.items():
         norm = norms.get(key)
         out = key in out_now
@@ -182,7 +182,7 @@ async def detect(session, now: datetime | None = None) -> dict:
                 _close(anomaly, span, now)
                 changed.append(anomaly)
                 closed += 1
-        elif out and level > norm.usual_peak:
+        elif out and level > norm.usual_height:
             changed.append(await _occur(session, key, level, norm, now, WINDOW))
             opened += 1
         elif out and departs(held.get(key, 0.0), norm, LEARNED[(key[0], key[2])].least_departure):

steward/core/norms.py

@@ -11,7 +11,12 @@ What a slot's normal is:
   (plugin_metrics_hourly.value_p95, #5738), at its 90th percentile across
   past days. A two-minute spike doesn't raise it; an hour spent high does.
 - usual_peak: the same for each hour's max, so a page can say how high the
-  slot usually goes.
+  slot usually goes. Most hours of a metric that bursts now and then see no
+  burst, so this can read 0 for a slot such a metric bursts in as much as any.
+- usual_anywhere: how high the metric has gone in most hours of the day, the
+  median across the 24 hours of each one's highest peak. Bursts that land at
+  any hour raise it. A job at a time of its own raises a few hours and leaves
+  the median alone, so the same level at another time still stands out.
 
 The slots:
 - An hour of the day (24 slots) once a key has DAY_MIN_DAYS of history.
@@ -116,6 +121,13 @@ class Norm:
     usual_hold: float
     usual_peak: float
     days_seen: int
+    usual_anywhere: float = 0.0
+
+    @property
+    def usual_height(self) -> float:
+        """The highest this metric usually goes at this time: a level past it
+        is news however briefly it lasted."""
+        return max(self.usual_peak, self.usual_anywhere)
 
 
 def slot_of(at: datetime, kind: str) -> int:
@@ -171,6 +183,12 @@ kinds AS (
     SELECT s, r, m, days, CASE WHEN first <= :week_ready THEN 'week' ELSE 'day' END AS kind
     FROM keyed WHERE days >= CAST(:day_min_days AS int)
 ),
+anywhere AS (
+    SELECT s, r, m, percentile_cont(0.5) WITHIN GROUP (ORDER BY highest) AS peak
+    FROM (SELECT s, r, m, max(peak) AS highest FROM hours
+          GROUP BY s, r, m, EXTRACT(HOUR FROM b AT TIME ZONE 'UTC')) by_hour
+    GROUP BY s, r, m
+),
 slotted AS (
     SELECT h.s, h.r, h.m, k.kind, k.days, h.hold, h.peak,
            CASE k.kind
@@ -183,13 +201,15 @@ slotted AS (
     WHERE k.kind = 'week' OR h.b >= :day_since
 )
 INSERT INTO metric_norms (source_module, resource_name, metric_name, slot, kind,
-                          usual_hold, usual_peak, days_seen, updated_at)
-SELECT s, r, m, (slot + n.off + slots) % slots, kind,
-       percentile_cont(CAST(:usual AS float8)) WITHIN GROUP (ORDER BY hold),
-       percentile_cont(CAST(:usual AS float8)) WITHIN GROUP (ORDER BY peak),
-       days, :now
-FROM slotted CROSS JOIN generate_series(-CAST(:neighbours AS int), CAST(:neighbours AS int)) AS n(off)
-GROUP BY s, r, m, (slot + n.off + slots) % slots, kind, days
+                          usual_hold, usual_peak, usual_anywhere, days_seen, updated_at)
+SELECT l.s, l.r, l.m, (l.slot + n.off + l.slots) % l.slots, l.kind,
+       percentile_cont(CAST(:usual AS float8)) WITHIN GROUP (ORDER BY l.hold),
+       percentile_cont(CAST(:usual AS float8)) WITHIN GROUP (ORDER BY l.peak),
+       a.peak, l.days, :now
+FROM slotted l
+JOIN anywhere a ON a.s = l.s AND a.r = l.r AND a.m = l.m
+CROSS JOIN generate_series(-CAST(:neighbours AS int), CAST(:neighbours AS int)) AS n(off)
+GROUP BY l.s, l.r, l.m, (l.slot + n.off + l.slots) % l.slots, l.kind, l.days, a.peak
 """).bindparams(bindparam("keys", type_=ARRAY(String)),
                bindparam("starts", type_=ARRAY(DateTime(timezone=True))),
                bindparam("parts", type_=ARRAY(Boolean)))
@@ -235,7 +255,7 @@ async def norms_at(session, keys: list[tuple[str, str, str]],
                | ((MetricNorm.kind == "week") & (MetricNorm.slot == week)))
     )).scalars().all()
     return {(n.source_module, n.resource_name, n.metric_name):
-            Norm(n.kind, n.slot, n.usual_hold, n.usual_peak, n.days_seen) for n in rows}
+            Norm(n.kind, n.slot, n.usual_hold, n.usual_peak, n.days_seen, n.usual_anywhere) for n in rows}
 
 
 async def run_norms(app) -> None:

steward/mcp/tools/anomalies.py

@@ -53,7 +53,7 @@ async def list_anomalies(review: str = "to_review", limit: int = 50) -> dict:
     """Metric anomalies: a learned metric whose 15-minute level went well
     above what is normal for it at that hour (its usual hold ×1.25, and by
     more than the metric's floor), and either past how high it usually goes
-    (`usual_peak`) or held there for most of an hour.
+    (see get_metric_normal) or held there for most of an hour.
 
     review: "to_review" (default) — waiting for someone to mark them; "normal"
     — accepted, their hours are learned; "real" — a real problem, not learned;
@@ -291,8 +291,10 @@ async def get_metric_normal(source_module: str, resource: str, metric: str) -> d
     90th percentile across past days), `usual_peak` (how high they usually
     went), `days_seen`, and `flag_above`: the 15-minute average that has
     departed there (usual hold ×1.25, and at least `floor` above it). A
-    departure opens an anomaly once it passes `usual_peak`, or when the median
-    of the last hour is over `flag_above` too.
+    departure opens an anomaly once it passes both `usual_peak` and
+    `usual_anywhere` (how high the metric has gone in most hours of the day,
+    one figure for the whole key), or when the median of the last hour is over
+    `flag_above` too.
     `floor` is the smallest departure worth calling an anomaly, and
     `floor_setting` the Settings → Thresholds floor it comes from. Hours an
     anomaly covers are learned only once someone marks it normal.
@@ -319,6 +321,7 @@ async def get_metric_normal(source_module: str, resource: str, metric: str) -> d
     out = {
         "source_module": source_module, "resource": resource, "metric": metric,
         "learned": bool(slots), "kind": kind,
+        "usual_anywhere": rnd(rows[0].usual_anywhere, 3) if rows else None,
         "floor": rnd(floor, 3), "floor_setting": learned.floor_name,
         "only_while": ({"metric": learned.gate[0],
                         "at_least": rnd(learned.gate[1]() if callable(learned.gate[1]) else learned.gate[1], 3),

steward/migrations/versions/0050_norm_usual_anywhere.py

@@ -0,0 +1,23 @@
+"""Metric normals: how high a metric has gone in most hours of the day.
+
+Revision ID: 0050_norm_usual_anywhere
+Revises: 0049_alert_on_a_rise
+Create Date: 2026-10-30
+"""
+from typing import Sequence, Union
+
+import sqlalchemy as sa
+from alembic import op
+
+revision: str = "0050_norm_usual_anywhere"
+down_revision: Union[str, None] = "0049_alert_on_a_rise"
+branch_labels: Union[str, Sequence[str], None] = None
+depends_on: Union[str, Sequence[str], None] = None
+
+
+def upgrade() -> None:
+    op.add_column("metric_norms", sa.Column("usual_anywhere", sa.Float, nullable=False, server_default="0"))
+
+
+def downgrade() -> None:
+    op.drop_column("metric_norms", "usual_anywhere")

steward/models/norms.py

@@ -24,6 +24,8 @@ class MetricNorm(Base):
     # peaked: the 90th percentile, across past days, of each hour's p95 and max.
     usual_hold: Mapped[float] = mapped_column(Float, nullable=False)
     usual_peak: Mapped[float] = mapped_column(Float, nullable=False)
+    # How high the metric has gone in most hours of the day, whatever the slot.
+    usual_anywhere: Mapped[float] = mapped_column(Float, nullable=False, default=0.0, server_default="0")
     days_seen: Mapped[int] = mapped_column(Integer, nullable=False)
     updated_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False)
 

tests/integration/test_anomalies.py

@@ -41,11 +41,13 @@ def src():
     norms.LEARNED.pop((name, METRIC), None)
 
 
-def _norms(src: str, res: str, hold: float = 10.0, peak: float = 20.0):
-    """A learned day normal for every hour: holds `hold`, peaks at `peak`."""
+def _norms(src: str, res: str, hold: float = 10.0, peak: float = 20.0, anywhere: float = 0.0):
+    """A learned day normal for every hour: holds `hold`, peaks at `peak`, and
+    reaches `anywhere` in most hours of the day."""
     from steward.models.norms import MetricNorm
     return [MetricNorm(source_module=src, resource_name=res, metric_name=METRIC, slot=h,
-                       kind="day", usual_hold=hold, usual_peak=peak, days_seen=7, updated_at=NOW)
+                       kind="day", usual_hold=hold, usual_peak=peak, usual_anywhere=anywhere,
+                       days_seen=7, updated_at=NOW)
             for h in range(24)]
 
 
@@ -104,7 +106,8 @@ def test_a_level_within_its_normal_is_no_anomaly(app, src):
     assert rows == []
 
 
-def _hour_of(app, src: str, res: str, *, level: float, for_minutes: int, then: float = 10.0):
+def _hour_of(app, src: str, res: str, *, level: float, for_minutes: int, then: float = 10.0,
+             anywhere: float = 0.0):
     """A detection pass after an hour reading `then`, with its last
     `for_minutes` at `level`, against the normal that holds 10 and peaks at 20."""
     from sqlalchemy import select
@@ -115,7 +118,7 @@ def _hour_of(app, src: str, res: str, *, level: float, for_minutes: int, then: f
     async def _go():
         async with app.db_sessionmaker() as s:
             async with s.begin():
-                s.add_all(_norms(src, res))
+                s.add_all(_norms(src, res, anywhere=anywhere))
                 s.add_all([PluginMetric(source_module=src, resource_name=res, metric_name=METRIC,
                                         value=level if m < for_minutes else then,
                                         recorded_at=NOW - timedelta(minutes=m)) for m in range(60)])
@@ -137,6 +140,15 @@ def test_a_burst_past_its_usual_peak_is_an_anomaly(app, src):
     assert a.started_at == NOW - timedelta(minutes=15)
 
 
+def test_a_burst_past_its_slots_peak_but_within_what_most_hours_reach_is_no_anomaly(app, src):
+    assert _hour_of(app, src, _res("anywhere"), level=30.0, for_minutes=15, anywhere=40.0) == []
+
+
+def test_a_level_within_what_most_hours_reach_is_an_anomaly_once_held(app, src):
+    (a,) = _hour_of(app, src, _res("anyheld"), level=30.0, for_minutes=45, anywhere=40.0)
+    assert a.peak_level == 30.0
+
+
 def test_a_level_within_its_usual_peak_is_an_anomaly_once_held(app, src):
     # The system running at its peaked rate: never past the peak, but not a burst.
     (a,) = _hour_of(app, src, _res("held"), level=18.0, for_minutes=45)
@@ -374,6 +386,42 @@ def test_a_metric_learns_only_from_its_since(app, src):
     assert held_back is None
 
 
+def test_bursts_at_any_hour_are_told_from_a_job_with_a_time_of_its_own(app, src):
+    """Fourteen days of a metric at 0. One burst to 50 in each hour of the day
+    leaves every slot's usual peak at 0 and is usual anywhere. The same burst
+    every night at 02:00 is usual at 02:00, and still stands out at 14:00."""
+    from steward.core.norms import norms_at, rebuild_norms
+    from steward.models.metrics import PluginMetricHourly
+    sporadic, nightly = _res("sporadic"), _res("nightly")
+    midnight = NOW.replace(hour=0)
+
+    def hours(res, burst):
+        return [PluginMetricHourly(source_module=src, resource_name=res, metric_name=METRIC,
+                                   bucket=midnight - timedelta(days=d) + timedelta(hours=h),
+                                   value_avg=0.0, value_p95=0.0, sample_count=60,
+                                   value_max=50.0 if burst(d, h) else 0.0)
+                for d in range(1, 15) for h in range(24)]
+
+    async def _go():
+        async with app.db_sessionmaker() as s:
+            async with s.begin():
+                s.add_all(hours(sporadic, lambda d, h: h % 14 == d - 1))
+                s.add_all(hours(nightly, lambda d, h: h == 2))
+            async with s.begin():
+                await rebuild_norms(s, NOW)
+            at = {}
+            for hour in (2, 14):
+                at[hour] = await norms_at(s, [(src, sporadic, METRIC), (src, nightly, METRIC)],
+                                          NOW.replace(hour=hour))
+            return at
+
+    at = asyncio.run(_go())
+    assert at[14][(src, sporadic, METRIC)].usual_peak == 0.0
+    assert at[14][(src, sporadic, METRIC)].usual_height == 50.0
+    assert at[2][(src, nightly, METRIC)].usual_height == 50.0
+    assert at[14][(src, nightly, METRIC)].usual_height == 0.0
+
+
 def test_a_metric_learned_without_its_parts_ignores_them(app, src):
     """A host's CPU learns a normal and each of its cores doesn't (#5742):
     with parts=False, "<host>:core0" is neither learned nor judged, and the