feat(traefik): routers are graded against their own normal (#5741)

#5741 "Traefik routers are graded against their own normal", milestone 513.

- A router's 4xx share and p95 learn their normal (learn_normal in the
  plugin's setup). Once a router has 5 days of history, a reading over the
  fixed cutoff but inside its normal is healthy, and the healthy table says
  it is usual for that router. One out of its normal is a warning that says
  what is usual, and a metric anomaly.
- The 5xx share is a failure: fixed cutoff always, whatever the history or
  traffic (System 182's ruling).
- A 4xx share over fewer requests than Settings -> Thresholds' new "Fewest
  requests to judge 4xx" (30 per 15 minutes) isn't graded on the page, and
  isn't judged for anomalies.
- Core: learn_normal(gate=(metric, least)) judges a key only while its
  resource's gate metric averages at least `least`; `least` may be a
  function, so it follows a setting. learn_normal(since=...) learns only
  from history after a moment: Traefik latency learns from #5755's
  per-scrape figures only.
- The Traefik page lists the routers' metric anomalies of the last 30 days
  (_anomaly_card.html). The dashboard widget keeps its why to problems.
- Thresholds and docs say when the cutoffs give way to a normal.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

steward/core/anomalies.py

@@ -36,6 +36,15 @@ REVIEWS = (TO_REVIEW, NORMAL, REAL, LAPSED)
 MARKS = (NORMAL, REAL)
 
 
+def _gate_open(learned, key: tuple[str, str, str], reported: dict) -> bool:
+    """Whether `key` may be judged: ungated, or its resource's gate metric
+    averaged at least the gate's least over the window."""
+    if learned.gate is None:
+        return True
+    metric, least = learned.gate
+    return reported.get((key[0], key[1], metric), 0.0) >= (least() if callable(least) else least)
+
+
 def departs(level: float, norm: Norm, floor: float) -> bool:
     """Whether `level` is out of `norm`: above its usual hold by DEPART times
     and by at least `floor`, so a 0.1% → 0.3% share is never news."""
@@ -60,8 +69,10 @@ WHERE l.source_module || '/' || l.metric_name = ANY(:keys)
 
 
 async def levels_now(session, now: datetime) -> dict[tuple[str, str, str], float]:
-    """{(source, resource, metric): its average over the last WINDOW}."""
-    keys = [f"{s}/{m}" for s, m in LEARNED]
+    """{(source, resource, metric): its average over the last WINDOW}, for
+    every learned metric and every metric one is gated on."""
+    keys = sorted({f"{s}/{m}" for s, m in LEARNED}
+                  | {f"{k.source_module}/{k.gate[0]}" for k in LEARNED.values() if k.gate})
     if not keys:
         return {}
     rows = await session.execute(_LEVELS, {"keys": keys, "since": now - WINDOW, "now": now})
@@ -72,7 +83,9 @@ async def detect(session, now: datetime | None = None) -> dict:
     """One pass: open, extend and close occurrences. Runs inside the caller's
     transaction. Returns {"opened", "closed"} counts."""
     now = now or datetime.now(timezone.utc)
-    levels = await levels_now(session, now)
+    reported = await levels_now(session, now)
+    levels = {key: level for key, level in reported.items()
+              if (key[0], key[2]) in LEARNED and _gate_open(LEARNED[(key[0], key[2])], key, reported)}
     norms = await norms_at(session, list(levels), now)
     open_now = {
         (a.source_module, a.resource_name, a.metric_name): (span, a)
@@ -97,8 +110,8 @@ async def detect(session, now: datetime | None = None) -> dict:
         elif out:
             await _occur(session, key, level, norm, now)
             opened += 1
-    # An open occurrence whose key stopped reporting, or lost its normal,
-    # can't be judged any more: it ends now.
+    # An open occurrence whose key stopped reporting, lost its normal or fell
+    # below its gate can't be judged any more: it ends now.
     for span, anomaly in open_now.values():
         _close(anomaly, span, now)
         closed += 1