feat(traefik): routers are graded against their own normal (#5741)
CI / lint (push) Successful in 3s
CI / integration (push) Successful in 1m18s
CI / unit (push) Successful in 1m31s
CI / publish (push) Successful in 1m3s

#5741 "Traefik routers are graded against their own normal", milestone 513.

- A router's 4xx share and p95 learn their normal (learn_normal in the
  plugin's setup). Once a router has 5 days of history, a reading over the
  fixed cutoff but inside its normal is healthy, and the healthy table says
  it is usual for that router. One out of its normal is a warning that says
  what is usual, and a metric anomaly.
- The 5xx share is a failure: fixed cutoff always, whatever the history or
  traffic (System 182's ruling).
- A 4xx share over fewer requests than Settings -> Thresholds' new "Fewest
  requests to judge 4xx" (30 per 15 minutes) isn't graded on the page, and
  isn't judged for anomalies.
- Core: learn_normal(gate=(metric, least)) judges a key only while its
  resource's gate metric averages at least `least`; `least` may be a
  function, so it follows a setting. learn_normal(since=...) learns only
  from history after a moment: Traefik latency learns from #5755's
  per-scrape figures only.
- The Traefik page lists the routers' metric anomalies of the last 30 days
  (_anomaly_card.html). The dashboard widget keeps its why to problems.
- Thresholds and docs say when the cutoffs give way to a normal.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-10-14 15:48:34 -04:00
co-authored by Claude Opus 5.5
parent 98a0ced5eb
commit 9f14620271
12 changed files with 321 additions and 46 deletions
+13
View File
@@ -20,6 +20,19 @@ def setup(app: "Quart") -> None:
prune_by_age(TraefikStatusCode.scraped_at)
prune_by_age(TraefikGlobalStat.scraped_at)
prune_by_age(TraefikAccessSummary.period_end)
# Each router's 4xx share and p95 learn their normal (#5741), judged only
# while the router answers the Thresholds minimum of requests a window
# (read at each pass, so a saved change applies at once). Latency is
# learned from the scrape-by-scrape figures only: before #5755 it was a
# percentile of every response since Traefik started.
from datetime import datetime, timezone
from steward.core.norms import learn_normal
from .routes import FLOOR_4XX, FLOOR_P95, STATE_WINDOW, min_requests
gate = ("request_rate",
lambda: min_requests(app.config.get("THRESHOLDS", {})) / STATE_WINDOW.total_seconds())
learn_normal("traefik", "error_rate_4xx_pct", FLOOR_4XX, gate=gate)
learn_normal("traefik", "latency_p95_ms", FLOOR_P95, gate=gate,
since=datetime(2026, 10, 14, 18, 47, tzinfo=timezone.utc))
def get_scheduled_tasks() -> list: