fix(traefik): count each Traefik process only against its own last scrape (#5844)
CI / lint (push) Successful in 2s
CI / unit (push) Successful in 1m28s
CI / integration (push) Successful in 1m33s
CI / publish (push) Successful in 20s

One metrics URL can reach several Traefik processes: a swarm service's
replicas behind the routing mesh or its VIP, a Kubernetes Service's pods,
a keepalived pair after failover, or the same Traefik after a restart.
Each keeps its own counters, so comparing one with another read the second
one's lifetime totals as new traffic: retired routers flashed 100% 502 and
an idle router showed 834 req/s whenever the scrape moved to the other
replica.

Each scrape is now keyed by Traefik's process_start_time_seconds and
compared only with the same process's previous scrape. A process seen for
the first time (or not in the last hour) sets its baseline and counts
nothing. Without a start time every scrape is one process, as before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-10-15 14:36:10 -04:00
co-authored by Claude Opus 5.5
parent 70eb253190
commit 98440d8de6
4 changed files with 116 additions and 12 deletions
+47
View File
@@ -39,6 +39,53 @@ def parse_prometheus(text: str) -> ParsedMetrics:
return metrics
# ──────────────────────────────────────────────────────────────────────────────
# Which Traefik answered (#5844)
# ──────────────────────────────────────────────────────────────────────────────
#
# One metrics URL can reach more than one Traefik: a swarm service's replicas
# behind the routing mesh or its VIP, a Kubernetes Service's pods, a keepalived
# pair after failover, or the same Traefik after a restart. Each process keeps
# its own counters, so comparing one process's scrape with another's reads the
# second one's lifetime totals as new traffic (a retired router flashing 100%
# 502, an idle one at 800 req/s). Each process is told apart by its start time,
# which Traefik has exposed since v2.0, and is only ever compared with itself.
# A process not heard from for this long has restarted or gone; its baseline
# is dropped rather than kept forever.
BASELINE_TTL_SECONDS = 3600
def instance_of(current: ParsedMetrics) -> str:
"""The Traefik process a scrape came from: its process_start_time_seconds,
or "" when the exposition doesn't carry one (then every scrape is taken to
be the same process, as before #5844)."""
start = current.get("process_start_time_seconds", {})
return f"{next(iter(start.values())):.2f}" if start else ""
class Baselines:
"""Each Traefik process's last scrape and when it was taken."""
def __init__(self) -> None:
self._seen: dict[str, tuple[ParsedMetrics, float]] = {}
def swap(self, current: ParsedMetrics, now: float) -> tuple[ParsedMetrics | None, float]:
"""(the previous scrape of the process `current` came from, seconds
since it), and keep `current` as that process's newest. A process seen
for the first time has no previous scrape: (None, 0.0), so nothing is
counted until it has a baseline of its own."""
key = instance_of(current)
prev = self._seen.get(key)
self._seen[key] = (current, now)
for k, (_, at) in list(self._seen.items()):
if now - at > BASELINE_TTL_SECONDS:
del self._seen[k]
if prev is None or now - prev[1] > BASELINE_TTL_SECONDS:
return None, 0.0
return prev[0], now - prev[1]
# ──────────────────────────────────────────────────────────────────────────────
# Per-router metric computation
# ──────────────────────────────────────────────────────────────────────────────