feat(traefik): scrape every Traefik replica behind the metrics URL and add them up (#5844)

Through one address Steward read whichever replica answered, and with
host-mode ports behind a keepalived VIP that can be the replica serving
nothing: since the 18:43 restart every scrape landed on Docker03's idle
Traefik, which has no traefik_service_* series, so no router was stored.

A metrics_url host name that resolves to several addresses (a swarm
service's tasks.<service>, a Kubernetes headless Service, round-robin DNS)
is now scraped address by address, with the original Host header and TLS
name. Answers from one process are kept once; a replica that doesn't answer
is skipped with a warning. Requests, bytes and latency buckets are summed
over the replicas that have a baseline, so the sum's rise is the sum of each
one's own rise; certificates are listed once; Traefik's own figures add
connections and memory and keep the newest reload. A name with one address
is one scrape, as before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

docs/plugins/traefik.md

@@ -14,7 +14,18 @@ On each scrape:
 
 Request rates and error rates are computed as deltas between the current and previous scrape, divided by elapsed time. Latency percentiles (p50, p95, p99) are approximated via linear interpolation over histogram buckets — these are estimates, not exact percentiles. They cover the responses since the previous scrape (each bucket's rise, summed over every status code and method), not every response since Traefik started. A scrape in which a router answered nothing records no latency, shown as "—"; one past the last bucket (5 s by default) reads as that bucket's edge.
 
-When the metrics URL can reach more than one Traefik (a swarm service's replicas behind the routing mesh or its VIP, a Kubernetes Service's pods, a keepalived pair after a failover, or the same Traefik after a restart), each process keeps its own counters. Steward tells them apart by `process_start_time_seconds` and compares each scrape only with the same process's previous one. The first scrape from a process it hasn't seen in the last hour sets that process's baseline and counts nothing. Through a single address the figures are those of whichever replica answered, not the sum of all of them. Scraping every replica and adding them up is not supported yet.
+When the metrics URL can reach more than one Traefik (a swarm service's replicas behind the routing mesh or its VIP, a Kubernetes Service's pods, a keepalived pair after a failover, or the same Traefik after a restart), each process keeps its own counters. Steward tells them apart by `process_start_time_seconds` and compares each scrape only with the same process's previous one. The first scrape from a process it hasn't seen in the last hour sets that process's baseline and counts nothing. When `metrics_url`'s host name resolves to several addresses, Steward scrapes each one and adds the replicas' requests, bytes and latency buckets together, so the figures are the whole proxy's. Each replica still counts only against its own last scrape; one that doesn't answer is left out with a warning in the log. Certificates are listed once however many replicas serve them; Traefik's own figures add open connections and memory and show the newest reload.
+
+Through a name with one address (a swarm service's VIP, a Kubernetes ClusterIP, a load balancer) you get whichever replica answered, which can be one that serves no traffic at all. Point `metrics_url` at a name that lists every replica instead:
+
+| Where Traefik runs | `metrics_url` |
+|---|---|
+| One container | its address, e.g. `http://traefik:8080/metrics` |
+| Docker Swarm service (Steward on a network it shares with Traefik) | `http://tasks.<service>:8080/metrics`, e.g. `http://tasks.traefik_router:8080/metrics` |
+| Kubernetes | a headless Service's name |
+| A pair behind a keepalived VIP | a name that resolves to both nodes |
+
+`api_url` can stay on one address: every replica serves the same configuration.
 
 ---
 
@@ -22,7 +33,7 @@ When the metrics URL can reach more than one Traefik (a swarm service's replicas
 
 | Key | Default | Description |
 |---|---|---|
-| `metrics_url` | `http://localhost:8080/metrics` | Traefik Prometheus endpoint |
+| `metrics_url` | `http://localhost:8080/metrics` | Traefik Prometheus endpoint. A host name with several addresses is scraped address by address and added up (see above) |
 | `scrape_interval_seconds` | `60` | How often to scrape |
 
 Enable the plugin by setting `enabled: true` in the app settings (Settings UI or directly in `app_settings` DB table under key `plugin.traefik`).

plugins/traefik/plugin.yaml

@@ -13,6 +13,9 @@ tags:
   - access-log
 
 config:
+  # A host name that resolves to several Traefiks (a swarm service's
+  # tasks.<service>, a Kubernetes headless Service) is scraped address by
+  # address and added up, so every replica's traffic counts.
   metrics_url: "http://localhost:8080/metrics"
   # Traefik's API, for which hosts each router serves (so a certificate's
   # expiry shows on its router). Empty: /api on the metrics URL's host.

plugins/traefik/scheduler.py

@@ -64,11 +64,13 @@ async def _do_scrape(app: "Quart") -> None:
     from .scraper import (
         Baselines,
         api_base,
+        combine,
         compute_global_stats,
         compute_router_metrics,
         compute_status_codes,
-        extract_certs,
-        fetch_metrics,
+        fetch_instances,
+        merge_global_stats,
+        unique_certs,
         fetch_error_page_services,
         fetch_router_domains,
     )
@@ -77,7 +79,7 @@ async def _do_scrape(app: "Quart") -> None:
     metrics_url: str = app.config["PLUGINS"]["traefik"]["metrics_url"]
 
     try:
-        current = await fetch_metrics(metrics_url)
+        scrapes = await fetch_instances(metrics_url)
     except Exception:
         logger.exception("Traefik scrape failed (url=%s)", metrics_url)
         return
@@ -95,11 +97,12 @@ async def _do_scrape(app: "Quart") -> None:
 
     if _baselines is None:
         _baselines = Baselines()
-    previous, elapsed = _baselines.swap(current, time.monotonic())
+    now = time.monotonic()
+    current, previous, elapsed = combine([(scrape, *_baselines.swap(scrape, now)) for scrape in scrapes])
     router_metrics = compute_router_metrics(current, previous, elapsed)
-    global_stats = compute_global_stats(current, previous)
+    global_stats = merge_global_stats([compute_global_stats(scrape, None) for scrape in scrapes])
     status_codes = compute_status_codes(current, previous)
-    cert_list = extract_certs(current)
+    cert_list = unique_certs(scrapes)
 
     scraped_at = datetime.now(timezone.utc)
 

plugins/traefik/scraper.py

@@ -11,8 +11,11 @@ Traefik exposes Prometheus metrics at /metrics. This module:
 5. Computes process-level and config health stats
 """
 from __future__ import annotations
+import asyncio
+import logging
 import math
 import re
+import socket
 from collections import defaultdict
 from datetime import datetime, timezone
 
@@ -20,6 +23,8 @@ import httpx
 
 from steward.core import prometheus
 
+logger = logging.getLogger(__name__)
+
 
 # ──────────────────────────────────────────────────────────────────────────────
 # Prometheus text format parser
@@ -86,6 +91,63 @@ class Baselines:
         return prev[0], now - prev[1]
 
 
+# The counter families the per-router figures are computed from. Across
+# several Traefiks these add up (requests, bytes and latency buckets are each
+# replica's share); everything else in an exposition is a gauge or a
+# timestamp, which summing would turn into nonsense.
+SUMMED = ("traefik_service_requests_total", "traefik_service_responses_bytes_total",
+          "traefik_service_request_duration_seconds_bucket")
+
+
+def _sum(scrapes: list[ParsedMetrics]) -> ParsedMetrics:
+    out: ParsedMetrics = {}
+    for scrape in scrapes:
+        for name in SUMMED:
+            into = out.setdefault(name, {})
+            for labels, value in scrape.get(name, {}).items():
+                into[labels] = into.get(labels, 0.0) + value
+    return out
+
+
+def combine(pairs: list[tuple[ParsedMetrics, ParsedMetrics | None, float]],
+            ) -> tuple[ParsedMetrics, ParsedMetrics | None, float]:
+    """One (current, previous, elapsed) for the router arithmetic out of each
+    Traefik's (current, its own previous scrape, seconds since it).
+
+    Only the Traefiks that have a baseline are added up, current and previous
+    alike, so the sum's rise is the sum of each one's own rise: a replica seen
+    for the first time contributes from its next scrape on, and is never read
+    as traffic. When none has a baseline yet, every current comes back with no
+    previous, so each router is listed at zero."""
+    based = [(cur, prev, el) for cur, prev, el in pairs if prev is not None]
+    if not based:
+        return _sum([cur for cur, _, _ in pairs]), None, 0.0
+    return (_sum([cur for cur, _, _ in based]), _sum([prev for _, prev, _ in based]),
+            sum(el for _, _, el in based) / len(based))
+
+
+def merge_global_stats(stats: list[dict]) -> dict:
+    """Traefik's own figures across its replicas: connections and memory add
+    up; reloads and the last reload are the newest any replica reports."""
+    def values(key):
+        return [s[key] for s in stats if s.get(key) is not None]
+    conns, mem = values("open_conns_total"), values("process_memory_bytes")
+    reloads, last = values("config_reloads_total"), values("config_last_reload_at")
+    return {"open_conns_total": sum(conns) if conns else None,
+            "config_reloads_total": max(reloads) if reloads else None,
+            "config_last_reload_at": max(last) if last else None,
+            "process_memory_bytes": sum(mem) if mem else None}
+
+
+def unique_certs(scrapes: list[ParsedMetrics]) -> list[dict]:
+    """The certificates any replica serves, each once (by serial)."""
+    seen: dict[str, dict] = {}
+    for scrape in scrapes:
+        for cert in extract_certs(scrape):
+            seen.setdefault(cert["serial"], cert)
+    return list(seen.values())
+
+
 # ──────────────────────────────────────────────────────────────────────────────
 # Per-router metric computation
 # ──────────────────────────────────────────────────────────────────────────────
@@ -339,6 +401,53 @@ async def fetch_metrics(metrics_url: str) -> ParsedMetrics:
     return parse_prometheus(response.text)
 
 
+async def _addresses(host: str, port: int) -> list[str]:
+    """Every address `host` resolves to, in the order given, each once."""
+    infos = await asyncio.get_running_loop().getaddrinfo(host, port, type=socket.SOCK_STREAM)
+    return list(dict.fromkeys(info[4][0] for info in infos))
+
+
+async def fetch_instances(metrics_url: str) -> list[ParsedMetrics]:
+    """A scrape of every Traefik the metrics URL's host name resolves to.
+
+    A name for several replicas (a swarm service's `tasks.<service>`, a
+    Kubernetes headless Service, round-robin DNS) is scraped address by
+    address, each with the original Host header and TLS name, so every
+    replica's traffic is counted (#5844). A name with one address is one
+    scrape, as before. Two answers from the same process (localhost over IPv4
+    and IPv6) are kept once. A replica that doesn't answer is left out with a
+    warning; none answering raises."""
+    url = httpx.URL(metrics_url)
+    host = url.host
+    port = url.port or (443 if url.scheme == "https" else 80)
+    try:
+        addresses = await _addresses(host, port)
+    except OSError:
+        addresses = [host]
+    if len(addresses) <= 1:
+        return [await fetch_metrics(metrics_url)]
+
+    async def one(address: str) -> ParsedMetrics:
+        target = url.copy_with(host=address)
+        async with httpx.AsyncClient(timeout=10.0) as client:
+            response = await client.get(target, headers={"Host": url.netloc.decode()},
+                                        extensions={"sni_hostname": host})
+            response.raise_for_status()
+        return parse_prometheus(response.text)
+
+    results = await asyncio.gather(*(one(a) for a in addresses), return_exceptions=True)
+    scrapes: dict[str, ParsedMetrics] = {}
+    for address, result in zip(addresses, results):
+        if isinstance(result, BaseException):
+            logger.warning("Traefik at %s (%s) didn't answer: %s", address, host, result)
+            continue
+        key = instance_of(result) or address
+        scrapes.setdefault(key, result)
+    if not scrapes:
+        raise RuntimeError(f"no Traefik behind {host} answered ({len(addresses)} addresses)")
+    return list(scrapes.values())
+
+
 # ──────────────────────────────────────────────────────────────────────────────
 # Router domains (Traefik API)
 # ──────────────────────────────────────────────────────────────────────────────

tests/plugins/traefik/test_baselines.py

@@ -1,7 +1,16 @@
 """#5844: one metrics URL can reach several Traefik processes (swarm replicas,
 pods, a keepalived pair, a restart). Each is counted only against its own last
 scrape, so one replica's lifetime totals never read as another's new traffic."""
-from plugins.traefik.scraper import BASELINE_TTL_SECONDS, Baselines, compute_router_metrics, instance_of
+from datetime import datetime, timezone
+from functools import partial
+
+import httpx
+import pytest
+
+from plugins.traefik import scraper
+from plugins.traefik.scraper import (
+    BASELINE_TTL_SECONDS, Baselines, combine, compute_router_metrics, instance_of, merge_global_stats, unique_certs,
+)
 
 REQ = "traefik_service_requests_total"
 
@@ -55,3 +64,95 @@ def test_a_baseline_older_than_the_ttl_is_dropped_not_compared():
     b = Baselines()
     b.swap(_scrape(1000.0, **{"200": 1}), 0.0)
     assert b.swap(_scrape(1000.0, **{"200": 900}), BASELINE_TTL_SECONDS + 1) == (None, 0.0)
+
+
+# ── Every replica, added up (#5844) ──────────────────────────────────────────
+
+def _router(started, service, **counts):
+    out = {REQ: {frozenset({"service": service, "code": c, "method": "GET"}.items()): v
+                 for c, v in counts.items()}}
+    out["process_start_time_seconds"] = {frozenset(): started}
+    return out
+
+
+def test_two_replicas_add_up_and_a_new_one_waits_for_its_baseline():
+    a0, a1 = _router(1.0, "app@swarm", **{"200": 100}), _router(1.0, "app@swarm", **{"200": 160})
+    b0, b1 = _router(2.0, "app@swarm", **{"200": 50}), _router(2.0, "app@swarm", **{"200": 70, "502": 4})
+    c1 = _router(3.0, "app@swarm", **{"200": 9000})                 # first seen: no baseline
+    current, previous, elapsed = combine([(a1, a0, 60.0), (b1, b0, 60.0), (c1, None, 0.0)])
+    got = compute_router_metrics(current, previous, elapsed)["app@swarm"]
+    # (160-100) + (70-50) + 4 new 502s = 84 requests in 60 s; C's 9000 are not traffic.
+    assert got["request_rate"] == pytest.approx(84 / 60)
+    assert got["error_rate_5xx_pct"] == pytest.approx(4 / 84 * 100)
+
+
+def test_the_busy_replica_counts_though_the_idle_one_has_no_routers():
+    """The live case: the VIP sends everything to one replica; the other has
+    no traefik_service_* series at all."""
+    busy0, busy1 = _router(1.0, "roundtable@swarm", **{"200": 10}), _router(1.0, "roundtable@swarm", **{"200": 40})
+    idle = {"process_start_time_seconds": {frozenset(): 2.0}}
+    current, previous, elapsed = combine([(busy1, busy0, 60.0), (idle, idle, 60.0)])
+    assert compute_router_metrics(current, previous, elapsed)["roundtable@swarm"]["request_rate"] == 0.5
+
+
+def test_with_no_baseline_yet_every_router_is_listed_at_zero():
+    current, previous, _ = combine([(_router(1.0, "a@swarm", **{"200": 5}), None, 0.0),
+                                    (_router(2.0, "b@swarm", **{"200": 7}), None, 0.0)])
+    assert previous is None
+    got = compute_router_metrics(current, previous, 0.0)
+    assert set(got) == {"a@swarm", "b@swarm"} and got["a@swarm"]["request_rate"] == 0.0
+
+
+def test_traefiks_own_figures_add_up_or_take_the_newest():
+    t1, t2 = datetime(2026, 10, 15, 18, 0, tzinfo=timezone.utc), datetime(2026, 10, 15, 18, 43, tzinfo=timezone.utc)
+    got = merge_global_stats([
+        {"open_conns_total": 40.0, "config_reloads_total": 70.0, "config_last_reload_at": t1, "process_memory_bytes": 2e8},
+        {"open_conns_total": 1.0, "config_reloads_total": 72.0, "config_last_reload_at": t2, "process_memory_bytes": 1e8},
+    ])
+    assert got == {"open_conns_total": 41.0, "config_reloads_total": 72.0, "config_last_reload_at": t2,
+                   "process_memory_bytes": 3e8}
+
+
+def test_a_certificate_both_replicas_serve_is_listed_once():
+    cert = {"traefik_tls_certs_not_after": {frozenset({"serial": "ab", "cn": "x.example", "sans": ""}.items()): 2e9}}
+    assert [c["serial"] for c in unique_certs([cert, cert])] == ["ab"]
+
+
+EXPO = {
+    "10.0.2.5": "process_start_time_seconds 1.0\ntraefik_service_requests_total{service=\"a@swarm\",code=\"200\",method=\"GET\"} 10\n",
+    "10.0.2.6": "process_start_time_seconds 2.0\n",
+    "10.0.2.7": "process_start_time_seconds 1.0\n",                # the same process again
+}
+
+
+async def test_a_name_for_several_replicas_is_scraped_address_by_address(monkeypatch):
+    seen_hosts = []
+
+    def answer(request: httpx.Request) -> httpx.Response:
+        seen_hosts.append(request.headers["host"])
+        if request.url.host == "10.0.2.8":
+            return httpx.Response(503)
+        return httpx.Response(200, text=EXPO[request.url.host])
+
+    async def addresses(host, port):
+        assert (host, port) == ("tasks.traefik_router", 8080)
+        return ["10.0.2.5", "10.0.2.6", "10.0.2.7", "10.0.2.8"]
+
+    monkeypatch.setattr(scraper, "_addresses", addresses)
+    monkeypatch.setattr(scraper.httpx, "AsyncClient",
+                        partial(httpx.AsyncClient, transport=httpx.MockTransport(answer)))
+    got = await scraper.fetch_instances("http://tasks.traefik_router:8080/metrics")
+    # 10.0.2.8 didn't answer and 10.0.2.7 is 10.0.2.5's process: two Traefiks.
+    assert sorted(instance_of(g) for g in got) == ["1.00", "2.00"]
+    assert set(seen_hosts) == {"tasks.traefik_router:8080"}
+
+
+async def test_a_name_with_one_address_is_one_scrape_as_before(monkeypatch):
+    async def addresses(host, port):
+        return ["10.0.2.5"]
+
+    monkeypatch.setattr(scraper, "_addresses", addresses)
+    monkeypatch.setattr(scraper.httpx, "AsyncClient", partial(
+        httpx.AsyncClient, transport=httpx.MockTransport(lambda r: httpx.Response(200, text=EXPO["10.0.2.5"]))))
+    [got] = await scraper.fetch_instances("http://trfk.traefik.internal/metrics")
+    assert instance_of(got) == "1.00"