fix(traefik): latency is each scrape's own percentile, every label set summed (#5755)
CI / lint (push) Successful in 2s
CI / integration (push) Successful in 1m23s
CI / unit (push) Successful in 1m27s
CI / publish (push) Successful in 17s

#5755 "Traefik latency was a percentile of every response since Traefik
started, over unsummed label sets". _percentile_ms read the duration
histogram as it stood, which counts from Traefik's start, so p50/p95/p99 were
lifetime figures that barely moved. It also walked every code/method label
set's buckets as one unsummed list, so equal counts from two sets returned
an edge exactly: nextcloud's flat 100.0 ms p95.

service_buckets() now takes each bucket's rise since the previous scrape,
summed per le over every label set. A scrape with no responses, and the
first scrape, record no latency: traefik_007_latency_nullable lets the
columns be NULL, the batch skips the figure, and the page shows "—".

Found while preparing #5741 "Traefik routers are graded against their own
normal", which grades p95 against a learned normal.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-10-14 14:45:07 -04:00
co-authored by Claude Opus 5.5
parent cd17def3ba
commit 98a0ced5eb
8 changed files with 143 additions and 39 deletions
+7 -3
View File
@@ -34,6 +34,10 @@
{% endif %}
{% endmacro %}
{# A router that has answered nothing since Steward started has no latency yet. #}
{% macro ms(v) %}{% if v is none %}—{% else %}{{ "%.0f"|format(v) }} ms{% endif %}{% endmacro %}
{% macro latency_cell(v) %}<td{% if v is not none %} class="tone-{{ status_info(status_class(v, 'latency'))[0] }}"{% endif %}>{{ ms(v) }}</td>{% endmacro %}
{% macro router_card(r) %}
<div class="card">
<div class="host-health-head">
@@ -54,8 +58,8 @@
<table class="table" style="font-size:0.85rem;">
<tr><td class="field-hint">Req/s</td><td>{{ "%.2f"|format(r.latest.request_rate) }}</td><td>{{ r.sparkline_req|safe }}</td></tr>
<tr><td class="field-hint">Bandwidth</td><td>{{ r.latest.response_bytes_rate|rate }}</td><td>{{ r.sparkline_bytes|safe }}</td></tr>
<tr><td class="field-hint">p95</td><td class="tone-{{ status_info(status_class(r.latest.latency_p95_ms, 'latency'))[0] }}">{{ "%.0f"|format(r.latest.latency_p95_ms) }} ms</td><td>{{ r.sparkline_p95|safe }}</td></tr>
<tr><td class="field-hint">p99</td><td>{{ "%.0f"|format(r.latest.latency_p99_ms) }} ms</td><td></td></tr>
<tr><td class="field-hint">p95</td>{{ latency_cell(r.latest.latency_p95_ms) }}<td>{{ r.sparkline_p95|safe }}</td></tr>
<tr><td class="field-hint">p99</td><td>{{ ms(r.latest.latency_p99_ms) }}</td><td></td></tr>
<tr><td class="field-hint">4xx</td><td class="{{ 'tone-warn' if r.latest.error_rate_4xx_pct > error_4xx else '' }}">{{ "%.1f"|format(r.latest.error_rate_4xx_pct) }}%</td><td>{{ r.sparkline_4xx|safe }}</td></tr>
<tr><td class="field-hint">5xx</td><td class="{{ 'tone-crit' if r.latest.error_rate_5xx_pct > error_5xx else '' }}">{{ "%.1f"|format(r.latest.error_rate_5xx_pct) }}%</td><td>{{ r.sparkline_5xx|safe }}</td></tr>
</table>
@@ -88,7 +92,7 @@
<td style="word-break:break-all;">{{ r.name }}</td>
<td>{{ "%.2f"|format(r.latest.request_rate) }}</td>
<td>{{ r.latest.response_bytes_rate|rate }}</td>
<td class="tone-{{ status_info(status_class(r.latest.latency_p95_ms, 'latency'))[0] }}">{{ "%.0f"|format(r.latest.latency_p95_ms) }} ms</td>
{{ latency_cell(r.latest.latency_p95_ms) }}
<td>{{ "%.1f"|format(r.latest.error_rate_4xx_pct) }}%</td>
<td>{{ "%.1f"|format(r.latest.error_rate_5xx_pct) }}%</td>
</tr>