bvandeusenandClaude Opus 5.5
9f14620271
#5741 "Traefik routers are graded against their own normal", milestone 513. - A router's 4xx share and p95 learn their normal (learn_normal in the plugin's setup). Once a router has 5 days of history, a reading over the fixed cutoff but inside its normal is healthy, and the healthy table says it is usual for that router. One out of its normal is a warning that says what is usual, and a metric anomaly. - The 5xx share is a failure: fixed cutoff always, whatever the history or traffic (System 182's ruling). - A 4xx share over fewer requests than Settings -> Thresholds' new "Fewest requests to judge 4xx" (30 per 15 minutes) isn't graded on the page, and isn't judged for anomalies. - Core: learn_normal(gate=(metric, least)) judges a key only while its resource's gate metric averages at least `least`; `least` may be a function, so it follows a setting. learn_normal(since=...) learns only from history after a moment: Traefik latency learns from #5755's per-scrape figures only. - The Traefik page lists the routers' metric anomalies of the last 30 days (_anomaly_card.html). The dashboard widget keeps its why to problems. - Thresholds and docs say when the cutoffs give way to a normal. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
126 lines
6.6 KiB
Markdown
126 lines
6.6 KiB
Markdown
# Traefik Plugin
|
||
|
||
The Traefik plugin scrapes the Traefik reverse proxy's Prometheus `/metrics` endpoint on a configurable interval, stores per-router metrics, and surfaces them on the dashboard and a dedicated detail page.
|
||
|
||
---
|
||
|
||
## What It Does
|
||
|
||
On each scrape:
|
||
1. Fetches raw Prometheus text from the configured `metrics_url`
|
||
2. Parses histogram and counter data to compute per-router rates and latency approximations
|
||
3. Writes computed metrics to the `traefik_metrics` history table
|
||
4. Calls `record_metric()` for each metric, making them available to alert rules
|
||
|
||
Request rates and error rates are computed as deltas between the current and previous scrape, divided by elapsed time. Latency percentiles (p50, p95, p99) are approximated via linear interpolation over histogram buckets — these are estimates, not exact percentiles. They cover the responses since the previous scrape (each bucket's rise, summed over every status code and method), not every response since Traefik started. A scrape in which a router answered nothing records no latency, shown as "—"; one past the last bucket (5 s by default) reads as that bucket's edge.
|
||
|
||
---
|
||
|
||
## Configuration
|
||
|
||
| Key | Default | Description |
|
||
|---|---|---|
|
||
| `metrics_url` | `http://localhost:8080/metrics` | Traefik Prometheus endpoint |
|
||
| `scrape_interval_seconds` | `60` | How often to scrape |
|
||
|
||
Enable the plugin by setting `enabled: true` in the app settings (Settings UI or directly in `app_settings` DB table under key `plugin.traefik`).
|
||
|
||
Traefik must have metrics enabled. In Traefik config:
|
||
|
||
```yaml
|
||
# traefik.yml
|
||
metrics:
|
||
prometheus: {}
|
||
```
|
||
|
||
---
|
||
|
||
## Metrics Collected
|
||
|
||
Per Traefik router, per scrape:
|
||
|
||
| Metric name | Description |
|
||
|---|---|
|
||
| `request_rate` | Requests per second (delta from previous scrape) |
|
||
| `error_rate_4xx_pct` | 4xx responses as % of total requests |
|
||
| `error_rate_5xx_pct` | 5xx responses as % of total requests |
|
||
| `latency_p50_ms` | Approximate p50 latency (ms) |
|
||
| `latency_p95_ms` | Approximate p95 latency (ms) |
|
||
| `latency_p99_ms` | Approximate p99 latency (ms) |
|
||
|
||
All metrics are available for alert rules with `source_module = "traefik"` and `resource_name = <router name>`.
|
||
|
||
---
|
||
|
||
## Alert Rule Examples
|
||
|
||
| Rule | source_module | resource_name | metric_name | operator | threshold |
|
||
|---|---|---|---|---|---|
|
||
| High 5xx rate on API router | `traefik` | `api@docker` | `error_rate_5xx_pct` | `>` | `1.0` |
|
||
| Slow API (p95 > 500ms) | `traefik` | `api@docker` | `latency_p95_ms` | `>` | `500` |
|
||
| Traffic spike | `traefik` | `web@docker` | `request_rate` | `>` | `1000` |
|
||
|
||
Router names are the Traefik router labels as reported in the Prometheus metrics (e.g. `api@docker`, `dashboard@internal`). Check the raw metrics at your `metrics_url` to see the exact names in use.
|
||
|
||
---
|
||
|
||
## UI
|
||
|
||
### Dashboard Widget
|
||
|
||
The widget appears on the dashboard when the Traefik plugin is enabled. It shows, per router:
|
||
|
||
- Router name
|
||
- Current req/s
|
||
- p95 latency (color-coded: green < 200ms, yellow < 500ms, red ≥ 500ms)
|
||
- 5xx error rate (shown only if > 0)
|
||
|
||
The widget auto-refreshes via HTMX polling every `poll_interval` seconds. Fragment endpoint: `GET /plugins/traefik/widget`.
|
||
|
||
### Detail Page
|
||
|
||
The full Traefik page at `/plugins/traefik/` reads top to bottom, for the time range picked at the top:
|
||
|
||
1. **The verdict.** It names the routers that need a look, says why, and says when the problem started ("api, 3.0% of responses are 5xx, started 2h ago"). When a router has been a problem for the whole range, it says so instead. Otherwise it reads "All N routers healthy".
|
||
2. **The request chart.** Requests per second across all routers and the share answered 5xx, the same chart as the dashboard widget.
|
||
3. **A card for each router that needs a look:** why, since when, the 4xx and 5xx status codes it answered in the last 15 minutes, most first ("404 ×1,812, 502 ×40"), and its request rate, bandwidth, p95/p99 latency and 4xx/5xx shares, each with its history. The code is what separates scanners probing for pages (404), a login wall (401/403) and an app that isn't answering (502/503/504). Steward records codes from the version that added them on; until then a card says they aren't recorded yet.
|
||
4. **Traefik's own figures:** when its config last reloaded, open connections, memory.
|
||
5. **The healthy routers,** folded into one table, then the routers Traefik no longer reports, then TLS certificates by soonest expiry, then Traffic Origin when the access log is on.
|
||
|
||
A router is graded on its last 15 minutes against Settings → Thresholds. When a problem started is read from the router's history over the range.
|
||
|
||
Once a router has 5 days of history, its 4xx share and p95 latency are judged against its own normal for that hour (see [Normals and anomalies](../architecture.md)) instead of the fixed cutoffs. One over the cutoff but within its normal is healthy, and the healthy table says it is usual for that router (an error-pages service that answers 4xx all day, a login wall's steady 401s). One out of its normal is a warning that says what is usual, and a metric anomaly to review; the Traefik page lists the routers' anomalies of the last 30 days. A 4xx share over fewer requests in the 15 minutes than Settings → Thresholds allows (30 by default) isn't judged at all. The 5xx share is a failure and always keeps its fixed cutoff. Latency is learned only from #5755's per-scrape figures, so its normal arrives 5 days after that release.
|
||
|
||
---
|
||
|
||
## File Locations
|
||
|
||
| File | Purpose |
|
||
|---|---|
|
||
| `plugins/traefik/__init__.py` | Plugin entry point: `setup()`, `get_scheduled_tasks()`, `get_blueprint()` |
|
||
| `plugins/traefik/plugin.yaml` | Plugin metadata and default config |
|
||
| `plugins/traefik/models.py` | `TraefikMetric` SQLAlchemy model |
|
||
| `plugins/traefik/scheduler.py` | `make_scrape_task()` and scrape logic |
|
||
| `plugins/traefik/scraper.py` | Prometheus text parsing and metric computation |
|
||
| `plugins/traefik/routes.py` | `GET /` (detail page) and `GET /widget` (HTMX fragment) |
|
||
| `plugins/traefik/migrations/` | Alembic migration for `traefik_metrics` table |
|
||
| `plugins/traefik/templates/traefik/` | `index.html` (detail) and `widget.html` (dashboard fragment) |
|
||
|
||
---
|
||
|
||
## Database Table
|
||
|
||
`traefik_metrics` (defined in `plugins/traefik/models.py`):
|
||
|
||
| Column | Type | Description |
|
||
|---|---|---|
|
||
| `id` | UUID | Primary key |
|
||
| `router_name` | str | Traefik router name |
|
||
| `scraped_at` | timestamp UTC | When this row was written |
|
||
| `request_rate` | float | req/s |
|
||
| `error_rate_4xx_pct` | float | % 4xx |
|
||
| `error_rate_5xx_pct` | float | % 5xx |
|
||
| `latency_p50_ms` | float | Approx p50 (ms) |
|
||
| `latency_p95_ms` | float | Approx p95 (ms) |
|
||
| `latency_p99_ms` | float | Approx p99 (ms) |
|