# Traefik Plugin

The Traefik plugin scrapes the Traefik reverse proxy's Prometheus `/metrics` endpoint on a configurable interval, stores per-router metrics, and surfaces them on the dashboard and a dedicated detail page.

---

## What It Does

On each scrape:
1. Fetches raw Prometheus text from the configured `metrics_url`
2. Parses histogram and counter data to compute per-router rates and latency approximations
3. Writes computed metrics to the `traefik_metrics` history table
4. Calls `record_metric()` for each metric, making them available to alert rules

Request rates and error rates are computed as deltas between the current and previous scrape, divided by elapsed time. Latency percentiles (p50, p95, p99) are approximated via linear interpolation over histogram buckets — these are estimates, not exact percentiles. They cover the responses since the previous scrape (each bucket's rise, summed over every status code and method), not every response since Traefik started. A scrape in which a router answered nothing records no latency, shown as "—"; one past the last bucket (5 s by default) reads as that bucket's edge.

When the metrics URL can reach more than one Traefik (a swarm service's replicas behind the routing mesh or its VIP, a Kubernetes Service's pods, a keepalived pair after a failover, or the same Traefik after a restart), each process keeps its own counters. Steward tells them apart by `process_start_time_seconds` and compares each scrape only with the same process's previous one. The first scrape from a process it hasn't seen in the last hour sets that process's baseline and counts nothing. When `metrics_url`'s host name resolves to several addresses, Steward scrapes each one and adds the replicas' requests, bytes and latency buckets together, so the figures are the whole proxy's. Each replica still counts only against its own last scrape; one that doesn't answer is left out with a warning in the log. Certificates are listed once however many replicas serve them; Traefik's own figures add open connections and memory and show the newest reload.

Through a name with one address (a swarm service's VIP, a Kubernetes ClusterIP, a load balancer) you get whichever replica answered, which can be one that serves no traffic at all. Point `metrics_url` at a name that lists every replica instead:

| Where Traefik runs | `metrics_url` |
|---|---|
| One container | its address, e.g. `http://traefik:8080/metrics` |
| Docker Swarm service (Steward on a network it shares with Traefik) | `http://tasks.<service>:8080/metrics`, e.g. `http://tasks.traefik_router:8080/metrics` |
| Kubernetes | a headless Service's name |
| A pair behind a keepalived VIP | a name that resolves to both nodes |

`api_url` can stay on one address: every replica serves the same configuration.

---

## Configuration

| Key | Default | Description |
|---|---|---|
| `metrics_url` | `http://localhost:8080/metrics` | Traefik Prometheus endpoint. A host name with several addresses is scraped address by address and added up (see above) |
| `scrape_interval_seconds` | `60` | How often to scrape |

Enable the plugin by setting `enabled: true` in the app settings (Settings UI or directly in `app_settings` DB table under key `plugin.traefik`).

Traefik must have metrics enabled. In Traefik config:

```yaml
# traefik.yml
metrics:
  prometheus: {}
```

---

## Metrics Collected

Per Traefik router, per scrape:

| Metric name | Description |
|---|---|
| `request_rate` | Requests per second (delta from previous scrape) |
| `error_rate_4xx_pct` | 4xx responses as % of total requests |
| `error_rate_5xx_pct` | 5xx responses as % of total requests |
| `latency_p50_ms` | Approximate p50 latency (ms) |
| `latency_p95_ms` | Approximate p95 latency (ms) |
| `latency_p99_ms` | Approximate p99 latency (ms) |

All metrics are available for alert rules with `source_module = "traefik"` and `resource_name = <router name>`.

---

## Alert Rule Examples

| Rule | source_module | resource_name | metric_name | operator | threshold |
|---|---|---|---|---|---|
| High 5xx rate on API router | `traefik` | `api@docker` | `error_rate_5xx_pct` | `>` | `1.0` |
| Slow API (p95 > 500ms) | `traefik` | `api@docker` | `latency_p95_ms` | `>` | `500` |
| Traffic spike | `traefik` | `web@docker` | `request_rate` | `>` | `1000` |

Router names are the Traefik router labels as reported in the Prometheus metrics (e.g. `api@docker`, `dashboard@internal`). Check the raw metrics at your `metrics_url` to see the exact names in use.

---

## UI

### Dashboard Widget

The widget appears on the dashboard when the Traefik plugin is enabled. It shows, per router:

- Router name
- Current req/s
- p95 latency (color-coded: green < 200ms, yellow < 500ms, red ≥ 500ms)
- 5xx error rate (shown only if > 0)

The widget auto-refreshes via HTMX polling every `poll_interval` seconds. Fragment endpoint: `GET /plugins/traefik/widget`.

### Detail Page

The full Traefik page at `/plugins/traefik/` reads top to bottom, for the time range picked at the top:

1. **The verdict.** It names the routers that need a look, says why, and says when the problem started ("api, 3.0% of responses are 5xx, started 2h ago"). When a router has been a problem for the whole range, it says so instead. Otherwise it reads "All N routers healthy".
2. **The request chart.** Requests per second across all routers and the share answered 5xx, the same chart as the dashboard widget.
3. **A card for each router that needs a look:** why, since when, the 4xx and 5xx status codes it answered in the last 15 minutes, most first ("404 ×1,812, 502 ×40"), and its request rate, bandwidth, p95/p99 latency and 4xx/5xx shares, each with its history. The code is what separates scanners probing for pages (404), a login wall (401/403) and an app that isn't answering (502/503/504). Steward records codes from the version that added them on; until then a card says they aren't recorded yet.
   Each card then says what its problem usually means and what to do (`plugins/traefik/meaning.py`, #5712):
   - **Server errors (5xx)** are almost always real. Mostly 500: the app is failing requests. Mostly 502/503/504: Traefik couldn't reach the app, or it timed out.
   - **Client errors (4xx)** are usually traffic rather than a fault. Mostly 404: scanners and bots. 401/403: a login wall. 429: rate limiting. Mostly 499 is the exception: clients giving up on a slow app.
   - **Slow responses** compare p95 with p50. When p95 is 4× p50 or more, a few requests are slow (one endpoint or a heavy page). When p50 is close behind, the whole app is slow.
   - **Next:** a fault links to the swarm service behind the router (its tasks and logs) when Steward can match the name ("codex@swarm" → `codex_app`, "radarr@swarm" → `arr_radarr`), otherwise to Docker → Swarm. Traffic links to Settings → Thresholds.
4. **Traefik's own figures:** when its config last reloaded, open connections, memory.
5. **The healthy routers,** folded into one table, then the routers Traefik no longer reports, then TLS certificates by soonest expiry, then Traffic Origin when the access log is on. A certificate Traefik has renewed is listed as old, not by its expiry, though a Traefik that hasn't restarted since keeps reporting it: one whose every name a later-expiring certificate covers.

A router is graded on its last 15 minutes against Settings → Thresholds. When a problem started is read from the router's history over the range.

Once a router has 5 days of history, its 4xx share and p95 latency are judged against its own normal for that hour (see [Normals and anomalies](../architecture.md)) instead of the fixed cutoffs. One over the cutoff but within its normal is healthy, and the healthy table says it is usual for that router (an error-pages service that answers 4xx all day, a login wall's steady 401s). One out of its normal is a warning that says what is usual, and a metric anomaly to review; the Traefik page lists the routers' anomalies of the last 30 days. How far out counts is set on Settings → Thresholds → Normals (2 points of 4xx share, 50 ms of p95 by default). A 4xx or 5xx share over fewer requests in the 15 minutes than Settings → Thresholds allows (30 by default) isn't judged at all: two 500s of sixteen requests would otherwise read 12% and critical. A 5xx share past its cutoff is failing only once the router answered at least 3 server errors in those 15 minutes (Settings → Thresholds, "Fewest server errors to fail"): one 502 among 70 requests is 1.4%, but it is one error. The 5xx share is a failure and keeps its fixed cutoff whatever the router's history. Latency is learned only from #5755's per-scrape figures, so its normal arrives 5 days after that release.

---

## File Locations

| File | Purpose |
|---|---|
| `plugins/traefik/__init__.py` | Plugin entry point: `setup()`, `get_scheduled_tasks()`, `get_blueprint()` |
| `plugins/traefik/plugin.yaml` | Plugin metadata and default config |
| `plugins/traefik/models.py` | `TraefikMetric` SQLAlchemy model |
| `plugins/traefik/scheduler.py` | `make_scrape_task()` and scrape logic |
| `plugins/traefik/scraper.py` | Prometheus text parsing and metric computation |
| `plugins/traefik/routes.py` | `GET /` (detail page) and `GET /widget` (HTMX fragment) |
| `plugins/traefik/migrations/` | Alembic migration for `traefik_metrics` table |
| `plugins/traefik/templates/traefik/` | `index.html` (detail) and `widget.html` (dashboard fragment) |

---

## Database Table

`traefik_metrics` (defined in `plugins/traefik/models.py`):

| Column | Type | Description |
|---|---|---|
| `id` | UUID | Primary key |
| `router_name` | str | Traefik router name |
| `scraped_at` | timestamp UTC | When this row was written |
| `request_rate` | float | req/s |
| `error_rate_4xx_pct` | float | % 4xx |
| `error_rate_5xx_pct` | float | % 5xx |
| `latency_p50_ms` | float | Approx p50 (ms) |
| `latency_p95_ms` | float | Approx p95 (ms) |
| `latency_p99_ms` | float | Approx p99 (ms) |