bvandeusenandClaude Opus 5.5
f3433fe3ce
Host CPU, memory and load per core, container CPU and memory, and every monitor's response time learn a normal per hour (milestone 513, System 182). A reading past its fixed cutoff but within its normal reads healthy and says what is usual at this hour; one well out of it keeps its grade and is a metric anomaly. Disk, a monitor down and an agent offline keep their cutoffs. - Each normal's floor, the smallest departure that counts, is a setting: Settings -> Thresholds -> Normals (norms.floor.*). Defaults: host CPU 10 points, host memory 5, load per core 10, container CPU 10 points of one core, container memory 5, monitor response 100 ms; Traefik's 2-point 4xx and 50 ms p95 floors move there from constants. learn_normal takes a floor function (floor_setting) read at each pass. - learn_normal(parts=False): a host learns, its cores don't. - Ingest records load_pct (load_1m / cores x 100), the figure pages grade. - Shared grading: anomalies.against / graded, worst_reading(normals=), hosts' host_grades, docker usage(thresholds=, norms=), monitors' latency_grade. Applied to host vitals, the Hosts widget, MCP overview, Docker usage, and the monitor list, widget and page. - The monitor and container pages list their anomalies. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
126 lines
6.7 KiB
Markdown
126 lines
6.7 KiB
Markdown
# Traefik Plugin
|
||
|
||
The Traefik plugin scrapes the Traefik reverse proxy's Prometheus `/metrics` endpoint on a configurable interval, stores per-router metrics, and surfaces them on the dashboard and a dedicated detail page.
|
||
|
||
---
|
||
|
||
## What It Does
|
||
|
||
On each scrape:
|
||
1. Fetches raw Prometheus text from the configured `metrics_url`
|
||
2. Parses histogram and counter data to compute per-router rates and latency approximations
|
||
3. Writes computed metrics to the `traefik_metrics` history table
|
||
4. Calls `record_metric()` for each metric, making them available to alert rules
|
||
|
||
Request rates and error rates are computed as deltas between the current and previous scrape, divided by elapsed time. Latency percentiles (p50, p95, p99) are approximated via linear interpolation over histogram buckets — these are estimates, not exact percentiles. They cover the responses since the previous scrape (each bucket's rise, summed over every status code and method), not every response since Traefik started. A scrape in which a router answered nothing records no latency, shown as "—"; one past the last bucket (5 s by default) reads as that bucket's edge.
|
||
|
||
---
|
||
|
||
## Configuration
|
||
|
||
| Key | Default | Description |
|
||
|---|---|---|
|
||
| `metrics_url` | `http://localhost:8080/metrics` | Traefik Prometheus endpoint |
|
||
| `scrape_interval_seconds` | `60` | How often to scrape |
|
||
|
||
Enable the plugin by setting `enabled: true` in the app settings (Settings UI or directly in `app_settings` DB table under key `plugin.traefik`).
|
||
|
||
Traefik must have metrics enabled. In Traefik config:
|
||
|
||
```yaml
|
||
# traefik.yml
|
||
metrics:
|
||
prometheus: {}
|
||
```
|
||
|
||
---
|
||
|
||
## Metrics Collected
|
||
|
||
Per Traefik router, per scrape:
|
||
|
||
| Metric name | Description |
|
||
|---|---|
|
||
| `request_rate` | Requests per second (delta from previous scrape) |
|
||
| `error_rate_4xx_pct` | 4xx responses as % of total requests |
|
||
| `error_rate_5xx_pct` | 5xx responses as % of total requests |
|
||
| `latency_p50_ms` | Approximate p50 latency (ms) |
|
||
| `latency_p95_ms` | Approximate p95 latency (ms) |
|
||
| `latency_p99_ms` | Approximate p99 latency (ms) |
|
||
|
||
All metrics are available for alert rules with `source_module = "traefik"` and `resource_name = <router name>`.
|
||
|
||
---
|
||
|
||
## Alert Rule Examples
|
||
|
||
| Rule | source_module | resource_name | metric_name | operator | threshold |
|
||
|---|---|---|---|---|---|
|
||
| High 5xx rate on API router | `traefik` | `api@docker` | `error_rate_5xx_pct` | `>` | `1.0` |
|
||
| Slow API (p95 > 500ms) | `traefik` | `api@docker` | `latency_p95_ms` | `>` | `500` |
|
||
| Traffic spike | `traefik` | `web@docker` | `request_rate` | `>` | `1000` |
|
||
|
||
Router names are the Traefik router labels as reported in the Prometheus metrics (e.g. `api@docker`, `dashboard@internal`). Check the raw metrics at your `metrics_url` to see the exact names in use.
|
||
|
||
---
|
||
|
||
## UI
|
||
|
||
### Dashboard Widget
|
||
|
||
The widget appears on the dashboard when the Traefik plugin is enabled. It shows, per router:
|
||
|
||
- Router name
|
||
- Current req/s
|
||
- p95 latency (color-coded: green < 200ms, yellow < 500ms, red ≥ 500ms)
|
||
- 5xx error rate (shown only if > 0)
|
||
|
||
The widget auto-refreshes via HTMX polling every `poll_interval` seconds. Fragment endpoint: `GET /plugins/traefik/widget`.
|
||
|
||
### Detail Page
|
||
|
||
The full Traefik page at `/plugins/traefik/` reads top to bottom, for the time range picked at the top:
|
||
|
||
1. **The verdict.** It names the routers that need a look, says why, and says when the problem started ("api, 3.0% of responses are 5xx, started 2h ago"). When a router has been a problem for the whole range, it says so instead. Otherwise it reads "All N routers healthy".
|
||
2. **The request chart.** Requests per second across all routers and the share answered 5xx, the same chart as the dashboard widget.
|
||
3. **A card for each router that needs a look:** why, since when, the 4xx and 5xx status codes it answered in the last 15 minutes, most first ("404 ×1,812, 502 ×40"), and its request rate, bandwidth, p95/p99 latency and 4xx/5xx shares, each with its history. The code is what separates scanners probing for pages (404), a login wall (401/403) and an app that isn't answering (502/503/504). Steward records codes from the version that added them on; until then a card says they aren't recorded yet.
|
||
4. **Traefik's own figures:** when its config last reloaded, open connections, memory.
|
||
5. **The healthy routers,** folded into one table, then the routers Traefik no longer reports, then TLS certificates by soonest expiry, then Traffic Origin when the access log is on.
|
||
|
||
A router is graded on its last 15 minutes against Settings → Thresholds. When a problem started is read from the router's history over the range.
|
||
|
||
Once a router has 5 days of history, its 4xx share and p95 latency are judged against its own normal for that hour (see [Normals and anomalies](../architecture.md)) instead of the fixed cutoffs. One over the cutoff but within its normal is healthy, and the healthy table says it is usual for that router (an error-pages service that answers 4xx all day, a login wall's steady 401s). One out of its normal is a warning that says what is usual, and a metric anomaly to review; the Traefik page lists the routers' anomalies of the last 30 days. How far out counts is set on Settings → Thresholds → Normals (2 points of 4xx share, 50 ms of p95 by default). A 4xx share over fewer requests in the 15 minutes than Settings → Thresholds allows (30 by default) isn't judged at all. The 5xx share is a failure and always keeps its fixed cutoff. Latency is learned only from #5755's per-scrape figures, so its normal arrives 5 days after that release.
|
||
|
||
---
|
||
|
||
## File Locations
|
||
|
||
| File | Purpose |
|
||
|---|---|
|
||
| `plugins/traefik/__init__.py` | Plugin entry point: `setup()`, `get_scheduled_tasks()`, `get_blueprint()` |
|
||
| `plugins/traefik/plugin.yaml` | Plugin metadata and default config |
|
||
| `plugins/traefik/models.py` | `TraefikMetric` SQLAlchemy model |
|
||
| `plugins/traefik/scheduler.py` | `make_scrape_task()` and scrape logic |
|
||
| `plugins/traefik/scraper.py` | Prometheus text parsing and metric computation |
|
||
| `plugins/traefik/routes.py` | `GET /` (detail page) and `GET /widget` (HTMX fragment) |
|
||
| `plugins/traefik/migrations/` | Alembic migration for `traefik_metrics` table |
|
||
| `plugins/traefik/templates/traefik/` | `index.html` (detail) and `widget.html` (dashboard fragment) |
|
||
|
||
---
|
||
|
||
## Database Table
|
||
|
||
`traefik_metrics` (defined in `plugins/traefik/models.py`):
|
||
|
||
| Column | Type | Description |
|
||
|---|---|---|
|
||
| `id` | UUID | Primary key |
|
||
| `router_name` | str | Traefik router name |
|
||
| `scraped_at` | timestamp UTC | When this row was written |
|
||
| `request_rate` | float | req/s |
|
||
| `error_rate_4xx_pct` | float | % 4xx |
|
||
| `error_rate_5xx_pct` | float | % 5xx |
|
||
| `latency_p50_ms` | float | Approx p50 (ms) |
|
||
| `latency_p95_ms` | float | Approx p95 (ms) |
|
||
| `latency_p99_ms` | float | Approx p99 (ms) |
|