FabledSteward/docs/plugins/traefik.md

Traefik Plugin

The Traefik plugin scrapes the Traefik reverse proxy's Prometheus /metrics endpoint on a configurable interval, stores per-router metrics, and surfaces them on the dashboard and a dedicated detail page.

---

What It Does

On each scrape:

1. Fetches raw Prometheus text from the configured metrics_url
2. Parses histogram and counter data to compute per-router rates and latency approximations
3. Writes computed metrics to the traefik_metrics history table
4. Calls record_metric() for each metric, making them available to alert rules

Request rates and error rates are computed as deltas between the current and previous scrape, divided by elapsed time. Latency percentiles (p50, p95, p99) are approximated via linear interpolation over histogram buckets — these are estimates, not exact percentiles. They cover the responses since the previous scrape (each bucket's rise, summed over every status code and method), not every response since Traefik started. A scrape in which a router answered nothing records no latency, shown as "—"; one past the last bucket (5 s by default) reads as that bucket's edge.

When the metrics URL can reach more than one Traefik (a swarm service's replicas behind the routing mesh or its VIP, a Kubernetes Service's pods, a keepalived pair after a failover, or the same Traefik after a restart), each process keeps its own counters. Steward tells them apart by process_start_time_seconds and compares each scrape only with the same process's previous one. The first scrape from a process it hasn't seen in the last hour sets that process's baseline and counts nothing. Through a single address the figures are those of whichever replica answered, not the sum of all of them. Scraping every replica and adding them up is not supported yet.

---

Configuration

Key	Default	Description
metrics_url	http://localhost:8080/metrics	Traefik Prometheus endpoint
scrape_interval_seconds	60	How often to scrape

Enable the plugin by setting enabled: true in the app settings (Settings UI or directly in app_settings DB table under key plugin.traefik).

Traefik must have metrics enabled. In Traefik config:

# traefik.yml
metrics:
  prometheus: {}

---

Metrics Collected

Per Traefik router, per scrape:

Metric name	Description
request_rate	Requests per second (delta from previous scrape)
error_rate_4xx_pct	4xx responses as % of total requests
error_rate_5xx_pct	5xx responses as % of total requests
latency_p50_ms	Approximate p50 latency (ms)
latency_p95_ms	Approximate p95 latency (ms)
latency_p99_ms	Approximate p99 latency (ms)

All metrics are available for alert rules with source_module = "traefik" and resource_name = <router name>.

---

Alert Rule Examples

Rule	source_module	resource_name	metric_name	operator	threshold
High 5xx rate on API router	traefik	api@docker	error_rate_5xx_pct	>	1.0
Slow API (p95 > 500ms)	traefik	api@docker	latency_p95_ms	>	500
Traffic spike	traefik	web@docker	request_rate	>	1000

Router names are the Traefik router labels as reported in the Prometheus metrics (e.g. api@docker, dashboard@internal). Check the raw metrics at your metrics_url to see the exact names in use.

---

UI

Dashboard Widget

The widget appears on the dashboard when the Traefik plugin is enabled. It shows, per router:

• Router name
• Current req/s
• p95 latency (color-coded: green < 200ms, yellow < 500ms, red ≥ 500ms)
• 5xx error rate (shown only if > 0)

The widget auto-refreshes via HTMX polling every poll_interval seconds. Fragment endpoint: GET /plugins/traefik/widget.

Detail Page

The full Traefik page at /plugins/traefik/ reads top to bottom, for the time range picked at the top:

1. The verdict. It names the routers that need a look, says why, and says when the problem started ("api, 3.0% of responses are 5xx, started 2h ago"). When a router has been a problem for the whole range, it says so instead. Otherwise it reads "All N routers healthy".
2. The request chart. Requests per second across all routers and the share answered 5xx, the same chart as the dashboard widget.
3. A card for each router that needs a look: why, since when, the 4xx and 5xx status codes it answered in the last 15 minutes, most first ("404 ×1,812, 502 ×40"), and its request rate, bandwidth, p95/p99 latency and 4xx/5xx shares, each with its history. The code is what separates scanners probing for pages (404), a login wall (401/403) and an app that isn't answering (502/503/504). Steward records codes from the version that added them on; until then a card says they aren't recorded yet. Each card then says what its problem usually means and what to do (plugins/traefik/meaning.py, #5712):
   • Server errors (5xx) are almost always real. Mostly 500: the app is failing requests. Mostly 502/503/504: Traefik couldn't reach the app, or it timed out.
   • Client errors (4xx) are usually traffic rather than a fault. Mostly 404: scanners and bots. 401/403: a login wall. 429: rate limiting. Mostly 499 is the exception: clients giving up on a slow app.
   • Slow responses compare p95 with p50. When p95 is 4× p50 or more, a few requests are slow (one endpoint or a heavy page). When p50 is close behind, the whole app is slow.
   • Next: a fault links to the swarm service behind the router (its tasks and logs) when Steward can match the name ("codex@swarm" → codex_app, "radarr@swarm" → arr_radarr), otherwise to Docker → Swarm. Traffic links to Settings → Thresholds.
4. Traefik's own figures: when its config last reloaded, open connections, memory.
5. The healthy routers, folded into one table, then the routers Traefik no longer reports, then TLS certificates by soonest expiry, then Traffic Origin when the access log is on.

A router is graded on its last 15 minutes against Settings → Thresholds. When a problem started is read from the router's history over the range.

Once a router has 5 days of history, its 4xx share and p95 latency are judged against its own normal for that hour (see Normals and anomalies) instead of the fixed cutoffs. One over the cutoff but within its normal is healthy, and the healthy table says it is usual for that router (an error-pages service that answers 4xx all day, a login wall's steady 401s). One out of its normal is a warning that says what is usual, and a metric anomaly to review; the Traefik page lists the routers' anomalies of the last 30 days. How far out counts is set on Settings → Thresholds → Normals (2 points of 4xx share, 50 ms of p95 by default). A 4xx share over fewer requests in the 15 minutes than Settings → Thresholds allows (30 by default) isn't judged at all. The 5xx share is a failure and always keeps its fixed cutoff. Latency is learned only from #5755's per-scrape figures, so its normal arrives 5 days after that release.

---

File Locations

File	Purpose
plugins/traefik/__init__.py	Plugin entry point: setup(), get_scheduled_tasks(), get_blueprint()
plugins/traefik/plugin.yaml	Plugin metadata and default config
plugins/traefik/models.py	TraefikMetric SQLAlchemy model
plugins/traefik/scheduler.py	make_scrape_task() and scrape logic
plugins/traefik/scraper.py	Prometheus text parsing and metric computation
plugins/traefik/routes.py	GET / (detail page) and GET /widget (HTMX fragment)
plugins/traefik/migrations/	Alembic migration for traefik_metrics table
plugins/traefik/templates/traefik/	index.html (detail) and widget.html (dashboard fragment)

---

Database Table

traefik_metrics (defined in plugins/traefik/models.py):

Column	Type	Description
id	UUID	Primary key
router_name	str	Traefik router name
scraped_at	timestamp UTC	When this row was written
request_rate	float	req/s
error_rate_4xx_pct	float	% 4xx
error_rate_5xx_pct	float	% 5xx
latency_p50_ms	float	Approx p50 (ms)
latency_p95_ms	float	Approx p95 (ms)
latency_p99_ms	float	Approx p99 (ms)