bvandeusenandClaude Opus 5.5
002181253e
- tests/test_breadcrumbs.py: the Prometheus target list is its section's own top page, like SNMP's, so it carries no trail. - The integration test's exposition has label braces, which str.format read as fields: the fake scrape raised, the target was recorded down and no series landed. It substitutes the counter with replace() now. - docs/plugins/prometheus.md: adding a target, filtering, how series are named and stored, alerting, and the Fabled Archives recipe. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
85 lines
4.6 KiB
Markdown
85 lines
4.6 KiB
Markdown
# Prometheus Plugin
|
|
|
|
The Prometheus plugin scrapes any service's Prometheus `/metrics` endpoint and stores what you choose to keep in Steward's metrics. Once stored, a series can be charted, read through the MCP `query_metrics` tool, and watched by an alert rule like any other metric.
|
|
|
|
It is on by default. Targets are added, tested and edited on its page, **Prometheus** in the sidebar (`/plugins/prometheus/`).
|
|
|
|
---
|
|
|
|
## Adding a target
|
|
|
|
| Field | What it is |
|
|
|---|---|
|
|
| Name | How the target's series are named in Steward (rule pickers, charts). No `/`, `:` or comma. |
|
|
| Metrics address | The full URL, e.g. `http://archives:8081/metrics`. |
|
|
| Bearer token | Sent as `Authorization: Bearer …`. Stored encrypted and never shown again; leave it blank to keep the saved one. |
|
|
| Scrape every | 10 s to an hour. Default 60 s. |
|
|
| Check its HTTPS certificate | Off only for a self-signed certificate you trust. |
|
|
| Keep / Leave out | Name patterns, one a line, `*` for any run of characters. Empty Keep keeps everything; Leave out wins over Keep. |
|
|
| Series cap | The most series stored from one scrape (default 1000). Past it, the rest are counted and shown on the target's page, not stored. |
|
|
|
|
**Test** scrapes the address once with what is typed, filters included, and says how many series would be stored and under which names. Nothing is saved.
|
|
|
|
The endpoint hands over everything; Steward chooses what to keep. Start narrow (`yourservice_*`) and widen it if you find you want more: every stored series is a row every scrape.
|
|
|
|
---
|
|
|
|
## How series are stored
|
|
|
|
| On the endpoint | In Steward |
|
|
|---|---|
|
|
| A gauge or an untyped sample | Stored as it is, under its own name. |
|
|
| A counter (`requests_total`) | Stored as a rate per second, `requests_per_sec`, from the second scrape on. A value lower than the last (the service restarted) is skipped rather than read as a negative rate. |
|
|
| A histogram or summary | Its `_sum` and `_count` are stored as rates (`x_sum_per_sec`, `x_count_per_sec`). Buckets and quantiles are not stored. |
|
|
| NaN or ±Inf | Not stored. |
|
|
|
|
A series without labels is stored under the target's name. A labelled one is stored under `<target>:<label>=<value>,…`, in the order the endpoint gives the labels, for example `Archives:target=Steward,state=late`.
|
|
|
|
Each target also records four metrics about itself, under its own name:
|
|
|
|
| Metric | Meaning |
|
|
|---|---|
|
|
| `is_up` | 1 when the last scrape got metrics, 0 when it didn't. |
|
|
| `scrape_ms` | How long the scrape took. |
|
|
| `series_kept` | Series stored from the last scrape. |
|
|
| `series_capped` | Series the last scrape left out because of the cap. |
|
|
|
|
A value's unit is read from Prometheus naming: `_bytes` as bytes, `_seconds` as a span, `_timestamp_seconds` as a moment ("3h ago"), `_per_sec` as a rate.
|
|
|
|
Renaming a target moves its stored history to the new name. Deleting one removes its newest values; its history ages out with retention (Settings → Retention).
|
|
|
|
---
|
|
|
|
## Alerting on a target
|
|
|
|
In **Alerts → New rule**, choose **Prometheus targets**, then the series (the resource) and the metric. Two rules worth having for any target:
|
|
|
|
- **It stopped answering:** resource `<target>`, metric `is_up`, `< 1`.
|
|
- **Something was cut off:** resource `<target>`, metric `series_capped`, `> 0`.
|
|
|
|
---
|
|
|
|
## Recipe: Fabled Archives
|
|
|
|
Archives serves `/metrics` on its backup listener (port 8081), which is reachable only on your LAN and the swarm's overlay network, behind a scrape token.
|
|
|
|
1. In Archives, **Settings → Metrics → Make a token**, named e.g. "Steward". Copy it; it is shown once.
|
|
2. In Steward, **Prometheus → Add target**:
|
|
- Name `Archives`
|
|
- Metrics address `http://<archives host>:8081/metrics` (on a shared overlay network, the service name, e.g. `http://archives_app:8081/metrics`)
|
|
- Bearer token: the one you copied
|
|
- Keep: `fabledarchives_*`
|
|
3. **Test**, then **Add target**.
|
|
|
|
Rules worth adding:
|
|
|
|
| Watch | Resource | Metric | Condition |
|
|
|---|---|---|---|
|
|
| A backup target is late | `Archives:target=<name>,target_id=<id>,state=late` | `fabledarchives_backup_target_state` | `>= 1` |
|
|
| A backup target is missing | `Archives:target=<name>,target_id=<id>,state=missing` | `fabledarchives_backup_target_state` | `>= 1` |
|
|
| A replication job is failing | `Archives:job=<name>,job_id=<id>,state=failing` | `fabledarchives_replication_job_state` | `>= 1` |
|
|
| Archives' data disk is filling | `Archives` | `fabledarchives_data_disk_free_bytes` | `<` your floor |
|
|
| Archives stopped answering | `Archives` | `is_up` | `< 1` |
|
|
|
|
The target's page lists every stored series with its exact resource name, so copy names from there rather than typing them.
|