bvandeusenandClaude Opus 5.5
5ca13f9278
#5691 "Generic Prometheus dashboard widget: pick a target and a template", step 3 of milestone 508 "Prometheus widget templates". - /plugins/prometheus/widget draws one target with one template in widget shape: the verdict (led by "isn't answering" when the target is down), then each block with problems first, figures beside a share bar or the last 24h of bars, and the healthy rest folding to "+N more healthy" past six. A missing target or template, or one that no longer checks, says so. - library.py: bundled templates from plugins/prometheus/widgets/*.yaml, the picker's target and template lists, and the default title. - widgets/fabled_archives.yaml: the first bundled template, against the series Archives reports (refined in #5693). - Dashboard: a widget setting may take `options_from` (read when the picker opens) with an `empty` sentence; `default_title`; a Details link may name settings ("/plugins/prometheus/targets/{target}"). Prometheus is grouped with the integrations. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
97 lines
6.2 KiB
Markdown
97 lines
6.2 KiB
Markdown
# Prometheus Plugin
|
|
|
|
The Prometheus plugin scrapes any service's Prometheus `/metrics` endpoint and stores what you choose to keep in Steward's metrics. Once stored, a series can be charted, read through the MCP `query_metrics` tool, and watched by an alert rule like any other metric.
|
|
|
|
It is on by default. Targets are added, tested and edited on its page, **Prometheus** in the sidebar (`/plugins/prometheus/`).
|
|
|
|
---
|
|
|
|
## Adding a target
|
|
|
|
| Field | What it is |
|
|
|---|---|
|
|
| Name | How the target's series are named in Steward (rule pickers, charts). No `/`, `:` or comma. |
|
|
| Metrics address | The full URL, e.g. `http://archives:8081/metrics`. |
|
|
| Bearer token | Sent as `Authorization: Bearer …`. Stored encrypted and never shown again; leave it blank to keep the saved one. |
|
|
| Scrape every | 10 s to an hour. Default 60 s. |
|
|
| Check its HTTPS certificate | Off only for a self-signed certificate you trust. |
|
|
| Keep / Leave out | Name patterns, one a line, `*` for any run of characters. Empty Keep keeps everything; Leave out wins over Keep. |
|
|
| Series cap | The most series stored from one scrape (default 1000). Past it, the rest are counted and shown on the target's page, not stored. |
|
|
|
|
**Test** scrapes the address once with what is typed, filters included, and says how many series would be stored and under which names. Nothing is saved.
|
|
|
|
The endpoint hands over everything; Steward chooses what to keep. Start narrow (`yourservice_*`) and widen it if you find you want more: every stored series is a row every scrape.
|
|
|
|
---
|
|
|
|
## How series are stored
|
|
|
|
| On the endpoint | In Steward |
|
|
|---|---|
|
|
| A gauge or an untyped sample | Stored as it is, under its own name. |
|
|
| A counter (`requests_total`) | Stored as a rate per second, `requests_per_sec`, from the second scrape on. A value lower than the last (the service restarted) is skipped rather than read as a negative rate. |
|
|
| A histogram or summary | Its `_sum` and `_count` are stored as rates (`x_sum_per_sec`, `x_count_per_sec`). Buckets and quantiles are not stored. |
|
|
| NaN or ±Inf | Not stored. |
|
|
|
|
A series without labels is stored under the target's name. A labelled one is stored under `<target>:<label>=<value>,…`, in the order the endpoint gives the labels, for example `Archives:target=Steward,state=late`.
|
|
|
|
Each target also records four metrics about itself, under its own name:
|
|
|
|
| Metric | Meaning |
|
|
|---|---|
|
|
| `is_up` | 1 when the last scrape got metrics, 0 when it didn't. |
|
|
| `scrape_ms` | How long the scrape took. |
|
|
| `series_kept` | Series stored from the last scrape. |
|
|
| `series_capped` | Series the last scrape left out because of the cap. |
|
|
|
|
A value's unit is read from Prometheus naming: `_bytes` as bytes, `_seconds` as a span, `_timestamp_seconds` as a moment ("3h ago"), `_per_sec` as a rate.
|
|
|
|
A **state set** is one series per possible state, the current one 1 and the rest 0, such as `fabledarchives_backup_target_state{target, state}`. The target page shows it as one row per target (or job), naming the current state; the other states are listed on hover. A metric is read as a state set when its name ends in `_state` and every series has a `state` label reading 0 or 1. The OpenMetrics form, where the label is named after the metric (`foo{foo="a"} 1`), counts too. Each state is still stored as its own series, so an alert rule can watch one state.
|
|
|
|
**Only what the endpoint reports now is current.** When a scrape succeeds, any series it no longer has loses its newest value. That happens when the endpoint stopped reporting it, for example an old `build_info` after an upgrade, or when a filter or the cap now leaves it out. Its history stays on the chart until retention removes it. A failed scrape leaves the newest values as they were.
|
|
|
|
Renaming a target moves its stored history to the new name. Deleting one removes its newest values; its history ages out with retention (Settings → Retention).
|
|
|
|
---
|
|
|
|
## The dashboard widget
|
|
|
|
Edit a dashboard, open **Add widget**, and pick **Prometheus** under Integrations. Choose a **target** and a **template**; the widget is named after the template unless you give it a title.
|
|
|
|
A template says which of the target's series to show and how: backup states, figures with their units, times, a table, the version running. The widget leads with one line naming the worst problem, then each section with its problems first and the healthy rest after. Its **Details** link opens the target's page.
|
|
|
|
Steward ships with one template, **Fabled Archives**. A target whose series don't match its template says so in the widget rather than showing blanks.
|
|
|
|
## Alerting on a target
|
|
|
|
In **Alerts → New rule**, choose **Prometheus targets**, then the series (the resource) and the metric. Two rules worth having for any target:
|
|
|
|
- **It stopped answering:** resource `<target>`, metric `is_up`, `< 1`.
|
|
- **Something was cut off:** resource `<target>`, metric `series_capped`, `> 0`.
|
|
|
|
---
|
|
|
|
## Recipe: Fabled Archives
|
|
|
|
Archives serves `/metrics` on its backup listener (port 8081), which is reachable only on your LAN and the swarm's overlay network, behind a scrape token.
|
|
|
|
1. In Archives, **Settings → Metrics → Make a token**, named e.g. "Steward". Copy it; it is shown once.
|
|
2. In Steward, **Prometheus → Add target**:
|
|
- Name `Archives`
|
|
- Metrics address `http://<archives host>:8081/metrics` (on a shared overlay network, the service name, e.g. `http://archives_app:8081/metrics`)
|
|
- Bearer token: the one you copied
|
|
- Keep: `fabledarchives_*`
|
|
3. **Test**, then **Add target**.
|
|
|
|
Rules worth adding:
|
|
|
|
| Watch | Resource | Metric | Condition |
|
|
|---|---|---|---|
|
|
| A backup target is late | `Archives:target=<name>,target_id=<id>,state=late` | `fabledarchives_backup_target_state` | `>= 1` |
|
|
| A backup target is missing | `Archives:target=<name>,target_id=<id>,state=missing` | `fabledarchives_backup_target_state` | `>= 1` |
|
|
| A replication job is failing | `Archives:job=<name>,job_id=<id>,state=failing` | `fabledarchives_replication_job_state` | `>= 1` |
|
|
| Archives' data disk is filling | `Archives` | `fabledarchives_data_disk_free_bytes` | `<` your floor |
|
|
| Archives stopped answering | `Archives` | `is_up` | `< 1` |
|
|
|
|
The target's page lists every stored series with its exact resource name, so copy names from there rather than typing them.
|