From 002181253ed394903b220eb39acdf4deeb649a46 Mon Sep 17 00:00:00 2001 From: Bryan Van Deusen Date: Mon, 12 Oct 2026 07:06:31 -0400 Subject: [PATCH] fix(prometheus): the list page is a section top; the scrape test builds its text without format(); docs (#5668, #5670) - tests/test_breadcrumbs.py: the Prometheus target list is its section's own top page, like SNMP's, so it carries no trail. - The integration test's exposition has label braces, which str.format read as fields: the fake scrape raised, the target was recorded down and no series landed. It substitutes the counter with replace() now. - docs/plugins/prometheus.md: adding a target, filtering, how series are named and stored, alerting, and the Fabled Archives recipe. Co-Authored-By: Claude Opus 5.5 --- README.md | 1 + docs/plugins/prometheus.md | 84 +++++++++++++++++++++ tests/integration/test_prometheus_scrape.py | 4 +- tests/test_breadcrumbs.py | 1 + 4 files changed, 88 insertions(+), 2 deletions(-) create mode 100644 docs/plugins/prometheus.md diff --git a/README.md b/README.md index d7e2999..3a4e09f 100644 --- a/README.md +++ b/README.md @@ -122,4 +122,5 @@ See `docs/plugins/` for a full plugin development guide. | `docs/plugins/overview.md` | Plugin system: how loading works, plugin.yaml schema, required exports | | `docs/plugins/writing-a-plugin.md` | Step-by-step plugin development guide | | `docs/plugins/traefik.md` | Traefik plugin: config, metrics, alert rule examples | +| `docs/plugins/prometheus.md` | Prometheus plugin: scraping any /metrics endpoint, how series are stored, the Fabled Archives recipe | | `docs/reference/code-map.md` | Where every key function, model, and route lives in the codebase | diff --git a/docs/plugins/prometheus.md b/docs/plugins/prometheus.md new file mode 100644 index 0000000..fa2bc8f --- /dev/null +++ b/docs/plugins/prometheus.md @@ -0,0 +1,84 @@ +# Prometheus Plugin + +The Prometheus plugin scrapes any service's Prometheus `/metrics` endpoint and stores what you choose to keep in Steward's metrics. Once stored, a series can be charted, read through the MCP `query_metrics` tool, and watched by an alert rule like any other metric. + +It is on by default. Targets are added, tested and edited on its page, **Prometheus** in the sidebar (`/plugins/prometheus/`). + +--- + +## Adding a target + +| Field | What it is | +|---|---| +| Name | How the target's series are named in Steward (rule pickers, charts). No `/`, `:` or comma. | +| Metrics address | The full URL, e.g. `http://archives:8081/metrics`. | +| Bearer token | Sent as `Authorization: Bearer …`. Stored encrypted and never shown again; leave it blank to keep the saved one. | +| Scrape every | 10 s to an hour. Default 60 s. | +| Check its HTTPS certificate | Off only for a self-signed certificate you trust. | +| Keep / Leave out | Name patterns, one a line, `*` for any run of characters. Empty Keep keeps everything; Leave out wins over Keep. | +| Series cap | The most series stored from one scrape (default 1000). Past it, the rest are counted and shown on the target's page, not stored. | + +**Test** scrapes the address once with what is typed, filters included, and says how many series would be stored and under which names. Nothing is saved. + +The endpoint hands over everything; Steward chooses what to keep. Start narrow (`yourservice_*`) and widen it if you find you want more: every stored series is a row every scrape. + +--- + +## How series are stored + +| On the endpoint | In Steward | +|---|---| +| A gauge or an untyped sample | Stored as it is, under its own name. | +| A counter (`requests_total`) | Stored as a rate per second, `requests_per_sec`, from the second scrape on. A value lower than the last (the service restarted) is skipped rather than read as a negative rate. | +| A histogram or summary | Its `_sum` and `_count` are stored as rates (`x_sum_per_sec`, `x_count_per_sec`). Buckets and quantiles are not stored. | +| NaN or ±Inf | Not stored. | + +A series without labels is stored under the target's name. A labelled one is stored under `: