diff --git a/README.md b/README.md index d7e2999..3a4e09f 100644 --- a/README.md +++ b/README.md @@ -122,4 +122,5 @@ See `docs/plugins/` for a full plugin development guide. | `docs/plugins/overview.md` | Plugin system: how loading works, plugin.yaml schema, required exports | | `docs/plugins/writing-a-plugin.md` | Step-by-step plugin development guide | | `docs/plugins/traefik.md` | Traefik plugin: config, metrics, alert rule examples | +| `docs/plugins/prometheus.md` | Prometheus plugin: scraping any /metrics endpoint, how series are stored, the Fabled Archives recipe | | `docs/reference/code-map.md` | Where every key function, model, and route lives in the codebase | diff --git a/docs/plugins/prometheus.md b/docs/plugins/prometheus.md new file mode 100644 index 0000000..fa2bc8f --- /dev/null +++ b/docs/plugins/prometheus.md @@ -0,0 +1,84 @@ +# Prometheus Plugin + +The Prometheus plugin scrapes any service's Prometheus `/metrics` endpoint and stores what you choose to keep in Steward's metrics. Once stored, a series can be charted, read through the MCP `query_metrics` tool, and watched by an alert rule like any other metric. + +It is on by default. Targets are added, tested and edited on its page, **Prometheus** in the sidebar (`/plugins/prometheus/`). + +--- + +## Adding a target + +| Field | What it is | +|---|---| +| Name | How the target's series are named in Steward (rule pickers, charts). No `/`, `:` or comma. | +| Metrics address | The full URL, e.g. `http://archives:8081/metrics`. | +| Bearer token | Sent as `Authorization: Bearer …`. Stored encrypted and never shown again; leave it blank to keep the saved one. | +| Scrape every | 10 s to an hour. Default 60 s. | +| Check its HTTPS certificate | Off only for a self-signed certificate you trust. | +| Keep / Leave out | Name patterns, one a line, `*` for any run of characters. Empty Keep keeps everything; Leave out wins over Keep. | +| Series cap | The most series stored from one scrape (default 1000). Past it, the rest are counted and shown on the target's page, not stored. | + +**Test** scrapes the address once with what is typed, filters included, and says how many series would be stored and under which names. Nothing is saved. + +The endpoint hands over everything; Steward chooses what to keep. Start narrow (`yourservice_*`) and widen it if you find you want more: every stored series is a row every scrape. + +--- + +## How series are stored + +| On the endpoint | In Steward | +|---|---| +| A gauge or an untyped sample | Stored as it is, under its own name. | +| A counter (`requests_total`) | Stored as a rate per second, `requests_per_sec`, from the second scrape on. A value lower than the last (the service restarted) is skipped rather than read as a negative rate. | +| A histogram or summary | Its `_sum` and `_count` are stored as rates (`x_sum_per_sec`, `x_count_per_sec`). Buckets and quantiles are not stored. | +| NaN or ±Inf | Not stored. | + +A series without labels is stored under the target's name. A labelled one is stored under `: