Files

T

bvandeusenandClaude Opus 5.5 002181253e fix(prometheus): the list page is a section top; the scrape test builds its text without format(); docs (#5668 , #5670 )

- tests/test_breadcrumbs.py: the Prometheus target list is its section's
  own top page, like SNMP's, so it carries no trail.
- The integration test's exposition has label braces, which str.format read
  as fields: the fake scrape raised, the target was recorded down and no
  series landed. It substitutes the counter with replace() now.
- docs/plugins/prometheus.md: adding a target, filtering, how series are
  named and stored, alerting, and the Fabled Archives recipe.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

2026-10-12 07:06:31 -04:00

4.6 KiB

Raw Blame History

Prometheus Plugin

The Prometheus plugin scrapes any service's Prometheus /metrics endpoint and stores what you choose to keep in Steward's metrics. Once stored, a series can be charted, read through the MCP query_metrics tool, and watched by an alert rule like any other metric.

It is on by default. Targets are added, tested and edited on its page, Prometheus in the sidebar (/plugins/prometheus/).

Adding a target

Field	What it is
Name	How the target's series are named in Steward (rule pickers, charts). No `/`, `:` or comma.
Metrics address	The full URL, e.g. `http://archives:8081/metrics`.
Bearer token	Sent as `Authorization: Bearer …`. Stored encrypted and never shown again; leave it blank to keep the saved one.
Scrape every	10 s to an hour. Default 60 s.
Check its HTTPS certificate	Off only for a self-signed certificate you trust.
Keep / Leave out	Name patterns, one a line, `*` for any run of characters. Empty Keep keeps everything; Leave out wins over Keep.
Series cap	The most series stored from one scrape (default 1000). Past it, the rest are counted and shown on the target's page, not stored.

Test scrapes the address once with what is typed, filters included, and says how many series would be stored and under which names. Nothing is saved.

The endpoint hands over everything; Steward chooses what to keep. Start narrow (yourservice_*) and widen it if you find you want more: every stored series is a row every scrape.

How series are stored

On the endpoint	In Steward
A gauge or an untyped sample	Stored as it is, under its own name.
A counter (`requests_total`)	Stored as a rate per second, `requests_per_sec`, from the second scrape on. A value lower than the last (the service restarted) is skipped rather than read as a negative rate.
A histogram or summary	Its `_sum` and `_count` are stored as rates (`x_sum_per_sec`, `x_count_per_sec`). Buckets and quantiles are not stored.
NaN or ±Inf	Not stored.

A series without labels is stored under the target's name. A labelled one is stored under <target>:<label>=<value>,…, in the order the endpoint gives the labels, for example Archives:target=Steward,state=late.

Each target also records four metrics about itself, under its own name:

Metric	Meaning
`is_up`	1 when the last scrape got metrics, 0 when it didn't.
`scrape_ms`	How long the scrape took.
`series_kept`	Series stored from the last scrape.
`series_capped`	Series the last scrape left out because of the cap.

A value's unit is read from Prometheus naming: _bytes as bytes, _seconds as a span, _timestamp_seconds as a moment ("3h ago"), _per_sec as a rate.

Renaming a target moves its stored history to the new name. Deleting one removes its newest values; its history ages out with retention (Settings → Retention).

Alerting on a target

In Alerts → New rule, choose Prometheus targets, then the series (the resource) and the metric. Two rules worth having for any target:

It stopped answering: resource <target>, metric is_up, < 1.
Something was cut off: resource <target>, metric series_capped, > 0.

Recipe: Fabled Archives

Archives serves /metrics on its backup listener (port 8081), which is reachable only on your LAN and the swarm's overlay network, behind a scrape token.

In Archives, Settings → Metrics → Make a token, named e.g. "Steward". Copy it; it is shown once.
In Steward, Prometheus → Add target:
- Name Archives
- Metrics address http://<archives host>:8081/metrics (on a shared overlay network, the service name, e.g. http://archives_app:8081/metrics)
- Bearer token: the one you copied
- Keep: fabledarchives_*
Test, then Add target.

Rules worth adding:

Watch	Resource	Metric	Condition
A backup target is late	`Archives:target=<name>,target_id=<id>,state=late`	`fabledarchives_backup_target_state`	`>= 1`
A backup target is missing	`Archives:target=<name>,target_id=<id>,state=missing`	`fabledarchives_backup_target_state`	`>= 1`
A replication job is failing	`Archives:job=<name>,job_id=<id>,state=failing`	`fabledarchives_replication_job_state`	`>= 1`
Archives' data disk is filling	`Archives`	`fabledarchives_data_disk_free_bytes`	`<` your floor
Archives stopped answering	`Archives`	`is_up`	`< 1`

The target's page lists every stored series with its exact resource name, so copy names from there rather than typing them.

4.6 KiB Raw Blame History