FabledSteward/docs/plugins/prometheus.md

Prometheus Plugin

The Prometheus plugin scrapes any service's Prometheus /metrics endpoint and stores what you choose to keep in Steward's metrics. Once stored, a series can be charted, read through the MCP query_metrics tool, and watched by an alert rule like any other metric.

It is on by default. Targets are added, tested and edited on its page, Prometheus in the sidebar (/plugins/prometheus/).

---

Adding a target

Field	What it is
Name	How the target's series are named in Steward (rule pickers, charts). No /, : or comma.
Metrics address	The full URL, e.g. http://archives:8081/metrics.
Bearer token	Sent as Authorization: Bearer …. Stored encrypted and never shown again; leave it blank to keep the saved one.
Scrape every	10 s to an hour. Default 60 s.
Check its HTTPS certificate	Off only for a self-signed certificate you trust.
Keep / Leave out	Name patterns, one a line, * for any run of characters. Empty Keep keeps everything; Leave out wins over Keep.
Series cap	The most series stored from one scrape (default 1000). Past it, the rest are counted and shown on the target's page, not stored.

Test scrapes the address once with what is typed, filters included, and says how many series would be stored and under which names. Nothing is saved.

The endpoint hands over everything; Steward chooses what to keep. Start narrow (yourservice_*) and widen it if you find you want more: every stored series is a row every scrape.

---

How series are stored

On the endpoint	In Steward
A gauge or an untyped sample	Stored as it is, under its own name.
A counter (requests_total)	Stored as a rate per second, requests_per_sec, from the second scrape on. A value lower than the last (the service restarted) is skipped rather than read as a negative rate.
A histogram or summary	Its _sum and _count are stored as rates (x_sum_per_sec, x_count_per_sec). Buckets and quantiles are not stored.
NaN or ±Inf	Not stored.

A series without labels is stored under the target's name. A labelled one is stored under <target>:<label>=<value>,…, in the order the endpoint gives the labels, for example Archives:target=Steward,state=late.

Each target also records four metrics about itself, under its own name:

Metric	Meaning
is_up	1 when the last scrape got metrics, 0 when it didn't.
scrape_ms	How long the scrape took.
series_kept	Series stored from the last scrape.
series_capped	Series the last scrape left out because of the cap.

A value's unit is read from Prometheus naming: _bytes as bytes, _seconds as a span, _timestamp_seconds as a moment ("3h ago"), _per_sec as a rate.

A state set is one series per possible state, the current one 1 and the rest 0, such as fabledarchives_backup_target_state{target, state}. The target page shows it as one row per target (or job), naming the current state; the other states are listed on hover. A metric is read as a state set when its name ends in _state and every series has a state label reading 0 or 1. The OpenMetrics form, where the label is named after the metric (foo{foo="a"} 1), counts too. Each state is still stored as its own series, so an alert rule can watch one state.

Only what the endpoint reports now is current. When a scrape succeeds, any series it no longer has loses its newest value. That happens when the endpoint stopped reporting it, for example an old build_info after an upgrade, or when a filter or the cap now leaves it out. Its history stays on the chart until retention removes it. A failed scrape leaves the newest values as they were.

Renaming a target moves its stored history to the new name. Deleting one removes its newest values; its history ages out with retention (Settings → Retention).

---

The dashboard widget

Edit a dashboard, open Add widget, and pick Prometheus under Integrations. Choose a target and a template; the widget is named after the template unless you give it a title.

A template says which of the target's series to show and how: backup states, figures with their units, times, a table, the version running. The widget leads with one line naming the worst problem, then each section with its problems first and the healthy rest after. Its Details link opens the target's page.

Steward ships with one template, Fabled Archives. A target whose series don't match its template says so in the widget rather than showing blanks.

Alerting on a target

In Alerts → New rule, choose Prometheus targets, then the series (the resource) and the metric. Two rules worth having for any target:

• It stopped answering: resource <target>, metric is_up, < 1.
• Something was cut off: resource <target>, metric series_capped, > 0.

---

Recipe: Fabled Archives

Archives serves /metrics on its backup listener (port 8081), which is reachable only on your LAN and the swarm's overlay network, behind a scrape token.

1. In Archives, Settings → Metrics → Make a token, named e.g. "Steward". Copy it; it is shown once.
2. In Steward, Prometheus → Add target:
   • Name Archives
   • Metrics address http://<archives host>:8081/metrics (on a shared overlay network, the service name, e.g. http://archives_app:8081/metrics)
   • Bearer token: the one you copied
   • Keep: fabledarchives_*
3. Test, then Add target.

Rules worth adding:

Watch	Resource	Metric	Condition
A backup target is late	Archives:target=<name>,target_id=<id>,state=late	fabledarchives_backup_target_state	>= 1
A backup target is missing	Archives:target=<name>,target_id=<id>,state=missing	fabledarchives_backup_target_state	>= 1
A replication job is failing	Archives:job=<name>,job_id=<id>,state=failing	fabledarchives_replication_job_state	>= 1
Archives' data disk is filling	Archives	fabledarchives_data_disk_free_bytes	< your floor
Archives stopped answering	Archives	is_up	< 1

The target's page lists every stored series with its exact resource name, so copy names from there rather than typing them.