FabledSteward/docs/plugins/prometheus.md

# Prometheus Plugin

The Prometheus plugin scrapes any service's Prometheus `/metrics` endpoint and stores what you choose to keep in Steward's metrics. Once stored, a series can be charted, read through the MCP `query_metrics` tool, and watched by an alert rule like any other metric.

It is on by default. Targets are added, tested and edited on its page, **Prometheus** in the sidebar (`/plugins/prometheus/`).

---

## Adding a target

| Field | What it is |
|---|---|
| Name | How the target's series are named in Steward (rule pickers, charts). No `/`, `:` or comma. |
| Metrics address | The full URL, e.g. `http://archives:8081/metrics`. |
| Bearer token | Sent as `Authorization: Bearer …`. Stored encrypted and never shown again; leave it blank to keep the saved one. |
| Scrape every | 10 s to an hour. Default 60 s. |
| Check its HTTPS certificate | Off only for a self-signed certificate you trust. |
| Keep / Leave out | Name patterns, one a line, `*` for any run of characters. Empty Keep keeps everything; Leave out wins over Keep. |
| Series cap | The most series stored from one scrape (default 1000). Past it, the rest are counted and shown on the target's page, not stored. |

**Test** scrapes the address once with what is typed, filters included, and says how many series would be stored and under which names. Nothing is saved.

The endpoint hands over everything; Steward chooses what to keep. Start narrow (`yourservice_*`) and widen it if you find you want more: every stored series is a row every scrape.

---

## How series are stored

| On the endpoint | In Steward |
|---|---|
| A gauge or an untyped sample | Stored as it is, under its own name. |
| A counter (`requests_total`) | Stored as a rate per second, `requests_per_sec`, from the second scrape on. A value lower than the last (the service restarted) is skipped rather than read as a negative rate. |
| A histogram or summary | Its `_sum` and `_count` are stored as rates (`x_sum_per_sec`, `x_count_per_sec`). Buckets and quantiles are not stored. |
| NaN or ±Inf | Not stored. |

A series without labels is stored under the target's name. A labelled one is stored under `<target>:<label>=<value>,…`, in the order the endpoint gives the labels, for example `Archives:target=Steward,state=late`.

Each target also records four metrics about itself, under its own name:

| Metric | Meaning |
|---|---|
| `is_up` | 1 when the last scrape got metrics, 0 when it didn't. |
| `scrape_ms` | How long the scrape took. |
| `series_kept` | Series stored from the last scrape. |
| `series_capped` | Series the last scrape left out because of the cap. |

A value's unit is read from Prometheus naming: `_bytes` as bytes, `_seconds` as a span, `_timestamp_seconds` as a moment ("3h ago"), `_per_sec` as a rate.

A **state set** is one series per possible state, the current one 1 and the rest 0, such as `fabledarchives_backup_target_state{target, state}`. The target page shows it as one row per target (or job), naming the current state; the other states are listed on hover. A metric is read as a state set when its name ends in `_state` and every series has a `state` label reading 0 or 1. The OpenMetrics form, where the label is named after the metric (`foo{foo="a"} 1`), counts too. Each state is still stored as its own series, so an alert rule can watch one state.

**Only what the endpoint reports now is current.** When a scrape succeeds, any series it no longer has loses its newest value. That happens when the endpoint stopped reporting it, for example an old `build_info` after an upgrade, or when a filter or the cap now leaves it out. Its history stays on the chart until retention removes it. A failed scrape leaves the newest values as they were.

Renaming a target moves its stored history to the new name. Deleting one removes its newest values; its history ages out with retention (Settings → Retention).

---

## The dashboard widget

Edit a dashboard, open **Add widget**, and pick **Prometheus** under Integrations. Choose a **target** and a **template**; the widget is named after the template unless you give it a title.

A template says which of the target's series to show and how: backup states, figures with their units, times, a table, the version running. The widget leads with one line naming the worst problem, then each section with its problems first and the healthy rest after. Its **Details** link opens the target's page.

Steward ships with one template, **Fabled Archives**. A target whose series don't match its template says so in the widget rather than showing blanks.

## Widget templates

**Prometheus → Widget templates** lists every template the widget can draw with: the ones built in, and your own. A template of yours that no longer passes the checker (a newer Steward may read the format differently) is listed first with what to change, and the widgets drawn with it say so too.

Open one to see its YAML beside a **preview**: the template drawn over a target you choose, exactly as the widget draws it, from the values Steward has stored. Nothing is fetched from the target to draw it.

- **New template**: write one, or paste in one someone shared. The preview redraws as you type, and lists each problem with its line until there are none.
- **Upload a file**: on the same page, read a `.yaml` file into the editor to look over before saving.
- **Duplicate**: copy any template, built-in ones included, under a new name, and change the copy. Built-in templates are read-only: they ship with Steward and change with it.
- **Export**: download a template's YAML, to keep or to share.

How to write one, every block, and the format reference: [Prometheus widget templates](prometheus-widgets.md).

Saving needs a template that passes the checker and a name no other template has. A widget remembers which template it draws with, so renaming a template keeps its widgets; deleting one leaves its widgets saying the template is gone. Templates are kept in Steward's backups, and adding, changing or deleting one is in the audit log.

## Alerting on a target

In **Alerts → New rule**, choose **Prometheus targets**, then the series (the resource) and the metric. Two rules worth having for any target:

- **It stopped answering:** resource `<target>`, metric `is_up`, `< 1`.
- **Something was cut off:** resource `<target>`, metric `series_capped`, `> 0`.

---

## Recipe: Fabled Archives

Archives serves `/metrics` on its backup listener (port 8081), which is reachable only on your LAN and the swarm's overlay network, behind a scrape token.

1. In Archives, **Settings → Metrics → Make a token**, named e.g. "Steward". Copy it; it is shown once.
2. In Steward, **Prometheus → Add target**:
   - Name `Archives`
   - Metrics address `http://<archives host>:8081/metrics` (on a shared overlay network, the service name, e.g. `http://archives_app:8081/metrics`)
   - Bearer token: the one you copied
   - Keep: `fabledarchives_*`
3. **Test**, then **Add target**.

Rules worth adding:

| Watch | Resource | Metric | Condition |
|---|---|---|---|
| A backup target is late | `Archives:target=<name>,target_id=<id>,state=late` | `fabledarchives_backup_target_state` | `>= 1` |
| A backup target is missing | `Archives:target=<name>,target_id=<id>,state=missing` | `fabledarchives_backup_target_state` | `>= 1` |
| A replication job is failing | `Archives:job=<name>,job_id=<id>,state=failing` | `fabledarchives_replication_job_state` | `>= 1` |
| Archives' data disk is filling | `Archives` | `fabledarchives_data_disk_free_bytes` | `<` your floor |
| Archives stopped answering | `Archives` | `is_up` | `< 1` |

The target's page lists every stored series with its exact resource name, so copy names from there rather than typing them.