FabledSteward/docs/plugins/prometheus-widgets.md

# Prometheus widget templates

The Prometheus dashboard widget draws one scrape target through a **template**: a short YAML file that says which of the target's series to show and how. Steward ships with one, **Fabled Archives**. You can write your own, paste in one someone shared, and share yours as a file.

A template is data, not code. It names series and lays them out in **blocks**, and Steward draws them. It cannot add a colour, a chart or a formula. Every widget it draws has the same shape:

1. **A verdict:** "All well.", or the worst thing and how many need a look ("2 need a look. Replication: FullNightly-Restic is running, 11h 53m overdue.").
2. **Each block in turn**, its problems first, saying why, then the healthy rest. After six healthy rows a block says "+N more healthy".
3. **The target's name and when it was scraped**, linking to its page.

This page covers where templates live, a walk through a first template, each block with an example, the format reference, and the Fabled Archives template.

---

## Where templates live

**Prometheus → Widget templates** lists every template: the built-in ones, and your own. Open one to see its YAML beside a **preview**, which draws it over a target you pick from the values Steward has already stored. The preview redraws as you type and lists each problem with its line number.

| You want to | Do this |
|---|---|
| Start from scratch | **New template**, then write or paste YAML. |
| Use one someone sent you | **New template**, then paste it or **upload** the `.yaml` file. Look at the preview, then **Save template**. |
| Change a built-in one | Open it, then **Duplicate**. Built-in templates are read-only and change with Steward, so you edit the copy. |
| Share yours | **Export** downloads the YAML as a file. |
| Use it on a dashboard | Edit the dashboard, **Add widget → Prometheus**, and pick the target and the template. |

A template saves only when it has no problems, and its `name` must differ from every other template's. A widget remembers which template it draws with, so renaming a template keeps its widgets. Deleting one leaves its widgets saying the template is gone. Your templates are in Steward's backups, and adding, changing or deleting one is recorded in the audit log.

Steward never fetches a template from anywhere. Templates arrive only by being pasted or uploaded.

---

## Your first template

Look at the target's page first (**Prometheus → the target**). It lists every series Steward stored, with its labels and newest value. A template names those series exactly as listed there.

```yaml
name: My service
version: 1
expects: myservice_
blocks:
  - values:
      title: Health
      items:
        - label: Queue depth
          metric: myservice_queue_depth
          unit: count
          warn_above: 100
        - label: Disk free
          metric: myservice_disk_free_bytes
          of: myservice_disk_size_bytes
          warn_below: 10%
          crit_below: 5%
  - times:
      items:
        - label: Last good run
          metric: myservice_last_success_timestamp_seconds
          warn_after: 26h
          crit_after: 2d
```

- `name` is what the widget is called, and what the template list and the widget picker show.
- `version` is the template's own version. Raise it when you change a template you have shared, so the people you sent it to can tell copies apart.
- `expects` is the start of the metric names it reads. If the target reports none, the widget says so ("is this the template for it?") instead of a column of blanks.

**Metric names as Steward stores them.** A gauge keeps its name. A counter is stored as a rate per second: `requests_total` becomes `requests_per_sec`. A histogram keeps only `_sum_per_sec` and `_count_per_sec`. See [How series are stored](prometheus.md#how-series-are-stored).

---

## The blocks

Every block takes an optional `title`. Without one, it gets the default title shown in its heading below.

### `states`: a state set, one row per thing (default title "States")

A state set has one series per possible state, with the current one at 1 and the rest at 0. Archives' `fabledarchives_backup_target_state{target, state}` is one. Each thing it describes becomes a row naming its current state.

```yaml
- states:
    title: Backup targets
    metric: fabledarchives_backup_target_state
    name: target                 # the label that names each row
    healthy: [fresh]
    quiet: [disabled, waiting]   # shown greyed, never a problem
    warning: [late]              # anything not listed is a problem
    columns:
      - {label: Last backup, metric: fabledarchives_backup_target_last_backup_timestamp_seconds}
      - {label: Stored, metric: fabledarchives_backup_target_stored_bytes}
```

- A state in `healthy` is fine, in `quiet` is shown greyed, and in `warning` is amber. **A state in none of the lists is red.** So a state the service adds later shows up as a problem rather than slipping by as healthy.
- A row reporting no current state is a warning.
- `columns` add the row's other series beside its state. Each column series is matched to its row by having the same labels. Failing that, it is matched by the label `name` points at.
- **Deadlines:** `warn_past` and `crit_past` each name a `*_timestamp_seconds` metric per row. Once that time has passed, a row in a healthy state reads as a warning, or a problem for `crit_past` ("is running, 11h 53m overdue"). Quiet and failing rows keep their own reading. Use this when a service ranks a state like "running" above lateness, so a run that never ends would otherwise read as healthy.

### `values`: figures with their units (default title "Figures")

```yaml
- values:
    title: Storage
    items:
      - label: Disk free
        metric: fabledarchives_data_disk_free_bytes
        of: fabledarchives_data_disk_size_bytes   # shown as "16.2 TiB of 41.9 TiB", with a bar
        warn_below: 10%                            # a share of `of`
        crit_below: 5%
        sparkline: true                            # the last day as small bars beside it
      - label: Stored
        metric: fabledarchives_person_stored_bytes
        name: person                               # one row per person
        where: {kind: files}                       # only the series with kind=files
```

- `warn_above`, `crit_above`, `warn_below` and `crit_below` take a number, or a share like `10%` when the item has `of`. The red thresholds are checked before the amber ones.
- Without `name`, an item must match exactly one series. If more match, the row says how many and asks you to narrow it with `where` or `name`.
- A series the target doesn't report reads "—", "not reported", as a warning.
- The sparkline's bars are scaled to that series' own peak over the last 24 hours.

### `times`: moments and how long ago (default title "Times")

```yaml
- times:
    items:
      - label: Last good run
        metric: myservice_last_success_timestamp_seconds
        warn_after: 26h
        crit_after: 2d
```

The metric is a `*_timestamp_seconds`. The row reads "3h ago" (exact time on hover) and turns amber or red once that long has passed. A timestamp of 0 reads "never", which is a warning when the item has a threshold.

### `table`: one metric, two labels (default title "Table")

```yaml
- table:
    title: Stored per person
    metric: fabledarchives_person_stored_bytes
    rows: person
    columns: kind
```

Rows and columns are the values of the two labels, in alphabetical order, under one header row. A missing cell reads "—". Tables show figures only; they don't judge them.

### `info`: labels worth reading (default title "About")

```yaml
- info:
    title: Running
    metric: fabledarchives_build_info
    labels: [version, channel, commit]   # which labels, in this order; all of them when left out
```

This block is for a `*_info` or `*_build_info` series, where the labels are the point and the value is always 1. It reads "version 2026.10.12.1717 · channel dev · commit 3605d0a".

---

## Reference

**Top level**

| Key | Needed | What it is |
|---|---|---|
| `name` | yes | At most 80 characters; unique among your templates and the built-in ones. |
| `version` | yes | A whole number from 1. |
| `blocks` | yes | One to 20 blocks. |
| `description` | | A line shown in the template list. |
| `expects` | | The prefix of the metric names it reads, like `fabledarchives_`. |
| `format` | | The template format; `1`, the only one so far. |

**Keys most blocks and items take**

| Key | What it is |
|---|---|
| `metric` | A series name as Steward stores it. |
| `where` | Labels a series must have, like `{kind: files, state: ok}`. |
| `name` | On `states`, `values` and `times`: the label that names each row (one row per value). |
| `unit` | On a `values` item, a `states` column or a `table`: overrides the unit read from the name: `bytes`, `seconds` (a span), `percent`, `per_sec`, `count` (thousands separated: 100,462), `timestamp`. |

Without `unit`, the name decides how a value reads: `_bytes` as bytes, `_seconds` as a span, `_timestamp_seconds` as a moment, `_per_sec` as a rate, `_pct` as a percentage. Anything else is a plain number.

**Spans** (`warn_after`, `crit_after`) are a number of seconds or a number with `s`, `m`, `h` or `d`: `90`, `15m`, `26h`, `2d`.

**Limits:** a template is at most 64 KB, with at most 20 blocks and 30 items in a block.

**What the checker refuses,** each with its line and what to change:
- an unknown key;
- a key given twice;
- anchors and aliases (`&name`, `*name`): write the value out;
- a share threshold (`10%`) without `of`;
- a state in two of `healthy`, `quiet` and `warning`;
- `crit_after` earlier than `warn_after`;
- a word YAML reads as true or false (`on`, `off`, `yes`, `no`) used as a state or label value. Quote it: `'off'`.

---

## The Fabled Archives template, walked through

The built-in template is a worked example of most of the above. **Duplicate** it to read or change it. It was written against the series Archives actually reports. On 14 Oct 2026 that turned up a nightly replication job that had read "running" for two days without a good copy; the deadlines below exist to catch exactly that.

| Block | What it shows | Why it is written this way |
|---|---|---|
| Backup targets (`states`) | Each target Archives receives backups from: fresh, late or missing, with its last backup and what it has stored. | `disabled` and `waiting` are quiet: a target you turned off, or one that hasn't sent its first backup, isn't a problem. `late` is a warning; `missing` (not listed) is a problem. |
| Replication (`states`) | Each replication job, its last good copy and its next run. | Archives ranks `running` above lateness, so `warn_past` and `crit_past` read the job's own late and missing times. A run that never finishes turns amber, then red. |
| Storage and integrity (`values`) | Disk free of its size (amber under 10%, red under 5%, last day as bars), files checked of files, malware held, failing notification channels. | Held malware and a failing channel each need you to do something, so any count above 0 is a warning. |
| Stored per person (`table`) | Each person's files, versions and trash. | Figures to read, not to judge. |
| Running (`info`) | The version, channel and commit. | |

Left out on purpose (the template's header says so too):
- `fabledarchives_section_up`: a section Archives couldn't read stops reporting its figures, and the rows that read them then say "not reported".
- The background queue, HTTP requests and audit counters: these suit charts and alert rules better than a glance.