docs(prometheus): writing a widget template, every block, and the format reference (#5694)

docs/plugins/prometheus-widgets.md covers:
- where templates live and how to copy one in, duplicate, export and share;
- a first template, walked through;
- each block (states with its deadlines, values, times, table, info) with an
  example;
- the format reference: keys, units, spans, limits, what the checker refuses;
- the Fabled Archives template, block by block, with what it leaves out and
  why.

Linked from the README, the plugin guide, the editor and template_format.
tests/plugins/prometheus/test_widget_guide.py checks every YAML example in the
guide against the checker, so the guide can't drift from the format.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

README.md

@@ -124,4 +124,5 @@ See `docs/plugins/` for a full plugin development guide.
 | `docs/plugins/writing-a-plugin.md` | Step-by-step plugin development guide |
 | `docs/plugins/traefik.md` | Traefik plugin: config, metrics, alert rule examples |
 | `docs/plugins/prometheus.md` | Prometheus plugin: scraping any /metrics endpoint, how series are stored, the Fabled Archives recipe |
+| `docs/plugins/prometheus-widgets.md` | Prometheus widget templates: writing, sharing and copying one in, every block, the format reference |
 | `docs/reference/code-map.md` | Where every key function, model, and route lives in the codebase |

docs/plugins/prometheus-widgets.md

@@ -0,0 +1,210 @@
+# Prometheus widget templates
+
+The Prometheus dashboard widget draws one scrape target through a **template**: a short YAML file that says which of the target's series to show and how. Steward ships with one, **Fabled Archives**. You can write your own, paste in one someone shared, and share yours as a file.
+
+A template is data, not code. It names series and lays them out in **blocks**, and Steward draws them. It cannot add a colour, a chart or a formula. Every widget it draws has the same shape:
+
+1. **A verdict:** "All well.", or the worst thing and how many need a look ("2 need a look. Replication: FullNightly-Restic is running, 11h 53m overdue.").
+2. **Each block in turn**, its problems first, saying why, then the healthy rest. After six healthy rows a block says "+N more healthy".
+3. **The target's name and when it was scraped**, linking to its page.
+
+This page covers where templates live, a walk through a first template, each block with an example, the format reference, and the Fabled Archives template.
+
+---
+
+## Where templates live
+
+**Prometheus → Widget templates** lists every template: the built-in ones, and your own. Open one to see its YAML beside a **preview**, which draws it over a target you pick from the values Steward has already stored. The preview redraws as you type and lists each problem with its line number.
+
+| You want to | Do this |
+|---|---|
+| Start from scratch | **New template**, then write or paste YAML. |
+| Use one someone sent you | **New template**, then paste it or **upload** the `.yaml` file. Look at the preview, then **Save template**. |
+| Change a built-in one | Open it, then **Duplicate**. Built-in templates are read-only and change with Steward, so you edit the copy. |
+| Share yours | **Export** downloads the YAML as a file. |
+| Use it on a dashboard | Edit the dashboard, **Add widget → Prometheus**, and pick the target and the template. |
+
+A template saves only when it has no problems, and its `name` must differ from every other template's. A widget remembers which template it draws with, so renaming a template keeps its widgets. Deleting one leaves its widgets saying the template is gone. Your templates are in Steward's backups, and adding, changing or deleting one is recorded in the audit log.
+
+Steward never fetches a template from anywhere. Templates arrive only by being pasted or uploaded.
+
+---
+
+## Your first template
+
+Look at the target's page first (**Prometheus → the target**). It lists every series Steward stored, with its labels and newest value. A template names those series exactly as listed there.
+
+```yaml
+name: My service
+version: 1
+expects: myservice_
+blocks:
+  - values:
+      title: Health
+      items:
+        - label: Queue depth
+          metric: myservice_queue_depth
+          unit: count
+          warn_above: 100
+        - label: Disk free
+          metric: myservice_disk_free_bytes
+          of: myservice_disk_size_bytes
+          warn_below: 10%
+          crit_below: 5%
+  - times:
+      items:
+        - label: Last good run
+          metric: myservice_last_success_timestamp_seconds
+          warn_after: 26h
+          crit_after: 2d
+```
+
+- `name` is what the widget is called, and what the template list and the widget picker show.
+- `version` is the template's own version. Raise it when you change a template you have shared, so the people you sent it to can tell copies apart.
+- `expects` is the start of the metric names it reads. If the target reports none, the widget says so ("is this the template for it?") instead of a column of blanks.
+
+**Metric names as Steward stores them.** A gauge keeps its name. A counter is stored as a rate per second: `requests_total` becomes `requests_per_sec`. A histogram keeps only `_sum_per_sec` and `_count_per_sec`. See [How series are stored](prometheus.md#how-series-are-stored).
+
+---
+
+## The blocks
+
+Every block takes an optional `title`. Without one, it gets the default title shown in its heading below.
+
+### `states`: a state set, one row per thing (default title "States")
+
+A state set has one series per possible state, with the current one at 1 and the rest at 0. Archives' `fabledarchives_backup_target_state{target, state}` is one. Each thing it describes becomes a row naming its current state.
+
+```yaml
+- states:
+    title: Backup targets
+    metric: fabledarchives_backup_target_state
+    name: target                 # the label that names each row
+    healthy: [fresh]
+    quiet: [disabled, waiting]   # shown greyed, never a problem
+    warning: [late]              # anything not listed is a problem
+    columns:
+      - {label: Last backup, metric: fabledarchives_backup_target_last_backup_timestamp_seconds}
+      - {label: Stored, metric: fabledarchives_backup_target_stored_bytes}
+```
+
+- A state in `healthy` is fine, in `quiet` is shown greyed, and in `warning` is amber. **A state in none of the lists is red.** So a state the service adds later shows up as a problem rather than slipping by as healthy.
+- A row reporting no current state is a warning.
+- `columns` add the row's other series beside its state. Each column series is matched to its row by having the same labels. Failing that, it is matched by the label `name` points at.
+- **Deadlines:** `warn_past` and `crit_past` each name a `*_timestamp_seconds` metric per row. Once that time has passed, a row in a healthy state reads as a warning, or a problem for `crit_past` ("is running, 11h 53m overdue"). Quiet and failing rows keep their own reading. Use this when a service ranks a state like "running" above lateness, so a run that never ends would otherwise read as healthy.
+
+### `values`: figures with their units (default title "Figures")
+
+```yaml
+- values:
+    title: Storage
+    items:
+      - label: Disk free
+        metric: fabledarchives_data_disk_free_bytes
+        of: fabledarchives_data_disk_size_bytes   # shown as "16.2 TiB of 41.9 TiB", with a bar
+        warn_below: 10%                            # a share of `of`
+        crit_below: 5%
+        sparkline: true                            # the last day as small bars beside it
+      - label: Stored
+        metric: fabledarchives_person_stored_bytes
+        name: person                               # one row per person
+        where: {kind: files}                       # only the series with kind=files
+```
+
+- `warn_above`, `crit_above`, `warn_below` and `crit_below` take a number, or a share like `10%` when the item has `of`. The red thresholds are checked before the amber ones.
+- Without `name`, an item must match exactly one series. If more match, the row says how many and asks you to narrow it with `where` or `name`.
+- A series the target doesn't report reads "—", "not reported", as a warning.
+- The sparkline's bars are scaled to that series' own peak over the last 24 hours.
+
+### `times`: moments and how long ago (default title "Times")
+
+```yaml
+- times:
+    items:
+      - label: Last good run
+        metric: myservice_last_success_timestamp_seconds
+        warn_after: 26h
+        crit_after: 2d
+```
+
+The metric is a `*_timestamp_seconds`. The row reads "3h ago" (exact time on hover) and turns amber or red once that long has passed. A timestamp of 0 reads "never", which is a warning when the item has a threshold.
+
+### `table`: one metric, two labels (default title "Table")
+
+```yaml
+- table:
+    title: Stored per person
+    metric: fabledarchives_person_stored_bytes
+    rows: person
+    columns: kind
+```
+
+Rows and columns are the values of the two labels, in alphabetical order, under one header row. A missing cell reads "—". Tables show figures only; they don't judge them.
+
+### `info`: labels worth reading (default title "About")
+
+```yaml
+- info:
+    title: Running
+    metric: fabledarchives_build_info
+    labels: [version, channel, commit]   # which labels, in this order; all of them when left out
+```
+
+This block is for a `*_info` or `*_build_info` series, where the labels are the point and the value is always 1. It reads "version 2026.10.12.1717 · channel dev · commit 3605d0a".
+
+---
+
+## Reference
+
+**Top level**
+
+| Key | Needed | What it is |
+|---|---|---|
+| `name` | yes | At most 80 characters; unique among your templates and the built-in ones. |
+| `version` | yes | A whole number from 1. |
+| `blocks` | yes | One to 20 blocks. |
+| `description` | | A line shown in the template list. |
+| `expects` | | The prefix of the metric names it reads, like `fabledarchives_`. |
+| `format` | | The template format; `1`, the only one so far. |
+
+**Keys most blocks and items take**
+
+| Key | What it is |
+|---|---|
+| `metric` | A series name as Steward stores it. |
+| `where` | Labels a series must have, like `{kind: files, state: ok}`. |
+| `name` | On `states`, `values` and `times`: the label that names each row (one row per value). |
+| `unit` | On a `values` item, a `states` column or a `table`: overrides the unit read from the name: `bytes`, `seconds` (a span), `percent`, `per_sec`, `count` (thousands separated: 100,462), `timestamp`. |
+
+Without `unit`, the name decides how a value reads: `_bytes` as bytes, `_seconds` as a span, `_timestamp_seconds` as a moment, `_per_sec` as a rate, `_pct` as a percentage. Anything else is a plain number.
+
+**Spans** (`warn_after`, `crit_after`) are a number of seconds or a number with `s`, `m`, `h` or `d`: `90`, `15m`, `26h`, `2d`.
+
+**Limits:** a template is at most 64 KB, with at most 20 blocks and 30 items in a block.
+
+**What the checker refuses,** each with its line and what to change:
+- an unknown key;
+- a key given twice;
+- anchors and aliases (`&name`, `*name`): write the value out;
+- a share threshold (`10%`) without `of`;
+- a state in two of `healthy`, `quiet` and `warning`;
+- `crit_after` earlier than `warn_after`;
+- a word YAML reads as true or false (`on`, `off`, `yes`, `no`) used as a state or label value. Quote it: `'off'`.
+
+---
+
+## The Fabled Archives template, walked through
+
+The built-in template is a worked example of most of the above. **Duplicate** it to read or change it. It was written against the series Archives actually reports. On 14 Oct 2026 that turned up a nightly replication job that had read "running" for two days without a good copy; the deadlines below exist to catch exactly that.
+
+| Block | What it shows | Why it is written this way |
+|---|---|---|
+| Backup targets (`states`) | Each target Archives receives backups from: fresh, late or missing, with its last backup and what it has stored. | `disabled` and `waiting` are quiet: a target you turned off, or one that hasn't sent its first backup, isn't a problem. `late` is a warning; `missing` (not listed) is a problem. |
+| Replication (`states`) | Each replication job, its last good copy and its next run. | Archives ranks `running` above lateness, so `warn_past` and `crit_past` read the job's own late and missing times. A run that never finishes turns amber, then red. |
+| Storage and integrity (`values`) | Disk free of its size (amber under 10%, red under 5%, last day as bars), files checked of files, malware held, failing notification channels. | Held malware and a failing channel each need you to do something, so any count above 0 is a warning. |
+| Stored per person (`table`) | Each person's files, versions and trash. | Figures to read, not to judge. |
+| Running (`info`) | The version, channel and commit. | |
+
+Left out on purpose (the template's header says so too):
+- `fabledarchives_section_up`: a section Archives couldn't read stops reporting its figures, and the rows that read them then say "not reported".
+- The background queue, HTTP requests and audit counters: these suit charts and alert rules better than a glance.

docs/plugins/prometheus.md

@@ -73,6 +73,8 @@ Open one to see its YAML beside a **preview**: the template drawn over a target
 - **Duplicate**: copy any template, built-in ones included, under a new name, and change the copy. Built-in templates are read-only: they ship with Steward and change with it.
 - **Export**: download a template's YAML, to keep or to share.
 
+How to write one, every block, and the format reference: [Prometheus widget templates](prometheus-widgets.md).
+
 Saving needs a template that passes the checker and a name no other template has. A widget remembers which template it draws with, so renaming a template keeps its widgets; deleting one leaves its widgets saying the template is gone. Templates are kept in Steward's backups, and adding, changing or deleting one is in the audit log.
 
 ## Alerting on a target

plugins/prometheus/template_format.py

@@ -3,7 +3,8 @@
 A template says which of a target's series a dashboard widget shows and
 how. It is data only: it names series and lays them out in blocks, and
 Steward draws them (template_view.py). It can't add a visual, a colour or a
-formula; a new block is a Steward release (System 181's ruling).
+formula; a new block is a Steward release (System 181's ruling). The
+format, for people writing one: docs/plugins/prometheus-widgets.md.
 
     name: Fabled Archives
     version: 1

plugins/prometheus/templates/prometheus/template.html

@@ -36,7 +36,7 @@
     <label class="field-label" for="tpl-text">YAML</label>
     <textarea class="input mono tpl-text" id="tpl-text" name="text" spellcheck="false" wrap="off"
               {%- if not editable %} readonly{% endif %}
-              placeholder="Paste a template here, or write one: name, version and blocks. The guide is docs/plugins/prometheus.md.">{{ text }}</textarea>
+              placeholder="Paste a template here, or write one: name, version and blocks. The guide is docs/plugins/prometheus-widgets.md.">{{ text }}</textarea>
     {% if editable %}
     <div class="form-actions">
       <button type="submit" class="btn">{{ "Save" if entry else "Save template" }}</button>

plugins/prometheus/widgets/fabled_archives.yaml

@@ -1,6 +1,6 @@
 # Fabled Archives, from its /metrics on the LAN-only backup listener (:8081).
-# Written against the series Steward stores for Archives (#5693); see
-# docs/plugins/prometheus.md.
+# Written against the series Steward stores for Archives (#5693); the format
+# is docs/plugins/prometheus-widgets.md.
 #
 # Left out on purpose:
 # - fabledarchives_section_up: a section Archives can't read stops reporting

tests/plugins/prometheus/test_widget_guide.py

@@ -0,0 +1,42 @@
+"""docs/plugins/prometheus-widgets.md (#5694): every YAML example in the
+guide passes the checker, so the guide can't drift from the format."""
+import re
+import textwrap
+
+from plugins.prometheus.template_format import check
+from tests._templates import REPO
+
+GUIDE = REPO / "docs" / "plugins" / "prometheus-widgets.md"
+_FENCE = re.compile(r"```yaml\n(.*?)```", re.S)
+
+
+def examples(text: str) -> list[str]:
+    """Each fenced YAML example as a whole template: a block on its own is
+    wrapped in a name, a version and a blocks list."""
+    out = []
+    for body in _FENCE.findall(text):
+        if body.startswith("name:"):
+            out.append(body)
+        else:
+            out.append("name: Example\nversion: 1\nblocks:\n" + textwrap.indent(body, "  "))
+    return out
+
+
+def test_the_guard_can_fail():
+    bad = examples("```yaml\n- states: {metric: x_state, healthy: [ok], colour: red}\n```")
+    assert len(bad) == 1
+    try:
+        check(bad[0])
+    except ValueError:
+        pass
+    else:
+        raise AssertionError("an unknown key passed the checker")
+
+
+def test_every_example_in_the_guide_is_a_template():
+    found = examples(GUIDE.read_text())
+    assert len(found) >= 6, len(found)       # the first template, then one per block kind
+    kinds = set()
+    for text in found:
+        kinds.update(b.kind for b in check(text).blocks)
+    assert kinds == {"states", "values", "times", "table", "info"}