feat(prometheus): scrape any /metrics endpoint into Steward's metrics, with pages (#5668, #5669)
CI / lint (push) Successful in 2s
CI / integration (push) Failing after 1m12s
CI / unit (push) Failing after 1m27s
CI / publish (push) Skipped

A bundled prometheus plugin, on by default like the other protocol plugins.
Targets are rows edited on its page: address, bearer token (encrypted),
interval, keep/leave-out name patterns and a series cap. Steward does the
filtering; the endpoint serves everything (Archives' ruling).

- A 5 s tick scrapes due targets at once, each within a 30 s deadline, and
  writes the outcome to its row and its series through record_metrics, so
  alert rules can watch any of them.
- Gauges are stored as they are; counters (and a histogram's or summary's
  _sum/_count) as <name>_per_sec rates from the second scrape. Buckets and
  quantiles are skipped. A labelled series is stored under
  "<target>:k=v,…". Each target also records is_up, scrape_ms, series_kept
  and series_capped.
- Pages: targets not answering first, with why; a target's series grouped
  by metric with their newest values; a chart per series; the editor with
  Test, which scrapes once and says what would be stored.
- The counter-to-rate keeper moves to steward/core/counter_rates.py; SNMP's
  RateKeeper now scales its rate on top of it.
- metric_value reads Prometheus's _seconds as a span and _per_sec as a rate.
- The catalog guard treats an empty list as a dynamic source (snmp,
  prometheus), as the rule form does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-10-12 07:04:01 -04:00
co-authored by Claude Opus 5.5
parent a9998681f9
commit 3cb2a95054
24 changed files with 1598 additions and 27 deletions
@@ -0,0 +1,111 @@
{# plugins/prometheus/templates/prometheus/form.html — add or edit a scrape
target (milestone 506): where its /metrics is, its token, how often, and
what to keep, with Test to scrape it once before saving. The token is
never sent back to the browser; a blank token field keeps the stored one. #}
{% extends "base.html" %}
{% from "_macros.html" import crumbs, page_header %}
{% block title %}{{ ("Edit " ~ target.name) if target else "Add Prometheus target" }} — Steward{% endblock %}
{% block breadcrumb %}
{%- if target -%}
{{ crumbs([("Prometheus", "/plugins/prometheus/"), (target.name, "/plugins/prometheus/targets/" ~ target.id), ("Edit", "")]) }}
{%- else -%}
{{ crumbs([("Prometheus", "/plugins/prometheus/"), ("Add target", "")]) }}
{%- endif -%}
{% endblock %}
{% block content %}
{{ page_header(("Edit " ~ target.name) if target else "Add Prometheus target") }}
<div class="form-page">
{% if error %}<p class="field-error" role="alert">{{ error }}</p>{% endif %}
<form method="post" id="prom-form" action="{{ '/plugins/prometheus/targets/' ~ target.id ~ '/edit' if target else '/plugins/prometheus/targets/new' }}">{{ csrf_field() }}
{% if target %}<input type="hidden" name="target_id" value="{{ target.id }}">{% endif %}
<section class="card form-section" aria-labelledby="prom-where">
<h2 id="prom-where" class="section-title">Endpoint</h2>
<div class="form-pair">
<div class="field">
<label class="field-label" for="prom-name">Name</label>
<input class="input" id="prom-name" name="name" required maxlength="100" value="{{ v.name or '' }}" placeholder="Archives" autocomplete="off">
<span class="field-hint">How its series are named in Steward: alert rules and charts show it.</span>
</div>
<div class="field">
<label class="field-label" for="prom-url">Metrics address</label>
<input class="input mono" id="prom-url" name="url" required maxlength="500" value="{{ v.url or '' }}" placeholder="http://archives:8081/metrics" autocomplete="off">
</div>
<div class="field">
<label class="field-label" for="prom-token">Bearer token</label>
<input class="input" id="prom-token" name="token" type="password" autocomplete="new-password"
placeholder="{{ 'kept; type to change' if v.has_token else 'none' }}">
<span class="field-hint">Sent as "Authorization: Bearer …". Blank {{ "keeps the saved one" if v.has_token else "sends none" }}.</span>
{% if v.has_token %}
<label style="display:flex;gap:0.5rem;align-items:center;margin-top:0.3rem;cursor:pointer;">
<input type="checkbox" name="clear_token" value="on"> <span class="field-hint">Remove the saved token</span>
</label>
{% endif %}
</div>
<div class="field">
<label class="field-label" for="prom-interval">Scrape every (seconds)</label>
<input class="input" id="prom-interval" name="interval_seconds" type="number" min="10" max="3600" value="{{ v.interval_seconds or 60 }}">
</div>
</div>
<div style="display:flex;gap:1.5rem;flex-wrap:wrap;margin-bottom:0.9rem;">
<label style="display:flex;gap:0.5rem;align-items:center;cursor:pointer;">
<input type="checkbox" name="enabled" value="on"{% if v.enabled %} checked{% endif %}> Scrape this target
</label>
<label style="display:flex;gap:0.5rem;align-items:center;cursor:pointer;">
<input type="checkbox" name="verify_tls" value="on"{% if v.verify_tls %} checked{% endif %}> Check its HTTPS certificate
</label>
</div>
<div style="display:flex;align-items:center;gap:0.75rem;flex-wrap:wrap;">
<button type="button" class="btn btn-sm btn-ghost"
hx-post="/plugins/prometheus/targets/test" hx-include="#prom-form" hx-target="#prom-test"
hx-indicator="#prom-testing">Test</button>
<span id="prom-testing" class="field-hint htmx-indicator">Scraping…</span>
<span class="field-hint">Scrapes the address once with what's typed here, filters included, and says what would be stored. Nothing is saved.</span>
</div>
<div id="prom-test" aria-live="polite"></div>
</section>
<section class="card form-section" aria-labelledby="prom-what">
<h2 id="prom-what" class="section-title">What to keep</h2>
<p class="field-hint" style="margin-top:0;">The endpoint hands over everything; Steward stores what you choose. Patterns match the endpoint's metric names, with <span class="mono">*</span> for any run of characters.</p>
<div class="form-pair">
<div class="field">
<label class="field-label" for="prom-keep">Keep</label>
<textarea class="input mono" id="prom-keep" name="keep" rows="4" placeholder="fabledarchives_*">{{ v.keep or '' }}</textarea>
<span class="field-hint">One a line. Empty keeps everything.</span>
</div>
<div class="field">
<label class="field-label" for="prom-drop">Leave out</label>
<textarea class="input mono" id="prom-drop" name="drop" rows="4" placeholder="go_*&#10;process_*">{{ v.drop or '' }}</textarea>
<span class="field-hint">One a line. Wins over Keep.</span>
</div>
<div class="field">
<label class="field-label" for="prom-cap">Series cap</label>
<input class="input" id="prom-cap" name="series_cap" type="number" min="1" max="10000" value="{{ v.series_cap or 1000 }}">
<span class="field-hint">The most series stored from one scrape. Past it, the rest are counted and shown, not stored.</span>
</div>
</div>
<p class="field-hint" style="margin:0;">Gauges are stored as they are. Counters are stored as a rate per second, named <span class="mono">…_per_sec</span>, from the second scrape on. Histogram buckets and summary quantiles aren't stored; their <span class="mono">_sum</span> and <span class="mono">_count</span> are, as rates.</p>
</section>
<div class="form-actions">
<button type="submit" class="btn">{{ "Save" if target else "Add target" }}</button>
<a href="{{ '/plugins/prometheus/targets/' ~ target.id if target else '/plugins/prometheus/' }}" class="btn btn-ghost">Cancel</a>
</div>
</form>
{% if target %}
<section class="card danger-zone" aria-labelledby="prom-delete">
<div>
<h2 id="prom-delete" class="section-title">Delete this target</h2>
<p class="field-hint">Steward stops scraping it. Its past series age out with the rest of the history.</p>
</div>
<form method="post" action="/plugins/prometheus/targets/{{ target.id }}/delete"
data-confirm="Delete Prometheus target {{ target.name }}? Steward stops scraping it.">{{ csrf_field() }}
<button class="btn btn-danger" type="submit">Delete target</button>
</form>
</section>
{% endif %}
</div>
{% endblock %}