feat(prometheus): scrape any /metrics endpoint into Steward's metrics, with pages (#5668, #5669)
CI / lint (push) Successful in 2s
CI / integration (push) Failing after 1m12s
CI / unit (push) Failing after 1m27s
CI / publish (push) Skipped

A bundled prometheus plugin, on by default like the other protocol plugins.
Targets are rows edited on its page: address, bearer token (encrypted),
interval, keep/leave-out name patterns and a series cap. Steward does the
filtering; the endpoint serves everything (Archives' ruling).

- A 5 s tick scrapes due targets at once, each within a 30 s deadline, and
  writes the outcome to its row and its series through record_metrics, so
  alert rules can watch any of them.
- Gauges are stored as they are; counters (and a histogram's or summary's
  _sum/_count) as <name>_per_sec rates from the second scrape. Buckets and
  quantiles are skipped. A labelled series is stored under
  "<target>:k=v,…". Each target also records is_up, scrape_ms, series_kept
  and series_capped.
- Pages: targets not answering first, with why; a target's series grouped
  by metric with their newest values; a chart per series; the editor with
  Test, which scrapes once and says what would be stored.
- The counter-to-rate keeper moves to steward/core/counter_rates.py; SNMP's
  RateKeeper now scales its rate on top of it.
- metric_value reads Prometheus's _seconds as a span and _per_sec as a rate.
- The catalog guard treats an empty list as a dynamic source (snmp,
  prometheus), as the rule form does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-10-12 07:04:01 -04:00
co-authored by Claude Opus 5.5
parent a9998681f9
commit 3cb2a95054
24 changed files with 1598 additions and 27 deletions
+4 -2
View File
@@ -9,7 +9,7 @@ import steward
from steward.alerts.service import CORE_METRICS
_REPO = pathlib.Path(steward.__file__).parent.parent
_PLUGINS = ("traefik", "unifi", "docker", "host_agent", "snmp")
_PLUGINS = ("traefik", "unifi", "docker", "host_agent", "snmp", "prometheus")
def catalog() -> dict[str, set[str]]:
@@ -98,5 +98,7 @@ def test_every_recorded_metric_is_in_the_catalog():
assert ("traefik", "error_rate_5xx_pct") in pairs and ("docker", "is_healthy") in pairs
assert ("host_agent", "seconds_since_seen") in pairs
cat = catalog()
missing = sorted(p for p in pairs if p[1] not in cat.get(p[0], set()))
# An empty list is a dynamic source (snmp, prometheus): the rule form
# offers whatever it has recorded, so any name it writes is offerable.
missing = sorted(p for p in pairs if cat.get(p[0]) != set() and p[1] not in cat.get(p[0], set()))
assert not missing, f"recorded but not in the alert catalog: {missing}"