feat(health): Steward records its own health and Settings → System shows it (#5467)
#5467 "Steward's own health: metrics, rules can watch them, a page that shows them". The self-monitor tick now records, under source "steward", resource "process": ingest_posts / ingest_failed / ingest_slowest_ms (the host-agent posts since the last tick, tallied in steward/core/health.py), db_roundtrip_ms (a timed SELECT 1), and unwatched_secs (the gap before this tick past watch.GAP_S, from Steward's own last sample). The hourly cleanup records cleanup_secs. The open-file pair is still recorded, and is now only skipped on its own where /proc is absent. All of these are in CORE_METRICS, so an alert rule can watch them under "Steward itself". Settings → System (/settings/system/, admin) shows a verdict first (failed posts, a gap in watching, open files near the limit), then each measure now and over the last day, with a sparkline scaled to its own peak. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -123,6 +123,24 @@ each covering rule's `hosts` and each host's rule's `part_of`.
|
||||
| `traefik` | `latency_p95_ms` | Approximate p95 latency (ms) |
|
||||
| `traefik` | `latency_p99_ms` | Approximate p99 latency (ms) |
|
||||
|
||||
#### Steward itself
|
||||
|
||||
Source `steward`, resource `process`, recorded once a minute by the
|
||||
self-monitor tick (`steward/core/self_monitor.py`; the ingest tally is in
|
||||
`steward/core/health.py`). Settings → System (`/settings/system/`, admin)
|
||||
shows each one now and over the last day, with a verdict above that names
|
||||
failed agent posts, a gap in watching, or open files near the limit.
|
||||
|
||||
| `metric_name` | Description |
|
||||
|---|---|
|
||||
| `ingest_posts` | Host-agent posts Steward took in the last minute |
|
||||
| `ingest_failed` | Posts in the last minute that didn't land (handler raised, or the agent gave up) |
|
||||
| `ingest_slowest_ms` | The slowest post in the last minute; agents give up at 10 s |
|
||||
| `db_roundtrip_ms` | A `SELECT 1`, connection checkout included |
|
||||
| `unwatched_secs` | The gap before this tick when it exceeds `watch.GAP_S` (a deploy or an outage); 0 when unbroken |
|
||||
| `cleanup_secs` | How long the last hourly retention and rollup run took |
|
||||
| `open_fds` / `open_fds_pct` | Steward's open files, and as a share of its limit (absent without `/proc`) |
|
||||
|
||||
---
|
||||
|
||||
## Notifications
|
||||
|
||||
Reference in New Issue
Block a user