feat(health): Steward records its own health and Settings → System shows it (#5467)
CI / lint (push) Successful in 2s
CI / integration (push) Successful in 1m15s
CI / unit (push) Failing after 1m29s
CI / publish (push) Skipped

#5467 "Steward's own health: metrics, rules can watch them, a page that shows them".

The self-monitor tick now records, under source "steward", resource "process":
ingest_posts / ingest_failed / ingest_slowest_ms (the host-agent posts since the
last tick, tallied in steward/core/health.py), db_roundtrip_ms (a timed SELECT 1),
and unwatched_secs (the gap before this tick past watch.GAP_S, from Steward's own
last sample). The hourly cleanup records cleanup_secs. The open-file pair is still
recorded, and is now only skipped on its own where /proc is absent. All of these
are in CORE_METRICS, so an alert rule can watch them under "Steward itself".

Settings → System (/settings/system/, admin) shows a verdict first (failed posts,
a gap in watching, open files near the limit), then each measure now and over the
last day, with a sparkline scaled to its own peak.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-10-09 14:02:50 -04:00
co-authored by Claude Opus 5.5
parent 656e127fd8
commit c8dbe811f8
12 changed files with 337 additions and 33 deletions
+41
View File
@@ -0,0 +1,41 @@
{# settings/system.html — Steward's own health (#5467): a verdict, then each of
its process metrics now and over the last day, from the self-monitor tick.
Each is a metric an alert rule can watch under "Steward itself". #}
{% extends "base.html" %}
{% from "_macros.html" import crumbs %}
{% from "settings/_tabs.html" import settings_tabs %}
{% block title %}Settings — System — Steward{% endblock %}
{% block section_tabs %}{{ settings_tabs("system") }}{% endblock %}
{% block breadcrumb %}{{ crumbs([("Settings", "/settings/"), ("System", "")]) }}{% endblock %}
{% block content %}
<p class="page-verdict">
{%- if not seen %}<span class="tone-dim">No readings yet: the first arrive within a minute of Steward starting.</span>
{%- elif problems %}{% set said = problems|join("; ") %}<span class="tone-warn">{{ said[:1]|upper }}{{ said[1:] }}.</span>
{%- else %}<span class="tone-ok">Steward is running normally.</span>{% endif -%}
</p>
<h2 class="section-title" style="margin:1.5rem 0 1rem;">Steward itself</h2>
<div class="card-flush">
<table class="table">
<thead><tr><th>Measure</th><th>Now</th><th>Last 24 h</th></tr></thead>
<tbody>
{% for r in rows %}
<tr>
<td class="cell-name"><span title="{{ metric_label(r.metric) }}">{{ metric_label(r.metric) }}</span>
<div class="cell-sub" title="{{ metric_about(r.metric) }}">{{ metric_about(r.metric) }}</div></td>
<td class="nowrap">{{ metric_value(r.metric, r.now) }}
{%- if r.at %}<div class="cell-sub" title="{{ r.at|when }}">{{ r.at|ago }}</div>{% endif %}</td>
<td class="nowrap">
{%- if r.low is not none %}{{ metric_value(r.metric, r.low) }} – {{ metric_value(r.metric, r.high) }}
<span class="metric-preview-spark" aria-hidden="true">{{ r.spark|safe }}</span>
{%- else %}<span class="tone-dim">—</span>{% endif -%}
</td>
</tr>
{% endfor %}
</tbody>
</table>
</div>
<p class="field-hint" style="margin-top:0.75rem;">
Recorded each minute (the cleanup each hour). Any of these can be watched by an
<a href="/alerts/rules/new">alert rule</a>: pick the source "Steward itself".</p>
{% endblock %}