feat(health): Steward records its own health and Settings → System shows it (#5467)
#5467 "Steward's own health: metrics, rules can watch them, a page that shows them". The self-monitor tick now records, under source "steward", resource "process": ingest_posts / ingest_failed / ingest_slowest_ms (the host-agent posts since the last tick, tallied in steward/core/health.py), db_roundtrip_ms (a timed SELECT 1), and unwatched_secs (the gap before this tick past watch.GAP_S, from Steward's own last sample). The hourly cleanup records cleanup_secs. The open-file pair is still recorded, and is now only skipped on its own where /proc is absent. All of these are in CORE_METRICS, so an alert rule can watch them under "Steward itself". Settings → System (/settings/system/, admin) shows a verdict first (failed posts, a gap in watching, open files near the limit), then each measure now and over the last day, with a sparkline scaled to its own peak. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -17,6 +17,7 @@ from sqlalchemy import select, or_, and_
|
||||
from datetime import timedelta
|
||||
|
||||
from steward.core.formatting import worst_reading
|
||||
from steward.core.health import ingest as ingest_tally
|
||||
from steward.core.metric_series import points as metric_points
|
||||
from steward.core.settings import get_setting, public_base_url
|
||||
from steward.core.time_range import parse_range, RANGE_OPTIONS
|
||||
@@ -368,7 +369,14 @@ async def ingest():
|
||||
# The agent hung up (its 10 s timeout) and the server dropped the
|
||||
# request: say how far it got before re-raising.
|
||||
phases.cancelled(token_hash[:8])
|
||||
ingest_tally.done(phases.elapsed_ms(), ok=False)
|
||||
raise
|
||||
except Exception:
|
||||
ingest_tally.done(phases.elapsed_ms(), ok=False)
|
||||
raise
|
||||
# Counted for Steward's own health (#5467). The early returns above (a
|
||||
# bad token, no valid samples) answer the agent and aren't counted.
|
||||
ingest_tally.done(phases.elapsed_ms())
|
||||
|
||||
phases.report(host.name, accepted, len(metric_rows),
|
||||
sum(len(c) for _, c in docker_snapshots),
|
||||
@@ -410,6 +418,9 @@ class _Phases:
|
||||
self.took: dict[str, float] = {}
|
||||
self.size = ""
|
||||
|
||||
def elapsed_ms(self) -> float:
|
||||
return (time.monotonic() - self.start) * 1000.0
|
||||
|
||||
def mark(self, name: str) -> None:
|
||||
now = time.monotonic()
|
||||
self.took[name] = self.took.get(name, 0.0) + (now - self.last)
|
||||
|
||||
Reference in New Issue
Block a user