#5060 "One evaluation path: record_metrics batch API; host-agent ingest
evaluates rules".
- record_metrics(session, rows) writes metric rows and evaluates the rules
that watch them, reading rules, states and maintenance windows once per
batch. Each rule sees its key's samples oldest first, so an agent catching
up after an outage counts consecutive checks as they happened.
- Host-agent ingest now goes through it. Before, it wrote plugin_metrics
directly and no host_agent rule could ever fire. The catalog gains the
per-mount disk_used_pct ("<host>:<mount>").
- MetricBatch gathers a transaction's samples; the docker ingest, traefik,
unifi, snmp, monitor and self-monitor writers record one batch each instead
of a rule query per sample.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Guardrails so the fd-leak class of bug (Errno 24 lockup; recent SNMP + UniFi
fixes) surfaces early or can't compound, instead of silently killing the app.
- Self-fd watchdog (steward/core/self_monitor.py): records open_fds and
open_fds_pct (% of soft RLIMIT_NOFILE) as "steward"/"process" metrics each
minute through the normal alert pipeline, so the operator can alert on them
via the existing alert-rules UI. Built-in WARNING floor at 80% gives a
zero-config early signal. Stdlib-only (/proc + resource); degrades to a no-op
off Linux. Registered as a core ScheduledTask in app.py.
- Poll-overlap guard (steward/core/scheduler.py): extract a pure _DueTracker
that skips a tick while a task's prior run is still in flight, so a hung poll
can't stack overlapping runs (which amplify per-poll resource/fd use). A
skipped task isn't penalised — it retries the next tick after it completes.
- fd-stability tests (tests/core/): _DueTracker overlap policy, the watchdog
metric/warning/degradation paths, and a real-fd canary that hammers tcp_check
and asserts /proc/self/fd doesn't grow.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>