feat(reliability): fd-leak rails — self-watchdog, poll-overlap guard, fd tests
CI / lint (push) Successful in 3s
CI / unit (push) Successful in 46s
CI / integration (push) Successful in 2m19s
CI / publish (push) Successful in 1m3s

Guardrails so the fd-leak class of bug (Errno 24 lockup; recent SNMP + UniFi
fixes) surfaces early or can't compound, instead of silently killing the app.

- Self-fd watchdog (steward/core/self_monitor.py): records open_fds and
  open_fds_pct (% of soft RLIMIT_NOFILE) as "steward"/"process" metrics each
  minute through the normal alert pipeline, so the operator can alert on them
  via the existing alert-rules UI. Built-in WARNING floor at 80% gives a
  zero-config early signal. Stdlib-only (/proc + resource); degrades to a no-op
  off Linux. Registered as a core ScheduledTask in app.py.

- Poll-overlap guard (steward/core/scheduler.py): extract a pure _DueTracker
  that skips a tick while a task's prior run is still in flight, so a hung poll
  can't stack overlapping runs (which amplify per-poll resource/fd use). A
  skipped task isn't penalised — it retries the next tick after it completes.

- fd-stability tests (tests/core/): _DueTracker overlap policy, the watchdog
  metric/warning/degradation paths, and a real-fd canary that hammers tcp_check
  and asserts /proc/self/fd doesn't grow.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-06-24 10:36:53 -04:00
parent 0c5a1573da
commit f40063a74d
6 changed files with 395 additions and 10 deletions
+13
View File
@@ -268,6 +268,10 @@ def _register_core_tasks(app: Quart) -> None:
from .core.reports import check_and_send
await check_and_send(app)
async def run_self_monitor():
from .core.self_monitor import record_self_metrics
await record_self_metrics(app)
app._task_registry.extend([
ScheduledTask(
name="weekly_report_check",
@@ -275,6 +279,15 @@ def _register_core_tasks(app: Quart) -> None:
interval_seconds=3600,
run_on_startup=False,
),
# Watch our own fd usage so a descriptor leak surfaces as a metric/alert
# instead of a silent Errno 24 lockup. Cheap; plugin_metrics roll up
# hourly so the 60s cadence is storage-bounded.
ScheduledTask(
name="self_monitor",
coro_factory=run_self_monitor,
interval_seconds=60,
run_on_startup=True,
),
ScheduledTask(
name="monitor_check",
coro_factory=run_monitor_checks,