feat(reliability): fd-leak rails — self-watchdog, poll-overlap guard, fd tests
Guardrails so the fd-leak class of bug (Errno 24 lockup; recent SNMP + UniFi fixes) surfaces early or can't compound, instead of silently killing the app. - Self-fd watchdog (steward/core/self_monitor.py): records open_fds and open_fds_pct (% of soft RLIMIT_NOFILE) as "steward"/"process" metrics each minute through the normal alert pipeline, so the operator can alert on them via the existing alert-rules UI. Built-in WARNING floor at 80% gives a zero-config early signal. Stdlib-only (/proc + resource); degrades to a no-op off Linux. Registered as a core ScheduledTask in app.py. - Poll-overlap guard (steward/core/scheduler.py): extract a pure _DueTracker that skips a tick while a task's prior run is still in flight, so a hung poll can't stack overlapping runs (which amplify per-poll resource/fd use). A skipped task isn't penalised — it retries the next tick after it completes. - fd-stability tests (tests/core/): _DueTracker overlap policy, the watchdog metric/warning/degradation paths, and a real-fd canary that hammers tcp_check and asserts /proc/self/fd doesn't grow. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -268,6 +268,10 @@ def _register_core_tasks(app: Quart) -> None:
|
||||
from .core.reports import check_and_send
|
||||
await check_and_send(app)
|
||||
|
||||
async def run_self_monitor():
|
||||
from .core.self_monitor import record_self_metrics
|
||||
await record_self_metrics(app)
|
||||
|
||||
app._task_registry.extend([
|
||||
ScheduledTask(
|
||||
name="weekly_report_check",
|
||||
@@ -275,6 +279,15 @@ def _register_core_tasks(app: Quart) -> None:
|
||||
interval_seconds=3600,
|
||||
run_on_startup=False,
|
||||
),
|
||||
# Watch our own fd usage so a descriptor leak surfaces as a metric/alert
|
||||
# instead of a silent Errno 24 lockup. Cheap; plugin_metrics roll up
|
||||
# hourly so the 60s cadence is storage-bounded.
|
||||
ScheduledTask(
|
||||
name="self_monitor",
|
||||
coro_factory=run_self_monitor,
|
||||
interval_seconds=60,
|
||||
run_on_startup=True,
|
||||
),
|
||||
ScheduledTask(
|
||||
name="monitor_check",
|
||||
coro_factory=run_monitor_checks,
|
||||
|
||||
Reference in New Issue
Block a user