feat(reliability): fd-leak rails — self-watchdog, poll-overlap guard, fd tests
Guardrails so the fd-leak class of bug (Errno 24 lockup; recent SNMP + UniFi fixes) surfaces early or can't compound, instead of silently killing the app. - Self-fd watchdog (steward/core/self_monitor.py): records open_fds and open_fds_pct (% of soft RLIMIT_NOFILE) as "steward"/"process" metrics each minute through the normal alert pipeline, so the operator can alert on them via the existing alert-rules UI. Built-in WARNING floor at 80% gives a zero-config early signal. Stdlib-only (/proc + resource); degrades to a no-op off Linux. Registered as a core ScheduledTask in app.py. - Poll-overlap guard (steward/core/scheduler.py): extract a pure _DueTracker that skips a tick while a task's prior run is still in flight, so a hung poll can't stack overlapping runs (which amplify per-poll resource/fd use). A skipped task isn't penalised — it retries the next tick after it completes. - fd-stability tests (tests/core/): _DueTracker overlap policy, the watchdog metric/warning/degradation paths, and a real-fd canary that hammers tcp_check and asserts /proc/self/fd doesn't grow. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,46 @@
|
||||
"""Real-fd regression canary.
|
||||
|
||||
Drives a real socket code path (the TCP reachability probe) in a tight loop and
|
||||
asserts the process's open-fd count doesn't grow. If someone reintroduces a
|
||||
socket leak in the probe path — the class of bug behind the Errno 24 lockups —
|
||||
this fails in CI instead of in production. Uses real descriptors, not fakes, so
|
||||
the assertion has teeth.
|
||||
"""
|
||||
import asyncio
|
||||
import os
|
||||
|
||||
import pytest
|
||||
|
||||
from steward.monitors.ping import tcp_check
|
||||
|
||||
_ITERATIONS = 100
|
||||
|
||||
|
||||
def _fd_count() -> int | None:
|
||||
try:
|
||||
return len(os.listdir("/proc/self/fd"))
|
||||
except OSError:
|
||||
return None
|
||||
|
||||
|
||||
async def _hammer_tcp_check() -> None:
|
||||
# 127.0.0.1:1 has no listener → connection refused immediately. Each call must
|
||||
# fully release its socket; a leak would add ~one fd per iteration.
|
||||
for _ in range(_ITERATIONS):
|
||||
await tcp_check("127.0.0.1", 1)
|
||||
|
||||
|
||||
def test_tcp_check_does_not_leak_fds():
|
||||
if _fd_count() is None:
|
||||
pytest.skip("/proc/self/fd unavailable on this platform")
|
||||
|
||||
asyncio.run(_hammer_tcp_check()) # warm up (lazy imports, caches)
|
||||
before = _fd_count()
|
||||
asyncio.run(_hammer_tcp_check())
|
||||
after = _fd_count()
|
||||
|
||||
# Small slack for interpreter-internal fds; a genuine leak over 100 iterations
|
||||
# would be far larger than this.
|
||||
assert after - before <= 5, (
|
||||
f"open fds grew {before}->{after} over {_ITERATIONS} tcp_check calls "
|
||||
"— possible socket leak")
|
||||
Reference in New Issue
Block a user