bvandeusen
f40063a74d
Guardrails so the fd-leak class of bug (Errno 24 lockup; recent SNMP + UniFi fixes) surfaces early or can't compound, instead of silently killing the app. - Self-fd watchdog (steward/core/self_monitor.py): records open_fds and open_fds_pct (% of soft RLIMIT_NOFILE) as "steward"/"process" metrics each minute through the normal alert pipeline, so the operator can alert on them via the existing alert-rules UI. Built-in WARNING floor at 80% gives a zero-config early signal. Stdlib-only (/proc + resource); degrades to a no-op off Linux. Registered as a core ScheduledTask in app.py. - Poll-overlap guard (steward/core/scheduler.py): extract a pure _DueTracker that skips a tick while a task's prior run is still in flight, so a hung poll can't stack overlapping runs (which amplify per-poll resource/fd use). A skipped task isn't penalised — it retries the next tick after it completes. - fd-stability tests (tests/core/): _DueTracker overlap policy, the watchdog metric/warning/degradation paths, and a real-fd canary that hammers tcp_check and asserts /proc/self/fd doesn't grow. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
47 lines
1.4 KiB
Python
47 lines
1.4 KiB
Python
"""Real-fd regression canary.
|
|
|
|
Drives a real socket code path (the TCP reachability probe) in a tight loop and
|
|
asserts the process's open-fd count doesn't grow. If someone reintroduces a
|
|
socket leak in the probe path — the class of bug behind the Errno 24 lockups —
|
|
this fails in CI instead of in production. Uses real descriptors, not fakes, so
|
|
the assertion has teeth.
|
|
"""
|
|
import asyncio
|
|
import os
|
|
|
|
import pytest
|
|
|
|
from steward.monitors.ping import tcp_check
|
|
|
|
_ITERATIONS = 100
|
|
|
|
|
|
def _fd_count() -> int | None:
|
|
try:
|
|
return len(os.listdir("/proc/self/fd"))
|
|
except OSError:
|
|
return None
|
|
|
|
|
|
async def _hammer_tcp_check() -> None:
|
|
# 127.0.0.1:1 has no listener → connection refused immediately. Each call must
|
|
# fully release its socket; a leak would add ~one fd per iteration.
|
|
for _ in range(_ITERATIONS):
|
|
await tcp_check("127.0.0.1", 1)
|
|
|
|
|
|
def test_tcp_check_does_not_leak_fds():
|
|
if _fd_count() is None:
|
|
pytest.skip("/proc/self/fd unavailable on this platform")
|
|
|
|
asyncio.run(_hammer_tcp_check()) # warm up (lazy imports, caches)
|
|
before = _fd_count()
|
|
asyncio.run(_hammer_tcp_check())
|
|
after = _fd_count()
|
|
|
|
# Small slack for interpreter-internal fds; a genuine leak over 100 iterations
|
|
# would be far larger than this.
|
|
assert after - before <= 5, (
|
|
f"open fds grew {before}->{after} over {_ITERATIONS} tcp_check calls "
|
|
"— possible socket leak")
|