bvandeusen
f40063a74d
Guardrails so the fd-leak class of bug (Errno 24 lockup; recent SNMP + UniFi fixes) surfaces early or can't compound, instead of silently killing the app. - Self-fd watchdog (steward/core/self_monitor.py): records open_fds and open_fds_pct (% of soft RLIMIT_NOFILE) as "steward"/"process" metrics each minute through the normal alert pipeline, so the operator can alert on them via the existing alert-rules UI. Built-in WARNING floor at 80% gives a zero-config early signal. Stdlib-only (/proc + resource); degrades to a no-op off Linux. Registered as a core ScheduledTask in app.py. - Poll-overlap guard (steward/core/scheduler.py): extract a pure _DueTracker that skips a tick while a task's prior run is still in flight, so a hung poll can't stack overlapping runs (which amplify per-poll resource/fd use). A skipped task isn't penalised — it retries the next tick after it completes. - fd-stability tests (tests/core/): _DueTracker overlap policy, the watchdog metric/warning/degradation paths, and a real-fd canary that hammers tcp_check and asserts /proc/self/fd doesn't grow. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
59 lines
2.1 KiB
Python
59 lines
2.1 KiB
Python
"""Unit tests for the scheduler's poll-overlap guard (_DueTracker).
|
|
|
|
The tracker is pure (clock passed in) so we can assert the overlap policy
|
|
deterministically: a task whose prior run is still in flight is never re-fired,
|
|
and a skipped task isn't penalised — it runs on the next tick once it finishes.
|
|
This is the rail that stops a hung poll from stacking overlapping runs (which
|
|
would amplify any per-poll resource/fd use).
|
|
"""
|
|
from steward.core.scheduler import ScheduledTask, _DueTracker
|
|
|
|
|
|
def _task(name: str, interval: int) -> ScheduledTask:
|
|
return ScheduledTask(name=name, coro_factory=lambda: None, interval_seconds=interval)
|
|
|
|
|
|
def test_due_only_after_interval_elapses():
|
|
t = _task("a", 60)
|
|
tr = _DueTracker()
|
|
assert tr.due([t], now=59) == [] # 59 < 60
|
|
assert tr.due([t], now=60) == [t] # interval elapsed
|
|
|
|
|
|
def test_in_flight_task_is_skipped_even_when_due():
|
|
t = _task("a", 0) # due every tick
|
|
tr = _DueTracker()
|
|
tr.mark_started(t, now=0)
|
|
assert tr.due([t], now=100) == [] # still running → skipped
|
|
tr.mark_done("a")
|
|
assert tr.due([t], now=100) == [t] # finished → eligible again
|
|
|
|
|
|
def test_skip_does_not_advance_last_run_so_it_retries():
|
|
t = _task("a", 10)
|
|
tr = _DueTracker()
|
|
tr.mark_started(t, now=0)
|
|
assert tr.due([t], now=100) == [] # due but in flight
|
|
assert tr.last_run["a"] == 0 # not advanced by the skip
|
|
tr.mark_done("a")
|
|
assert tr.due([t], now=100) == [t] # retried promptly after completion
|
|
|
|
|
|
def test_mark_started_advances_last_run_and_marks_in_flight():
|
|
t = _task("a", 10)
|
|
tr = _DueTracker()
|
|
tr.mark_started(t, now=50)
|
|
assert tr.last_run["a"] == 50
|
|
assert tr.in_flight == {"a"}
|
|
assert tr.due([t], now=55) == [] # 5 < 10 (not yet due)
|
|
assert tr.due([t], now=61) == [] # due by interval, but still in flight
|
|
tr.mark_done("a")
|
|
assert tr.due([t], now=61) == [t]
|
|
|
|
|
|
def test_independent_tasks_do_not_block_each_other():
|
|
a, b = _task("a", 0), _task("b", 0)
|
|
tr = _DueTracker()
|
|
tr.mark_started(a, now=0) # a hangs
|
|
assert tr.due([a, b], now=10) == [b] # b still fires
|