Commit Graph
2 Commits
Author SHA1 Message Date
bvandeusenandClaude Opus 5.5 401bfb5251 fix(scheduler): a task marked not to run on startup waits an interval after boot (#5431)
CI / lint (push) Successful in 3s
CI / integration (push) Successful in 1m57s
CI / unit (push) Successful in 2m8s
CI / publish (push) Successful in 1m33s
_DueTracker counted a task that had never run from 0 on the loop clock,
which is host uptime, so every run_on_startup=False task (hourly data
cleanup with the Docker log prune, the weekly report check, alert
reconcile, agent age, Ansible schedules) fired on the first tick after
each deploy, while agents were flushing their backlog. Agent ingest was
slow or dropped for a few minutes after every deploy.

start_scheduler now calls tracker.begin(): such a task is first due one
interval after the loop starts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-08 21:52:08 -04:00
bvandeusenandClaude Opus 4.8 f40063a74d feat(reliability): fd-leak rails — self-watchdog, poll-overlap guard, fd tests
CI / lint (push) Successful in 3s
CI / unit (push) Successful in 46s
CI / integration (push) Successful in 2m19s
CI / publish (push) Successful in 1m3s
Guardrails so the fd-leak class of bug (Errno 24 lockup; recent SNMP + UniFi
fixes) surfaces early or can't compound, instead of silently killing the app.

- Self-fd watchdog (steward/core/self_monitor.py): records open_fds and
  open_fds_pct (% of soft RLIMIT_NOFILE) as "steward"/"process" metrics each
  minute through the normal alert pipeline, so the operator can alert on them
  via the existing alert-rules UI. Built-in WARNING floor at 80% gives a
  zero-config early signal. Stdlib-only (/proc + resource); degrades to a no-op
  off Linux. Registered as a core ScheduledTask in app.py.

- Poll-overlap guard (steward/core/scheduler.py): extract a pure _DueTracker
  that skips a tick while a task's prior run is still in flight, so a hung poll
  can't stack overlapping runs (which amplify per-poll resource/fd use). A
  skipped task isn't penalised — it retries the next tick after it completes.

- fd-stability tests (tests/core/): _DueTracker overlap policy, the watchdog
  metric/warning/degradation paths, and a real-fd canary that hammers tcp_check
  and asserts /proc/self/fd doesn't grow.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-24 10:36:53 -04:00