fix(audit): chunk + self-resume library scans (stop the 2h queue-hog timeouts)

scan_library_for_rule ran one 2-hour pass that timed out on large libraries and
held the concurrency-1 maintenance queue the whole time, starving vacuum/backup/
normalize (operator-flagged — it was the dominant entry in the 24h failures).

It now runs ~10-min chunks and re-enqueues itself until the library is
exhausted, matching the operator's preferred pattern (reasonable timeout → retry
queued → other things process between). New columns (alembic 0039):
resume_after_id persists the keyset cursor so a chunk continues where the last
left off; last_progress_at lets the recovery sweep tell a progressing multi-
chunk audit from a dead one (it now measures staleness from last_progress_at,
not started_at). Matches accumulate across chunks. soft/hard limits dropped
2h→15/16.7 min so the in-chunk budget fires first; a soft-limit backstop
re-enqueues to resume instead of erroring the whole run.

Tests: time-box → re-enqueue (status stays running); resume carries prior
matches and appends new ones. Existing full-scan tests unchanged (small sets
finish in one chunk).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-06-07 00:08:19 -04:00
parent d9d502a60d
commit f2e9ae07dc
5 changed files with 174 additions and 12 deletions
+13 -2
View File
@@ -647,19 +647,30 @@ def recover_stalled_library_audit_runs() -> int:
guard in start_audit_run — a SIGKILL'd run would block all future
audits until manual DB surgery. (The guard is now age-aware, but
this sweep is what makes that work in practice.)
Measures staleness from last_progress_at (alembic 0039), NOT started_at:
a chunked scan stays 'running' across many re-enqueued chunks and can
legitimately run for hours on a big library — only flag one that hasn't
made progress in the threshold window (a dead chunk that never re-enqueued).
Falls back to started_at for pre-0039 / never-ticked rows.
"""
SessionLocal = _sync_session_factory()
now = datetime.now(UTC)
cutoff = now - timedelta(minutes=LIBRARY_AUDIT_STALL_THRESHOLD_MINUTES)
msg = (
f"stranded by recovery sweep (no terminal status after "
f"stranded by recovery sweep (no progress for "
f"{LIBRARY_AUDIT_STALL_THRESHOLD_MINUTES} min)"
)
with SessionLocal() as session:
result = session.execute(
update(LibraryAuditRun)
.where(LibraryAuditRun.status == "running")
.where(LibraryAuditRun.started_at < cutoff)
.where(
func.coalesce(
LibraryAuditRun.last_progress_at,
LibraryAuditRun.started_at,
) < cutoff
)
.values(status="error", finished_at=now, error=msg)
)
session.commit()