fix(audit): chunk + self-resume library scans (stop the 2h queue-hog timeouts)
scan_library_for_rule ran one 2-hour pass that timed out on large libraries and held the concurrency-1 maintenance queue the whole time, starving vacuum/backup/ normalize (operator-flagged — it was the dominant entry in the 24h failures). It now runs ~10-min chunks and re-enqueues itself until the library is exhausted, matching the operator's preferred pattern (reasonable timeout → retry queued → other things process between). New columns (alembic 0039): resume_after_id persists the keyset cursor so a chunk continues where the last left off; last_progress_at lets the recovery sweep tell a progressing multi- chunk audit from a dead one (it now measures staleness from last_progress_at, not started_at). Matches accumulate across chunks. soft/hard limits dropped 2h→15/16.7 min so the in-chunk budget fires first; a soft-limit backstop re-enqueues to resume instead of erroring the whole run. Tests: time-box → re-enqueue (status stays running); resume carries prior matches and appends new ones. Existing full-scan tests unchanged (small sets finish in one chunk). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -647,19 +647,30 @@ def recover_stalled_library_audit_runs() -> int:
|
||||
guard in start_audit_run — a SIGKILL'd run would block all future
|
||||
audits until manual DB surgery. (The guard is now age-aware, but
|
||||
this sweep is what makes that work in practice.)
|
||||
|
||||
Measures staleness from last_progress_at (alembic 0039), NOT started_at:
|
||||
a chunked scan stays 'running' across many re-enqueued chunks and can
|
||||
legitimately run for hours on a big library — only flag one that hasn't
|
||||
made progress in the threshold window (a dead chunk that never re-enqueued).
|
||||
Falls back to started_at for pre-0039 / never-ticked rows.
|
||||
"""
|
||||
SessionLocal = _sync_session_factory()
|
||||
now = datetime.now(UTC)
|
||||
cutoff = now - timedelta(minutes=LIBRARY_AUDIT_STALL_THRESHOLD_MINUTES)
|
||||
msg = (
|
||||
f"stranded by recovery sweep (no terminal status after "
|
||||
f"stranded by recovery sweep (no progress for "
|
||||
f"{LIBRARY_AUDIT_STALL_THRESHOLD_MINUTES} min)"
|
||||
)
|
||||
with SessionLocal() as session:
|
||||
result = session.execute(
|
||||
update(LibraryAuditRun)
|
||||
.where(LibraryAuditRun.status == "running")
|
||||
.where(LibraryAuditRun.started_at < cutoff)
|
||||
.where(
|
||||
func.coalesce(
|
||||
LibraryAuditRun.last_progress_at,
|
||||
LibraryAuditRun.started_at,
|
||||
) < cutoff
|
||||
)
|
||||
.values(status="error", finished_at=now, error=msg)
|
||||
)
|
||||
session.commit()
|
||||
|
||||
Reference in New Issue
Block a user