fix(audit): chunk + self-resume library scans (stop the 2h queue-hog timeouts)
scan_library_for_rule ran one 2-hour pass that timed out on large libraries and held the concurrency-1 maintenance queue the whole time, starving vacuum/backup/ normalize (operator-flagged — it was the dominant entry in the 24h failures). It now runs ~10-min chunks and re-enqueues itself until the library is exhausted, matching the operator's preferred pattern (reasonable timeout → retry queued → other things process between). New columns (alembic 0039): resume_after_id persists the keyset cursor so a chunk continues where the last left off; last_progress_at lets the recovery sweep tell a progressing multi- chunk audit from a dead one (it now measures staleness from last_progress_at, not started_at). Matches accumulate across chunks. soft/hard limits dropped 2h→15/16.7 min so the in-chunk budget fires first; a soft-limit backstop re-enqueues to resume instead of erroring the whole run. Tests: time-box → re-enqueue (status stays running); resume carries prior matches and appends new ones. Existing full-scan tests unchanged (small sets finish in one chunk). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,40 @@
|
||||
"""library_audit_run: resume cursor + progress timestamp for chunked scans
|
||||
|
||||
Revision ID: 0039
|
||||
Revises: 0038
|
||||
Create Date: 2026-06-07
|
||||
|
||||
scan_library_for_rule used to run one 2h pass that timed out on large libraries
|
||||
and monopolized the concurrency-1 maintenance queue (operator-flagged). It now
|
||||
runs short time-boxed chunks that re-enqueue: `resume_after_id` persists the
|
||||
keyset cursor so the next chunk continues where it left off, and
|
||||
`last_progress_at` lets the recovery sweep tell a progressing multi-chunk audit
|
||||
from a genuinely stuck one.
|
||||
"""
|
||||
from typing import Sequence, Union
|
||||
|
||||
import sqlalchemy as sa
|
||||
from alembic import op
|
||||
|
||||
revision: str = "0039"
|
||||
down_revision: Union[str, None] = "0038"
|
||||
branch_labels: Union[str, Sequence[str], None] = None
|
||||
depends_on: Union[str, Sequence[str], None] = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
op.add_column(
|
||||
"library_audit_run",
|
||||
sa.Column(
|
||||
"resume_after_id", sa.Integer, nullable=False, server_default="0"
|
||||
),
|
||||
)
|
||||
op.add_column(
|
||||
"library_audit_run",
|
||||
sa.Column("last_progress_at", sa.DateTime(timezone=True), nullable=True),
|
||||
)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
op.drop_column("library_audit_run", "last_progress_at")
|
||||
op.drop_column("library_audit_run", "resume_after_id")
|
||||
Reference in New Issue
Block a user