bvandeusen
f2e9ae07dc
scan_library_for_rule ran one 2-hour pass that timed out on large libraries and held the concurrency-1 maintenance queue the whole time, starving vacuum/backup/ normalize (operator-flagged — it was the dominant entry in the 24h failures). It now runs ~10-min chunks and re-enqueues itself until the library is exhausted, matching the operator's preferred pattern (reasonable timeout → retry queued → other things process between). New columns (alembic 0039): resume_after_id persists the keyset cursor so a chunk continues where the last left off; last_progress_at lets the recovery sweep tell a progressing multi- chunk audit from a dead one (it now measures staleness from last_progress_at, not started_at). Matches accumulate across chunks. soft/hard limits dropped 2h→15/16.7 min so the in-chunk budget fires first; a soft-limit backstop re-enqueues to resume instead of erroring the whole run. Tests: time-box → re-enqueue (status stays running); resume carries prior matches and appends new ones. Existing full-scan tests unchanged (small sets finish in one chunk). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
45 lines
1.9 KiB
Python
45 lines
1.9 KiB
Python
"""LibraryAuditRun — async transparency / single_color audit lifecycle.
|
|
|
|
State machine: running → ready → applied / cancelled / error.
|
|
matched_ids JSONB is appended-to by scan_library_for_rule; apply_audit_run
|
|
reads it and routes through cleanup_service.delete_images.
|
|
"""
|
|
|
|
from datetime import datetime
|
|
from typing import Any
|
|
|
|
from sqlalchemy import DateTime, Integer, String, Text, func
|
|
from sqlalchemy.dialects.postgresql import JSONB
|
|
from sqlalchemy.orm import Mapped, mapped_column
|
|
|
|
from .base import Base
|
|
|
|
|
|
class LibraryAuditRun(Base):
|
|
__tablename__ = "library_audit_run"
|
|
|
|
id: Mapped[int] = mapped_column(Integer, primary_key=True)
|
|
rule: Mapped[str] = mapped_column(String(32), nullable=False, index=True)
|
|
params: Mapped[dict[str, Any]] = mapped_column(JSONB, nullable=False)
|
|
status: Mapped[str] = mapped_column(
|
|
String(16), nullable=False, default="running", index=True,
|
|
)
|
|
# running | ready | applied | cancelled | error
|
|
started_at: Mapped[datetime] = mapped_column(
|
|
DateTime(timezone=True), nullable=False, server_default=func.now(),
|
|
)
|
|
finished_at: Mapped[datetime | None] = mapped_column(
|
|
DateTime(timezone=True), nullable=True,
|
|
)
|
|
scanned_count: Mapped[int] = mapped_column(Integer, nullable=False, default=0)
|
|
matched_count: Mapped[int] = mapped_column(Integer, nullable=False, default=0)
|
|
matched_ids: Mapped[list[int]] = mapped_column(JSONB, nullable=False, default=list)
|
|
error: Mapped[str | None] = mapped_column(Text, nullable=True)
|
|
# Chunked-scan state (alembic 0039): keyset cursor the next chunk resumes
|
|
# from, and the last time a chunk made progress (so the recovery sweep can
|
|
# tell a progressing multi-chunk audit from a stuck one).
|
|
resume_after_id: Mapped[int] = mapped_column(Integer, nullable=False, default=0)
|
|
last_progress_at: Mapped[datetime | None] = mapped_column(
|
|
DateTime(timezone=True), nullable=True,
|
|
)
|