fix(audit-g3): lifecycle batch — recovery sweeps, retention, timeouts
Plugs the FC long-running-entity discipline gaps the 2026-06-02 audit flagged: every entity that can get stuck now has recovery + retention + timeout, and the long-runners no longer collide with the FC-3i sweep. Recovery sweeps (every 5 min): - recover_stalled_backup_runs — flips BackupRun stuck in running/restoring past 7h (covers the 6.5h images-backup hard limit) to error. prune_backups docstring corrected — the FC-3i TaskRun sweep never touched BackupRun rows. - recover_stalled_library_audit_runs — flips LibraryAuditRun stuck past 135 min (10-min buffer above scan_library_for_rule's 2h5m hard limit) to error. Previously a SIGKILL'd row blocked all future audits until manual DB surgery. - recover_stalled_import_batches — finalizes ImportBatch rows stuck running >2h whose child tasks are all terminal (orphan case where the orchestrator crashed before the closing UPDATE). Uses the same EXISTS predicate /api/system/stats already had. Retention (daily): - prune_library_audit_runs — 30-day window. Audit rows carry matched_ids JSONB blobs that can hold tens of thousands of ids. - prune_import_batches — 30-day window. Cascades to ImportTask via the model relationship. time_limits on five long-runners that previously had none (the audit's headline finding — every one of these collided with the recover_stalled_task_runs 5-min default and could be marked 'error' mid-flight): - scan_directory: 60m soft / 70m hard - verify_integrity: 60m / 70m - backfill_phash: 30m / 35m - apply_allowlist_tags: 30m / 35m - recompute_centroids: 30m / 35m QUEUE_STUCK_THRESHOLD_MINUTES now covers maintenance (75) and scan (75) — above the longest task on each — with per-task overrides for the outliers (backup_images_task 420, restore_images_task 420, scan_library_for_rule 130). start_audit_run guard is now age-aware: a 'running' row older than the audit hard limit doesn't block a new run (the sweep will catch it within 5 min). Previously a SIGKILL'd row blocked forever. /api/import/status now uses the same EXISTS predicate /api/system/stats does, so the two endpoints no longer disagree on the active-batch question. DownloadEvent.started_at resets on pending→running so a freshly- promoted event from a busy queue isn't measured against its original enqueue time (was racing recover_stalled_download_events on heavy-queue days).
This commit is contained in:
@@ -11,7 +11,7 @@ the one-and-done GS/IR migration tooling.)
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
from datetime import UTC, datetime
|
||||
from datetime import UTC, datetime, timedelta
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
@@ -517,6 +517,9 @@ class ConfirmTokenMismatch(Exception):
|
||||
_VALID_RULES = ("transparency", "single_color")
|
||||
|
||||
|
||||
_AUDIT_GUARD_THRESHOLD_MINUTES = 135 # matches LIBRARY_AUDIT_STALL_THRESHOLD_MINUTES
|
||||
|
||||
|
||||
def start_audit_run(
|
||||
session: Session, *, rule: str, params: dict[str, Any],
|
||||
) -> int:
|
||||
@@ -524,11 +527,21 @@ def start_audit_run(
|
||||
scan_library_for_rule Celery task. Returns the new audit_id.
|
||||
|
||||
Concurrent-runs guard: raises AuditAlreadyRunning if any audit_run
|
||||
has status='running'. Operator must cancel or wait."""
|
||||
has status='running' AND started recently. Audit 2026-06-02 made
|
||||
the guard age-aware: a SIGKILL'd run leaves a row in 'running'
|
||||
that the recovery sweep flips on its next pass (~5 min), but a
|
||||
fresh start_audit_run between the SIGKILL and the sweep would
|
||||
previously block forever. Past the threshold, treat the running
|
||||
row as stale and let the sweep clean it up — the new run still
|
||||
gets to start.
|
||||
"""
|
||||
if rule not in _VALID_RULES:
|
||||
raise ValueError(f"unknown rule {rule!r}; expected one of {_VALID_RULES}")
|
||||
cutoff = datetime.now(UTC) - timedelta(minutes=_AUDIT_GUARD_THRESHOLD_MINUTES)
|
||||
existing = session.execute(
|
||||
select(LibraryAuditRun.id).where(LibraryAuditRun.status == "running")
|
||||
select(LibraryAuditRun.id)
|
||||
.where(LibraryAuditRun.status == "running")
|
||||
.where(LibraryAuditRun.started_at >= cutoff)
|
||||
).scalar_one_or_none()
|
||||
if existing is not None:
|
||||
raise AuditAlreadyRunning(existing)
|
||||
|
||||
@@ -177,6 +177,13 @@ class DownloadService:
|
||||
return {"status": "in_flight", "event_id": existing.id}
|
||||
if existing and existing.status == "pending":
|
||||
existing.status = "running"
|
||||
# Reset started_at on the pending→running transition so the
|
||||
# recovery sweep (DOWNLOAD_STALL_THRESHOLD_MINUTES, 30 min)
|
||||
# measures from real start, not from enqueue. On heavy-queue
|
||||
# days a freshly-promoted event whose original started_at
|
||||
# predated the cutoff would otherwise get swept mid-flight,
|
||||
# racing phase3's commit. Audit 2026-06-02.
|
||||
existing.started_at = datetime.now(UTC)
|
||||
await self.async_session.commit()
|
||||
event_id = existing.id
|
||||
else:
|
||||
|
||||
Reference in New Issue
Block a user