fix(audit-g3): lifecycle batch — recovery sweeps, retention, timeouts
CI / lint (push) Successful in 3s
CI / backend-lint-and-test (push) Successful in 18s
CI / frontend-build (push) Successful in 33s
CI / intimp (push) Successful in 3m30s
CI / intapi (push) Successful in 7m30s
CI / intcore (push) Successful in 8m8s

Plugs the FC long-running-entity discipline gaps the 2026-06-02 audit
flagged: every entity that can get stuck now has recovery + retention +
timeout, and the long-runners no longer collide with the FC-3i sweep.

Recovery sweeps (every 5 min):
- recover_stalled_backup_runs — flips BackupRun stuck in
  running/restoring past 7h (covers the 6.5h images-backup hard
  limit) to error. prune_backups docstring corrected — the FC-3i
  TaskRun sweep never touched BackupRun rows.
- recover_stalled_library_audit_runs — flips LibraryAuditRun stuck
  past 135 min (10-min buffer above scan_library_for_rule's 2h5m
  hard limit) to error. Previously a SIGKILL'd row blocked all
  future audits until manual DB surgery.
- recover_stalled_import_batches — finalizes ImportBatch rows
  stuck running >2h whose child tasks are all terminal (orphan case
  where the orchestrator crashed before the closing UPDATE). Uses
  the same EXISTS predicate /api/system/stats already had.

Retention (daily):
- prune_library_audit_runs — 30-day window. Audit rows carry
  matched_ids JSONB blobs that can hold tens of thousands of ids.
- prune_import_batches — 30-day window. Cascades to ImportTask via
  the model relationship.

time_limits on five long-runners that previously had none (the
audit's headline finding — every one of these collided with the
recover_stalled_task_runs 5-min default and could be marked
'error' mid-flight):
- scan_directory: 60m soft / 70m hard
- verify_integrity: 60m / 70m
- backfill_phash: 30m / 35m
- apply_allowlist_tags: 30m / 35m
- recompute_centroids: 30m / 35m

QUEUE_STUCK_THRESHOLD_MINUTES now covers maintenance (75) and scan
(75) — above the longest task on each — with per-task overrides
for the outliers (backup_images_task 420, restore_images_task 420,
scan_library_for_rule 130).

start_audit_run guard is now age-aware: a 'running' row older than
the audit hard limit doesn't block a new run (the sweep will catch
it within 5 min). Previously a SIGKILL'd row blocked forever.

/api/import/status now uses the same EXISTS predicate
/api/system/stats does, so the two endpoints no longer disagree on
the active-batch question.

DownloadEvent.started_at resets on pending→running so a freshly-
promoted event from a busy queue isn't measured against its
original enqueue time (was racing recover_stalled_download_events
on heavy-queue days).
This commit is contained in:
2026-06-02 14:30:46 -04:00
parent e66987f092
commit e30f50e6fe
8 changed files with 298 additions and 11 deletions
+25
View File
@@ -105,6 +105,31 @@ def make_celery() -> Celery:
"task": "backend.app.tasks.backup.prune_backups",
"schedule": 86400.0, # daily
},
# Audit 2026-06-02 — three new per-entity recovery sweeps.
# Each runs every 5 min like the other recover_stalled_*
# sweeps; each is a no-op when nothing is stuck.
"recover-stalled-backup-runs": {
"task": "backend.app.tasks.maintenance.recover_stalled_backup_runs",
"schedule": 300.0,
},
"recover-stalled-library-audit-runs": {
"task": "backend.app.tasks.maintenance.recover_stalled_library_audit_runs",
"schedule": 300.0,
},
"recover-stalled-import-batches": {
"task": "backend.app.tasks.maintenance.recover_stalled_import_batches",
"schedule": 300.0,
},
# Audit 2026-06-02 — daily retention for two entities
# whose terminal rows otherwise accumulate forever.
"prune-library-audit-runs": {
"task": "backend.app.tasks.maintenance.prune_library_audit_runs",
"schedule": 86400.0,
},
"prune-import-batches": {
"task": "backend.app.tasks.maintenance.prune_import_batches",
"schedule": 86400.0,
},
},
timezone="UTC",
)