audit-g3: lifecycle batch — recovery sweeps, retention, timeouts #51

Merged

bvandeusen merged 1 commits from dev into main 2026-06-02 14:49:28 -04:00

Conversation 0 Commits 1 Files Changed 8

+298 -11

bvandeusen commented 2026-06-02 14:49:22 -04:00

Owner

Copy Link

Copy Source

Summary

Group 3 of the 2026-06-02 multi-system drift audit. Plugs the FC long-running-entity discipline gaps the audit flagged: every entity that can get stuck now has recovery + retention + timeout, and the long-runners no longer collide with the FC-3i recover_stalled_task_runs 5-min default.

Recovery sweeps (every 5 min, beat schedule)

• recover_stalled_backup_runs — flips BackupRun stuck in running/restoring past 7h (covers the 6.5h images-backup hard limit) to error. prune_backups docstring corrected — the FC-3i TaskRun sweep never touched BackupRun rows; the dashboard was showing phantom in-flight backups.
• recover_stalled_library_audit_runs — flips LibraryAuditRun stuck past 135 min (10-min buffer above scan_library_for_rule's 2h5m hard limit) to error. Previously a SIGKILL'd row blocked all future audits until manual DB surgery.
• recover_stalled_import_batches — finalizes ImportBatch rows stuck running >2h whose child tasks are all terminal (orphan case where the orchestrator crashed before the closing UPDATE). Uses the same EXISTS predicate /api/system/stats already had.

Retention (daily, beat schedule)

• prune_library_audit_runs — 30-day window. Audit rows carry matched_ids JSONB blobs that can hold tens of thousands of ids.
• prune_import_batches — 30-day window. Cascades to ImportTask via the model relationship.

Time limits on five long-runners

Every one of these previously had no time_limit and collided with the recover-stalled-task-runs 5-min default — they could be marked error mid-flight:

• scan_directory: soft 60m / hard 70m
• verify_integrity: 60m / 70m
• backfill_phash: 30m / 35m
• apply_allowlist_tags: 30m / 35m
• recompute_centroids: 30m / 35m

QUEUE_STUCK_THRESHOLD_MINUTES now covers maintenance (75) and scan (75) — above the longest task on each — with per-task overrides for the outliers (backup_images_task: 420, restore_images_task: 420, scan_library_for_rule: 130).

Smaller fixes

• start_audit_run is now age-aware: a running row older than the audit hard limit no longer blocks a new run (the sweep will catch it within 5 min). Previously a SIGKILL'd row blocked forever.
• /api/import/status now uses the same EXISTS predicate /api/system/stats does. The UI banner no longer says "Scanning…" indefinitely while the stats card says idle.
• DownloadEvent.started_at resets on pending→running so a freshly-promoted event from a busy queue isn't measured against its original enqueue time. Was racing recover_stalled_download_events on heavy-queue days.

Test Plan

• Kill a backup job mid-flight (SIGKILL the worker, or stop the container) — wait 5 min, confirm the BackupRun row flips to error instead of staying running forever
• Same for a library audit run — confirm a new audit can be started afterward without manual cleanup
• Trigger a scan that creates an ImportBatch, then kill the orchestrator after all child tasks complete — wait 5 min, confirm the batch finalizes to complete and the UI banner clears
• Trigger a verify_integrity on a large library — confirm it runs to completion without being marked error by the 5-min sweep
• Wait for the daily retention runs (or trigger them manually) — confirm old terminal LibraryAuditRun / ImportBatch rows are pruned

Audit reference

Closes Group 3 of plan task Scribe #551. Group 4 (status-enum miss batch: download_service handling of failed/refreshed, FE handlers for deferred, VALIDATION_FAILED backfill auto-complete, archive dispatch artist/source threading, celery_signals queue mapping) next.

🤖 Generated with Claude Code

## Summary Group 3 of the 2026-06-02 multi-system drift audit. Plugs the FC long-running-entity discipline gaps the audit flagged: every entity that can get stuck now has recovery + retention + timeout, and the long-runners no longer collide with the FC-3i `recover_stalled_task_runs` 5-min default. ### Recovery sweeps (every 5 min, beat schedule) - **`recover_stalled_backup_runs`** — flips `BackupRun` stuck in `running`/`restoring` past 7h (covers the 6.5h images-backup hard limit) to `error`. `prune_backups` docstring corrected — the FC-3i TaskRun sweep never touched BackupRun rows; the dashboard was showing phantom in-flight backups. - **`recover_stalled_library_audit_runs`** — flips `LibraryAuditRun` stuck past 135 min (10-min buffer above `scan_library_for_rule`'s 2h5m hard limit) to `error`. Previously a SIGKILL'd row blocked all future audits until manual DB surgery. - **`recover_stalled_import_batches`** — finalizes `ImportBatch` rows stuck running >2h whose child tasks are all terminal (orphan case where the orchestrator crashed before the closing UPDATE). Uses the same EXISTS predicate `/api/system/stats` already had. ### Retention (daily, beat schedule) - **`prune_library_audit_runs`** — 30-day window. Audit rows carry `matched_ids` JSONB blobs that can hold tens of thousands of ids. - **`prune_import_batches`** — 30-day window. Cascades to `ImportTask` via the model relationship. ### Time limits on five long-runners Every one of these previously had no `time_limit` and collided with the recover-stalled-task-runs 5-min default — they could be marked `error` mid-flight: - `scan_directory`: soft 60m / hard 70m - `verify_integrity`: 60m / 70m - `backfill_phash`: 30m / 35m - `apply_allowlist_tags`: 30m / 35m - `recompute_centroids`: 30m / 35m `QUEUE_STUCK_THRESHOLD_MINUTES` now covers `maintenance` (75) and `scan` (75) — above the longest task on each — with per-task overrides for the outliers (`backup_images_task`: 420, `restore_images_task`: 420, `scan_library_for_rule`: 130). ### Smaller fixes - **`start_audit_run` is now age-aware**: a `running` row older than the audit hard limit no longer blocks a new run (the sweep will catch it within 5 min). Previously a SIGKILL'd row blocked forever. - **`/api/import/status`** now uses the same EXISTS predicate `/api/system/stats` does. The UI banner no longer says "Scanning…" indefinitely while the stats card says idle. - **`DownloadEvent.started_at`** resets on `pending→running` so a freshly-promoted event from a busy queue isn't measured against its original enqueue time. Was racing `recover_stalled_download_events` on heavy-queue days. ## Test Plan - [ ] Kill a backup job mid-flight (SIGKILL the worker, or stop the container) — wait 5 min, confirm the `BackupRun` row flips to `error` instead of staying `running` forever - [ ] Same for a library audit run — confirm a new audit can be started afterward without manual cleanup - [ ] Trigger a scan that creates an `ImportBatch`, then kill the orchestrator after all child tasks complete — wait 5 min, confirm the batch finalizes to `complete` and the UI banner clears - [ ] Trigger a `verify_integrity` on a large library — confirm it runs to completion without being marked `error` by the 5-min sweep - [ ] Wait for the daily retention runs (or trigger them manually) — confirm old terminal `LibraryAuditRun` / `ImportBatch` rows are pruned ## Audit reference Closes Group 3 of plan task `Scribe #551`. Group 4 (status-enum miss batch: download_service handling of `failed`/`refreshed`, FE handlers for `deferred`, VALIDATION_FAILED backfill auto-complete, archive dispatch artist/source threading, celery_signals queue mapping) next. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

bvandeusen added 1 commit 2026-06-02 14:49:22 -04:00

fix(audit-g3): lifecycle batch — recovery sweeps, retention, timeouts ... e30f50e6fe

Plugs the FC long-running-entity discipline gaps the 2026-06-02 audit
flagged: every entity that can get stuck now has recovery + retention +
timeout, and the long-runners no longer collide with the FC-3i sweep.

Recovery sweeps (every 5 min):
- recover_stalled_backup_runs — flips BackupRun stuck in
  running/restoring past 7h (covers the 6.5h images-backup hard
  limit) to error. prune_backups docstring corrected — the FC-3i
  TaskRun sweep never touched BackupRun rows.
- recover_stalled_library_audit_runs — flips LibraryAuditRun stuck
  past 135 min (10-min buffer above scan_library_for_rule's 2h5m
  hard limit) to error. Previously a SIGKILL'd row blocked all
  future audits until manual DB surgery.
- recover_stalled_import_batches — finalizes ImportBatch rows
  stuck running >2h whose child tasks are all terminal (orphan case
  where the orchestrator crashed before the closing UPDATE). Uses
  the same EXISTS predicate /api/system/stats already had.

Retention (daily):
- prune_library_audit_runs — 30-day window. Audit rows carry
  matched_ids JSONB blobs that can hold tens of thousands of ids.
- prune_import_batches — 30-day window. Cascades to ImportTask via
  the model relationship.

time_limits on five long-runners that previously had none (the
audit's headline finding — every one of these collided with the
recover_stalled_task_runs 5-min default and could be marked
'error' mid-flight):
- scan_directory: 60m soft / 70m hard
- verify_integrity: 60m / 70m
- backfill_phash: 30m / 35m
- apply_allowlist_tags: 30m / 35m
- recompute_centroids: 30m / 35m

QUEUE_STUCK_THRESHOLD_MINUTES now covers maintenance (75) and scan
(75) — above the longest task on each — with per-task overrides
for the outliers (backup_images_task 420, restore_images_task 420,
scan_library_for_rule 130).

start_audit_run guard is now age-aware: a 'running' row older than
the audit hard limit doesn't block a new run (the sweep will catch
it within 5 min). Previously a SIGKILL'd row blocked forever.

/api/import/status now uses the same EXISTS predicate
/api/system/stats does, so the two endpoints no longer disagree on
the active-batch question.

DownloadEvent.started_at resets on pending→running so a freshly-
promoted event from a busy queue isn't measured against its
original enqueue time (was racing recover_stalled_download_events
on heavy-queue days).

bvandeusen merged commit d60e0b9494 into main 2026-06-02 14:49:28 -04:00

bvandeusen referenced this pull request 2026-06-02 19:09:44 -04:00

audit-g5 final + ML threshold default + kebab menu fix #54

Sign in to join this conversation.

Reviewers

No Reviewers

Labels

No items

No Label

Milestone

No items

No Milestone

Projects

Clear projects

No project

Assignees

Clear assignees

bvandeusen

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: bvandeusen/FabledCurator#51