fix(maintenance): time-box + self-resume the archive re-extract task
CI / lint (push) Successful in 3s
CI / frontend-build (push) Successful in 17s
CI / backend-lint-and-test (push) Successful in 25s
CI / integration (push) Failing after 3m3s

reextract_archive_attachments loaded ALL PostAttachments and ran in one pass up
to a 30-min soft limit, then died without re-enqueueing — a large archive
backlog would only ever partially process. And a naive re-run can't advance: an
already-extracted archive is still an archive on disk, so it'd re-extract the
same first batch forever.

Give it a real cursor + time-box + self-resume (mirrors normalize_tags_task,
operator-asked 2026-06-07: reasonable timeout, then re-queue so other work keeps
flowing):
- service scans attachments with id > after_id in ascending order, time-boxes
  the chunk, and reports partial=True + resume_after_id (last scanned id).
- task passes a 600s budget and re-enqueues itself from the cursor until the
  scan is exhausted. Routes on the maintenance_long lane.
- This is independent of the maintenance_long lane isolation (already shipped) —
  that stops long tasks starving the quick maintenance queue; this stops the
  re-extract itself dying on a big backlog.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-06-07 12:03:13 -04:00
parent 5201fab088
commit a73d9327d8
3 changed files with 125 additions and 6 deletions
+26 -3
View File
@@ -57,6 +57,14 @@ def bulk_delete_images_task(self, *, image_ids: list[int]) -> dict:
)
# Time-box one chunk well under the soft limit so a large archive back-catalog
# can't run the task into the Celery time limit (or hog the maintenance_long
# lane). The task re-enqueues itself with the resume cursor until the scan is
# exhausted — mirrors normalize_tags_task (operator-asked 2026-06-07: reasonable
# timeout, then re-queue so other work keeps flowing).
_REEXTRACT_CHUNK_SECONDS = 600
@celery.task(
name="backend.app.tasks.admin.reextract_archive_attachments_task",
bind=True,
@@ -64,15 +72,30 @@ def bulk_delete_images_task(self, *, image_ids: list[int]) -> dict:
retry_backoff=15, retry_backoff_max=180, max_retries=1,
soft_time_limit=1800, time_limit=2400, # 30 min / 40 min
)
def reextract_archive_attachments_task(self) -> dict:
def reextract_archive_attachments_task(self, after_id: int = 0) -> dict:
"""Wraps cleanup_service.reextract_archive_attachments (#713 part 2):
re-extract PostAttachments that are actually archives but were filed
opaquely before the magic-byte gate, and link their members to the post."""
opaquely before the magic-byte gate, and link their members to the post.
Time-boxed + self-resuming: scans attachments after ``after_id`` and, on a
chunk cut, re-enqueues from where it stopped so a big backlog finishes across
chunks instead of dying at the soft limit."""
SessionLocal = _sync_session_factory()
with SessionLocal() as session:
return cleanup_service.reextract_archive_attachments(
summary = cleanup_service.reextract_archive_attachments(
session, images_root=IMAGES_ROOT,
time_budget_seconds=_REEXTRACT_CHUNK_SECONDS, after_id=after_id,
)
# More attachments past this chunk's cursor — continue in the next.
if summary.get("partial") and summary.get("resume_after_id", 0) > after_id:
log.info(
"reextract chunk done (%d scanned, %d archives, resume after id %s) "
"— re-enqueuing to continue",
summary.get("scanned", 0), summary.get("archives", 0),
summary["resume_after_id"],
)
reextract_archive_attachments_task.delay(summary["resume_after_id"])
return summary
# Time-box one chunk well under the soft limit so a large back-catalog (the