fix: a media backup or restore cut off by a restart carries on where it stopped
CI and images / lint (push) Failing after 1s
CI and images / extension-version (push) Successful in 2s
CI and images / extension-test (push) Successful in 19s
CI and images / frontend-build (push) Successful in 20s
CI and images / backend-lint-and-test (push) Successful in 46s
CI and images / integration (push) Failing after 3m8s
CI and images / sign-extension (push) Skipped
CI and images / build-web (push) Skipped
CI and images / smoke-web (push) Skipped
CI and images / promote (push) Skipped
CI and images / refresh-dev (push) Skipped
CI and images / unattended-failure (push) Skipped
CI and images / build-agent (push) Skipped

A media backup and a restore's file phase run as chains of 45-minute chunks,
each queuing the next when it ends. A restart killed the chunk in flight, so
nothing queued the next one; the run's row stayed "running", "Back up now"
skipped it as already running, and only the 7-hour stall sweep ended it.

- When the long-maintenance lane starts, it queues the next chunk of every
  media run still running and every restore fetching its files.
- Each chunk writes its token to the run and hands it to the chunk it queues.
  A chunk queued by any other chunk is dropped (a killed chunk's message
  redelivered by acks_late, say), so a resume never doubles the chain; a chunk
  taken over stops at its next file.
- The integration job installs postgresql-client when the CI image lacks it:
  the backup round trip in 375e7e2 runs the real pg_dump and pg_restore.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-11-07 14:53:57 -05:00
co-authored by Claude Opus 5.5
parent 375e7e2eda
commit 1e63f70bf7
7 changed files with 281 additions and 45 deletions
+27 -6
View File
@@ -232,17 +232,38 @@ def _consumed_queues(consumer) -> set[str]:
@worker_ready.connect
def _on_worker_ready(sender=None, **_):
"""The download lane clears what its previous process left behind (#4433).
"""A lane picks up what its previous process left behind. Only the lane
that runs the work does it; the other lanes booting beside it must not,
or a lane restarting on its own would disturb work still running.
Best-effort: a failure is logged and the worker still starts."""
queues = _consumed_queues(sender)
if "download" in queues:
_clear_download_leftovers()
if "maintenance_long" in queues:
_resume_offsite_runs()
def _resume_offsite_runs() -> None:
"""The long-maintenance lane carries on an off-site backup or restore that
a restart cut off mid-chunk. With nothing queuing its next chunk, the run
would read "running" for hours, and block every new run, until the stall
sweep ended it."""
try:
from .tasks.backup import resume_offsite_runs
resume_offsite_runs()
except Exception: # noqa: BLE001 — never block the worker from starting
log.exception("long-maintenance lane boot: resuming off-site runs failed")
def _clear_download_leftovers() -> None:
"""The download lane clears what its previous process left behind.
A restart SIGKILLs any walk that outlives the stop grace, so its event is
never finalized and its platform lock is never released. Both used to wait
out timers — a 30-min stall sweep that then blamed the source, and a 27-min
lock TTL that stalled every other source on the platform. Only the process
consuming `download` does this; the other lanes booting beside it must not.
Best-effort: a failure here is logged and the worker still starts.
lock TTL that stalled every other source on the platform.
"""
if "download" not in _consumed_queues(sender):
return
booted_at = datetime.now(UTC)
try:
from .services.download_recovery import interrupt_orphaned_download_events