fix: a media backup or restore cut off by a restart carries on where it stopped
CI and images / lint (push) Failing after 1s
CI and images / extension-version (push) Successful in 2s
CI and images / extension-test (push) Successful in 19s
CI and images / frontend-build (push) Successful in 20s
CI and images / backend-lint-and-test (push) Successful in 46s
CI and images / integration (push) Failing after 3m8s
CI and images / sign-extension (push) Skipped
CI and images / build-web (push) Skipped
CI and images / smoke-web (push) Skipped
CI and images / promote (push) Skipped
CI and images / refresh-dev (push) Skipped
CI and images / unattended-failure (push) Skipped
CI and images / build-agent (push) Skipped
CI and images / lint (push) Failing after 1s
CI and images / extension-version (push) Successful in 2s
CI and images / extension-test (push) Successful in 19s
CI and images / frontend-build (push) Successful in 20s
CI and images / backend-lint-and-test (push) Successful in 46s
CI and images / integration (push) Failing after 3m8s
CI and images / sign-extension (push) Skipped
CI and images / build-web (push) Skipped
CI and images / smoke-web (push) Skipped
CI and images / promote (push) Skipped
CI and images / refresh-dev (push) Skipped
CI and images / unattended-failure (push) Skipped
CI and images / build-agent (push) Skipped
A media backup and a restore's file phase run as chains of 45-minute chunks,
each queuing the next when it ends. A restart killed the chunk in flight, so
nothing queued the next one; the run's row stayed "running", "Back up now"
skipped it as already running, and only the 7-hour stall sweep ended it.
- When the long-maintenance lane starts, it queues the next chunk of every
media run still running and every restore fetching its files.
- Each chunk writes its token to the run and hands it to the chunk it queues.
A chunk queued by any other chunk is dropped (a killed chunk's message
redelivered by acks_late, say), so a resume never doubles the chain; a chunk
taken over stops at its next file.
- The integration job installs postgresql-client when the CI image lacks it:
the backup round trip in 375e7e2 runs the real pg_dump and pg_restore.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -232,17 +232,38 @@ def _consumed_queues(consumer) -> set[str]:
|
||||
|
||||
@worker_ready.connect
|
||||
def _on_worker_ready(sender=None, **_):
|
||||
"""The download lane clears what its previous process left behind (#4433).
|
||||
"""A lane picks up what its previous process left behind. Only the lane
|
||||
that runs the work does it; the other lanes booting beside it must not,
|
||||
or a lane restarting on its own would disturb work still running.
|
||||
Best-effort: a failure is logged and the worker still starts."""
|
||||
queues = _consumed_queues(sender)
|
||||
if "download" in queues:
|
||||
_clear_download_leftovers()
|
||||
if "maintenance_long" in queues:
|
||||
_resume_offsite_runs()
|
||||
|
||||
|
||||
def _resume_offsite_runs() -> None:
|
||||
"""The long-maintenance lane carries on an off-site backup or restore that
|
||||
a restart cut off mid-chunk. With nothing queuing its next chunk, the run
|
||||
would read "running" for hours, and block every new run, until the stall
|
||||
sweep ended it."""
|
||||
try:
|
||||
from .tasks.backup import resume_offsite_runs
|
||||
|
||||
resume_offsite_runs()
|
||||
except Exception: # noqa: BLE001 — never block the worker from starting
|
||||
log.exception("long-maintenance lane boot: resuming off-site runs failed")
|
||||
|
||||
|
||||
def _clear_download_leftovers() -> None:
|
||||
"""The download lane clears what its previous process left behind.
|
||||
|
||||
A restart SIGKILLs any walk that outlives the stop grace, so its event is
|
||||
never finalized and its platform lock is never released. Both used to wait
|
||||
out timers — a 30-min stall sweep that then blamed the source, and a 27-min
|
||||
lock TTL that stalled every other source on the platform. Only the process
|
||||
consuming `download` does this; the other lanes booting beside it must not.
|
||||
Best-effort: a failure here is logged and the worker still starts.
|
||||
lock TTL that stalled every other source on the platform.
|
||||
"""
|
||||
if "download" not in _consumed_queues(sender):
|
||||
return
|
||||
booted_at = datetime.now(UTC)
|
||||
try:
|
||||
from .services.download_recovery import interrupt_orphaned_download_events
|
||||
|
||||
Reference in New Issue
Block a user