fix: a media backup or restore cut off by a restart carries on where it stopped
CI and images / lint (push) Failing after 1s
CI and images / extension-version (push) Successful in 2s
CI and images / extension-test (push) Successful in 19s
CI and images / frontend-build (push) Successful in 20s
CI and images / backend-lint-and-test (push) Successful in 46s
CI and images / integration (push) Failing after 3m8s
CI and images / sign-extension (push) Skipped
CI and images / build-web (push) Skipped
CI and images / smoke-web (push) Skipped
CI and images / promote (push) Skipped
CI and images / refresh-dev (push) Skipped
CI and images / unattended-failure (push) Skipped
CI and images / build-agent (push) Skipped

A media backup and a restore's file phase run as chains of 45-minute chunks,
each queuing the next when it ends. A restart killed the chunk in flight, so
nothing queued the next one; the run's row stayed "running", "Back up now"
skipped it as already running, and only the 7-hour stall sweep ended it.

- When the long-maintenance lane starts, it queues the next chunk of every
  media run still running and every restore fetching its files.
- Each chunk writes its token to the run and hands it to the chunk it queues.
  A chunk queued by any other chunk is dropped (a killed chunk's message
  redelivered by acks_late, say), so a resume never doubles the chain; a chunk
  taken over stops at its next file.
- The integration job installs postgresql-client when the CI image lacks it:
  the backup round trip in 375e7e2 runs the real pg_dump and pg_restore.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-11-07 14:53:57 -05:00
co-authored by Claude Opus 5.5
parent 375e7e2eda
commit 1e63f70bf7
7 changed files with 281 additions and 45 deletions
+23 -4
View File
@@ -1,7 +1,9 @@
"""#4433: only the process consuming `download` clears a restart's leftovers.
"""Each lane picks up only its own restart's leftovers: the process consuming
`download` clears orphaned walks and platform locks, and the one consuming
`maintenance_long` resumes off-site backups and restores.
The consolidated container boots four lanes side by side. If the scheduler or
ml lane ran this too, a lane restarting on its own (supervisord restarts a
ml lane ran these too, a lane restarting on its own (supervisord restarts a
crashed program) would close the events of walks that are still running.
"""
from __future__ import annotations
@@ -42,6 +44,10 @@ def calls(monkeypatch):
"backend.app.services.platform_lock.release_all_platform_locks",
lambda: seen.append("release") or 0,
)
monkeypatch.setattr(
"backend.app.tasks.backup.resume_offsite_runs",
lambda: seen.append("resume") or {},
)
return seen
@@ -50,12 +56,25 @@ def test_the_download_lane_clears_orphans_and_locks(calls):
assert calls == ["interrupt", "commit", "release"]
@pytest.mark.parametrize("queues", [("maintenance", "scan"), ("ml",), ("maintenance_long",)])
def test_other_lanes_leave_downloads_alone(calls, queues):
@pytest.mark.parametrize("queues", [("maintenance", "scan"), ("ml",)])
def test_other_lanes_leave_downloads_and_backups_alone(calls, queues):
celery_signals._on_worker_ready(sender=_consumer(*queues))
assert calls == []
def test_the_long_maintenance_lane_resumes_offsite_runs_and_nothing_else(calls):
celery_signals._on_worker_ready(sender=_consumer("maintenance_long"))
assert calls == ["resume"]
def test_a_failed_resume_does_not_stop_the_worker_starting(monkeypatch, calls):
def boom():
raise RuntimeError("db down")
monkeypatch.setattr("backend.app.tasks.backup.resume_offsite_runs", boom)
celery_signals._on_worker_ready(sender=_consumer("maintenance_long")) # must not raise
def test_unreadable_queues_do_nothing_rather_than_guess(calls):
celery_signals._on_worker_ready(sender=SimpleNamespace())
assert calls == []