fix: a media backup or restore cut off by a restart carries on where it stopped
CI and images / lint (push) Failing after 1s
CI and images / extension-version (push) Successful in 2s
CI and images / extension-test (push) Successful in 19s
CI and images / frontend-build (push) Successful in 20s
CI and images / backend-lint-and-test (push) Successful in 46s
CI and images / integration (push) Failing after 3m8s
CI and images / sign-extension (push) Skipped
CI and images / build-web (push) Skipped
CI and images / smoke-web (push) Skipped
CI and images / promote (push) Skipped
CI and images / refresh-dev (push) Skipped
CI and images / unattended-failure (push) Skipped
CI and images / build-agent (push) Skipped
CI and images / lint (push) Failing after 1s
CI and images / extension-version (push) Successful in 2s
CI and images / extension-test (push) Successful in 19s
CI and images / frontend-build (push) Successful in 20s
CI and images / backend-lint-and-test (push) Successful in 46s
CI and images / integration (push) Failing after 3m8s
CI and images / sign-extension (push) Skipped
CI and images / build-web (push) Skipped
CI and images / smoke-web (push) Skipped
CI and images / promote (push) Skipped
CI and images / refresh-dev (push) Skipped
CI and images / unattended-failure (push) Skipped
CI and images / build-agent (push) Skipped
A media backup and a restore's file phase run as chains of 45-minute chunks,
each queuing the next when it ends. A restart killed the chunk in flight, so
nothing queued the next one; the run's row stayed "running", "Back up now"
skipped it as already running, and only the 7-hour stall sweep ended it.
- When the long-maintenance lane starts, it queues the next chunk of every
media run still running and every restore fetching its files.
- Each chunk writes its token to the run and hands it to the chunk it queues.
A chunk queued by any other chunk is dropped (a killed chunk's message
redelivered by acks_late, say), so a resume never doubles the chain; a chunk
taken over stops at its next file.
- The integration job installs postgresql-client when the CI image lacks it:
the backup round trip in 375e7e2 runs the real pg_dump and pg_restore.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -1,7 +1,9 @@
|
||||
"""#4433: only the process consuming `download` clears a restart's leftovers.
|
||||
"""Each lane picks up only its own restart's leftovers: the process consuming
|
||||
`download` clears orphaned walks and platform locks, and the one consuming
|
||||
`maintenance_long` resumes off-site backups and restores.
|
||||
|
||||
The consolidated container boots four lanes side by side. If the scheduler or
|
||||
ml lane ran this too, a lane restarting on its own (supervisord restarts a
|
||||
ml lane ran these too, a lane restarting on its own (supervisord restarts a
|
||||
crashed program) would close the events of walks that are still running.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
@@ -42,6 +44,10 @@ def calls(monkeypatch):
|
||||
"backend.app.services.platform_lock.release_all_platform_locks",
|
||||
lambda: seen.append("release") or 0,
|
||||
)
|
||||
monkeypatch.setattr(
|
||||
"backend.app.tasks.backup.resume_offsite_runs",
|
||||
lambda: seen.append("resume") or {},
|
||||
)
|
||||
return seen
|
||||
|
||||
|
||||
@@ -50,12 +56,25 @@ def test_the_download_lane_clears_orphans_and_locks(calls):
|
||||
assert calls == ["interrupt", "commit", "release"]
|
||||
|
||||
|
||||
@pytest.mark.parametrize("queues", [("maintenance", "scan"), ("ml",), ("maintenance_long",)])
|
||||
def test_other_lanes_leave_downloads_alone(calls, queues):
|
||||
@pytest.mark.parametrize("queues", [("maintenance", "scan"), ("ml",)])
|
||||
def test_other_lanes_leave_downloads_and_backups_alone(calls, queues):
|
||||
celery_signals._on_worker_ready(sender=_consumer(*queues))
|
||||
assert calls == []
|
||||
|
||||
|
||||
def test_the_long_maintenance_lane_resumes_offsite_runs_and_nothing_else(calls):
|
||||
celery_signals._on_worker_ready(sender=_consumer("maintenance_long"))
|
||||
assert calls == ["resume"]
|
||||
|
||||
|
||||
def test_a_failed_resume_does_not_stop_the_worker_starting(monkeypatch, calls):
|
||||
def boom():
|
||||
raise RuntimeError("db down")
|
||||
|
||||
monkeypatch.setattr("backend.app.tasks.backup.resume_offsite_runs", boom)
|
||||
celery_signals._on_worker_ready(sender=_consumer("maintenance_long")) # must not raise
|
||||
|
||||
|
||||
def test_unreadable_queues_do_nothing_rather_than_guess(calls):
|
||||
celery_signals._on_worker_ready(sender=SimpleNamespace())
|
||||
assert calls == []
|
||||
|
||||
Reference in New Issue
Block a user