fix: a media backup or restore cut off by a restart carries on where it stopped
CI and images / lint (push) Failing after 1s
CI and images / extension-version (push) Successful in 2s
CI and images / extension-test (push) Successful in 19s
CI and images / frontend-build (push) Successful in 20s
CI and images / backend-lint-and-test (push) Successful in 46s
CI and images / integration (push) Failing after 3m8s
CI and images / sign-extension (push) Skipped
CI and images / build-web (push) Skipped
CI and images / smoke-web (push) Skipped
CI and images / promote (push) Skipped
CI and images / refresh-dev (push) Skipped
CI and images / unattended-failure (push) Skipped
CI and images / build-agent (push) Skipped
CI and images / lint (push) Failing after 1s
CI and images / extension-version (push) Successful in 2s
CI and images / extension-test (push) Successful in 19s
CI and images / frontend-build (push) Successful in 20s
CI and images / backend-lint-and-test (push) Successful in 46s
CI and images / integration (push) Failing after 3m8s
CI and images / sign-extension (push) Skipped
CI and images / build-web (push) Skipped
CI and images / smoke-web (push) Skipped
CI and images / promote (push) Skipped
CI and images / refresh-dev (push) Skipped
CI and images / unattended-failure (push) Skipped
CI and images / build-agent (push) Skipped
A media backup and a restore's file phase run as chains of 45-minute chunks,
each queuing the next when it ends. A restart killed the chunk in flight, so
nothing queued the next one; the run's row stayed "running", "Back up now"
skipped it as already running, and only the 7-hour stall sweep ended it.
- When the long-maintenance lane starts, it queues the next chunk of every
media run still running and every restore fetching its files.
- Each chunk writes its token to the run and hands it to the chunk it queues.
A chunk queued by any other chunk is dropped (a killed chunk's message
redelivered by acks_late, say), so a resume never doubles the chain; a chunk
taken over stops at its next file.
- The integration job installs postgresql-client when the CI image lacks it:
the backup round trip in 375e7e2 runs the real pg_dump and pg_restore.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -188,10 +188,39 @@ def test_a_restore_out_of_time_queues_its_next_chunk(db_sync, tmp_path, run_env,
|
||||
result = tasks.restore_offsite_task(files_only=True)
|
||||
queued, _ = run_env
|
||||
assert result["continues"] is True
|
||||
assert queued == [{"run_id": result["backup_run_id"]}]
|
||||
row = db_sync.get(BackupRun, result["backup_run_id"])
|
||||
db_sync.refresh(row)
|
||||
assert row.status == "restoring" and row.last_progress_at is not None
|
||||
# The next chunk carries the token of the chunk that queued it.
|
||||
assert queued == [{"run_id": row.id, "after": row.manifest["chunk"]}]
|
||||
|
||||
|
||||
def test_a_restart_resumes_the_file_phase_and_drops_the_stale_chunk(
|
||||
db_sync, tmp_path, run_env,
|
||||
):
|
||||
from backend.app.tasks.backup import resume_offsite_runs, restore_offsite_task
|
||||
_file(tmp_path, "a.jpg", b"aa")
|
||||
_image(db_sync, tmp_path / "a.jpg", b"aa")
|
||||
now = datetime.now(UTC)
|
||||
files = BackupRun(kind="restore", status="restoring", triggered_by="restore",
|
||||
started_at=now, manifest={"phase": "files", "cursor": "",
|
||||
"failures": [], "chunk": "killed"})
|
||||
loading = BackupRun(kind="restore", status="restoring", triggered_by="restore",
|
||||
started_at=now, manifest={"phase": "database"})
|
||||
db_sync.add_all([files, loading])
|
||||
db_sync.commit()
|
||||
queued, _ = run_env
|
||||
|
||||
# A restore stopped while loading its database isn't resumed.
|
||||
resume_offsite_runs()
|
||||
assert queued == [{"run_id": files.id, "resume": True}]
|
||||
|
||||
# The killed chunk's own message, delivered again, is dropped.
|
||||
assert "another chunk" in restore_offsite_task(run_id=files.id, after="earlier")["skipped"]
|
||||
done = restore_offsite_task(run_id=files.id, resume=True)
|
||||
assert done["continues"] is False
|
||||
db_sync.refresh(files)
|
||||
assert files.status == "restored" and files.manifest["present"] == 1
|
||||
|
||||
|
||||
def test_a_restore_waits_for_a_backup_that_is_running(db_sync, run_env):
|
||||
|
||||
Reference in New Issue
Block a user