fix: a media backup or restore cut off by a restart carries on where it stopped
CI and images / lint (push) Failing after 1s
CI and images / extension-version (push) Successful in 2s
CI and images / extension-test (push) Successful in 19s
CI and images / frontend-build (push) Successful in 20s
CI and images / backend-lint-and-test (push) Successful in 46s
CI and images / integration (push) Failing after 3m8s
CI and images / sign-extension (push) Skipped
CI and images / build-web (push) Skipped
CI and images / smoke-web (push) Skipped
CI and images / promote (push) Skipped
CI and images / refresh-dev (push) Skipped
CI and images / unattended-failure (push) Skipped
CI and images / build-agent (push) Skipped

A media backup and a restore's file phase run as chains of 45-minute chunks,
each queuing the next when it ends. A restart killed the chunk in flight, so
nothing queued the next one; the run's row stayed "running", "Back up now"
skipped it as already running, and only the 7-hour stall sweep ended it.

- When the long-maintenance lane starts, it queues the next chunk of every
  media run still running and every restore fetching its files.
- Each chunk writes its token to the run and hands it to the chunk it queues.
  A chunk queued by any other chunk is dropped (a killed chunk's message
  redelivered by acks_late, say), so a resume never doubles the chain; a chunk
  taken over stops at its next file.
- The integration job installs postgresql-client when the CI image lacks it:
  the backup round trip in 375e7e2 runs the real pg_dump and pg_restore.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-11-07 14:53:57 -05:00
co-authored by Claude Opus 5.5
parent 375e7e2eda
commit 1e63f70bf7
7 changed files with 281 additions and 45 deletions
+30 -1
View File
@@ -188,10 +188,39 @@ def test_a_restore_out_of_time_queues_its_next_chunk(db_sync, tmp_path, run_env,
result = tasks.restore_offsite_task(files_only=True)
queued, _ = run_env
assert result["continues"] is True
assert queued == [{"run_id": result["backup_run_id"]}]
row = db_sync.get(BackupRun, result["backup_run_id"])
db_sync.refresh(row)
assert row.status == "restoring" and row.last_progress_at is not None
# The next chunk carries the token of the chunk that queued it.
assert queued == [{"run_id": row.id, "after": row.manifest["chunk"]}]
def test_a_restart_resumes_the_file_phase_and_drops_the_stale_chunk(
db_sync, tmp_path, run_env,
):
from backend.app.tasks.backup import resume_offsite_runs, restore_offsite_task
_file(tmp_path, "a.jpg", b"aa")
_image(db_sync, tmp_path / "a.jpg", b"aa")
now = datetime.now(UTC)
files = BackupRun(kind="restore", status="restoring", triggered_by="restore",
started_at=now, manifest={"phase": "files", "cursor": "",
"failures": [], "chunk": "killed"})
loading = BackupRun(kind="restore", status="restoring", triggered_by="restore",
started_at=now, manifest={"phase": "database"})
db_sync.add_all([files, loading])
db_sync.commit()
queued, _ = run_env
# A restore stopped while loading its database isn't resumed.
resume_offsite_runs()
assert queued == [{"run_id": files.id, "resume": True}]
# The killed chunk's own message, delivered again, is dropped.
assert "another chunk" in restore_offsite_task(run_id=files.id, after="earlier")["skipped"]
done = restore_offsite_task(run_id=files.id, resume=True)
assert done["continues"] is False
db_sync.refresh(files)
assert files.status == "restored" and files.manifest["present"] == 1
def test_a_restore_waits_for_a_backup_that_is_running(db_sync, run_env):