bvandeusenandClaude Opus 5.5
1e63f70bf7
CI and images / lint (push) Failing after 1s
CI and images / extension-version (push) Successful in 2s
CI and images / extension-test (push) Successful in 19s
CI and images / frontend-build (push) Successful in 20s
CI and images / backend-lint-and-test (push) Successful in 46s
CI and images / integration (push) Failing after 3m8s
CI and images / sign-extension (push) Skipped
CI and images / build-web (push) Skipped
CI and images / smoke-web (push) Skipped
CI and images / promote (push) Skipped
CI and images / refresh-dev (push) Skipped
CI and images / unattended-failure (push) Skipped
CI and images / build-agent (push) Skipped
A media backup and a restore's file phase run as chains of 45-minute chunks,
each queuing the next when it ends. A restart killed the chunk in flight, so
nothing queued the next one; the run's row stayed "running", "Back up now"
skipped it as already running, and only the 7-hour stall sweep ended it.
- When the long-maintenance lane starts, it queues the next chunk of every
media run still running and every restore fetching its files.
- Each chunk writes its token to the run and hands it to the chunk it queues.
A chunk queued by any other chunk is dropped (a killed chunk's message
redelivered by acks_late, say), so a resume never doubles the chain; a chunk
taken over stops at its next file.
- The integration job installs postgresql-client when the CI image lacks it:
the backup round trip in 375e7e2 runs the real pg_dump and pg_restore.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
91 lines
2.9 KiB
Python
91 lines
2.9 KiB
Python
"""Each lane picks up only its own restart's leftovers: the process consuming
|
|
`download` clears orphaned walks and platform locks, and the one consuming
|
|
`maintenance_long` resumes off-site backups and restores.
|
|
|
|
The consolidated container boots four lanes side by side. If the scheduler or
|
|
ml lane ran these too, a lane restarting on its own (supervisord restarts a
|
|
crashed program) would close the events of walks that are still running.
|
|
"""
|
|
from __future__ import annotations
|
|
|
|
from types import SimpleNamespace
|
|
|
|
import pytest
|
|
|
|
from backend.app import celery_signals
|
|
|
|
|
|
def _consumer(*queues: str):
|
|
return SimpleNamespace(
|
|
task_consumer=SimpleNamespace(queues=[SimpleNamespace(name=q) for q in queues])
|
|
)
|
|
|
|
|
|
@pytest.fixture
|
|
def calls(monkeypatch):
|
|
seen: list[str] = []
|
|
|
|
class _Session:
|
|
def __enter__(self):
|
|
return self
|
|
|
|
def __exit__(self, *exc):
|
|
return False
|
|
|
|
def commit(self):
|
|
seen.append("commit")
|
|
|
|
monkeypatch.setattr(celery_signals, "sync_session_factory", lambda: _Session)
|
|
monkeypatch.setattr(
|
|
"backend.app.services.download_recovery.interrupt_orphaned_download_events",
|
|
lambda session, *, booted_at: seen.append("interrupt") or 0,
|
|
)
|
|
monkeypatch.setattr(
|
|
"backend.app.services.platform_lock.release_all_platform_locks",
|
|
lambda: seen.append("release") or 0,
|
|
)
|
|
monkeypatch.setattr(
|
|
"backend.app.tasks.backup.resume_offsite_runs",
|
|
lambda: seen.append("resume") or {},
|
|
)
|
|
return seen
|
|
|
|
|
|
def test_the_download_lane_clears_orphans_and_locks(calls):
|
|
celery_signals._on_worker_ready(sender=_consumer("default", "download", "import"))
|
|
assert calls == ["interrupt", "commit", "release"]
|
|
|
|
|
|
@pytest.mark.parametrize("queues", [("maintenance", "scan"), ("ml",)])
|
|
def test_other_lanes_leave_downloads_and_backups_alone(calls, queues):
|
|
celery_signals._on_worker_ready(sender=_consumer(*queues))
|
|
assert calls == []
|
|
|
|
|
|
def test_the_long_maintenance_lane_resumes_offsite_runs_and_nothing_else(calls):
|
|
celery_signals._on_worker_ready(sender=_consumer("maintenance_long"))
|
|
assert calls == ["resume"]
|
|
|
|
|
|
def test_a_failed_resume_does_not_stop_the_worker_starting(monkeypatch, calls):
|
|
def boom():
|
|
raise RuntimeError("db down")
|
|
|
|
monkeypatch.setattr("backend.app.tasks.backup.resume_offsite_runs", boom)
|
|
celery_signals._on_worker_ready(sender=_consumer("maintenance_long")) # must not raise
|
|
|
|
|
|
def test_unreadable_queues_do_nothing_rather_than_guess(calls):
|
|
celery_signals._on_worker_ready(sender=SimpleNamespace())
|
|
assert calls == []
|
|
|
|
|
|
def test_a_failure_does_not_stop_the_worker_starting(monkeypatch, calls):
|
|
def boom(session, *, booted_at):
|
|
raise RuntimeError("db down")
|
|
|
|
monkeypatch.setattr(
|
|
"backend.app.services.download_recovery.interrupt_orphaned_download_events", boom
|
|
)
|
|
celery_signals._on_worker_ready(sender=_consumer("download")) # must not raise
|