feat: with an off-site destination set up, database backups stream to it
CI and images / lint (push) Successful in 2s
CI and images / extension-version (push) Successful in 1s
CI and images / extension-test (push) Successful in 20s
CI and images / frontend-build (push) Successful in 23s
CI and images / backend-lint-and-test (push) Successful in 41s
CI and images / integration (push) Successful in 2m25s
CI and images / sign-extension (push) Successful in 2s
CI and images / build-agent (push) Successful in 5s
CI and images / build-web (push) Successful in 1m50s
CI and images / smoke-web (push) Successful in 49s
CI and images / promote (push) Successful in 2s
CI and images / refresh-dev (push) Skipped
CI and images / unattended-failure (push) Skipped

The nightly database backup (and "Run DB backup now") now sends pg_dump's
output straight to the off-site bucket as a multipart upload, with no copy
staged on this server. The object is named database/fabledcurator-<UTC
time>.dump and appears only once pg_dump has exited cleanly and every part
has landed; a failed dump aborts the upload and records pg_dump's last line.
Without a destination, dumps are written to /images/_backups as before.

- backup_s3: Upload (start / add part / complete / abort), with a busy or
  silent bucket rested and asked again (15 s, 60 s, Retry-After obeyed), and
  a Pacer that holds uploads to the new bandwidth limit.
- Settings: "Upload limit (Mbit/s)" on the destination card, 200 by default,
  0 for none (migration 0120).
- pg_dump runs under nice (and ionice where installed), locally too.
- Backups card says where dumps go; the runs table gains "Saved to" and
  shows a failed run's error. Off-site rows have no restore/tag/delete here:
  restoring from the bucket is a later step (#6880).

Task #6878, milestone 586.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-11-06 21:11:12 -05:00
co-authored by Claude Opus 5.5
parent bb10d30b2d
commit 772482d794
14 changed files with 749 additions and 29 deletions
+36 -11
View File
@@ -18,7 +18,7 @@ from sqlalchemy.exc import DBAPIError, OperationalError
from ..celery_app import celery
from ..config import get_config
from ..models import BackupRun, ImportSettings
from ..services import backup_service
from ..services import backup_destination, backup_service
from ._sync_engine import sync_session_factory as _sync_session_factory
log = logging.getLogger(__name__)
@@ -49,7 +49,12 @@ def _mark_failed(session, row: BackupRun, exc: BaseException) -> None:
)
def backup_db_task(self, *, tag: str | None = None,
triggered_by: str = "manual") -> dict:
"""Create one DB backup. Returns {'backup_run_id': N}."""
"""Create one DB backup. Returns {'backup_run_id': N}.
With an off-site destination set up, the dump streams to that bucket and
nothing is written here; the row's manifest says where it went
(`offsite`: endpoint, bucket, key). Without one, the dump is written to
/images/_backups."""
SessionLocal = _sync_session_factory()
cfg = get_config()
now = datetime.now(UTC)
@@ -64,10 +69,19 @@ def backup_db_task(self, *, tag: str | None = None,
run_id = row.id
try:
result = backup_service.backup_db(
db_url=cfg.database_url_sync, images_root=IMAGES_ROOT,
tag=tag, triggered_by=triggered_by,
)
with SessionLocal() as session:
settings = ImportSettings.load_sync(session)
target = backup_destination.saved_target(settings)
limit_mbps = settings.backup_upload_limit_mbps
if target is not None:
result = backup_service.backup_db_offsite(
db_url=cfg.database_url_sync, target=target, limit_mbps=limit_mbps,
)
else:
result = backup_service.backup_db(
db_url=cfg.database_url_sync, images_root=IMAGES_ROOT,
tag=tag, triggered_by=triggered_by,
)
except (SoftTimeLimitExceeded, Exception) as exc:
with SessionLocal() as session:
row = session.get(BackupRun, run_id)
@@ -79,12 +93,23 @@ def backup_db_task(self, *, tag: str | None = None,
row = session.get(BackupRun, run_id)
row.status = "ok"
row.finished_at = datetime.now(UTC)
row.sql_path = result["sql_path"]
row.size_bytes = result["size_bytes"]
row.manifest = {
"manifest_path": result["manifest_path"],
"ts": result["ts"],
}
if target is not None:
row.manifest = {
"ts": result["ts"],
"offsite": {
"endpoint": result["endpoint"],
"bucket": result["bucket"],
"key": result["key"],
"parts": result["parts"],
},
}
else:
row.sql_path = result["sql_path"]
row.manifest = {
"manifest_path": result["manifest_path"],
"ts": result["ts"],
}
session.commit()
return {"backup_run_id": run_id}