feat: with an off-site destination set up, database backups stream to it
CI and images / lint (push) Successful in 2s
CI and images / extension-version (push) Successful in 1s
CI and images / extension-test (push) Successful in 20s
CI and images / frontend-build (push) Successful in 23s
CI and images / backend-lint-and-test (push) Successful in 41s
CI and images / integration (push) Successful in 2m25s
CI and images / sign-extension (push) Successful in 2s
CI and images / build-agent (push) Successful in 5s
CI and images / build-web (push) Successful in 1m50s
CI and images / smoke-web (push) Successful in 49s
CI and images / promote (push) Successful in 2s
CI and images / refresh-dev (push) Skipped
CI and images / unattended-failure (push) Skipped
CI and images / lint (push) Successful in 2s
CI and images / extension-version (push) Successful in 1s
CI and images / extension-test (push) Successful in 20s
CI and images / frontend-build (push) Successful in 23s
CI and images / backend-lint-and-test (push) Successful in 41s
CI and images / integration (push) Successful in 2m25s
CI and images / sign-extension (push) Successful in 2s
CI and images / build-agent (push) Successful in 5s
CI and images / build-web (push) Successful in 1m50s
CI and images / smoke-web (push) Successful in 49s
CI and images / promote (push) Successful in 2s
CI and images / refresh-dev (push) Skipped
CI and images / unattended-failure (push) Skipped
The nightly database backup (and "Run DB backup now") now sends pg_dump's output straight to the off-site bucket as a multipart upload, with no copy staged on this server. The object is named database/fabledcurator-<UTC time>.dump and appears only once pg_dump has exited cleanly and every part has landed; a failed dump aborts the upload and records pg_dump's last line. Without a destination, dumps are written to /images/_backups as before. - backup_s3: Upload (start / add part / complete / abort), with a busy or silent bucket rested and asked again (15 s, 60 s, Retry-After obeyed), and a Pacer that holds uploads to the new bandwidth limit. - Settings: "Upload limit (Mbit/s)" on the destination card, 200 by default, 0 for none (migration 0120). - pg_dump runs under nice (and ionice where installed), locally too. - Backups card says where dumps go; the runs table gains "Saved to" and shows a failed run's error. Off-site rows have no restore/tag/delete here: restoring from the bucket is a later step (#6880). Task #6878, milestone 586. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -49,6 +49,9 @@ def _row_to_dict(r: BackupRun) -> dict:
|
||||
"error": r.error,
|
||||
"restored_from_id": r.restored_from_id,
|
||||
"manifest": r.manifest or {},
|
||||
# Where an off-site dump went ({endpoint, bucket, key}); None for a
|
||||
# backup kept on this server.
|
||||
"offsite": (r.manifest or {}).get("offsite"),
|
||||
}
|
||||
|
||||
|
||||
@@ -180,6 +183,12 @@ async def trigger_restore(run_id: int):
|
||||
"not_restorable",
|
||||
detail=f"source backup status={row.status!r}; only 'ok' rows are restorable",
|
||||
)
|
||||
if (row.manifest or {}).get("offsite"):
|
||||
return _bad(
|
||||
"not_restorable",
|
||||
detail=("this dump is kept in the off-site bucket, not on this server, "
|
||||
"so it can't be restored from here"),
|
||||
)
|
||||
expected = f"restore-{row.kind}-{row.id}"
|
||||
if supplied != expected:
|
||||
return _bad(
|
||||
@@ -250,7 +259,8 @@ async def get_destination():
|
||||
@system_backup_bp.route("/destination", methods=["PATCH"])
|
||||
async def patch_destination():
|
||||
"""Save the destination. Body {endpoint, bucket, access_key, region,
|
||||
secret_key}; an empty field clears it. A missing or empty secret_key
|
||||
secret_key, upload_limit_mbps}; an empty field clears it, and a missing
|
||||
upload_limit_mbps keeps the saved limit. A missing or empty secret_key
|
||||
keeps the saved one, so the screen can save the other fields without the
|
||||
secret being typed again."""
|
||||
from ..services import backup_destination as dest
|
||||
@@ -264,9 +274,17 @@ async def patch_destination():
|
||||
secret = body.get("secret_key")
|
||||
if secret is not None and not isinstance(secret, str):
|
||||
return _bad("invalid_destination", detail="secret_key must be text")
|
||||
limit = body.get("upload_limit_mbps")
|
||||
if limit is not None and (
|
||||
not isinstance(limit, int) or isinstance(limit, bool) or not 0 <= limit <= 100_000
|
||||
):
|
||||
return _bad("invalid_destination",
|
||||
detail="upload_limit_mbps must be a whole number from 0 (no limit) up")
|
||||
|
||||
async with get_session() as session:
|
||||
row = await ImportSettings.load(session)
|
||||
if limit is not None:
|
||||
row.backup_upload_limit_mbps = limit
|
||||
row.backup_s3_endpoint = fields.endpoint
|
||||
row.backup_s3_bucket = fields.bucket
|
||||
row.backup_s3_access_key = fields.access_key
|
||||
|
||||
Reference in New Issue
Block a user