refactor: retire the images tarball backup
CI and images / lint (push) Successful in 2s
CI and images / extension-version (push) Successful in 2s
CI and images / extension-test (push) Successful in 20s
CI and images / frontend-build (push) Successful in 25s
CI and images / backend-lint-and-test (push) Successful in 50s
CI and images / integration (push) Successful in 2m38s
CI and images / build-agent (push) Successful in 5s
CI and images / sign-extension (push) Successful in 8s
CI and images / build-web (push) Successful in 4m10s
CI and images / smoke-web (push) Successful in 1m3s
CI and images / promote (push) Successful in 1s
CI and images / refresh-dev (push) Skipped
CI and images / unattended-failure (push) Skipped
CI and images / lint (push) Successful in 2s
CI and images / extension-version (push) Successful in 2s
CI and images / extension-test (push) Successful in 20s
CI and images / frontend-build (push) Successful in 25s
CI and images / backend-lint-and-test (push) Successful in 50s
CI and images / integration (push) Successful in 2m38s
CI and images / build-agent (push) Successful in 5s
CI and images / sign-extension (push) Successful in 8s
CI and images / build-web (push) Successful in 4m10s
CI and images / smoke-web (push) Successful in 1m3s
CI and images / promote (push) Successful in 1s
CI and images / refresh-dev (push) Skipped
CI and images / unattended-failure (push) Skipped
Images and attachments are backed up off-site, one object per file, so the tar+zstd of /images goes: its service and restore, both tasks, the trigger and restore routes, the Keep last setting, the screen's Images section, and the tests. Migration 0122 drops backup_run.tar_path and import_settings.backup_images_keep_last_n and deletes any images runs. The local database dump stays for an install with no off-site destination, and the Backups card now says plainly that images and attachments are backed up only off-site. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -1,14 +1,11 @@
|
||||
"""FC-3h: first-class backup/restore service for FC.
|
||||
|
||||
Two independent backup kinds:
|
||||
- 'db' — pg_dump only; fast; nightly via Beat (settings-gated)
|
||||
- 'images' — tar+zstd of /images; slow; manual trigger only
|
||||
|
||||
With an off-site destination set up, a database dump streams straight to that
|
||||
bucket instead (backup_db_offsite). Otherwise files live under
|
||||
<images_root>/_backups/. Each local backup writes:
|
||||
fc_<kind>_<ts>.{sql|tar.zst} — the artifact
|
||||
fc_<kind>_<ts>.json — manifest (kind/tag/triggered_by)
|
||||
The database dump (pg_dump), nightly via Beat when the setting is on. With an
|
||||
off-site destination set up, the dump streams straight to that bucket
|
||||
(backup_db_offsite), and images and attachments go there too (backup_media).
|
||||
Without one, the dump is written under <images_root>/_backups/:
|
||||
fc_db_<ts>.dump — the dump (custom format)
|
||||
fc_db_<ts>.json — manifest (tag, trigger, row counts)
|
||||
|
||||
Service functions are sync (subprocess-bound). Celery tasks in
|
||||
backend.app.tasks.backup wrap each one with task_run-tracked
|
||||
@@ -31,25 +28,6 @@ log = logging.getLogger(__name__)
|
||||
|
||||
_BACKUPS_DIRNAME = "_backups"
|
||||
|
||||
# Excluded from the images tarball, and each for its own reason (#4233, #4234):
|
||||
#
|
||||
# _backups — the archive would otherwise contain every previous archive.
|
||||
# This is not hypothetical: the 2026-05-23/24 runs, taken before
|
||||
# this exclude existed, grew 43G -> 107G -> ... -> 2123G as each
|
||||
# swallowed its predecessors, and cost 4.3T of the images
|
||||
# filesystem until they were reclaimed on 2026-09-21.
|
||||
# _quarantine — holds files deliberately pulled OUT of the library.
|
||||
# secrets — `credential_key.b64`, the key that decrypts the stored
|
||||
# Patreon/SubscribeStar session credentials.
|
||||
# cookies — those session cookies themselves.
|
||||
#
|
||||
# The last two are the ones worth stating plainly: an images tarball is a media
|
||||
# archive, and a media archive that carries the key to the operator's accounts
|
||||
# is a credential leak wearing a backup's name. Encryption at rest buys nothing
|
||||
# when the key rides along in the same file. A restore therefore does NOT
|
||||
# re-establish credentials — you sign in again, which is the correct outcome.
|
||||
_IMAGES_EXCLUDED_DIRNAMES = ("_backups", "_quarantine", "secrets", "cookies")
|
||||
|
||||
# Subprocess-level guardrails BEYOND the Celery soft_time_limit. The Celery
|
||||
# soft limit signals the Python process; subprocess.Popen in a blocking syscall
|
||||
# ignores that signal, so these bound the worst case directly. Each sits just
|
||||
@@ -58,9 +36,7 @@ _IMAGES_EXCLUDED_DIRNAMES = ("_backups", "_quarantine", "secrets", "cookies")
|
||||
# matters because an NFS D-state hang defeats even Celery's SIGKILL (the failure
|
||||
# that wedged the maintenance lane for hours, #739).
|
||||
# backup_db_task: soft=1800s / hard=2100s → 1700s
|
||||
# backup_images_task: soft=21600s / hard=23400s → 21000s
|
||||
_DB_SUBPROCESS_TIMEOUT_S = 1700 # ~28 min, under the 30-min DB soft limit
|
||||
_IMAGES_SUBPROCESS_TIMEOUT_S = 21000 # ~5.8 hr, under the 6-hr images soft limit
|
||||
# Grace after SIGKILL to reap the child. If it can't be reaped in this window
|
||||
# (an uninterruptible NFS D-state — the failure mode that wedged the
|
||||
# concurrency-1 maintenance lane for hours, operator-flagged 2026-06-07), we
|
||||
@@ -259,7 +235,6 @@ def backup_db(
|
||||
"kind": "db",
|
||||
"ts": ts,
|
||||
"sql_path": str(sql_path),
|
||||
"tar_path": None,
|
||||
"manifest_path": str(manifest_path),
|
||||
"size_bytes": _file_size_or_none(sql_path),
|
||||
"counts": counts,
|
||||
@@ -348,42 +323,6 @@ def _stream_dump(db_url: str, target, ts: str, limit_mbps: int | None, snapshot:
|
||||
}
|
||||
|
||||
|
||||
def backup_images(
|
||||
*, images_root: Path,
|
||||
tag: str | None = None, triggered_by: str = "manual",
|
||||
) -> dict:
|
||||
"""Run tar --zstd over images_root; write .tar.zst + manifest."""
|
||||
ts = _now_ts()
|
||||
out_dir = _backups_dir(images_root)
|
||||
tar_path = out_dir / f"fc_images_{ts}.tar.zst"
|
||||
# No local-temp here (the archive is hundreds of GB — it can't stage in
|
||||
# /tmp), but bounded-kill still applies so a tar wedged on NFS fails fast
|
||||
# rather than holding the lane for hours.
|
||||
_run_bounded(
|
||||
[
|
||||
"tar", "--zstd", "-cf", str(tar_path),
|
||||
"-C", str(images_root.parent), images_root.name,
|
||||
*(
|
||||
f"--exclude={images_root.name}/{name}"
|
||||
for name in _IMAGES_EXCLUDED_DIRNAMES
|
||||
),
|
||||
],
|
||||
_IMAGES_SUBPROCESS_TIMEOUT_S,
|
||||
)
|
||||
manifest_path = _write_manifest(
|
||||
out_dir, kind="images", ts=ts, tag=tag, triggered_by=triggered_by,
|
||||
artifact_path=tar_path,
|
||||
)
|
||||
return {
|
||||
"kind": "images",
|
||||
"ts": ts,
|
||||
"sql_path": None,
|
||||
"tar_path": str(tar_path),
|
||||
"manifest_path": str(manifest_path),
|
||||
"size_bytes": _file_size_or_none(tar_path),
|
||||
}
|
||||
|
||||
|
||||
class RestoreFailed(Exception):
|
||||
"""psql or pg_restore exited with an error; the message is its last line."""
|
||||
|
||||
@@ -483,24 +422,9 @@ def restore_db(*, db_url: str, sql_path: Path) -> None:
|
||||
else f"pg_restore exited with {proc.returncode}")
|
||||
|
||||
|
||||
def restore_images(*, images_root: Path, tar_path: Path) -> None:
|
||||
"""Untar over images_root.parent. Additive — files NOT in the
|
||||
tarball are NOT removed. Caller wipes first if a clean restore
|
||||
is needed."""
|
||||
subprocess.run(
|
||||
[
|
||||
"tar", "--zstd", "-xf", str(tar_path),
|
||||
"-C", str(images_root.parent),
|
||||
],
|
||||
capture_output=True, check=True,
|
||||
timeout=_IMAGES_SUBPROCESS_TIMEOUT_S,
|
||||
)
|
||||
|
||||
|
||||
def unlink_artifact_files(
|
||||
*,
|
||||
sql_path: str | None,
|
||||
tar_path: str | None,
|
||||
manifest_path: str | None,
|
||||
) -> dict:
|
||||
"""Best-effort unlink of all on-disk files for a BackupRun row.
|
||||
@@ -509,7 +433,6 @@ def unlink_artifact_files(
|
||||
deleted: dict = {}
|
||||
for label, p in (
|
||||
("sql", sql_path),
|
||||
("tar", tar_path),
|
||||
("manifest", manifest_path),
|
||||
):
|
||||
if not p:
|
||||
|
||||
Reference in New Issue
Block a user