refactor: retire the images tarball backup
CI and images / lint (push) Successful in 2s
CI and images / extension-version (push) Successful in 2s
CI and images / extension-test (push) Successful in 20s
CI and images / frontend-build (push) Successful in 25s
CI and images / backend-lint-and-test (push) Successful in 50s
CI and images / integration (push) Successful in 2m38s
CI and images / build-agent (push) Successful in 5s
CI and images / sign-extension (push) Successful in 8s
CI and images / build-web (push) Successful in 4m10s
CI and images / smoke-web (push) Successful in 1m3s
CI and images / promote (push) Successful in 1s
CI and images / refresh-dev (push) Skipped
CI and images / unattended-failure (push) Skipped

Images and attachments are backed up off-site, one object per file, so the
tar+zstd of /images goes: its service and restore, both tasks, the trigger and
restore routes, the Keep last setting, the screen's Images section, and the
tests. Migration 0122 drops backup_run.tar_path and
import_settings.backup_images_keep_last_n and deletes any images runs.

The local database dump stays for an install with no off-site destination,
and the Backups card now says plainly that images and attachments are backed
up only off-site.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-11-07 22:07:44 -05:00
co-authored by Claude Opus 5.5
parent 794b04a64e
commit 14936b62dc
14 changed files with 92 additions and 393 deletions
+6 -83
View File
@@ -1,14 +1,11 @@
"""FC-3h: first-class backup/restore service for FC.
Two independent backup kinds:
- 'db' — pg_dump only; fast; nightly via Beat (settings-gated)
- 'images' — tar+zstd of /images; slow; manual trigger only
With an off-site destination set up, a database dump streams straight to that
bucket instead (backup_db_offsite). Otherwise files live under
<images_root>/_backups/. Each local backup writes:
fc_<kind>_<ts>.{sql|tar.zst} — the artifact
fc_<kind>_<ts>.json — manifest (kind/tag/triggered_by)
The database dump (pg_dump), nightly via Beat when the setting is on. With an
off-site destination set up, the dump streams straight to that bucket
(backup_db_offsite), and images and attachments go there too (backup_media).
Without one, the dump is written under <images_root>/_backups/:
fc_db_<ts>.dump — the dump (custom format)
fc_db_<ts>.json — manifest (tag, trigger, row counts)
Service functions are sync (subprocess-bound). Celery tasks in
backend.app.tasks.backup wrap each one with task_run-tracked
@@ -31,25 +28,6 @@ log = logging.getLogger(__name__)
_BACKUPS_DIRNAME = "_backups"
# Excluded from the images tarball, and each for its own reason (#4233, #4234):
#
# _backups — the archive would otherwise contain every previous archive.
# This is not hypothetical: the 2026-05-23/24 runs, taken before
# this exclude existed, grew 43G -> 107G -> ... -> 2123G as each
# swallowed its predecessors, and cost 4.3T of the images
# filesystem until they were reclaimed on 2026-09-21.
# _quarantine — holds files deliberately pulled OUT of the library.
# secrets — `credential_key.b64`, the key that decrypts the stored
# Patreon/SubscribeStar session credentials.
# cookies — those session cookies themselves.
#
# The last two are the ones worth stating plainly: an images tarball is a media
# archive, and a media archive that carries the key to the operator's accounts
# is a credential leak wearing a backup's name. Encryption at rest buys nothing
# when the key rides along in the same file. A restore therefore does NOT
# re-establish credentials — you sign in again, which is the correct outcome.
_IMAGES_EXCLUDED_DIRNAMES = ("_backups", "_quarantine", "secrets", "cookies")
# Subprocess-level guardrails BEYOND the Celery soft_time_limit. The Celery
# soft limit signals the Python process; subprocess.Popen in a blocking syscall
# ignores that signal, so these bound the worst case directly. Each sits just
@@ -58,9 +36,7 @@ _IMAGES_EXCLUDED_DIRNAMES = ("_backups", "_quarantine", "secrets", "cookies")
# matters because an NFS D-state hang defeats even Celery's SIGKILL (the failure
# that wedged the maintenance lane for hours, #739).
# backup_db_task: soft=1800s / hard=2100s → 1700s
# backup_images_task: soft=21600s / hard=23400s → 21000s
_DB_SUBPROCESS_TIMEOUT_S = 1700 # ~28 min, under the 30-min DB soft limit
_IMAGES_SUBPROCESS_TIMEOUT_S = 21000 # ~5.8 hr, under the 6-hr images soft limit
# Grace after SIGKILL to reap the child. If it can't be reaped in this window
# (an uninterruptible NFS D-state — the failure mode that wedged the
# concurrency-1 maintenance lane for hours, operator-flagged 2026-06-07), we
@@ -259,7 +235,6 @@ def backup_db(
"kind": "db",
"ts": ts,
"sql_path": str(sql_path),
"tar_path": None,
"manifest_path": str(manifest_path),
"size_bytes": _file_size_or_none(sql_path),
"counts": counts,
@@ -348,42 +323,6 @@ def _stream_dump(db_url: str, target, ts: str, limit_mbps: int | None, snapshot:
}
def backup_images(
*, images_root: Path,
tag: str | None = None, triggered_by: str = "manual",
) -> dict:
"""Run tar --zstd over images_root; write .tar.zst + manifest."""
ts = _now_ts()
out_dir = _backups_dir(images_root)
tar_path = out_dir / f"fc_images_{ts}.tar.zst"
# No local-temp here (the archive is hundreds of GB — it can't stage in
# /tmp), but bounded-kill still applies so a tar wedged on NFS fails fast
# rather than holding the lane for hours.
_run_bounded(
[
"tar", "--zstd", "-cf", str(tar_path),
"-C", str(images_root.parent), images_root.name,
*(
f"--exclude={images_root.name}/{name}"
for name in _IMAGES_EXCLUDED_DIRNAMES
),
],
_IMAGES_SUBPROCESS_TIMEOUT_S,
)
manifest_path = _write_manifest(
out_dir, kind="images", ts=ts, tag=tag, triggered_by=triggered_by,
artifact_path=tar_path,
)
return {
"kind": "images",
"ts": ts,
"sql_path": None,
"tar_path": str(tar_path),
"manifest_path": str(manifest_path),
"size_bytes": _file_size_or_none(tar_path),
}
class RestoreFailed(Exception):
"""psql or pg_restore exited with an error; the message is its last line."""
@@ -483,24 +422,9 @@ def restore_db(*, db_url: str, sql_path: Path) -> None:
else f"pg_restore exited with {proc.returncode}")
def restore_images(*, images_root: Path, tar_path: Path) -> None:
"""Untar over images_root.parent. Additive — files NOT in the
tarball are NOT removed. Caller wipes first if a clean restore
is needed."""
subprocess.run(
[
"tar", "--zstd", "-xf", str(tar_path),
"-C", str(images_root.parent),
],
capture_output=True, check=True,
timeout=_IMAGES_SUBPROCESS_TIMEOUT_S,
)
def unlink_artifact_files(
*,
sql_path: str | None,
tar_path: str | None,
manifest_path: str | None,
) -> dict:
"""Best-effort unlink of all on-disk files for a BackupRun row.
@@ -509,7 +433,6 @@ def unlink_artifact_files(
deleted: dict = {}
for label, p in (
("sql", sql_path),
("tar", tar_path),
("manifest", manifest_path),
):
if not p: