feat: images and attachments go to the off-site bucket once each, by fingerprint
CI and images / lint (push) Successful in 2s
CI and images / extension-version (push) Successful in 2s
CI and images / frontend-build (push) Successful in 25s
CI and images / extension-test (push) Successful in 18s
CI and images / backend-lint-and-test (push) Failing after 44s
CI and images / integration (push) Failing after 2m28s
CI and images / sign-extension (push) Skipped
CI and images / build-web (push) Skipped
CI and images / smoke-web (push) Skipped
CI and images / promote (push) Skipped
CI and images / refresh-dev (push) Skipped
CI and images / unattended-failure (push) Skipped
CI and images / build-agent (push) Skipped

With an off-site destination set up, the library's files are sent to the
bucket's media/ folder as media/ab/cd/<sha256><ext>, beside the database
dumps under database/. Each file goes once: a new backup_blob table records
what each destination holds (migration 0121), and a run's work list is every
image and attachment sha with no row yet, so runs are incremental and a new
bucket starts its copy from the beginning.

- The bucket checks each file against the sha recorded at import; a file
  over 1 GiB goes in parts and is checked before the upload is completed.
  A file missing or changed on disk is reported, not stored.
- A run is chunked: each 45-minute chunk commits after every file and queues
  the next, so other long-maintenance jobs get turns. One run at a time.
  The stall sweep judges a media run by its progress heartbeat
  (backup_run.last_progress_at), not its start.
- A file that can't be sent is listed on the run (status ok, with the error
  line) and tried again by the next run. A refused put for an object
  already in an append-only bucket counts as sent.
- Nightly, with a destination set, runs the dump and then the media.
- Weekly reconcile lists media/ and re-sends what the bucket lost, and
  adopts what it already holds (a restored database sends nothing twice).
- Uploads use the bandwidth limit; read transactions end before transfers.
- Backups card: with a destination, an "Images and attachments" section
  shows coverage (files and bytes in the bucket), "Send new files now", and
  the media runs. Without one, the local tarball section is unchanged.
- Media run history keeps the latest 60 rows.

Task #6879, milestone 586.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-11-06 22:07:15 -05:00
co-authored by Claude Opus 5.5
parent 772482d794
commit 0cf8edaa10
15 changed files with 925 additions and 24 deletions
+2
View File
@@ -4,6 +4,7 @@ from .app_setting import AppSetting
from .artist import Artist
from .artist_membership_suggestion import ArtistMembershipSuggestion
from .artist_visit import ArtistVisit
from .backup_blob import BackupBlob
from .backup_run import BackupRun
from .base import Base
from .character_prototype import CcipPrototypeState, CharacterPrototype
@@ -55,6 +56,7 @@ __all__ = [
"Artist",
"ArtistMembershipSuggestion",
"ArtistVisit",
"BackupBlob",
"BackupRun",
"Source",
"Credential",
+26
View File
@@ -0,0 +1,26 @@
"""backup_blob: a library file already sent to the off-site bucket.
One row per (destination, sha256). `target` names the destination as
"<endpoint>/<bucket>", so pointing backups at another bucket leaves these rows
inert and that bucket's copy starts from the beginning. `key` is where the
file sits in the bucket (`media/ab/cd/<sha256><ext>`).
"""
from datetime import datetime
from sqlalchemy import BigInteger, DateTime, String, Text, func
from sqlalchemy.orm import Mapped, mapped_column
from .base import Base
class BackupBlob(Base):
__tablename__ = "backup_blob"
target: Mapped[str] = mapped_column(Text, primary_key=True)
sha256: Mapped[str] = mapped_column(String(64), primary_key=True)
key: Mapped[str] = mapped_column(Text, nullable=False)
size_bytes: Mapped[int] = mapped_column(BigInteger, nullable=False)
uploaded_at: Mapped[datetime] = mapped_column(
DateTime(timezone=True), nullable=False, server_default=func.now(),
)
+7 -1
View File
@@ -1,6 +1,7 @@
"""FC-3h: backup_run — operator-facing artifact record for a backup run.
One row per backup attempt (kind='db' or 'images'). Lifecycle
One row per backup attempt (kind='db', 'images', or 'media' for the off-site
upload of images and attachments, which spans many task chunks). Lifecycle
tracking (started_at/finished_at/duration_ms/exception text) lives
in task_run from FC-3i — this row records artifact metadata: file
paths, sizes, tag (retention protection), and restore lineage via
@@ -62,6 +63,11 @@ class BackupRun(Base):
)
# Self-referential FK, unindexed until 0089 (#3300): SET NULL has to find
# the rows pointing at a deleted run before it can null them.
# Heartbeat of a run that spans hours (the media backup): set as it makes
# progress, and what the stall sweep measures its silence from.
last_progress_at: Mapped[datetime | None] = mapped_column(
DateTime(timezone=True), nullable=True,
)
restored_from_id: Mapped[int | None] = mapped_column(
ForeignKey("backup_run.id", ondelete="SET NULL"),
nullable=True, index=True,