feat: images and attachments go to the off-site bucket once each, by fingerprint
CI and images / lint (push) Successful in 2s
CI and images / extension-version (push) Successful in 2s
CI and images / frontend-build (push) Successful in 25s
CI and images / extension-test (push) Successful in 18s
CI and images / backend-lint-and-test (push) Failing after 44s
CI and images / integration (push) Failing after 2m28s
CI and images / sign-extension (push) Skipped
CI and images / build-web (push) Skipped
CI and images / smoke-web (push) Skipped
CI and images / promote (push) Skipped
CI and images / refresh-dev (push) Skipped
CI and images / unattended-failure (push) Skipped
CI and images / build-agent (push) Skipped
CI and images / lint (push) Successful in 2s
CI and images / extension-version (push) Successful in 2s
CI and images / frontend-build (push) Successful in 25s
CI and images / extension-test (push) Successful in 18s
CI and images / backend-lint-and-test (push) Failing after 44s
CI and images / integration (push) Failing after 2m28s
CI and images / sign-extension (push) Skipped
CI and images / build-web (push) Skipped
CI and images / smoke-web (push) Skipped
CI and images / promote (push) Skipped
CI and images / refresh-dev (push) Skipped
CI and images / unattended-failure (push) Skipped
CI and images / build-agent (push) Skipped
With an off-site destination set up, the library's files are sent to the bucket's media/ folder as media/ab/cd/<sha256><ext>, beside the database dumps under database/. Each file goes once: a new backup_blob table records what each destination holds (migration 0121), and a run's work list is every image and attachment sha with no row yet, so runs are incremental and a new bucket starts its copy from the beginning. - The bucket checks each file against the sha recorded at import; a file over 1 GiB goes in parts and is checked before the upload is completed. A file missing or changed on disk is reported, not stored. - A run is chunked: each 45-minute chunk commits after every file and queues the next, so other long-maintenance jobs get turns. One run at a time. The stall sweep judges a media run by its progress heartbeat (backup_run.last_progress_at), not its start. - A file that can't be sent is listed on the run (status ok, with the error line) and tried again by the next run. A refused put for an object already in an append-only bucket counts as sent. - Nightly, with a destination set, runs the dump and then the media. - Weekly reconcile lists media/ and re-sends what the bucket lost, and adopts what it already holds (a restored database sends nothing twice). - Uploads use the bandwidth limit; read transactions end before transfers. - Backups card: with a destination, an "Images and attachments" section shows coverage (files and bytes in the bucket), "Send new files now", and the media runs. Without one, the local tarball section is unchanged. - Media run history keeps the latest 60 rows. Task #6879, milestone 586. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,46 @@
|
||||
"""backup_blob: which library files are already in the off-site bucket.
|
||||
|
||||
One row per (destination, sha256) sent. The media backup's work list is every
|
||||
image and attachment sha with no row for the current destination, so a run is
|
||||
incremental and resumable by construction, and pointing backups at a new
|
||||
bucket starts that bucket's copy from the beginning.
|
||||
|
||||
backup_run.last_progress_at is the heartbeat of a run that spans hours; the
|
||||
stall sweep measures a media run's silence from it, not from its start.
|
||||
|
||||
Revision ID: 0121
|
||||
Revises: 0120
|
||||
Create Date: 2026-11-07
|
||||
|
||||
"""
|
||||
import sqlalchemy as sa
|
||||
from alembic import op
|
||||
|
||||
revision = "0121"
|
||||
down_revision = "0120"
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
op.create_table(
|
||||
"backup_blob",
|
||||
sa.Column("target", sa.Text(), nullable=False),
|
||||
sa.Column("sha256", sa.String(64), nullable=False),
|
||||
sa.Column("key", sa.Text(), nullable=False),
|
||||
sa.Column("size_bytes", sa.BigInteger(), nullable=False),
|
||||
sa.Column(
|
||||
"uploaded_at", sa.DateTime(timezone=True), nullable=False,
|
||||
server_default=sa.text("now()"),
|
||||
),
|
||||
sa.PrimaryKeyConstraint("target", "sha256", name=op.f("pk_backup_blob")),
|
||||
)
|
||||
op.add_column(
|
||||
"backup_run",
|
||||
sa.Column("last_progress_at", sa.DateTime(timezone=True), nullable=True),
|
||||
)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
op.drop_column("backup_run", "last_progress_at")
|
||||
op.drop_table("backup_blob")
|
||||
Reference in New Issue
Block a user