feat: images and attachments go to the off-site bucket once each, by fingerprint
CI and images / lint (push) Successful in 2s
CI and images / extension-version (push) Successful in 2s
CI and images / frontend-build (push) Successful in 25s
CI and images / extension-test (push) Successful in 18s
CI and images / backend-lint-and-test (push) Failing after 44s
CI and images / integration (push) Failing after 2m28s
CI and images / sign-extension (push) Skipped
CI and images / build-web (push) Skipped
CI and images / smoke-web (push) Skipped
CI and images / promote (push) Skipped
CI and images / refresh-dev (push) Skipped
CI and images / unattended-failure (push) Skipped
CI and images / build-agent (push) Skipped
CI and images / lint (push) Successful in 2s
CI and images / extension-version (push) Successful in 2s
CI and images / frontend-build (push) Successful in 25s
CI and images / extension-test (push) Successful in 18s
CI and images / backend-lint-and-test (push) Failing after 44s
CI and images / integration (push) Failing after 2m28s
CI and images / sign-extension (push) Skipped
CI and images / build-web (push) Skipped
CI and images / smoke-web (push) Skipped
CI and images / promote (push) Skipped
CI and images / refresh-dev (push) Skipped
CI and images / unattended-failure (push) Skipped
CI and images / build-agent (push) Skipped
With an off-site destination set up, the library's files are sent to the bucket's media/ folder as media/ab/cd/<sha256><ext>, beside the database dumps under database/. Each file goes once: a new backup_blob table records what each destination holds (migration 0121), and a run's work list is every image and attachment sha with no row yet, so runs are incremental and a new bucket starts its copy from the beginning. - The bucket checks each file against the sha recorded at import; a file over 1 GiB goes in parts and is checked before the upload is completed. A file missing or changed on disk is reported, not stored. - A run is chunked: each 45-minute chunk commits after every file and queues the next, so other long-maintenance jobs get turns. One run at a time. The stall sweep judges a media run by its progress heartbeat (backup_run.last_progress_at), not its start. - A file that can't be sent is listed on the run (status ok, with the error line) and tried again by the next run. A refused put for an object already in an append-only bucket counts as sent. - Nightly, with a destination set, runs the dump and then the media. - Weekly reconcile lists media/ and re-sends what the bucket lost, and adopts what it already holds (a restored database sends nothing twice). - Uploads use the bandwidth limit; read transactions end before transfers. - Backups card: with a destination, an "Images and attachments" section shows coverage (files and bytes in the bucket), "Send new files now", and the media runs. Without one, the local tarball section is unchanged. - Media run history keeps the latest 60 rows. Task #6879, milestone 586. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -20,7 +20,7 @@ system_backup_bp = Blueprint(
|
||||
"system_backup", __name__, url_prefix="/api/system/backup",
|
||||
)
|
||||
|
||||
_KINDS = frozenset({"db", "images"})
|
||||
_KINDS = frozenset({"db", "images", "media"})
|
||||
_TAG_MAX_LEN = 64
|
||||
_BACKUP_SETTINGS_FIELDS = (
|
||||
"backup_db_nightly_enabled",
|
||||
@@ -110,6 +110,38 @@ async def trigger_images_backup():
|
||||
return jsonify({"status": "dispatched"}), 202
|
||||
|
||||
|
||||
@system_backup_bp.route("/media", methods=["POST"])
|
||||
async def trigger_media_backup():
|
||||
"""Send the images and attachments not yet in the off-site bucket. Refused
|
||||
when no destination is set up; a run already going is left to finish."""
|
||||
from ..services import backup_destination as dest
|
||||
|
||||
async with get_session() as session:
|
||||
row = await ImportSettings.load(session)
|
||||
if not dest.describe(row)["configured"]:
|
||||
return _bad("no_destination",
|
||||
detail="set up an off-site destination before sending files")
|
||||
from ..tasks.backup import backup_media_task
|
||||
backup_media_task.delay(triggered_by="manual")
|
||||
return jsonify({"status": "dispatched"}), 202
|
||||
|
||||
|
||||
@system_backup_bp.route("/media", methods=["GET"])
|
||||
async def media_coverage():
|
||||
"""How much of the library is in the off-site bucket: files and bytes,
|
||||
sent and total, each file counted once however many posts hold it."""
|
||||
from ..services import backup_destination as dest
|
||||
from ..services import backup_media
|
||||
|
||||
async with get_session() as session:
|
||||
row = await ImportSettings.load(session)
|
||||
f = dest.fields_of(row)
|
||||
configured = dest.describe(row)["configured"]
|
||||
tid = backup_media.destination_id(f.endpoint, f.bucket) if configured else None
|
||||
counts = await session.run_sync(lambda s: backup_media.coverage(s, tid))
|
||||
return jsonify({"configured": configured, **counts})
|
||||
|
||||
|
||||
@system_backup_bp.route("/runs", methods=["GET"])
|
||||
async def list_runs():
|
||||
try:
|
||||
|
||||
Reference in New Issue
Block a user