feat: images and attachments go to the off-site bucket once each, by fingerprint
CI and images / lint (push) Successful in 2s
CI and images / extension-version (push) Successful in 2s
CI and images / frontend-build (push) Successful in 25s
CI and images / extension-test (push) Successful in 18s
CI and images / backend-lint-and-test (push) Failing after 44s
CI and images / integration (push) Failing after 2m28s
CI and images / sign-extension (push) Skipped
CI and images / build-web (push) Skipped
CI and images / smoke-web (push) Skipped
CI and images / promote (push) Skipped
CI and images / refresh-dev (push) Skipped
CI and images / unattended-failure (push) Skipped
CI and images / build-agent (push) Skipped
CI and images / lint (push) Successful in 2s
CI and images / extension-version (push) Successful in 2s
CI and images / frontend-build (push) Successful in 25s
CI and images / extension-test (push) Successful in 18s
CI and images / backend-lint-and-test (push) Failing after 44s
CI and images / integration (push) Failing after 2m28s
CI and images / sign-extension (push) Skipped
CI and images / build-web (push) Skipped
CI and images / smoke-web (push) Skipped
CI and images / promote (push) Skipped
CI and images / refresh-dev (push) Skipped
CI and images / unattended-failure (push) Skipped
CI and images / build-agent (push) Skipped
With an off-site destination set up, the library's files are sent to the bucket's media/ folder as media/ab/cd/<sha256><ext>, beside the database dumps under database/. Each file goes once: a new backup_blob table records what each destination holds (migration 0121), and a run's work list is every image and attachment sha with no row yet, so runs are incremental and a new bucket starts its copy from the beginning. - The bucket checks each file against the sha recorded at import; a file over 1 GiB goes in parts and is checked before the upload is completed. A file missing or changed on disk is reported, not stored. - A run is chunked: each 45-minute chunk commits after every file and queues the next, so other long-maintenance jobs get turns. One run at a time. The stall sweep judges a media run by its progress heartbeat (backup_run.last_progress_at), not its start. - A file that can't be sent is listed on the run (status ok, with the error line) and tried again by the next run. A refused put for an object already in an append-only bucket counts as sent. - Nightly, with a destination set, runs the dump and then the media. - Weekly reconcile lists media/ and re-sends what the bucket lost, and adopts what it already holds (a restored database sends nothing twice). - Uploads use the bandwidth limit; read transactions end before transfers. - Backups card: with a destination, an "Images and attachments" section shows coverage (files and bytes in the bucket), "Send new files now", and the media runs. Without one, the local tarball section is unchanged. - Media run history keeps the latest 60 rows. Task #6879, milestone 586. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -322,3 +322,42 @@ def test_the_pacer_holds_bytes_to_the_limit(monkeypatch):
|
||||
def test_no_limit_never_waits(monkeypatch):
|
||||
monkeypatch.setattr(s3.time, "sleep", lambda secs: pytest.fail("waited without a limit"))
|
||||
assert b"".join(s3.Pacer(0).chunks(b"y" * (2 * s3.CHUNK + 1))) == b"y" * (2 * s3.CHUNK + 1)
|
||||
|
||||
|
||||
def test_a_large_file_is_sent_in_parts_and_checked_against_its_sha(tmp_path, monkeypatch):
|
||||
monkeypatch.setattr(s3, "PART_SIZE", 4)
|
||||
f = tmp_path / "big.zip"
|
||||
f.write_bytes(b"0123456789")
|
||||
with _Server(_bucket()) as srv:
|
||||
sent = s3.put_file_in_parts(srv.target(), "media/x.zip", f,
|
||||
sha256=hashlib.sha256(b"0123456789").hexdigest())
|
||||
assert sent == 10
|
||||
assert [r["body"] for r in srv.seen if r["method"] == "PUT"] == [b"0123", b"4567", b"89"]
|
||||
assert srv.seen[-1]["method"] == "POST" # completed
|
||||
|
||||
|
||||
def test_a_file_that_changed_since_import_is_not_completed(tmp_path, monkeypatch):
|
||||
monkeypatch.setattr(s3, "PART_SIZE", 4)
|
||||
f = tmp_path / "big.zip"
|
||||
f.write_bytes(b"0123456789")
|
||||
with _Server(_bucket()) as srv:
|
||||
with pytest.raises(s3.S3Error, match="recorded checksum"):
|
||||
s3.put_file_in_parts(srv.target(), "media/x.zip", f, sha256="0" * 64)
|
||||
assert srv.seen[-1]["method"] == "DELETE" # aborted, nothing stored
|
||||
|
||||
|
||||
def test_a_paced_put_sends_the_same_bytes(tmp_path):
|
||||
f = tmp_path / "a.bin"
|
||||
f.write_bytes(b"z" * (s3.CHUNK + 5))
|
||||
with _Server(lambda h: (200, b"")) as srv:
|
||||
s3.put_file(srv.target(), "a.bin", f, pacer=s3.Pacer(0))
|
||||
assert srv.seen[0]["body"] == b"z" * (s3.CHUNK + 5)
|
||||
|
||||
|
||||
def test_presence_is_the_exact_key_at_the_exact_size():
|
||||
listing = (b"<ListBucketResult><Contents><Key>media/a.jpg</Key><Size>3</Size>"
|
||||
b"</Contents></ListBucketResult>")
|
||||
with _Server(lambda h: (200, listing)) as srv:
|
||||
assert s3.object_present(srv.target(), "media/a.jpg", 3)
|
||||
assert not s3.object_present(srv.target(), "media/a.jpg", 4)
|
||||
assert not s3.object_present(srv.target(), "media/a.jp", 3)
|
||||
|
||||
Reference in New Issue
Block a user