CI and images / lint (push) Successful in 2s
CI and images / extension-version (push) Successful in 1s
CI and images / extension-test (push) Successful in 17s
CI and images / frontend-build (push) Successful in 19s
CI and images / backend-lint-and-test (push) Successful in 45s
CI and images / integration (push) Failing after 2m26s
CI and images / sign-extension (push) Skipped
CI and images / build-web (push) Skipped
CI and images / smoke-web (push) Skipped
CI and images / promote (push) Skipped
CI and images / refresh-dev (push) Skipped
CI and images / unattended-failure (push) Skipped
CI and images / build-agent (push) Skipped
Settings → Maintenance → Backups gains "Restore from the bucket": pick one of the bucket's dated dumps, type its confirmation, and FabledCurator saves the current database to the bucket, replaces it with the dump, migrates it to this version, then fetches every image and attachment back to the path its row records. Each file is checked against its sha256 on arrival and moved into place only then; a file already on disk is left alone, so "Fetch missing files" repairs a library that lost files without touching the database. - services/backup_restore.py: list/find/fetch dumps, migrate, keep this install's destination (the dump's sealed secret can't be opened here), library_files work list, restore_file (one fetch per sha, copies for duplicate paths, refuses paths outside /images). - restore_offsite_task: the database phase, then 45-minute file chunks with a heartbeat; a failed load leaves an empty migrated database and says why. The dump's own run row is marked ok, other runs then going "Interrupted". - backup_s3.get_file: sha/size check on arrival, ObjectMissing for a 404, download pacing. - backup_service: every dump leaves the credential table's rows out; restore_db reports pg_restore's last line, stops at the first error, ends other connections and waits at most 60 s for locks. - Backups and orphan repair wait while a restore runs; the stall sweep judges restores by heartbeat; restore runs are pruned to 20. - CLI: python -m backend.app.scripts.restore list|database|files. - README: backup and restore section; logins are in no backup. - Tests: a round trip through an in-process S3 bucket (tests/fake_s3.py) with real pg_dump/pg_restore, plus file, run, API and get_file cases. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
111 lines
4.4 KiB
Python
111 lines
4.4 KiB
Python
"""An in-process S3 bucket for round-trip tests: put, multipart upload,
|
|
get, and ListObjectsV2, kept in memory. It doesn't check signatures; the
|
|
signing has its own tests against AWS's published vectors. Like Fabled
|
|
Archives, it refuses to replace an object that is already there.
|
|
|
|
with FakeS3() as s3:
|
|
target = s3.target()
|
|
...
|
|
s3.objects # {key: bytes}
|
|
"""
|
|
from __future__ import annotations
|
|
|
|
import threading
|
|
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
|
|
from urllib.parse import parse_qs, unquote, urlsplit
|
|
from xml.sax.saxutils import escape
|
|
|
|
from backend.app.services.backup_s3 import S3Target
|
|
|
|
BUCKET = "curator"
|
|
|
|
|
|
class FakeS3:
|
|
def __init__(self):
|
|
self.objects: dict[str, bytes] = {}
|
|
self.uploads: dict[str, dict[int, bytes]] = {}
|
|
outer = self
|
|
|
|
class Handler(BaseHTTPRequestHandler):
|
|
def _parts(self):
|
|
u = urlsplit(self.path)
|
|
path = unquote(u.path).lstrip("/")
|
|
bucket, _, key = path.partition("/")
|
|
query = parse_qs(u.query, keep_blank_values=True) # `?uploads` has no value
|
|
return bucket, key, {k: v[0] for k, v in query.items()}
|
|
|
|
def _reply(self, status, body=b"", headers=None):
|
|
self.send_response(status)
|
|
for name, value in (headers or {}).items():
|
|
self.send_header(name, value)
|
|
self.send_header("Content-Length", str(len(body)))
|
|
self.end_headers()
|
|
if self.command != "HEAD":
|
|
self.wfile.write(body)
|
|
|
|
def _body(self):
|
|
length = int(self.headers.get("Content-Length") or 0)
|
|
return self.rfile.read(length) if length else b""
|
|
|
|
def do_PUT(self):
|
|
_, key, q = self._parts()
|
|
body = self._body()
|
|
if "uploadId" in q:
|
|
outer.uploads[q["uploadId"]][int(q["partNumber"])] = body
|
|
return self._reply(200, headers={"ETag": f'"p{q["partNumber"]}"'})
|
|
if key in outer.objects:
|
|
return self._reply(403, b"<Error><Code>AccessDenied</Code></Error>")
|
|
outer.objects[key] = body
|
|
return self._reply(200)
|
|
|
|
def do_POST(self):
|
|
_, key, q = self._parts()
|
|
self._body()
|
|
if "uploads" in q:
|
|
upload_id = f"u{len(outer.uploads) + 1}"
|
|
outer.uploads[upload_id] = {}
|
|
return self._reply(200, (
|
|
"<InitiateMultipartUploadResult>"
|
|
f"<UploadId>{upload_id}</UploadId>"
|
|
"</InitiateMultipartUploadResult>").encode())
|
|
parts = outer.uploads.pop(q["uploadId"])
|
|
outer.objects[key] = b"".join(parts[n] for n in sorted(parts))
|
|
return self._reply(200, b"<CompleteMultipartUploadResult/>")
|
|
|
|
def do_DELETE(self):
|
|
_, _, q = self._parts()
|
|
outer.uploads.pop(q.get("uploadId", ""), None)
|
|
return self._reply(204)
|
|
|
|
def do_GET(self):
|
|
_, key, q = self._parts()
|
|
if q.get("list-type") == "2":
|
|
prefix = q.get("prefix", "")
|
|
items = "".join(
|
|
f"<Contents><Key>{escape(k)}</Key><Size>{len(v)}</Size>"
|
|
"<LastModified>2026-01-01T00:00:00Z</LastModified></Contents>"
|
|
for k, v in sorted(outer.objects.items()) if k.startswith(prefix)
|
|
)
|
|
return self._reply(200, f"<ListBucketResult>{items}</ListBucketResult>".encode())
|
|
if key not in outer.objects:
|
|
return self._reply(404, b"<Error><Code>NoSuchKey</Code></Error>")
|
|
return self._reply(200, outer.objects[key])
|
|
|
|
def log_message(self, *args):
|
|
pass
|
|
|
|
self.httpd = ThreadingHTTPServer(("127.0.0.1", 0), Handler)
|
|
self.thread = threading.Thread(target=self.httpd.serve_forever, daemon=True)
|
|
|
|
def __enter__(self):
|
|
self.thread.start()
|
|
return self
|
|
|
|
def __exit__(self, *exc):
|
|
self.httpd.shutdown()
|
|
self.httpd.server_close()
|
|
|
|
def target(self) -> S3Target:
|
|
host, port = self.httpd.server_address
|
|
return S3Target(f"http://{host}:{port}", BUCKET, "AK", "SK")
|