feat(backup): check a backup against what is in place, changing nothing (#5823)
#5823 "Check a backup: compare what it would restore with what is in place, without restoring". - steward/backup/check.py: pg_restore writes the backup's rows as COPY text with no database involved. The rows in place go out through COPY in the same form. Tables are compared on their primary keys, over the columns both sides have, and grouped by area. - Sealed values are opened with each side's own key and compared as plaintext digests, so a secret sealed twice still matches. They are never shown. A backup whose key can't open its own secrets fails the check. - What Steward records as it runs (last-seen times, a schedule's last run, backup history, scraped UniFi fields) is left out of the comparison: contents.ACTIVITY, STATE_SETTINGS and HAND_SET_ONLY. - Settings → Backups: - a Check button on each sent backup, which fetches it from the bucket (new s3.get_file); - "Check a file" to upload one, with the size limit raised before the CSRF check reads the form; - a results page. - Each backup Steward sends checks itself once written, and its history row says how that went. - New CLI command `steward check-backup FILE`. CI's image smoke step checks a real backup against its source and expects a match. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
+33
-3
@@ -3,10 +3,11 @@ path-style addressing (endpoint/bucket/key), the three calls a backup needs.
|
||||
|
||||
Fabled Archives speaks S3 with path-style buckets and append-only keys by
|
||||
default (System 180 ruling: backups go there with an append-only key), so
|
||||
Steward only ever puts and lists; it never deletes. Retention is the
|
||||
target's business.
|
||||
Steward puts, lists and reads back (to check or restore a backup); it never
|
||||
deletes. Retention is the target's business. A key that may not read says
|
||||
so as AccessDenied, and the backup can be uploaded by hand instead.
|
||||
|
||||
Written by hand rather than with boto3: three calls don't justify a large
|
||||
Written by hand rather than with boto3: four calls don't justify a large
|
||||
dependency, and signing is a pure function tests can hold to AWS's own
|
||||
published examples.
|
||||
"""
|
||||
@@ -189,3 +190,32 @@ async def list_objects(target: S3Target, prefix: str = "", limit: int = 1000, *,
|
||||
out.append(StoredObject(fields.get("Key", ""), int(fields.get("Size") or 0),
|
||||
fields.get("LastModified", "")))
|
||||
return out
|
||||
|
||||
|
||||
async def get_file(target: S3Target, key: str, path: Path, *,
|
||||
transport: httpx.AsyncBaseTransport | None = None,
|
||||
now: datetime | None = None) -> int:
|
||||
"""Download one object to `path`, streamed; returns its size. A partial
|
||||
download is removed."""
|
||||
url = target.url(key)
|
||||
headers = sign("GET", url, {}, EMPTY_SHA256, target.access_key, target.secret_key,
|
||||
target.region, now or datetime.now(timezone.utc))
|
||||
size = 0
|
||||
try:
|
||||
async with httpx.AsyncClient(verify=target.verify_tls, timeout=UPLOAD_TIMEOUT_SECONDS,
|
||||
transport=transport) as client:
|
||||
async with client.stream("GET", url, headers=headers) as response:
|
||||
if response.status_code != 200:
|
||||
await response.aread()
|
||||
raise S3Error(f"Reading {key} refused: {_says(response)}")
|
||||
with open(path, "wb") as f:
|
||||
async for chunk in response.aiter_bytes(CHUNK):
|
||||
f.write(chunk)
|
||||
size += len(chunk)
|
||||
except httpx.HTTPError as exc:
|
||||
path.unlink(missing_ok=True)
|
||||
raise _reach_error(target, exc) from None
|
||||
except S3Error:
|
||||
path.unlink(missing_ok=True)
|
||||
raise
|
||||
return size
|
||||
|
||||
Reference in New Issue
Block a user