4 Commits

Include renames

Author	SHA1	Message	Date
bvandeusenandClaude Opus 5.5	375e7e2eda	feat: restore a new install from the off-site bucket, database first, then every file ... Settings → Maintenance → Backups gains "Restore from the bucket": pick one of the bucket's dated dumps, type its confirmation, and FabledCurator saves the current database to the bucket, replaces it with the dump, migrates it to this version, then fetches every image and attachment back to the path its row records. Each file is checked against its sha256 on arrival and moved into place only then; a file already on disk is left alone, so "Fetch missing files" repairs a library that lost files without touching the database. - services/backup_restore.py: list/find/fetch dumps, migrate, keep this install's destination (the dump's sealed secret can't be opened here), library_files work list, restore_file (one fetch per sha, copies for duplicate paths, refuses paths outside /images). - restore_offsite_task: the database phase, then 45-minute file chunks with a heartbeat; a failed load leaves an empty migrated database and says why. The dump's own run row is marked ok, other runs then going "Interrupted". - backup_s3.get_file: sha/size check on arrival, ObjectMissing for a 404, download pacing. - backup_service: every dump leaves the credential table's rows out; restore_db reports pg_restore's last line, stops at the first error, ends other connections and waits at most 60 s for locks. - Backups and orphan repair wait while a restore runs; the stall sweep judges restores by heartbeat; restore runs are pruned to 20. - CLI: python -m backend.app.scripts.restore list|database|files. - README: backup and restore section; logins are in no backup. - Tests: a round trip through an in-process S3 bucket (tests/fake_s3.py) with real pg_dump/pg_restore, plus file, run, API and get_file cases. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-07 14:34:47 -05:00
bvandeusenandClaude Opus 5.5	0cf8edaa10	feat: images and attachments go to the off-site bucket once each, by fingerprint ... With an off-site destination set up, the library's files are sent to the bucket's media/ folder as media/ab/cd/<sha256><ext>, beside the database dumps under database/. Each file goes once: a new backup_blob table records what each destination holds (migration 0121), and a run's work list is every image and attachment sha with no row yet, so runs are incremental and a new bucket starts its copy from the beginning. - The bucket checks each file against the sha recorded at import; a file over 1 GiB goes in parts and is checked before the upload is completed. A file missing or changed on disk is reported, not stored. - A run is chunked: each 45-minute chunk commits after every file and queues the next, so other long-maintenance jobs get turns. One run at a time. The stall sweep judges a media run by its progress heartbeat (backup_run.last_progress_at), not its start. - A file that can't be sent is listed on the run (status ok, with the error line) and tried again by the next run. A refused put for an object already in an append-only bucket counts as sent. - Nightly, with a destination set, runs the dump and then the media. - Weekly reconcile lists media/ and re-sends what the bucket lost, and adopts what it already holds (a restored database sends nothing twice). - Uploads use the bandwidth limit; read transactions end before transfers. - Backups card: with a destination, an "Images and attachments" section shows coverage (files and bytes in the bucket), "Send new files now", and the media runs. Without one, the local tarball section is unchanged. - Media run history keeps the latest 60 rows. Task #6879, milestone 586. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-06 22:07:15 -05:00
bvandeusenandClaude Opus 5.5	772482d794	feat: with an off-site destination set up, database backups stream to it ... The nightly database backup (and "Run DB backup now") now sends pg_dump's output straight to the off-site bucket as a multipart upload, with no copy staged on this server. The object is named database/fabledcurator-<UTC time>.dump and appears only once pg_dump has exited cleanly and every part has landed; a failed dump aborts the upload and records pg_dump's last line. Without a destination, dumps are written to /images/_backups as before. - backup_s3: Upload (start / add part / complete / abort), with a busy or silent bucket rested and asked again (15 s, 60 s, Retry-After obeyed), and a Pacer that holds uploads to the new bandwidth limit. - Settings: "Upload limit (Mbit/s)" on the destination card, 200 by default, 0 for none (migration 0120). - pg_dump runs under nice (and ionice where installed), locally too. - Backups card says where dumps go; the runs table gains "Saved to" and shows a failed run's error. Off-site rows have no restore/tag/delete here: restoring from the bucket is a later step (#6880). Task #6878, milestone 586. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-06 21:11:12 -05:00
bvandeusenandClaude Opus 5.5	4477d7d14f	feat: Settings can point backups at an off-site S3 bucket, with a Test destination check ... Settings → Maintenance → Storage gains "Off-site backup destination": the address (with its port), bucket, region, access key and secret key of an S3 bucket such as Fabled Archives. The secret key is sealed with the credential key, never returned by the API, and kept when the field is left blank. Test destination lists the bucket with the fields as typed and writes nothing, and names the cause when it fails ("nothing is listening there (check the port)", "the secret key is wrong", "answered, but not as S3"). services/backup_s3 is a standard-library SigV4 client (put, get, paged list) ported from Grimoire's; it is held to AWS's published signing examples and exercised against a local HTTP server. Migration 0118 adds the columns; no destination is set up until someone saves one. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-06 16:04:57 -05:00