Scheduled backups to a folder or S3 bucket, and forge tokens sealed at rest (milestone 523) #210

Merged
bvandeusen merged 7 commits from dev into main 2026-10-15 18:19:26 -04:00
7 Commits
Author SHA1 Message Date
bvandeusenandClaude Opus 5.5 07bc51b890 feat(523): forge-connection tokens are stored sealed under SECRET_KEY, like the backup S3 secret
CI & Build / Python lint (push) Successful in 3s
CI & Build / Plugin hooks (push) Successful in 16s
CI & Build / TypeScript typecheck (push) Successful in 56s
CI & Build / integration (push) Successful in 1m8s
CI & Build / Python tests (push) Successful in 1m54s
CI & Build / Build & push image (push) Successful in 36s
The column becomes token_sealed (migration 0122 seals existing rows with a
frozen copy of the v1 key), every reader opens it through connection_token,
a row this instance cannot open is skipped with a warning, and Settings asks
for that token again.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-15 18:08:11 -04:00
bvandeusenandClaude Opus 5.5 3008d29f89 fix(523): restore CLI imports the engine from scribe.models, where it lives
CI & Build / Python lint (push) Successful in 3s
CI & Build / Plugin hooks (push) Successful in 13s
CI & Build / TypeScript typecheck (push) Successful in 52s
CI & Build / integration (push) Successful in 1m2s
CI & Build / Python tests (push) Successful in 1m54s
CI & Build / Build & push image (push) Successful in 32s
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-15 17:42:54 -04:00
bvandeusenandClaude Opus 5.5 b45fc9efba feat(523): restore reads the backup tar; python -m scribe.restore brings a fresh instance back
CI & Build / Python lint (push) Successful in 4s
CI & Build / Plugin hooks (push) Successful in 16s
CI & Build / TypeScript typecheck (push) Successful in 56s
CI & Build / integration (push) Failing after 1m8s
CI & Build / Python tests (push) Successful in 1m54s
CI & Build / Build & push image (push) Skipped
The restore route takes the tar as a raw body beside the JSON download.
scribe/restore.py reads either format, refuses a database that already
has an account (a restore re-creates everything with new ids, so into a
live instance it would duplicate rather than replace), has no size
limit, and says to restart so search indexes rebuild. An integration
test restores a written tar through the real restore_full_backup and
checks every id is remapped. docs/backups.md explains the file, why it
is uncompressed, both destinations, Fabled Archives as the worked
example, and restoring.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-15 17:40:24 -04:00
bvandeusenandClaude Opus 5.5 be59829dde feat(523): Settings -> Data -> Scheduled backups - the verdict first, then destination, schedule, Test, Back up now and recent runs
BackupScheduleSettings.vue leads with whether the data is backed up
(last good backup, or Overdue), then what is wrong (last failure in
words, an S3 secret this SECRET_KEY cannot open, incomplete config),
then the form: daily toggle and UTC hour, folder or S3 bucket, Test
destination with unsaved values, Back up now (polls while running,
re-armed in finally), and the last runs. The secret is write-only.
formatBytes moves to utils/bytes.ts for both views. Restore from
Backup now also takes the .tar, and says to use the CLI on a 413.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-15 17:40:24 -04:00
bvandeusenandClaude Opus 5.5 4248b450ea feat(523): the backup job - an hourly tick that runs at the chosen hour, Back up now, and the last 30 runs
CI & Build / Python lint (push) Successful in 3s
CI & Build / Plugin hooks (push) Successful in 13s
CI & Build / TypeScript typecheck (push) Successful in 53s
CI & Build / integration (push) Successful in 1m9s
CI & Build / Python tests (push) Successful in 1m54s
CI & Build / Build & push image (push) Successful in 23s
services/backup_job.py reads its config when it fires, so startup reads
nothing (#4181) and a changed hour needs no reschedule. A run is held to
a 60-minute deadline (a hung NFS mount would otherwise hold the lock and
skip every later backup), never runs twice at once, and keeps its outcome
- file, size, duration, or the reason in words - in the bounded
backup_runs setting, which no backup carries. summarise() gives the page
its verdict: overdue when on and no success for 26h.

Admin API: GET/PUT /api/admin/backups, POST /backups/check (tests unsaved
form values, 45s deadline), POST /backups/run (202, background).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-15 17:34:01 -04:00
bvandeusenandClaude Opus 5.5 ff4f2ee5c2 feat(523): backup destinations - a folder or an S3 bucket, with the S3 secret sealed and kept out of backups
CI & Build / Python lint (push) Successful in 2s
CI & Build / Plugin hooks (push) Successful in 15s
CI & Build / TypeScript typecheck (push) Successful in 53s
CI & Build / integration (push) Successful in 1m8s
CI & Build / Python tests (push) Successful in 1m53s
CI & Build / Build & push image (push) Successful in 36s
services/backup_s3.py is FabledSteward's SigV4 client ported (put + list,
never delete, so an append-only key is enough), with its tests against
AWS's published signatures. services/backup_destination.py holds the
admin config (backup_config JSON) and delivers a dated tar to a mounted
folder or a bucket, and checks either without sending a backup.
services/secrets.py seals the S3 secret with Fernet under a key derived
from SECRET_KEY; backup._SETTINGS_NOT_EXPORTED keeps it out of every
backup. cryptography becomes a direct dependency (already in the image
through pyjwt).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-15 17:31:08 -04:00
bvandeusenandClaude Opus 5.5 1942a7a336 feat(523): the backup as an uncompressed tar of per-table JSON Lines, ordered so restic dedupes it
CI & Build / Python lint (push) Successful in 4s
CI & Build / Plugin hooks (push) Successful in 13s
CI & Build / TypeScript typecheck (push) Successful in 55s
CI & Build / integration (push) Successful in 1m9s
CI & Build / Python tests (push) Successful in 1m55s
CI & Build / Build & push image (push) Successful in 1m36s
export_full_backup orders every query by primary key: an UPDATE moves a
row to the end of the heap, so an unordered export shifted every byte
after an edited row. services/backup_file.py writes the payload as one
dated tar per run - manifest.json plus <section>.jsonl, sorted keys,
fixed member headers - atomically via a dot-prefixed .part file, and
reads it back into the dict restore_full_backup takes, refusing a
missing or short section.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-15 16:30:16 -04:00