Scheduled backups to a folder or S3 bucket, and forge tokens sealed at rest (milestone 523) #210

Merged

bvandeusen merged 7 commits from dev into main 2026-10-15 18:19:26 -04:00

Conversation 0 Commits 7 Files Changed 31

+2998 -63

bvandeusen commented 2026-10-15 18:19:15 -04:00

Owner

Copy Link

Copy Source

Merged on the operator's request (2026-10-15: "then merge to main").

Scheduled backups (milestone 523):

• Each day, at an hour the admin chooses, Scribe writes one dated, uncompressed scribe-<UTC>.tar to a folder or an S3-compatible bucket. The tar holds a manifest plus one JSON Lines file per table, written in a fixed order so restic can dedupe it. Scribe only adds files and never deletes.
• Settings → Data → Scheduled backups: a verdict (or Overdue), the destination and schedule, Test destination, Back up now, and recent runs.
• The S3 secret is stored sealed under a key derived from SECRET_KEY. It is never returned to the client and never included in a backup.
• The web restore accepts the .tar. python -m scribe.restore <file> restores into an empty instance, with no size limit.
• Docs: docs/backups.md.

Forge-connection tokens are sealed at rest:

• Migration 0122 renames forge_connections.token to token_sealed and seals the existing rows.
• Keep the same SECRET_KEY across this deploy. A token sealed under a different key can't be opened; Settings flags that connection so its token can be entered again.

Commits: 1942a7a, ff4f2ee, 4248b45, be59829, b45fc9e, 3008d29, 07bc51b. CI is green on dev's head, 07bc51b (run 9225).

🤖 Generated with Claude Code

Merged on the operator's request (2026-10-15: "then merge to main"). **Scheduled backups (milestone 523):** - Each day, at an hour the admin chooses, Scribe writes one dated, uncompressed `scribe-<UTC>.tar` to a folder or an S3-compatible bucket. The tar holds a manifest plus one JSON Lines file per table, written in a fixed order so restic can dedupe it. Scribe only adds files and never deletes. - Settings → Data → Scheduled backups: a verdict (or Overdue), the destination and schedule, Test destination, Back up now, and recent runs. - The S3 secret is stored sealed under a key derived from SECRET_KEY. It is never returned to the client and never included in a backup. - The web restore accepts the `.tar`. `python -m scribe.restore <file>` restores into an empty instance, with no size limit. - Docs: `docs/backups.md`. **Forge-connection tokens are sealed at rest:** - Migration 0122 renames `forge_connections.token` to `token_sealed` and seals the existing rows. - **Keep the same SECRET_KEY across this deploy.** A token sealed under a different key can't be opened; Settings flags that connection so its token can be entered again. Commits: 1942a7a, ff4f2ee, 4248b45, be59829, b45fc9e, 3008d29, 07bc51b. CI is green on dev's head, 07bc51b (run 9225). 🤖 Generated with [Claude Code](https://claude.com/claude-code)

bvandeusen added 7 commits 2026-10-15 18:19:16 -04:00

feat(523): the backup as an uncompressed tar of per-table JSON Lines, ordered so restic dedupes it ... 1942a7a336

export_full_backup orders every query by primary key: an UPDATE moves a
row to the end of the heap, so an unordered export shifted every byte
after an edited row. services/backup_file.py writes the payload as one
dated tar per run - manifest.json plus <section>.jsonl, sorted keys,
fixed member headers - atomically via a dot-prefixed .part file, and
reads it back into the dict restore_full_backup takes, refusing a
missing or short section.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

feat(523): backup destinations - a folder or an S3 bucket, with the S3 secret sealed and kept out of backups ... ff4f2ee5c2

services/backup_s3.py is FabledSteward's SigV4 client ported (put + list,
never delete, so an append-only key is enough), with its tests against
AWS's published signatures. services/backup_destination.py holds the
admin config (backup_config JSON) and delivers a dated tar to a mounted
folder or a bucket, and checks either without sending a backup.
services/secrets.py seals the S3 secret with Fernet under a key derived
from SECRET_KEY; backup._SETTINGS_NOT_EXPORTED keeps it out of every
backup. cryptography becomes a direct dependency (already in the image
through pyjwt).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

feat(523): the backup job - an hourly tick that runs at the chosen hour, Back up now, and the last 30 runs ... 4248b450ea

services/backup_job.py reads its config when it fires, so startup reads
nothing (#4181) and a changed hour needs no reschedule. A run is held to
a 60-minute deadline (a hung NFS mount would otherwise hold the lock and
skip every later backup), never runs twice at once, and keeps its outcome
- file, size, duration, or the reason in words - in the bounded
backup_runs setting, which no backup carries. summarise() gives the page
its verdict: overdue when on and no success for 26h.

Admin API: GET/PUT /api/admin/backups, POST /backups/check (tests unsaved
form values, 45s deadline), POST /backups/run (202, background).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

feat(523): Settings -> Data -> Scheduled backups - the verdict first, then destination, schedule, Test, Back up now and recent runs ... be59829dde

BackupScheduleSettings.vue leads with whether the data is backed up
(last good backup, or Overdue), then what is wrong (last failure in
words, an S3 secret this SECRET_KEY cannot open, incomplete config),
then the form: daily toggle and UTC hour, folder or S3 bucket, Test
destination with unsaved values, Back up now (polls while running,
re-armed in finally), and the last runs. The secret is write-only.
formatBytes moves to utils/bytes.ts for both views. Restore from
Backup now also takes the .tar, and says to use the CLI on a 413.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

feat(523): restore reads the backup tar; python -m scribe.restore brings a fresh instance back ... b45fc9efba

The restore route takes the tar as a raw body beside the JSON download.
scribe/restore.py reads either format, refuses a database that already
has an account (a restore re-creates everything with new ids, so into a
live instance it would duplicate rather than replace), has no size
limit, and says to restart so search indexes rebuild. An integration
test restores a written tar through the real restore_full_backup and
checks every id is remapped. docs/backups.md explains the file, why it
is uncompressed, both destinations, Fabled Archives as the worked
example, and restoring.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

fix(523): restore CLI imports the engine from scribe.models, where it lives ... 3008d29f89

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

feat(523): forge-connection tokens are stored sealed under SECRET_KEY, like the backup S3 secret ... 07bc51b890

The column becomes token_sealed (migration 0122 seals existing rows with a
frozen copy of the v1 key), every reader opens it through connection_token,
a row this instance cannot open is skipped with a warning, and Settings asks
for that token again.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

bvandeusen merged commit cef173c0a6 into main 2026-10-15 18:19:26 -04:00

Sign in to join this conversation.

Reviewers

No Reviewers

Labels

No items

No labels

Milestone

No items

No Milestone

Projects

Clear projects

No projects

Assignees

Clear assignees

renovate-bot bvandeusen

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: bvandeusen/FabledScribe#210