feat(523): restore reads the backup tar; python -m scribe.restore brings a fresh instance back
CI & Build / Python lint (push) Successful in 4s
CI & Build / Plugin hooks (push) Successful in 16s
CI & Build / TypeScript typecheck (push) Successful in 56s
CI & Build / integration (push) Failing after 1m8s
CI & Build / Python tests (push) Successful in 1m54s
CI & Build / Build & push image (push) Skipped
CI & Build / Python lint (push) Successful in 4s
CI & Build / Plugin hooks (push) Successful in 16s
CI & Build / TypeScript typecheck (push) Successful in 56s
CI & Build / integration (push) Failing after 1m8s
CI & Build / Python tests (push) Successful in 1m54s
CI & Build / Build & push image (push) Skipped
The restore route takes the tar as a raw body beside the JSON download. scribe/restore.py reads either format, refuses a database that already has an account (a restore re-creates everything with new ids, so into a live instance it would duplicate rather than replace), has no size limit, and says to restart so search indexes rebuild. An integration test restores a written tar through the real restore_full_backup and checks every id is remapped. docs/backups.md explains the file, why it is uncompressed, both destinations, Fabled Archives as the worked example, and restoring. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
+158
@@ -0,0 +1,158 @@
|
||||
# Backups
|
||||
|
||||
Scribe can back itself up every day to a **folder** or an **S3-compatible
|
||||
bucket**. Each run adds one file; Scribe never deletes an old one. Keeping and
|
||||
thinning the history is the destination's job.
|
||||
|
||||
Set it up under **Settings → Data → Scheduled backups** (admins only).
|
||||
|
||||
## What a backup is
|
||||
|
||||
One file per run, named for the moment it was taken in UTC:
|
||||
|
||||
```
|
||||
scribe-20261015T020000Z.tar
|
||||
```
|
||||
|
||||
It is a plain, **uncompressed** tar:
|
||||
|
||||
- `manifest.json` comes first. It holds the backup format version, the Scribe
|
||||
version, when the backup was taken, and how many rows each table has.
|
||||
- Then there is one `<table>.jsonl` per table, one row per line.
|
||||
|
||||
It holds everything the admin **Full Backup** download holds, laid out
|
||||
differently. It does **not** hold:
|
||||
|
||||
- **Search indexes.** They are rebuilt from the records when Scribe starts.
|
||||
- **Sharing:** groups, project shares and note shares.
|
||||
- **Credentials:** API keys and forge connections. After a restore, issue new
|
||||
API keys and add forge connections again.
|
||||
- **Logs and notifications.**
|
||||
- **The backup destination's own S3 secret.**
|
||||
|
||||
The file contains hashed passwords, so store it somewhere only trusted people
|
||||
can read.
|
||||
|
||||
### Why it is shaped this way
|
||||
|
||||
The file is meant to be copied off-site by a deduplicating tool such as
|
||||
restic, which stores a chunk of data only once no matter how many backups
|
||||
contain it. Every choice serves that:
|
||||
|
||||
- **It is never compressed.** Compression changes every byte after the first
|
||||
edit, so nothing would match the day before. restic compresses its own
|
||||
repository after deduplicating, so nothing is lost.
|
||||
- **Rows are in primary-key order, with their keys sorted.** A row that did
|
||||
not change is the same bytes in the same place every day. An edited row
|
||||
changes one line, and new rows land at the end of their table's file.
|
||||
- **Each table is its own file**, so a busy table never shifts the bytes of a
|
||||
quiet one.
|
||||
- **The parts that change every run are kept in `manifest.json`**: the time
|
||||
and the counts.
|
||||
- **Each run gets a new, dated name.** Overwriting one name would keep one
|
||||
copy and no history.
|
||||
|
||||
Day to day, the off-site copy then stores roughly what changed, not the whole
|
||||
file again.
|
||||
|
||||
## Destinations
|
||||
|
||||
### A folder
|
||||
|
||||
A path **inside the Scribe container**, normally a share mounted there:
|
||||
|
||||
```yaml
|
||||
services:
|
||||
scribe:
|
||||
volumes:
|
||||
- /nfs/backup/scribe:/backups
|
||||
```
|
||||
|
||||
Scribe writes each file under a hidden name (`.scribe-….tar.part`) and renames
|
||||
it once it is complete. A program watching the folder never sees half a
|
||||
backup.
|
||||
|
||||
### An S3-compatible bucket
|
||||
|
||||
You need four things:
|
||||
|
||||
- **Address:** the service itself, like `https://backup.example.com`. Don't
|
||||
include the bucket; Scribe addresses buckets by path.
|
||||
- **Bucket.**
|
||||
- **Access key.**
|
||||
- **Secret key.**
|
||||
|
||||
Scribe uploads and lists; it never deletes. A key that may **only add
|
||||
objects** is therefore enough, and it is the safer key to hand out: a stolen
|
||||
one cannot wipe the history.
|
||||
|
||||
The secret is stored encrypted, under a key derived from `SECRET_KEY`. It is
|
||||
never shown again, and never included in a backup. If an instance's
|
||||
`SECRET_KEY` changes, for example after a restore onto a new instance, the
|
||||
page asks for the secret again.
|
||||
|
||||
**Test destination** checks the destination without sending a backup:
|
||||
|
||||
- a folder: it writes and removes a hidden file;
|
||||
- a bucket: it only lists, because an add-only key could never remove a test
|
||||
file.
|
||||
|
||||
## Schedule and history
|
||||
|
||||
- Backups run once a day at the hour you choose, in UTC. The default is 02:00,
|
||||
before the nightly trash purge and database maintenance.
|
||||
- **Back up now** runs one straight away.
|
||||
- The page leads with when the last good backup was taken. It says
|
||||
**Overdue** when backups are on and none has worked for 26 hours.
|
||||
- It lists the last 30 runs, and gives the reason for each failure in words.
|
||||
- A run that hasn't finished after an hour is abandoned and recorded as
|
||||
failed.
|
||||
|
||||
## Example: Fabled Archives
|
||||
|
||||
[Fabled Archives](https://git.fabledsword.com/bvandeusen/fabledarchives)
|
||||
keeps service backups as "backup targets". Either destination works with it.
|
||||
|
||||
1. On Archives' **Backups** page, make a target for Scribe, named `scribe`
|
||||
for example.
|
||||
2. Choose how Scribe's files arrive:
|
||||
- **S3**: turn on the S3 way in for the target and make an **append-only**
|
||||
S3 key. In Scribe, choose **An S3 bucket**:
|
||||
- address: Archives' backup address (`http://<archives>:8081`);
|
||||
- bucket: the target's name;
|
||||
- access key and secret key: the key you just made.
|
||||
- **A folder**: mount the share Archives watches into Scribe as
|
||||
`/backups/scribe`, and point the target at that folder. In Scribe, choose
|
||||
**A folder** with `/backups/scribe`.
|
||||
3. Set the target's **Retention** to **Archives**. Archives then keeps the
|
||||
newest backups and thins the rest by day, week and month. Scribe only adds.
|
||||
4. Give the target an expected cadence of **daily**, so a missed backup turns
|
||||
it late in Archives as well as overdue in Scribe.
|
||||
5. Add an off-site **restic** replication job for the target. The uncompressed
|
||||
layout above is what lets that job send only what changed each day.
|
||||
|
||||
## Restoring
|
||||
|
||||
### Into a fresh instance
|
||||
|
||||
Start Scribe on an empty database, copy the backup file into the container,
|
||||
and run:
|
||||
|
||||
```sh
|
||||
docker exec -it scribe python -m scribe.restore /backups/scribe-20261015T020000Z.tar
|
||||
```
|
||||
|
||||
- The command refuses a database that already has an account. A restore
|
||||
re-creates everything with new ids, so on a live instance it would add a
|
||||
second copy instead of replacing anything.
|
||||
- Nothing limits the file's size.
|
||||
- It also reads the JSON from the admin **Full Backup** download.
|
||||
|
||||
Afterwards, **restart Scribe** so the search indexes are rebuilt. Then issue
|
||||
new API keys, add forge connections again, and set up any sharing.
|
||||
|
||||
### From the web
|
||||
|
||||
**Settings → Data → Restore from Backup** takes the `.tar` as well as the
|
||||
JSON download. The upload is held to the server's request-size limit. For a
|
||||
bigger file, use the command above.
|
||||
Reference in New Issue
Block a user