FabledGrimoire/docs/backups.md

Backups

Grimoire can back itself up every day to a folder or an S3-compatible bucket. Each run adds one file; Grimoire never deletes an old one. Keeping and thinning the history is the destination's job.

Set it up under Settings → Data → Scheduled backups (admins only).

What a backup is

One file per run, named for the moment it was taken in UTC:

scribe-20261015T020000Z.tar

It is a plain, uncompressed tar:

• manifest.json comes first. It holds the backup format version, the Grimoire version, when the backup was taken, and how many rows each table has.
• Then there is one <table>.jsonl per table, one row per line.

It holds everything the admin Full Backup download holds, laid out differently. It does not hold:

• Search indexes. They are rebuilt from the records when Grimoire starts.
• Sharing: groups, project shares and note shares.
• Credentials: API keys and forge connections. After a restore, issue new API keys and add forge connections again. (Forge tokens are also sealed under SECRET_KEY in the database itself, so a copy of the database doesn't carry them in readable form either.)
• Logs and notifications.
• The backup destination's own S3 secret.

The file contains hashed passwords, so store it somewhere only trusted people can read.

Why it is shaped this way

The file is meant to be copied off-site by a deduplicating tool such as restic, which stores a chunk of data only once no matter how many backups contain it. Every choice serves that:

• It is never compressed. Compression changes every byte after the first edit, so nothing would match the day before. restic compresses its own repository after deduplicating, so nothing is lost.
• Rows are in primary-key order, with their keys sorted. A row that did not change is the same bytes in the same place every day. An edited row changes one line, and new rows land at the end of their table's file.
• Each table is its own file, so a busy table never shifts the bytes of a quiet one.
• The parts that change every run are kept in manifest.json: the time and the counts.
• Each run gets a new, dated name. Overwriting one name would keep one copy and no history.

Day to day, the off-site copy then stores roughly what changed, not the whole file again.

Destinations

A folder

A path inside the Grimoire container, normally a share mounted there:

services:
  scribe:
    volumes:
      - /nfs/backup/scribe:/backups

Grimoire writes each file under a hidden name (.scribe-….tar.part) and renames it once it is complete. A program watching the folder never sees half a backup.

An S3-compatible bucket

You need four things:

• Address: the service itself, like https://backup.example.com. Don't include the bucket; Grimoire addresses buckets by path.
• Bucket.
• Access key.
• Secret key.

Grimoire uploads and lists; it never deletes. A key that may only add objects is therefore enough, and it is the safer key to hand out: a stolen one cannot wipe the history.

The secret is stored encrypted, under a key derived from SECRET_KEY. It is never shown again, and never included in a backup. If an instance's SECRET_KEY changes, for example after a restore onto a new instance, the page asks for the secret again.

Test destination checks the destination without sending a backup:

• a folder: it writes and removes a hidden file;
• a bucket: it only lists, because an add-only key could never remove a test file.

Schedule and history

• Backups run once a day at the hour you choose, in UTC. The default is 02:00, before the nightly trash purge and database maintenance.
• Back up now runs one straight away.
• The page leads with when the last good backup was taken. It says Overdue when backups are on and none has worked for 26 hours.
• It lists the last 30 runs, and gives the reason for each failure in words.
• A run that hasn't finished after an hour is abandoned and recorded as failed.

Example: Fabled Archives

Fabled Archives keeps service backups as "backup targets". Either destination works with it.

1. On Archives' Backups page, make a target for Grimoire, named scribe for example.
2. Choose how Grimoire's files arrive:
   • S3: turn on the S3 way in for the target and make an append-only S3 key. In Grimoire, choose An S3 bucket:
     • address: Archives' backup address (http://<archives>:8081);
     • bucket: the target's name;
     • access key and secret key: the key you just made.
   • A folder: mount the share Archives watches into Grimoire as /backups/scribe, and point the target at that folder. In Grimoire, choose A folder with /backups/scribe.
3. Set the target's Retention to Archives. Archives then keeps the newest backups and thins the rest by day, week and month. Grimoire only adds.
4. Give the target an expected cadence of daily, so a missed backup turns it late in Archives as well as overdue in Grimoire.
5. Add an off-site restic replication job for the target. The uncompressed layout above is what lets that job send only what changed each day.

Restoring

Into a fresh instance

Start Grimoire on an empty database, copy the backup file into the container, and run:

docker exec -it scribe python -m scribe.restore /backups/scribe-20261015T020000Z.tar

• The command refuses a database that already has an account. A restore re-creates everything with new ids, so on a live instance it would add a second copy instead of replacing anything.
• Nothing limits the file's size.
• It also reads the JSON from the admin Full Backup download.

Afterwards, restart Grimoire so the search indexes are rebuilt. Then issue new API keys, add forge connections again, and set up any sharing.

From the web

Settings → Data → Restore from Backup takes the .tar as well as the JSON download. The upload is held to the server's request-size limit. For a bigger file, use the command above.