FabledGrimoire/docs/backups.md

# Backups

Grimoire can back itself up every day to a **folder** or an **S3-compatible
bucket**. Each run adds one file; Grimoire never deletes an old one. Keeping and
thinning the history is the destination's job.

Set it up under **Settings → Data → Scheduled backups** (admins only).

## What a backup is

One file per run, named for the moment it was taken in UTC:

```
grimoire-20261015T020000Z.tar
```

Backups written while the app was called Scribe are named `scribe-…tar`.
They are counted with the rest and restore the same way.

It is a plain, **uncompressed** tar:

- `manifest.json` comes first. It holds the backup format version, the Grimoire
  version, when the backup was taken, and how many rows each table has.
- Then there is one `<table>.jsonl` per table, one row per line.

It holds everything the admin **Full Backup** download holds, laid out
differently. It does **not** hold:

- **Search indexes.** They are rebuilt from the records when Grimoire starts.
- **Sharing:** groups, project shares and note shares.
- **Credentials:** API keys and forge connections. After a restore, issue new
  API keys and add forge connections again. (Forge tokens are also sealed under
  `SECRET_KEY` in the database itself, so a copy of the database doesn't carry
  them in readable form either. The key lives in `/data/secret_key`, outside
  the database.)
- **Logs and notifications.**
- **The backup destination's own S3 secret.**

The file contains hashed passwords, so store it somewhere only trusted people
can read.

### Why it is shaped this way

The file is meant to be copied off-site by a deduplicating tool such as
restic, which stores a chunk of data only once no matter how many backups
contain it. Every choice serves that:

- **It is never compressed.** Compression changes every byte after the first
  edit, so nothing would match the day before. restic compresses its own
  repository after deduplicating, so nothing is lost.
- **Rows are in primary-key order, with their keys sorted.** A row that did
  not change is the same bytes in the same place every day. An edited row
  changes one line, and new rows land at the end of their table's file.
- **Each table is its own file**, so a busy table never shifts the bytes of a
  quiet one.
- **The parts that change every run are kept in `manifest.json`**: the time
  and the counts.
- **Each run gets a new, dated name.** Overwriting one name would keep one
  copy and no history.

Day to day, the off-site copy then stores roughly what changed, not the whole
file again.

## Destinations

### A folder

A path **inside the Grimoire container**, normally a share mounted there:

```yaml
services:
  grimoire:
    volumes:
      - /nfs/backup/grimoire:/backups
```

Grimoire writes each file under a hidden name (`.grimoire-….tar.part`) and renames
it once it is complete. A program watching the folder never sees half a
backup.

### An S3-compatible bucket

You need four things:

- **Address:** the service itself, like `https://backup.example.com`. Don't
  include the bucket; Grimoire addresses buckets by path.
- **Bucket.**
- **Access key.**
- **Secret key.**

Grimoire uploads and lists, and reads a backup back when you restore from the
bucket; it never deletes. A key that may **add and read objects but not
delete them** is therefore enough, and it is the safer key to hand out: a
stolen one cannot wipe the history. A key that may only add objects still
takes backups; restore then works by uploading a file or from the command
line.

The secret is stored encrypted, under a key derived from `SECRET_KEY`. It is
never shown again, and never included in a backup. If an instance's
`SECRET_KEY` changes, for example after a restore onto a new instance, the
page asks for the secret again.

**Test destination** checks the destination without sending a backup:

- a folder: it writes and removes a hidden file;
- a bucket: it only lists, because an add-only key could never remove a test
  file.

## Schedule and history

- Backups run once a day at the hour you choose, in UTC. The default is 02:00,
  before the nightly trash purge and database maintenance.
- **Back up now** runs one straight away.
- The page leads with when the last good backup was taken. It says
  **Overdue** when backups are on and none has worked for 26 hours.
- It lists the last 30 runs, and gives the reason for each failure in words.
- A run that hasn't finished after an hour is abandoned and recorded as
  failed.

## Example: Fabled Archives

[Fabled Archives](https://git.fabledsword.com/bvandeusen/fabledarchives)
keeps service backups as "backup targets". Either destination works with it.

1. On Archives' **Backups** page, make a target for Grimoire, named `grimoire`
   for example.
2. Choose how Grimoire's files arrive:
   - **S3**: turn on the S3 way in for the target and make an S3 key that
     may add and read objects but not delete them (an **append-only** key
     takes backups too, but then Grimoire can't restore from the bucket). In
     Grimoire, choose **An S3 bucket**:
     - address: Archives' backup address (`http://<archives>:8081`);
     - bucket: the target's name;
     - access key and secret key: the key you just made.
   - **A folder**: mount the share Archives watches into Grimoire as
     `/backups/grimoire`, and point the target at that folder. In Grimoire, choose
     **A folder** with `/backups/grimoire`.
3. Set the target's **Retention** to **Archives**. Archives then keeps the
   newest backups and thins the rest by day, week and month. Grimoire only adds.
4. Give the target an expected cadence of **daily**, so a missed backup turns
   it late in Archives as well as overdue in Grimoire.
5. Add an off-site **restic** replication job for the target. The uncompressed
   layout above is what lets that job send only what changed each day.

## Restoring

A restore puts the **whole instance** back to the moment a backup was taken,
for every user:

- records edited since go back to how they were;
- records deleted since come back;
- anything made since is removed.

It matches records by their ids, so a live instance only restores a backup **of
its own**. Each instance has an id, and every backup records the one it came
from. A backup from another instance, or one made before backups recorded it,
only goes into a new, empty instance, which takes it as it is.

Before changing anything, the restore saves the current state as one more
backup. It goes to the backup destination when scheduled backups are on, and
otherwise to `/data/pre-restore`. The result says which file it is, so a
restore can itself be undone.

What a backup doesn't carry stays as it is wherever its owner survives the
restore: API keys, forge connections, sharing, logs, and the destination's S3
secret. The trash isn't in backups, so anything in the trash when the backup
was taken is gone afterwards. Everyone is signed out, and search is rebuilt.

### From the web

**Settings → Data → Restore from Backup** lists the backups at the backup
destination. You can pick one of those, or upload a `.tar` or the JSON
**Full Backup** download (uploads are held to the server's request-size
limit). The restore then shows what would change: when the backup was taken,
and how many accounts, projects, plans, notes and rules there are now and will
be afterwards. Type `roll back` to run it. Then sign in again.

To read backups back from an S3 bucket, the key needs read access as well as
write. Grimoire never deletes from the bucket.

### From the command line

For a file with no size limit, or a new instance with no backup destination
yet, copy the file into the container and run:

```sh
docker exec -it grimoire python -m grimoire.restore /backups/grimoire-20261015T020000Z.tar
```

It prints what would change and stops. Add `--yes` to restore. It reads the
JSON download too. Afterwards, **restart Grimoire** so the search indexes are
rebuilt.

After restoring into a new instance, issue new API keys, add forge connections
again, and set up any sharing: none of them travel in a backup.