FabledGrimoire/docs/backups.md

# Backups

Grimoire can back itself up every day to a **folder** or an **S3-compatible
bucket**. Each run adds one file; Grimoire never deletes an old one. Keeping and
thinning the history is the destination's job.

Set it up under **Settings → Data → Scheduled backups** (admins only).

## What a backup is

One file per run, named for the moment it was taken in UTC:

```
grimoire-20261015T020000Z.tar
```

Backups written while the app was called Scribe are named `scribe-…tar`.
They are counted with the rest and restore the same way.

It is a plain, **uncompressed** tar:

- `manifest.json` comes first. It holds the backup format version, the Grimoire
  version, when the backup was taken, and how many rows each table has.
- Then there is one `<table>.jsonl` per table, one row per line.

It holds everything the admin **Full Backup** download holds, laid out
differently. It does **not** hold:

- **Search indexes.** They are rebuilt from the records when Grimoire starts.
- **Sharing:** groups, project shares and note shares.
- **Credentials:** API keys and forge connections. After a restore, issue new
  API keys and add forge connections again. (Forge tokens are also sealed under
  `SECRET_KEY` in the database itself, so a copy of the database doesn't carry
  them in readable form either. The key lives in `/data/secret_key`, outside
  the database.)
- **Logs and notifications.**
- **The backup destination's own S3 secret.**

The file contains hashed passwords, so store it somewhere only trusted people
can read.

### Why it is shaped this way

The file is meant to be copied off-site by a deduplicating tool such as
restic, which stores a chunk of data only once no matter how many backups
contain it. Every choice serves that:

- **It is never compressed.** Compression changes every byte after the first
  edit, so nothing would match the day before. restic compresses its own
  repository after deduplicating, so nothing is lost.
- **Rows are in primary-key order, with their keys sorted.** A row that did
  not change is the same bytes in the same place every day. An edited row
  changes one line, and new rows land at the end of their table's file.
- **Each table is its own file**, so a busy table never shifts the bytes of a
  quiet one.
- **The parts that change every run are kept in `manifest.json`**: the time
  and the counts.
- **Each run gets a new, dated name.** Overwriting one name would keep one
  copy and no history.

Day to day, the off-site copy then stores roughly what changed, not the whole
file again.

## Destinations

### A folder

A path **inside the Grimoire container**, normally a share mounted there:

```yaml
services:
  grimoire:
    volumes:
      - /nfs/backup/grimoire:/backups
```

Grimoire writes each file under a hidden name (`.grimoire-….tar.part`) and renames
it once it is complete. A program watching the folder never sees half a
backup.

### An S3-compatible bucket

You need four things:

- **Address:** the service itself, like `https://backup.example.com`. Don't
  include the bucket; Grimoire addresses buckets by path.
- **Bucket.**
- **Access key.**
- **Secret key.**

Grimoire uploads and lists; it never deletes. A key that may **only add
objects** is therefore enough, and it is the safer key to hand out: a stolen
one cannot wipe the history.

The secret is stored encrypted, under a key derived from `SECRET_KEY`. It is
never shown again, and never included in a backup. If an instance's
`SECRET_KEY` changes, for example after a restore onto a new instance, the
page asks for the secret again.

**Test destination** checks the destination without sending a backup:

- a folder: it writes and removes a hidden file;
- a bucket: it only lists, because an add-only key could never remove a test
  file.

## Schedule and history

- Backups run once a day at the hour you choose, in UTC. The default is 02:00,
  before the nightly trash purge and database maintenance.
- **Back up now** runs one straight away.
- The page leads with when the last good backup was taken. It says
  **Overdue** when backups are on and none has worked for 26 hours.
- It lists the last 30 runs, and gives the reason for each failure in words.
- A run that hasn't finished after an hour is abandoned and recorded as
  failed.

## Example: Fabled Archives

[Fabled Archives](https://git.fabledsword.com/bvandeusen/fabledarchives)
keeps service backups as "backup targets". Either destination works with it.

1. On Archives' **Backups** page, make a target for Grimoire, named `grimoire`
   for example.
2. Choose how Grimoire's files arrive:
   - **S3**: turn on the S3 way in for the target and make an **append-only**
     S3 key. In Grimoire, choose **An S3 bucket**:
     - address: Archives' backup address (`http://<archives>:8081`);
     - bucket: the target's name;
     - access key and secret key: the key you just made.
   - **A folder**: mount the share Archives watches into Grimoire as
     `/backups/grimoire`, and point the target at that folder. In Grimoire, choose
     **A folder** with `/backups/grimoire`.
3. Set the target's **Retention** to **Archives**. Archives then keeps the
   newest backups and thins the rest by day, week and month. Grimoire only adds.
4. Give the target an expected cadence of **daily**, so a missed backup turns
   it late in Archives as well as overdue in Grimoire.
5. Add an off-site **restic** replication job for the target. The uncompressed
   layout above is what lets that job send only what changed each day.

## Restoring

### Into a fresh instance

Start Grimoire on an empty database, copy the backup file into the container,
and run:

```sh
docker exec -it grimoire python -m grimoire.restore /backups/grimoire-20261015T020000Z.tar
```

- The command refuses a database that already has an account. A restore
  re-creates everything with new ids, so on a live instance it would add a
  second copy instead of replacing anything.
- Nothing limits the file's size.
- It also reads the JSON from the admin **Full Backup** download.

Afterwards, **restart Grimoire** so the search indexes are rebuilt. Then issue
new API keys, add forge connections again, and set up any sharing.

### From the web

**Settings → Data → Restore from Backup** takes the `.tar` as well as the
JSON download. The upload is held to the server's request-size limit. For a
bigger file, use the command above.