FabledSteward/docs/core/backups.md

# Backups: Steward backing itself up, and coming back

Steward sends a backup of itself to an S3 bucket once a day. Fabled Archives
is the intended target. A backup is one file, and `steward restore` brings
Steward back from it into an empty database.

## What a backup holds

`steward-<UTC time>.tar`, containing:

| Member | What it is |
|---|---|
| `steward.dump` | `pg_dump --format=custom` of Steward's database |
| `secret.key` | The key Steward's stored passwords and tokens are sealed with |
| `manifest.json` | Format, Steward version, migration heads, time, and the tables whose rows were left out |

**Kept:**
- configuration: settings, users, API tokens, hosts, monitors, alert rules, maintenance windows, Ansible inventory and schedules, dashboards, SNMP devices, Prometheus targets, UniFi client names;
- the audit log;
- alert history.

**Left out** (the tables come back empty and fill again as Steward runs):
- metrics and their rollups;
- monitor check results;
- Docker, Traefik and UniFi history;
- Ansible run records;
- the email queue;
- password-reset links.

The list is `steward/backup/contents.py`. A test fails when a new table
appears in neither list, so whoever adds a table decides.

**The secret key is in the file.** That makes a restore complete on its own.
It also means anyone holding a copy can read every password and key Steward
keeps. Keep the bucket private.

## Setting it up with Fabled Archives

1. In Archives, make a backup target for Steward with an S3 key. The default
   append-only key is enough: Steward only uploads and never deletes.
   How long backups are kept is set on the Archives side.
2. In Steward, open **Settings → Backups** and fill in:
   - **Endpoint**: the S3 address Archives shows for the target, without the
     bucket (for example `http://archives:8081`). Requests are path-style:
     `endpoint/bucket/file`.
   - **Bucket**: the target's bucket name.
   - **Access key** and **Secret key**: from Archives. The secret is sealed at
     rest; a blank box on save keeps the stored one.
3. Press **Test the bucket**. A key that may not list ("AccessDenied") still
   counts as reached: an append-only key can't list, and uploads work anyway.
4. Tick **Back up every day**, pick the hour (on your clock), and save.
   **Back up now** makes one straight away.

A day missed while Steward was down is made up when it comes back. A failed
backup is tried again an hour later, never sooner. The last 30 runs are
listed on the page, and each one is in the audit log (`backup.made` /
`backup.failed`).

### Alert when backups stop

While backups are on, Steward records two metrics under the source
**Steward itself**, resource `backup`:

| Metric | Meaning |
|---|---|
| `backup_age_secs` ("Newest backup's age") | Seconds since the last backup was sent, updated every 15 minutes |
| `backup_bytes` ("Backup size") | The size of the last backup sent |

A rule such as `backup_age_secs > 93600` (26 hours) fires when a daily backup
has been missed.

## Making a backup by hand

```sh
docker exec -u app <steward container> steward backup --to /tmp/backups
```

This writes the file and sends it nowhere. It needs the same database access
as Steward itself.

## Restoring

`steward restore` writes into an **empty** database. Pass `--wipe` to drop
everything in the database's public schema first.

1. Stop Steward (scale its service to 0) so nothing writes during the restore.
2. Download the backup file from Archives.
3. Run the restore with the image, pointed at the database:

   ```sh
   docker run --rm \
     -e STEWARD_DATABASE_URL=postgresql+asyncpg://steward:…@db:5432/steward \
     -v steward_data:/data -v "$PWD":/restore \
     git.fabledsword.com/bvandeusen/steward:latest \
     steward restore /restore/steward-20261012T030000Z.tar
   ```

   - The backup's secret key is written to `/data/secret.key`.
   - If `STEWARD_SECRET_KEY` is set instead, it must equal the backup's key,
     or the restore stops and says so. Read the key with
     `tar -xOf <file> secret.key`.
4. Start Steward. On start it runs any migrations the backup is behind on.

**How the restore works.** pg_restore runs in its three sections: schema,
then data, then constraints and indexes. Between the data and the
constraints, a kept row that points at a left-out row is put right by its
foreign key's ON DELETE:
- an alert event's link to an Ansible run is cleared;
- a row whose foreign key would cascade or block is removed.

pg_restore 17 writes each section as a script for psql. It runs against
Postgres 16 with one setting line dropped (`transaction_timeout`, which 16
doesn't know).

CI checks the whole path on every build. The built image backs up its own
database, restores into a second one, checks a setting came back and the
metrics table is empty, then starts Steward on it.