FabledSteward/docs/core/backups.md

# Backups: Steward backing itself up, and coming back

Steward sends a backup of itself to an S3 bucket once a day. Fabled Archives
is the intended target. A backup is one file. Settings → Backups restores it
on a running Steward, first-run setup restores it on a new one, and
`steward restore` brings Steward back from it when Steward can't start.

## What a backup holds

`steward-<UTC time>.tar`, containing:

| Member | What it is |
|---|---|
| `steward.dump` | `pg_dump --format=custom` of Steward's database |
| `secret.key` | The key Steward's stored passwords and tokens are sealed with |
| `manifest.json` | Format, Steward version, migration heads, time, and the tables whose rows were left out |

A backup is for **system recovery**: it holds what you would otherwise have
to set up again, and no logs or history.

**Kept:**
- settings, with the passwords and keys they hold;
- users and API tokens;
- hosts and their agent enrolments, so agents don't need enrolling again;
- monitors, alert rules, maintenance windows and maintenance cadences;
- Ansible inventory and schedules;
- dashboards and share links;
- SNMP devices, Prometheus targets and widget templates, UniFi client names.

**Left out** (the tables come back empty and fill again as Steward runs):
- the audit log;
- alert history and alert state. Each rule starts inactive and is watched
  again within a minute;
- metric anomalies and their reviews;
- metrics, their rollups and learned normals;
- monitor check results;
- Docker, Traefik and UniFi history;
- Ansible run records and maintenance job history;
- the email queue, pending invites and password-reset links.

The list is `steward/backup/contents.py`. A test fails when a new table
appears in neither list, so whoever adds a table decides.

**The secret key is in the file.** That makes a restore complete on its own.
It also means anyone holding a copy can read every password and key Steward
keeps. Keep the bucket private.

## Setting it up with Fabled Archives

1. In Archives, make a backup target for Steward with an S3 key. The default
   append-only key is enough: Steward only uploads and never deletes.
   How long backups are kept is set on the Archives side.
2. In Steward, open **Settings → Backups** and fill in:
   - **Endpoint**: the S3 address Archives shows for the target, without the
     bucket (for example `http://archives:8081`). Requests are path-style:
     `endpoint/bucket/file`.
   - **Bucket**: the target's bucket name.
   - **Access key** and **Secret key**: from Archives. The secret is sealed at
     rest; a blank box on save keeps the stored one.
3. Press **Test the bucket**. A key that may not list ("AccessDenied") still
   counts as reached: an append-only key can't list, and uploads work anyway.
4. Tick **Back up every day**, pick the hour (on your clock), and save.
   **Back up now** makes one straight away.

A day missed while Steward was down is made up when it comes back. A failed
backup is tried again an hour later, never sooner. The last 30 runs are
listed on the page, and each one is in the audit log (`backup.made` /
`backup.failed`).

### Alert when backups stop

While backups are on, Steward records two metrics under the source
**Steward itself**, resource `backup`:

| Metric | Meaning |
|---|---|
| `backup_age_secs` ("Newest backup's age") | Seconds since the last backup was sent, updated every 15 minutes |
| `backup_bytes` ("Backup size") | The size of the last backup sent |

A rule such as `backup_age_secs > 93600` (26 hours) fires when a daily backup
has been missed.

An agent on the MCP sees the same without a rule. `get_overview` names a
scheduled backup that failed, is two hours past its slot with none made, or has
no bucket set up. `list_backups`, `check_backup` and `make_backup` are this
page's admin tools (see [MCP](mcp.md)); restoring isn't among them.

## Checking a backup

**Settings → Backups → Check** compares a backup with what is in place, without
changing anything. It shows what a restore would bring back, change or remove.
Each row under Recent backups has its own Check button, and **Check a file**
takes a file downloaded from the bucket. From a shell:

```sh
docker exec -u app <steward container> steward check-backup /tmp/steward-20261015T030000Z.tar
```

The check reports:
- for each area (Settings, Users and API tokens, Hosts and agents, Monitors,
  Alert rules, Ansible, Dashboards, Plugins), how many entries are the same,
  differ, are only in the backup or are only here;
- each setting that differs, with both values. A password or key shows only
  "differs": sealed values are opened with their own key and compared, never
  shown;
- whether the backup's secret key is the one in use, and whether it opens the
  backup's own sealed values. A backup whose key can't open them fails the
  check, because its secrets would not come back;
- whether the backup was made by this Steward's schema, an older one or a
  newer one.

What Steward records as it runs is left out of the comparison: when an agent
or monitor was last seen, a schedule's last run, the backup history. These
change by the minute. The list is `ACTIVITY` and `STATE_SETTINGS` in
`steward/backup/contents.py`.

Each backup Steward sends is checked once it is written, and its row under
Recent backups says so. A backup is sent even when its check can't run.

Checking a backup on the bucket reads it back from there. An append-only key
may not be allowed to; the page then says AccessDenied, and the file can be
downloaded and checked with **Check a file** instead.

## Making a backup by hand

```sh
docker exec -u app <steward container> steward backup --to /tmp/backups
```

This writes the file and sends it nowhere. It needs the same database access
as Steward itself.

## Restoring

### From the web UI

**On a running Steward: Settings → Backups.** Press **Restore** on one of
the recent backups, or choose a file on the "Check or restore a file" card.
Steward first shows what the backup would change: the same comparison as
Check. Then it says what a restore does and asks you to type `restore`.

- **A backup of this Steward's schema goes back in place.** Settings,
  users, hosts, monitors, alert rules and dashboards become the backup's.
  The audit log and the history of everything still here stay. History of
  anything the backup doesn't have goes with it. It is one transaction: if
  anything fails part-way, nothing changes, and Backups says why.
- **A backup from an older Steward replaces everything**, history included,
  and you tick "Replace everything" as well. Steward brings it up to date
  as it starts.
- **A backup from a newer Steward isn't offered.** Update Steward first.

Before anything changes, Steward makes a **safety backup** of itself as it
stands. It goes to `/data/backups` (the newest five are kept) and to the
bucket, and it is listed under Recent backups as "before restore". A
restore can be undone by restoring that.

While the restore runs:
- every page says Steward is restoring;
- agents get a 503 and keep their readings until Steward is back;
- scheduled work holds.

Then Steward restarts (it exits with code 3 so Swarm, compose or systemd
starts it again), and you sign in as a user from the backup. If an older
backup's full replace fails after the database was emptied, Steward starts
at first-run setup, where the safety backup can be uploaded.

**On a new Steward: first-run setup.** Next to "Create the initial admin
account" is **Or restore from a backup**: upload a backup file instead,
and Steward comes up with its users, settings and key. There is nothing to
keep yet, so no safety backup is made.

The backup's secret key is put in place as `/data/secret.key`. If
`STEWARD_SECRET_KEY` is set instead, it must equal the backup's key, or the
restore stops before anything changes.

### From the command line

`steward restore` writes into an **empty** database. Pass `--wipe` to drop
everything in the database's public schema first. Use it when Steward
can't start at all.

`steward restore --in-place` does what Settings → Backups does for a backup
of the same schema, from the shell: it replaces what the backup carries and
keeps the audit log and history, in one transaction. Restart Steward after
it so the restored settings load.

1. Stop Steward (scale its service to 0) so nothing writes during the restore.
2. Download the backup file from Archives.
3. Run the restore with the image, pointed at the database:

   ```sh
   docker run --rm \
     -e STEWARD_DATABASE_URL=postgresql+asyncpg://steward:…@db:5432/steward \
     -v steward_data:/data -v "$PWD":/restore \
     git.fabledsword.com/bvandeusen/steward:latest \
     steward restore /restore/steward-20261012T030000Z.tar
   ```

   - The backup's secret key is written to `/data/secret.key`.
   - If `STEWARD_SECRET_KEY` is set instead, it must equal the backup's key,
     or the restore stops and says so. Read the key with
     `tar -xOf <file> secret.key`.
4. Start Steward. On start it runs any migrations the backup is behind on.

**How the restore works.** pg_restore runs in its three sections: schema,
then data, then constraints and indexes. Between the data and the
constraints, a kept row that points at a left-out row is put right by its
foreign key's ON DELETE:
- a SET NULL link is cleared and the row kept;
- a row whose foreign key would cascade or block is removed.

pg_restore 17 writes each section as a script for psql. It runs against
Postgres 16 with one setting line dropped (`transaction_timeout`, which 16
doesn't know).

CI checks the whole path on every build. The built image backs up its own
database and checks the backup against it. It changes a setting and puts
the backup back in place: the setting returns and the audit log stays.
Then it restores into a second, empty database, checks a setting came back
and the metrics table and audit log are empty, and starts Steward on it.