bvandeusenandClaude Opus 5.5
e6da0ff299
#5822 "Backup contents: settings and keys only". The operator's ruling (System 180, 2026-10-15) overturns 2026-10-12's "Config + audit + alerts": a backup is for system recovery and carries what would otherwise have to be set up again, with no logs or history. - contents.py: audit_events, alert_events, alert_states, metric_anomalies, metric_anomaly_spans and user_invites move to EXCLUDED. - Alert state is no longer restored, and evaluation skips a rule with no state row. ensure_rule_states (alerts/service.py) gives every single-resource rule lacking one an inactive row; alert_reconcile runs it each minute. - Docs, the Backups page and the CLI help say what a backup holds now. CI's image check also finds the audit log empty after a restore. - tests/integration/test_backup_restore_fixes.py is removed: its case (an alert event naming a left-out Ansible run) no longer arises. The SET NULL fix-up is covered by a unit test on synthetic tables. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
128 lines
5.2 KiB
Markdown
128 lines
5.2 KiB
Markdown
# Backups: Steward backing itself up, and coming back
|
|
|
|
Steward sends a backup of itself to an S3 bucket once a day. Fabled Archives
|
|
is the intended target. A backup is one file, and `steward restore` brings
|
|
Steward back from it into an empty database.
|
|
|
|
## What a backup holds
|
|
|
|
`steward-<UTC time>.tar`, containing:
|
|
|
|
| Member | What it is |
|
|
|---|---|
|
|
| `steward.dump` | `pg_dump --format=custom` of Steward's database |
|
|
| `secret.key` | The key Steward's stored passwords and tokens are sealed with |
|
|
| `manifest.json` | Format, Steward version, migration heads, time, and the tables whose rows were left out |
|
|
|
|
A backup is for **system recovery**: it holds what you would otherwise have
|
|
to set up again, and no logs or history.
|
|
|
|
**Kept:**
|
|
- settings, with the passwords and keys they hold;
|
|
- users and API tokens;
|
|
- hosts and their agent enrolments, so agents don't need enrolling again;
|
|
- monitors, alert rules, maintenance windows and maintenance cadences;
|
|
- Ansible inventory and schedules;
|
|
- dashboards and share links;
|
|
- SNMP devices, Prometheus targets and widget templates, UniFi client names.
|
|
|
|
**Left out** (the tables come back empty and fill again as Steward runs):
|
|
- the audit log;
|
|
- alert history and alert state. Each rule starts inactive and is watched
|
|
again within a minute;
|
|
- metric anomalies and their reviews;
|
|
- metrics, their rollups and learned normals;
|
|
- monitor check results;
|
|
- Docker, Traefik and UniFi history;
|
|
- Ansible run records and maintenance job history;
|
|
- the email queue, pending invites and password-reset links.
|
|
|
|
The list is `steward/backup/contents.py`. A test fails when a new table
|
|
appears in neither list, so whoever adds a table decides.
|
|
|
|
**The secret key is in the file.** That makes a restore complete on its own.
|
|
It also means anyone holding a copy can read every password and key Steward
|
|
keeps. Keep the bucket private.
|
|
|
|
## Setting it up with Fabled Archives
|
|
|
|
1. In Archives, make a backup target for Steward with an S3 key. The default
|
|
append-only key is enough: Steward only uploads and never deletes.
|
|
How long backups are kept is set on the Archives side.
|
|
2. In Steward, open **Settings → Backups** and fill in:
|
|
- **Endpoint**: the S3 address Archives shows for the target, without the
|
|
bucket (for example `http://archives:8081`). Requests are path-style:
|
|
`endpoint/bucket/file`.
|
|
- **Bucket**: the target's bucket name.
|
|
- **Access key** and **Secret key**: from Archives. The secret is sealed at
|
|
rest; a blank box on save keeps the stored one.
|
|
3. Press **Test the bucket**. A key that may not list ("AccessDenied") still
|
|
counts as reached: an append-only key can't list, and uploads work anyway.
|
|
4. Tick **Back up every day**, pick the hour (on your clock), and save.
|
|
**Back up now** makes one straight away.
|
|
|
|
A day missed while Steward was down is made up when it comes back. A failed
|
|
backup is tried again an hour later, never sooner. The last 30 runs are
|
|
listed on the page, and each one is in the audit log (`backup.made` /
|
|
`backup.failed`).
|
|
|
|
### Alert when backups stop
|
|
|
|
While backups are on, Steward records two metrics under the source
|
|
**Steward itself**, resource `backup`:
|
|
|
|
| Metric | Meaning |
|
|
|---|---|
|
|
| `backup_age_secs` ("Newest backup's age") | Seconds since the last backup was sent, updated every 15 minutes |
|
|
| `backup_bytes` ("Backup size") | The size of the last backup sent |
|
|
|
|
A rule such as `backup_age_secs > 93600` (26 hours) fires when a daily backup
|
|
has been missed.
|
|
|
|
## Making a backup by hand
|
|
|
|
```sh
|
|
docker exec -u app <steward container> steward backup --to /tmp/backups
|
|
```
|
|
|
|
This writes the file and sends it nowhere. It needs the same database access
|
|
as Steward itself.
|
|
|
|
## Restoring
|
|
|
|
`steward restore` writes into an **empty** database. Pass `--wipe` to drop
|
|
everything in the database's public schema first.
|
|
|
|
1. Stop Steward (scale its service to 0) so nothing writes during the restore.
|
|
2. Download the backup file from Archives.
|
|
3. Run the restore with the image, pointed at the database:
|
|
|
|
```sh
|
|
docker run --rm \
|
|
-e STEWARD_DATABASE_URL=postgresql+asyncpg://steward:…@db:5432/steward \
|
|
-v steward_data:/data -v "$PWD":/restore \
|
|
git.fabledsword.com/bvandeusen/steward:latest \
|
|
steward restore /restore/steward-20261012T030000Z.tar
|
|
```
|
|
|
|
- The backup's secret key is written to `/data/secret.key`.
|
|
- If `STEWARD_SECRET_KEY` is set instead, it must equal the backup's key,
|
|
or the restore stops and says so. Read the key with
|
|
`tar -xOf <file> secret.key`.
|
|
4. Start Steward. On start it runs any migrations the backup is behind on.
|
|
|
|
**How the restore works.** pg_restore runs in its three sections: schema,
|
|
then data, then constraints and indexes. Between the data and the
|
|
constraints, a kept row that points at a left-out row is put right by its
|
|
foreign key's ON DELETE:
|
|
- a SET NULL link is cleared and the row kept;
|
|
- a row whose foreign key would cascade or block is removed.
|
|
|
|
pg_restore 17 writes each section as a script for psql. It runs against
|
|
Postgres 16 with one setting line dropped (`transaction_timeout`, which 16
|
|
doesn't know).
|
|
|
|
CI checks the whole path on every build. The built image backs up its own
|
|
database, restores into a second one, checks a setting came back and the
|
|
metrics table is empty, then starts Steward on it.
|