feat(backup): steward backup and steward restore; CI restores a dump and boots on it (#5672, #5674)
CI / lint (push) Successful in 3s
CI / integration (push) Successful in 1m14s
CI / unit (push) Failing after 1m28s
CI / publish (push) Skipped

The backup file (#5672 "Backup file: pg_dump of config + audit + alert
history, the secret key and a manifest"): steward-<UTC>.tar holding a
pg_dump --format=custom with metrics', check results' and Docker/Traefik/
UniFi rows left out, the secret key, and a manifest. Every table is named
KEPT or EXCLUDED in steward/backup/contents.py, and a guard test fails on a
new one. The image gains postgresql-client-17 (approved, System 180).

The restore (#5674 "Restore: `steward restore` and a CI test that a dump
brings Steward back"): pg_restore in its three sections. Between data and
constraints, a kept row pointing at a left-out row gets what its foreign
key's ON DELETE says (alert_events.ansible_run_id is cleared). It refuses
a non-empty database unless --wipe, and puts the backup's key in
/data/secret.key. Migrations run on Steward's next start.

`steward` stays the server command: it is now a click group that serves
when given no subcommand.

The publish smoke step backs up the booted image's database, restores it
into a second one, checks a marker setting came back and plugin_metrics
is empty, and starts Steward on it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-10-12 07:14:04 -04:00
co-authored by Claude Opus 5.5
parent eea29123f9
commit b875160c40
10 changed files with 639 additions and 4 deletions
+32 -1
View File
@@ -221,6 +221,36 @@ jobs:
exit 1
fi
# A backup made by the image restores into an empty database, and
# Steward starts on it (milestone 507). It carries configuration
# (the marker setting) and leaves metrics' rows out.
sql() { docker exec "$SMOKE-pg" psql -U steward -v ON_ERROR_STOP=1 -tA "$@"; }
sql -d steward -c "INSERT INTO app_settings (key, value_json, updated_at) VALUES ('smoke.backup', '\"kept\"', now())"
sql -d steward -c "INSERT INTO plugin_metrics (id, source_module, resource_name, metric_name, value, recorded_at) VALUES ('smoke-metric', 'smoke', 'smoke', 'smoke', 1, now())"
docker exec -u app "$SMOKE-app" steward backup --to /tmp/backups
sql -d steward -c "CREATE DATABASE restored"
docker exec -u app \
-e "STEWARD_DATABASE_URL=postgresql+asyncpg://steward:steward@$SMOKE-pg:5432/restored" \
"$SMOKE-app" sh -c 'steward restore /tmp/backups/steward-*.tar'
test "$(sql -d restored -c "SELECT value_json FROM app_settings WHERE key = 'smoke.backup'")" = '"kept"'
test "$(sql -d restored -c "SELECT count(*) FROM plugin_metrics")" = "0"
test "$(sql -d restored -c "SELECT count(*) FROM alembic_version")" -ge 1
docker run -d --name "$SMOKE-restored" --network "$SMOKE-net" \
-e "STEWARD_DATABASE_URL=postgresql+asyncpg://steward:steward@$SMOKE-pg:5432/restored" \
-e STEWARD_SECRET_KEY=smoke-test-placeholder \
"$IMAGE"
up=""
for _ in $(seq 1 60); do
if docker exec "$SMOKE-restored" python -c 'import urllib.request
urllib.request.urlopen("http://127.0.0.1:5000/health", timeout=5)' >/dev/null 2>&1; then up=1; break; fi
sleep 2
done
if [ -z "$up" ]; then
echo "FATAL: Steward never started on the restored database"
docker logs "$SMOKE-restored" 2>&1 | tail -80
exit 1
fi
# A stop is a clean shutdown: SIGTERM through tini to Hypercorn, which
# finishes within its graceful timeout and exits 0.
docker stop -t 30 "$SMOKE-app"
@@ -246,7 +276,8 @@ jobs:
- name: Smoke clean-up
if: always()
run: |
docker rm -f "steward-smoke-${{ github.run_id }}-app" "steward-smoke-${{ github.run_id }}-pg" >/dev/null 2>&1 || true
docker rm -f "steward-smoke-${{ github.run_id }}-app" "steward-smoke-${{ github.run_id }}-restored" \
"steward-smoke-${{ github.run_id }}-pg" >/dev/null 2>&1 || true
docker network rm "steward-smoke-${{ github.run_id }}-net" >/dev/null 2>&1 || true
- name: Prune dangling layers
if: always()