fix(db): survive a database restart and a not-yet-ready database
CI / lint (push) Successful in 4s
CI / unit (push) Successful in 50s
CI / integration (push) Successful in 2m21s
CI / publish (push) Successful in 1m2s

Two connection-recovery gaps, both surfaced by a Postgres restart that left
the app throwing tracebacks while the DB itself was healthy.

pool_pre_ping + pool_recycle on the app engine [#2626]: when the database
restarts, every connection already in the pool is dead at the socket level.
SQLAlchemy only discovered that by failing a real query, so the first
operation after a restart errored out on whatever triggered it. pre_ping
checks liveness on checkout and swaps the dead connection transparently;
pool_recycle caps connection age so a socket stranded by a NAT/conntrack
timeout or a Docker network rebuild is retired on a timer instead.

wait_for_database() gate at startup [#2627]: create_app touches the DB
synchronously (migrations, secret re-encryption, settings load) and assumed
it was both resolvable and accepting connections on the first try. Neither
holds after a host reboot (Docker DNS not yet serving `db` -> gaierror -2)
or an unclean shutdown (Postgres still replaying WAL -> "not yet accepting
connections"). Both are transient, so retry with capped backoff behind one
gate ahead of the first DB touch. Credential and missing-database errors
are classified by SQLSTATE and still fail immediately -- waiting cannot fix
those. Budget is bootstrap-configurable (STEWARD_DB_CONNECT_TIMEOUT /
database.connect_timeout, default 60s) since it governs reaching the DB and
so cannot live in the DB-backed settings.

Tests drive the retry loop off a fake clock, so backoff and timeout
behaviour are deterministic rather than wall-clock dependent.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
2026-08-12 23:21:58 -04:00
co-authored by Claude Opus 5
parent 4b97bd01ea
commit 6c9b89390a
6 changed files with 464 additions and 2 deletions
+15 -1
View File
@@ -5,7 +5,7 @@ import logging
from pathlib import Path
from quart import Quart, render_template
from .config import load_bootstrap
from .database import init_db
from .database import init_db, DB_CONNECT_TIMEOUT_SECONDS
logger = logging.getLogger(__name__)
@@ -23,6 +23,7 @@ def create_app(
bootstrap = {
"database_url": "postgresql+asyncpg://test/test",
"secret_key": "test-secret-key",
"db_connect_timeout": DB_CONNECT_TIMEOUT_SECONDS,
"plugin_dirs": ["plugins"],
"plugin_install_dir": "plugins",
}
@@ -42,6 +43,19 @@ def create_app(
from unittest.mock import MagicMock
app.db_sessionmaker = MagicMock()
# ── 2b. Block until the database is actually reachable ────────────────────
# Everything from here down touches the DB synchronously (migrations,
# secret re-encryption, settings load). Gate all of it behind one readiness
# check so a DB that is merely slow to come up — WAL recovery after an
# unclean shutdown, or Docker DNS not yet serving the `db` name after a host
# reboot — is waited out instead of crashing the container on a traceback.
if not testing:
from .database import wait_for_database
wait_for_database(
app.config["DATABASE_URL"],
timeout_seconds=bootstrap["db_connect_timeout"],
)
# ── 3. Core migrations only (creates app_settings table) ──────────────────
if not testing:
from .core.migration_runner import run_core_migrations