fix(db): survive a database restart and a not-yet-ready database
Two connection-recovery gaps, both surfaced by a Postgres restart that left the app throwing tracebacks while the DB itself was healthy. pool_pre_ping + pool_recycle on the app engine [#2626]: when the database restarts, every connection already in the pool is dead at the socket level. SQLAlchemy only discovered that by failing a real query, so the first operation after a restart errored out on whatever triggered it. pre_ping checks liveness on checkout and swaps the dead connection transparently; pool_recycle caps connection age so a socket stranded by a NAT/conntrack timeout or a Docker network rebuild is retired on a timer instead. wait_for_database() gate at startup [#2627]: create_app touches the DB synchronously (migrations, secret re-encryption, settings load) and assumed it was both resolvable and accepting connections on the first try. Neither holds after a host reboot (Docker DNS not yet serving `db` -> gaierror -2) or an unclean shutdown (Postgres still replaying WAL -> "not yet accepting connections"). Both are transient, so retry with capped backoff behind one gate ahead of the first DB touch. Credential and missing-database errors are classified by SQLSTATE and still fail immediately -- waiting cannot fix those. Budget is bootstrap-configurable (STEWARD_DB_CONNECT_TIMEOUT / database.connect_timeout, default 60s) since it governs reaching the DB and so cannot live in the DB-backed settings. Tests drive the retry loop off a fake clock, so backoff and timeout behaviour are deterministic rather than wall-clock dependent. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
+15
-1
@@ -5,7 +5,7 @@ import logging
|
||||
from pathlib import Path
|
||||
from quart import Quart, render_template
|
||||
from .config import load_bootstrap
|
||||
from .database import init_db
|
||||
from .database import init_db, DB_CONNECT_TIMEOUT_SECONDS
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
@@ -23,6 +23,7 @@ def create_app(
|
||||
bootstrap = {
|
||||
"database_url": "postgresql+asyncpg://test/test",
|
||||
"secret_key": "test-secret-key",
|
||||
"db_connect_timeout": DB_CONNECT_TIMEOUT_SECONDS,
|
||||
"plugin_dirs": ["plugins"],
|
||||
"plugin_install_dir": "plugins",
|
||||
}
|
||||
@@ -42,6 +43,19 @@ def create_app(
|
||||
from unittest.mock import MagicMock
|
||||
app.db_sessionmaker = MagicMock()
|
||||
|
||||
# ── 2b. Block until the database is actually reachable ────────────────────
|
||||
# Everything from here down touches the DB synchronously (migrations,
|
||||
# secret re-encryption, settings load). Gate all of it behind one readiness
|
||||
# check so a DB that is merely slow to come up — WAL recovery after an
|
||||
# unclean shutdown, or Docker DNS not yet serving the `db` name after a host
|
||||
# reboot — is waited out instead of crashing the container on a traceback.
|
||||
if not testing:
|
||||
from .database import wait_for_database
|
||||
wait_for_database(
|
||||
app.config["DATABASE_URL"],
|
||||
timeout_seconds=bootstrap["db_connect_timeout"],
|
||||
)
|
||||
|
||||
# ── 3. Core migrations only (creates app_settings table) ──────────────────
|
||||
if not testing:
|
||||
from .core.migration_runner import run_core_migrations
|
||||
|
||||
Reference in New Issue
Block a user