fix(db): survive a database restart and a not-yet-ready database
Two connection-recovery gaps, both surfaced by a Postgres restart that left the app throwing tracebacks while the DB itself was healthy. pool_pre_ping + pool_recycle on the app engine [#2626]: when the database restarts, every connection already in the pool is dead at the socket level. SQLAlchemy only discovered that by failing a real query, so the first operation after a restart errored out on whatever triggered it. pre_ping checks liveness on checkout and swaps the dead connection transparently; pool_recycle caps connection age so a socket stranded by a NAT/conntrack timeout or a Docker network rebuild is retired on a timer instead. wait_for_database() gate at startup [#2627]: create_app touches the DB synchronously (migrations, secret re-encryption, settings load) and assumed it was both resolvable and accepting connections on the first try. Neither holds after a host reboot (Docker DNS not yet serving `db` -> gaierror -2) or an unclean shutdown (Postgres still replaying WAL -> "not yet accepting connections"). Both are transient, so retry with capped backoff behind one gate ahead of the first DB touch. Credential and missing-database errors are classified by SQLSTATE and still fail immediately -- waiting cannot fix those. Budget is bootstrap-configurable (STEWARD_DB_CONNECT_TIMEOUT / database.connect_timeout, default 60s) since it governs reaching the DB and so cannot live in the DB-backed settings. Tests drive the retry loop off a fake clock, so backoff and timeout behaviour are deterministic rather than wall-clock dependent. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
@@ -13,6 +13,14 @@
|
||||
database:
|
||||
url: "postgresql+asyncpg://steward:password@localhost/steward"
|
||||
|
||||
# Optional: seconds to wait at startup for the database to become reachable
|
||||
# before giving up (default: 60). Steward retries with backoff rather than
|
||||
# crashing when the DB is merely slow to arrive — replaying WAL after an
|
||||
# unclean shutdown, or container DNS not yet resolving after a host reboot.
|
||||
# Raise it if your database is routinely slower than this to accept
|
||||
# connections. Env var: STEWARD_DB_CONNECT_TIMEOUT
|
||||
# connect_timeout: 60
|
||||
|
||||
# Optional: override the auto-generated secret key.
|
||||
# If not set, a key is auto-generated on first run and saved to /data/secret.key.
|
||||
# secret_key: "change-me-to-a-random-string"
|
||||
|
||||
Reference in New Issue
Block a user