fix(db): survive a database restart and a not-yet-ready database
Two connection-recovery gaps, both surfaced by a Postgres restart that left the app throwing tracebacks while the DB itself was healthy. pool_pre_ping + pool_recycle on the app engine [#2626]: when the database restarts, every connection already in the pool is dead at the socket level. SQLAlchemy only discovered that by failing a real query, so the first operation after a restart errored out on whatever triggered it. pre_ping checks liveness on checkout and swaps the dead connection transparently; pool_recycle caps connection age so a socket stranded by a NAT/conntrack timeout or a Docker network rebuild is retired on a timer instead. wait_for_database() gate at startup [#2627]: create_app touches the DB synchronously (migrations, secret re-encryption, settings load) and assumed it was both resolvable and accepting connections on the first try. Neither holds after a host reboot (Docker DNS not yet serving `db` -> gaierror -2) or an unclean shutdown (Postgres still replaying WAL -> "not yet accepting connections"). Both are transient, so retry with capped backoff behind one gate ahead of the first DB touch. Credential and missing-database errors are classified by SQLSTATE and still fail immediately -- waiting cannot fix those. Budget is bootstrap-configurable (STEWARD_DB_CONNECT_TIMEOUT / database.connect_timeout, default 60s) since it governs reaching the DB and so cannot live in the DB-backed settings. Tests drive the retry loop off a fake clock, so backoff and timeout behaviour are deterministic rather than wall-clock dependent. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
@@ -65,3 +65,50 @@ def test_secret_key_unpersistable_raises_instead_of_ephemeral(tmp_path, monkeypa
|
||||
monkeypatch.delenv("STEWARD_SECRET_KEY", raising=False)
|
||||
with pytest.raises(RuntimeError, match="could not persist"):
|
||||
_resolve_secret_key({})
|
||||
|
||||
|
||||
# ── database connect timeout (bootstrap-only: it governs reaching the DB) ────
|
||||
|
||||
|
||||
def test_db_connect_timeout_defaults_when_unset(tmp_path, monkeypatch):
|
||||
from steward.database import DB_CONNECT_TIMEOUT_SECONDS
|
||||
cfg_file = tmp_path / "config.yaml"
|
||||
cfg_file.write_text("database:\n url: x\nsecret_key: s\n")
|
||||
monkeypatch.delenv("STEWARD_DB_CONNECT_TIMEOUT", raising=False)
|
||||
cfg = load_bootstrap(cfg_file)
|
||||
assert cfg["db_connect_timeout"] == DB_CONNECT_TIMEOUT_SECONDS
|
||||
|
||||
|
||||
def test_db_connect_timeout_from_yaml(tmp_path, monkeypatch):
|
||||
cfg_file = tmp_path / "config.yaml"
|
||||
cfg_file.write_text(
|
||||
"database:\n url: x\n connect_timeout: 120\nsecret_key: s\n")
|
||||
monkeypatch.delenv("STEWARD_DB_CONNECT_TIMEOUT", raising=False)
|
||||
assert load_bootstrap(cfg_file)["db_connect_timeout"] == 120.0
|
||||
|
||||
|
||||
def test_db_connect_timeout_env_overrides_yaml(tmp_path, monkeypatch):
|
||||
cfg_file = tmp_path / "config.yaml"
|
||||
cfg_file.write_text(
|
||||
"database:\n url: x\n connect_timeout: 120\nsecret_key: s\n")
|
||||
monkeypatch.setenv("STEWARD_DB_CONNECT_TIMEOUT", "5")
|
||||
assert load_bootstrap(cfg_file)["db_connect_timeout"] == 5.0
|
||||
|
||||
|
||||
def test_db_connect_timeout_garbage_falls_back_to_default(tmp_path, monkeypatch):
|
||||
from steward.database import DB_CONNECT_TIMEOUT_SECONDS
|
||||
cfg_file = tmp_path / "config.yaml"
|
||||
cfg_file.write_text("database:\n url: x\nsecret_key: s\n")
|
||||
monkeypatch.setenv("STEWARD_DB_CONNECT_TIMEOUT", "not-a-number")
|
||||
cfg = load_bootstrap(cfg_file)
|
||||
assert cfg["db_connect_timeout"] == DB_CONNECT_TIMEOUT_SECONDS
|
||||
|
||||
|
||||
def test_db_connect_timeout_non_positive_falls_back_to_default(tmp_path, monkeypatch):
|
||||
# 0 would mean "never wait", reinstating the crash this setting exists to fix.
|
||||
from steward.database import DB_CONNECT_TIMEOUT_SECONDS
|
||||
cfg_file = tmp_path / "config.yaml"
|
||||
cfg_file.write_text("database:\n url: x\nsecret_key: s\n")
|
||||
monkeypatch.setenv("STEWARD_DB_CONNECT_TIMEOUT", "0")
|
||||
cfg = load_bootstrap(cfg_file)
|
||||
assert cfg["db_connect_timeout"] == DB_CONNECT_TIMEOUT_SECONDS
|
||||
|
||||
Reference in New Issue
Block a user