fix(db): ride out a brief database outage mid-request; explain it if it persists
CI / lint (push) Successful in 2s
CI / unit (push) Successful in 40s
CI / integration (push) Successful in 2m21s
CI / publish (push) Successful in 1m2s

The operator's 500 was NOT the startup gap I attributed it to. They found it by
logging in and getting Steward's own error page -- which means create_app had
completed and the app was serving, so the gaierror came from a request handler
acquiring a connection, not from boot. Neither prior fix covers that: the
startup retry never runs, and pool_pre_ping only helps once the database is
back, since its replacement connect fails too while the container is gone.

Adds a before_request gate that acquires a pooled connection with a short
bounded retry (~1.75s over 4 tries) so a database restart is ridden out
invisibly, and renders a distinct 503 "database unavailable" page when the
budget is exhausted. The request budget is deliberately far shorter than the
startup one: nobody watches a container boot, but somebody is watching this
page load, and a page that hangs is worse than one that says what is wrong.

/health and static are exempt -- a liveness probe must stay answerable while
the database is down, or a dependency outage triggers a restart loop.

Credential and missing-database errors still propagate rather than being
reported as "unavailable", which would send the operator chasing the wrong
problem entirely.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
2026-08-13 08:23:01 -04:00
co-authored by Claude Opus 5
parent 59fece855d
commit 8c50ff242c
4 changed files with 193 additions and 7 deletions
+28 -2
View File
@@ -3,9 +3,12 @@ from __future__ import annotations
import asyncio
import logging
from pathlib import Path
from quart import Quart, render_template
from quart import Quart, render_template, request
from .config import load_bootstrap
from .database import init_db, DB_CONNECT_TIMEOUT_SECONDS
from .database import (
init_db, ensure_database_reachable, DatabaseUnavailable,
DB_CONNECT_TIMEOUT_SECONDS,
)
logger = logging.getLogger(__name__)
@@ -245,6 +248,29 @@ def create_app(
async def health():
return {"status": "ok"}
# ── 11b. Database availability gate ────────────────────────────────────────
# Every page in Steward reads the database, so a database that has gone away
# under a running app turns each request into an opaque 500 (this is how a
# login attempt surfaced a bare gaierror). Acquire a connection up front,
# retrying briefly to ride out a restart, and answer honestly if it stays
# down rather than failing deep inside a handler with a generic error.
if not testing:
@app.before_request
async def _database_gate():
# /health is a liveness probe for the container itself — it must
# stay answerable while the database is down, or a restart loop
# gets triggered by a dependency outage. Static files need no DB.
if request.endpoint in ("health", "static"):
return None
try:
await ensure_database_reachable(app._db_engine)
except DatabaseUnavailable as exc:
logger.error("Database unreachable while serving %s: %s",
request.path, exc)
return await render_template(
"errors/database_unavailable.html"), 503
return None
# ── 12. Error handlers ─────────────────────────────────────────────────────
@app.errorhandler(404)
async def not_found(_):