fix(db): ride out a brief database outage mid-request; explain it if it persists
The operator's 500 was NOT the startup gap I attributed it to. They found it by logging in and getting Steward's own error page -- which means create_app had completed and the app was serving, so the gaierror came from a request handler acquiring a connection, not from boot. Neither prior fix covers that: the startup retry never runs, and pool_pre_ping only helps once the database is back, since its replacement connect fails too while the container is gone. Adds a before_request gate that acquires a pooled connection with a short bounded retry (~1.75s over 4 tries) so a database restart is ridden out invisibly, and renders a distinct 503 "database unavailable" page when the budget is exhausted. The request budget is deliberately far shorter than the startup one: nobody watches a container boot, but somebody is watching this page load, and a page that hangs is worse than one that says what is wrong. /health and static are exempt -- a liveness probe must stay answerable while the database is down, or a dependency outage triggers a restart loop. Credential and missing-database errors still propagate rather than being reported as "unavailable", which would send the operator chasing the wrong problem entirely. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
+28
-2
@@ -3,9 +3,12 @@ from __future__ import annotations
|
||||
import asyncio
|
||||
import logging
|
||||
from pathlib import Path
|
||||
from quart import Quart, render_template
|
||||
from quart import Quart, render_template, request
|
||||
from .config import load_bootstrap
|
||||
from .database import init_db, DB_CONNECT_TIMEOUT_SECONDS
|
||||
from .database import (
|
||||
init_db, ensure_database_reachable, DatabaseUnavailable,
|
||||
DB_CONNECT_TIMEOUT_SECONDS,
|
||||
)
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
@@ -245,6 +248,29 @@ def create_app(
|
||||
async def health():
|
||||
return {"status": "ok"}
|
||||
|
||||
# ── 11b. Database availability gate ────────────────────────────────────────
|
||||
# Every page in Steward reads the database, so a database that has gone away
|
||||
# under a running app turns each request into an opaque 500 (this is how a
|
||||
# login attempt surfaced a bare gaierror). Acquire a connection up front,
|
||||
# retrying briefly to ride out a restart, and answer honestly if it stays
|
||||
# down rather than failing deep inside a handler with a generic error.
|
||||
if not testing:
|
||||
@app.before_request
|
||||
async def _database_gate():
|
||||
# /health is a liveness probe for the container itself — it must
|
||||
# stay answerable while the database is down, or a restart loop
|
||||
# gets triggered by a dependency outage. Static files need no DB.
|
||||
if request.endpoint in ("health", "static"):
|
||||
return None
|
||||
try:
|
||||
await ensure_database_reachable(app._db_engine)
|
||||
except DatabaseUnavailable as exc:
|
||||
logger.error("Database unreachable while serving %s: %s",
|
||||
request.path, exc)
|
||||
return await render_template(
|
||||
"errors/database_unavailable.html"), 503
|
||||
return None
|
||||
|
||||
# ── 12. Error handlers ─────────────────────────────────────────────────────
|
||||
@app.errorhandler(404)
|
||||
async def not_found(_):
|
||||
|
||||
Reference in New Issue
Block a user