bvandeusenandClaude Opus 5.5
cba2c24612
Brings the three compose files in line with rule 198 (milestone 544, step 6203, at the operator's request): - no healthcheck on the app or the database, so a slow pg_isready on stalled storage can no longer get a working database killed; - no deploy.restart_policy in the Swarm file, so Swarm's default (condition: any) restarts clean exits too, instead of on-failure with max_attempts leaving the app down; - depends_on is plain start order; restart: unless-stopped stays on the plain-Compose files (Swarm ignores it), and the dev app gains it. The orchestrator used to be what retried a container that started before Postgres was ready. That is now the app's job: alembic/env.py calls services/db_ready.wait_for_database before migrating. It retries refused connections, unresolved hosts, connect timeouts and "the database system is starting up" with 1-2-4-...-10s spacing for up to 300s, then exits non-zero; a wrong password or missing database fails at once. Per-request recovery is the engine's pool_pre_ping. Also fixes the dev compose database image: postgres:16-alpine has no vector extension, so migrations could not run against it. It now uses pgvector/pgvector:pg17 like the other two files. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
78 lines
2.6 KiB
Python
78 lines
2.6 KiB
Python
"""Wait for the database before the app starts.
|
|
|
|
The container runs migrations first, and the database may still be starting
|
|
(or its host may not resolve yet) when it does. Waiting here, with a deadline,
|
|
lets the app start on an orchestrator's defaults instead of relying on a
|
|
healthcheck or a restart policy to retry it.
|
|
"""
|
|
|
|
import asyncio
|
|
import logging
|
|
import time
|
|
from collections.abc import Awaitable, Callable
|
|
|
|
import asyncpg
|
|
from sqlalchemy.engine import URL, make_url
|
|
|
|
logger = logging.getLogger(__name__)
|
|
|
|
# Long enough for a first boot that initialises the data directory on slow
|
|
# storage; past it the process exits non-zero and the orchestrator restarts it.
|
|
DEFAULT_DEADLINE_S = 300.0
|
|
|
|
# Errors that mean "not up yet". Anything else (a wrong password, a missing
|
|
# database) fails at once, because waiting would not change the answer.
|
|
_TRANSIENT = (
|
|
OSError, # refused, unreachable, DNS not resolving yet, connect timeout
|
|
asyncpg.CannotConnectNowError, # "the database system is starting up"
|
|
asyncpg.TooManyConnectionsError,
|
|
)
|
|
|
|
|
|
def asyncpg_dsn(database_url: str | URL) -> str:
|
|
"""The SQLAlchemy URL as a plain DSN asyncpg accepts."""
|
|
url = make_url(database_url).set(drivername="postgresql")
|
|
return url.render_as_string(hide_password=False)
|
|
|
|
|
|
async def wait_for_database(
|
|
database_url: str | URL,
|
|
deadline_s: float = DEFAULT_DEADLINE_S,
|
|
*,
|
|
connect: Callable[..., Awaitable] = asyncpg.connect,
|
|
sleep: Callable[[float], Awaitable] = asyncio.sleep,
|
|
clock: Callable[[], float] = time.monotonic,
|
|
) -> None:
|
|
"""Return once a connection succeeds; raise the last error after the deadline.
|
|
|
|
Retries are spaced 1, 2, 4 … seconds apart, capped at 10, so a database
|
|
that is down for minutes sees a handful of attempts rather than a flood.
|
|
"""
|
|
dsn = asyncpg_dsn(database_url)
|
|
started = clock()
|
|
delay = 1.0
|
|
attempt = 0
|
|
while True:
|
|
attempt += 1
|
|
try:
|
|
conn = await connect(dsn, timeout=10)
|
|
except _TRANSIENT as exc:
|
|
waited = clock() - started
|
|
if waited + delay > deadline_s:
|
|
logger.error(
|
|
"Database still unreachable after %.0fs (%d attempts): %s",
|
|
waited, attempt, exc,
|
|
)
|
|
raise
|
|
logger.warning(
|
|
"Waiting for the database (attempt %d, %.0fs so far): %s",
|
|
attempt, waited, exc,
|
|
)
|
|
await sleep(delay)
|
|
delay = min(delay * 2, 10.0)
|
|
continue
|
|
await conn.close()
|
|
if attempt > 1:
|
|
logger.info("Database reachable after %d attempts", attempt)
|
|
return
|