feat(544): shipped compose files run on the orchestrator's defaults; the app waits for its database itself
Brings the three compose files in line with rule 198 (milestone 544, step 6203, at the operator's request): - no healthcheck on the app or the database, so a slow pg_isready on stalled storage can no longer get a working database killed; - no deploy.restart_policy in the Swarm file, so Swarm's default (condition: any) restarts clean exits too, instead of on-failure with max_attempts leaving the app down; - depends_on is plain start order; restart: unless-stopped stays on the plain-Compose files (Swarm ignores it), and the dev app gains it. The orchestrator used to be what retried a container that started before Postgres was ready. That is now the app's job: alembic/env.py calls services/db_ready.wait_for_database before migrating. It retries refused connections, unresolved hosts, connect timeouts and "the database system is starting up" with 1-2-4-...-10s spacing for up to 300s, then exits non-zero; a wrong password or missing database fails at once. Per-request recovery is the engine's pool_pre_ping. Also fixes the dev compose database image: postgres:16-alpine has no vector extension, so migrations could not run against it. It now uses pgvector/pgvector:pg17 like the other two files. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,77 @@
|
||||
"""Wait for the database before the app starts.
|
||||
|
||||
The container runs migrations first, and the database may still be starting
|
||||
(or its host may not resolve yet) when it does. Waiting here, with a deadline,
|
||||
lets the app start on an orchestrator's defaults instead of relying on a
|
||||
healthcheck or a restart policy to retry it.
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import logging
|
||||
import time
|
||||
from collections.abc import Awaitable, Callable
|
||||
|
||||
import asyncpg
|
||||
from sqlalchemy.engine import URL, make_url
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# Long enough for a first boot that initialises the data directory on slow
|
||||
# storage; past it the process exits non-zero and the orchestrator restarts it.
|
||||
DEFAULT_DEADLINE_S = 300.0
|
||||
|
||||
# Errors that mean "not up yet". Anything else (a wrong password, a missing
|
||||
# database) fails at once, because waiting would not change the answer.
|
||||
_TRANSIENT = (
|
||||
OSError, # refused, unreachable, DNS not resolving yet, connect timeout
|
||||
asyncpg.CannotConnectNowError, # "the database system is starting up"
|
||||
asyncpg.TooManyConnectionsError,
|
||||
)
|
||||
|
||||
|
||||
def asyncpg_dsn(database_url: str | URL) -> str:
|
||||
"""The SQLAlchemy URL as a plain DSN asyncpg accepts."""
|
||||
url = make_url(database_url).set(drivername="postgresql")
|
||||
return url.render_as_string(hide_password=False)
|
||||
|
||||
|
||||
async def wait_for_database(
|
||||
database_url: str | URL,
|
||||
deadline_s: float = DEFAULT_DEADLINE_S,
|
||||
*,
|
||||
connect: Callable[..., Awaitable] = asyncpg.connect,
|
||||
sleep: Callable[[float], Awaitable] = asyncio.sleep,
|
||||
clock: Callable[[], float] = time.monotonic,
|
||||
) -> None:
|
||||
"""Return once a connection succeeds; raise the last error after the deadline.
|
||||
|
||||
Retries are spaced 1, 2, 4 … seconds apart, capped at 10, so a database
|
||||
that is down for minutes sees a handful of attempts rather than a flood.
|
||||
"""
|
||||
dsn = asyncpg_dsn(database_url)
|
||||
started = clock()
|
||||
delay = 1.0
|
||||
attempt = 0
|
||||
while True:
|
||||
attempt += 1
|
||||
try:
|
||||
conn = await connect(dsn, timeout=10)
|
||||
except _TRANSIENT as exc:
|
||||
waited = clock() - started
|
||||
if waited + delay > deadline_s:
|
||||
logger.error(
|
||||
"Database still unreachable after %.0fs (%d attempts): %s",
|
||||
waited, attempt, exc,
|
||||
)
|
||||
raise
|
||||
logger.warning(
|
||||
"Waiting for the database (attempt %d, %.0fs so far): %s",
|
||||
attempt, waited, exc,
|
||||
)
|
||||
await sleep(delay)
|
||||
delay = min(delay * 2, 10.0)
|
||||
continue
|
||||
await conn.close()
|
||||
if attempt > 1:
|
||||
logger.info("Database reachable after %d attempts", attempt)
|
||||
return
|
||||
Reference in New Issue
Block a user