feat: wait for Postgres and Redis before starting work (4295)
CI / lint (push) Successful in 2s
CI / extension-version (push) Successful in 2s
Build images / sign-extension (push) Successful in 3s
Build images / build-agent (push) Successful in 6s
CI / frontend-build (push) Successful in 23s
CI / backend-lint-and-test (push) Failing after 32s
Build images / build-web (push) Successful in 1m43s
CI / integration (push) Successful in 2m12s
Build images / smoke-web (push) Successful in 57s
Build images / promote (push) Skipped

Operator, 2026-09-23: *"it's a single container that need to connect
successfully to redis and postgres before starting work shouldn't that simply
be a check (with retries) at the start of the container."*

Yes, and the consolidated layout makes it necessary rather than tidy.

Swarm has no ordering primitive — it ignores `depends_on` outright — so every
service in a stack starts at once and this container has always raced its own
database on a cold deploy. The multi-service stack hid how sharp that is: a
`web` task that failed `alembic upgrade head` against a still-initialising
Postgres simply died, and Swarm restarted it until it worked. Nobody ever saw
a problem worth naming.

Consolidation removes that safety net. Each supervisord program gets
`startretries=3`, so three quick failures put the program in FATAL and leave
it there — supervisord keeps running, the container keeps running, and the
application never starts. It would present as a permanently unhealthy
container whose image was fine and whose database merely took twenty seconds
to initialise, which is a miserable thing to debug on a first deploy.

A TCP connect, not a query: the same probe ci.yml's integration lane and the
build smoke already use. It answers the question actually being asked — is
something listening — and cannot fail for a reason that retrying will never
fix. A real query would be a stronger readiness signal and a worse gate,
since a wrong password or a missing database is not transient, and a loop
waiting for one to heal turns a five-second misconfiguration into a
two-minute timeout with a misleading message. Those belong to alembic, which
runs seconds later and says exactly what is wrong.

Targets are derived from the same env the application reads, so the wait
cannot drift from what the app will actually connect to — a gate checking a
different host than the app uses is worse than no gate.

Bounded at 120s (rule 156), reporting every few attempts so `docker logs` on
a waiting container says what it is waiting for. Skipped for `shell`, which
exists precisely for when something else is broken and you want a prompt
rather than a gate.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR
This commit is contained in:
2026-09-23 09:17:58 -04:00
co-authored by Claude Opus 5
parent b09ee87255
commit b2da3acce9
3 changed files with 259 additions and 0 deletions
+22
View File
@@ -51,6 +51,28 @@ if [ -z "${FC_ROLE:-}" ]; then
export FC_ROLE="$ROLE"
printf '%s\n' "$ROLE" > "${FC_ROLE_FILE:-/tmp/fc-role}" 2>/dev/null || true
fi
# WAIT FOR POSTGRES AND REDIS before doing anything that needs them.
#
# Swarm has no ordering primitive — it ignores `depends_on` entirely — so
# every service in a stack starts at once and this container races its own
# database on every cold deploy.
#
# The multi-service stack hid how sharp that is: a `web` task that failed
# `alembic upgrade head` against a still-initialising Postgres simply died,
# and Swarm restarted it until it worked. Consolidation removes that. Each
# supervisord program gets `startretries=3`, so three quick failures put the
# program in FATAL and leave it there — supervisord keeps running, the
# container keeps running, and the application never starts. It would present
# as a permanently unhealthy container whose image was fine and whose
# database merely took twenty seconds to come up.
#
# Skipped for `shell`, which exists precisely for the case where something
# else is broken and you want a prompt rather than a gate.
case "$ROLE" in
shell|bash) ;;
*) python -m backend.app.scripts.wait_for_deps ;;
esac
shift || true
case "$ROLE" in