Files
FabledCurator/backend/app/scripts/wait_for_deps.py
T
bvandeusenandClaude Opus 5 274f7ffe21
CI and images / lint (push) Successful in 3s
CI and images / extension-version (push) Successful in 3s
extension / lint (push) Successful in 21s
CI and images / frontend-build (push) Successful in 21s
CI and images / backend-lint-and-test (push) Successful in 31s
CI and images / integration (push) Successful in 2m10s
CI and images / sign-extension (push) Successful in 3s
CI and images / build-agent (push) Successful in 6s
CI and images / build-web (push) Successful in 1m51s
CI and images / smoke-web (push) Successful in 57s
CI and images / promote (push) Skipped
ci: the tests gate the publish — ci.yml folds into build.yml
Operator, 2026-09-23: *"tighten the gate so :dev can't publish on red tests"*,
then *"I don't want failing builds to publish anywhere going forward."*

Run 7348 is the worked example. The backend unit lane went red on `2f8f0bc`
and `build-web` pushed `:dev` in the same minute, because the lanes and the
build were SEPARATE WORKFLOWS on the same push trigger. Neither could see the
other's verdict. `:dev` was a "it built" signal, never a "it passed" one, and
nothing about that was visible from either run.

Two workflows cannot express the gate. A `needs:` edge only exists inside one
graph. So `ci.yml`'s five lanes move into `build.yml` and `ci.yml` is deleted;
`sign-extension`, `build-web` and `build-agent` now need all five.

Nothing here is a new mechanism — it is the same edge that has gated `promote`
since milestone 362 step 4, and it keeps that step's hardest-won property:
**not running is not the same as passing.** `needs` treats a SKIPPED
dependency as unsatisfied, so a lane that silently skips itself blocks the
publish exactly as a failing one does. Run 5290 is why that is worth stating.

Scope, said plainly rather than implied:

- Gated: every image tag (`:dev`, `:latest`, `:c-<sha>`), the weekly base
  refresh, and the `ext-<version>` signed-XPI release asset — `sign-extension`
  publishes too, so it is gated with the rest.
- Not gated, deliberately: `extension.yml` publishes nothing, and
  `release.yml` runs on a `v*` tag, generates notes rather than an artifact,
  and its commit already went through main's gated build.
- `pull_request` (Renovate bumps into `dev`) comes across with the lanes. Its
  runs are the lanes and nothing else, via an `if:` on each publishing job
  rather than an inference from the `needs` chain.

The cost, accepted knowingly: this workflow queues per branch and never
cancels, so on two pushes in quick succession the second's lint feedback waits
out the first's build. A slower red beats a fast red that ships.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR
2026-09-23 11:15:42 -04:00

147 lines
5.4 KiB
Python

"""Block until Postgres and Redis accept connections. Exit 0 ready, 1 timed out.
## Why the container has to do this itself
Compose has `depends_on: {condition: service_healthy}`, and **Swarm ignores
it**. `docker stack deploy` has no ordering primitive at all: every service in
the stack starts at once, so FabledCurator races Postgres on every cold
deploy and always has.
The multi-service stack hid how sharp that is. `web` ran `alembic upgrade
head`, failed against a Postgres that was still doing `initdb`, and the task
died — but Swarm restarts a failed task forever, so the service came up a few
seconds later and nobody saw a problem worth naming.
Consolidation removes that safety net. supervisord gives each program
`startretries=3`, so a web program that fails three times in the first
seconds goes FATAL and **stays** FATAL: supervisord keeps running, the
container keeps running, and the application never starts. The healthcheck
catches it — but as a container that is permanently unhealthy for a reason
that has nothing to do with the image, on a stack whose database simply took
twenty seconds to initialise.
Operator, 2026-09-23: *"it's a single container that need to connect
successfully to redis and postgres before starting work shouldn't that simply
be a check (with retries) at the start of the container."* Yes.
## A TCP connect, not a query
The same probe `build.yml`'s integration lane and the build smoke already use.
It answers the question that is actually being asked — is something listening
— and it cannot fail for a reason that retrying will never fix.
A real query would be a stronger readiness signal and a worse gate: a wrong
password or a missing database is not a transient condition, and a loop that
waits for one to heal turns a five-second misconfiguration into a two-minute
timeout with a misleading message. Those belong to alembic, which runs
seconds later and says exactly what is wrong.
The Postgres image is well behaved here: during `initdb` it serves on a unix
socket only and opens TCP when it is ready for clients, so the connect is a
good proxy for "ready" rather than merely "process exists".
"""
from __future__ import annotations
import argparse
import os
import socket
import sys
import time
from urllib.parse import urlparse
# Long enough for a first-ever `initdb` on a slow disk, which is the worst
# case this exists for and is measured in tens of seconds, not minutes. A
# deploy that is genuinely misconfigured should fail while someone is still
# watching it rather than hold the container open for a quarter of an hour.
DEFAULT_TIMEOUT = 120.0
CONNECT_TIMEOUT = 2.0
RETRY_DELAY = 1.0
# Progress every N attempts. `docker logs` on a container that is waiting must
# say what it is waiting for — silence is indistinguishable from a hang.
REPORT_EVERY = 5
def _target(url: str | None, default_port: int) -> tuple[str, int] | None:
"""(host, port) from a connection URL, or None if there is nothing to wait for."""
if not url:
return None
parsed = urlparse(url)
if not parsed.hostname:
return None
return parsed.hostname, parsed.port or default_port
def targets() -> list[tuple[str, tuple[str, int]]]:
"""What this container must reach, read from the same env the app reads.
Derived rather than passed in, so the wait cannot drift from what the
application will actually connect to — a gate that checks a different
host than the app uses is worse than no gate.
"""
out: list[tuple[str, tuple[str, int]]] = []
host = os.environ.get("DB_HOST")
if host:
out.append(("postgres", (host, int(os.environ.get("DB_PORT") or 5432))))
broker = _target(os.environ.get("CELERY_BROKER_URL"), 6379)
if broker:
out.append(("redis", broker))
return out
def _accepts(host: str, port: int) -> bool:
try:
with socket.create_connection((host, port), timeout=CONNECT_TIMEOUT):
return True
except OSError:
return False
def wait(
name: str, host: str, port: int, deadline: float, now=time.monotonic,
) -> bool:
attempt = 0
while True:
if _accepts(host, port):
print(f"[wait] {name} at {host}:{port} is accepting connections")
return True
attempt += 1
if now() >= deadline:
print(
f"[wait] TIMEOUT: {name} at {host}:{port} never accepted a "
f"connection ({attempt} attempts)",
file=sys.stderr,
)
return False
if attempt % REPORT_EVERY == 0:
left = int(deadline - now())
print(f"[wait] {name} at {host}:{port} not ready yet, {left}s left")
time.sleep(RETRY_DELAY)
def main(argv: list[str] | None = None) -> int:
ap = argparse.ArgumentParser(description="Wait for Postgres and Redis.")
ap.add_argument("--timeout", type=float, default=DEFAULT_TIMEOUT)
args = ap.parse_args(argv)
wanted = targets()
if not wanted:
# Nothing configured to wait for. Not an error: `shell` and one-off
# runs are legitimate, and refusing to start would make this gate the
# reason a debugging container will not boot.
print("[wait] no database or broker configured; nothing to wait for")
return 0
deadline = time.monotonic() + args.timeout
for name, (host, port) in wanted:
if not wait(name, host, port, deadline):
return 1
return 0
if __name__ == "__main__":
raise SystemExit(main())