CI and images / lint (push) Successful in 3s
CI and images / extension-version (push) Successful in 3s
extension / lint (push) Successful in 21s
CI and images / frontend-build (push) Successful in 21s
CI and images / backend-lint-and-test (push) Successful in 31s
CI and images / integration (push) Successful in 2m10s
CI and images / sign-extension (push) Successful in 3s
CI and images / build-agent (push) Successful in 6s
CI and images / build-web (push) Successful in 1m51s
CI and images / smoke-web (push) Successful in 57s
CI and images / promote (push) Skipped
Operator, 2026-09-23: *"tighten the gate so :dev can't publish on red tests"*, then *"I don't want failing builds to publish anywhere going forward."* Run 7348 is the worked example. The backend unit lane went red on `2f8f0bc` and `build-web` pushed `:dev` in the same minute, because the lanes and the build were SEPARATE WORKFLOWS on the same push trigger. Neither could see the other's verdict. `:dev` was a "it built" signal, never a "it passed" one, and nothing about that was visible from either run. Two workflows cannot express the gate. A `needs:` edge only exists inside one graph. So `ci.yml`'s five lanes move into `build.yml` and `ci.yml` is deleted; `sign-extension`, `build-web` and `build-agent` now need all five. Nothing here is a new mechanism — it is the same edge that has gated `promote` since milestone 362 step 4, and it keeps that step's hardest-won property: **not running is not the same as passing.** `needs` treats a SKIPPED dependency as unsatisfied, so a lane that silently skips itself blocks the publish exactly as a failing one does. Run 5290 is why that is worth stating. Scope, said plainly rather than implied: - Gated: every image tag (`:dev`, `:latest`, `:c-<sha>`), the weekly base refresh, and the `ext-<version>` signed-XPI release asset — `sign-extension` publishes too, so it is gated with the rest. - Not gated, deliberately: `extension.yml` publishes nothing, and `release.yml` runs on a `v*` tag, generates notes rather than an artifact, and its commit already went through main's gated build. - `pull_request` (Renovate bumps into `dev`) comes across with the lanes. Its runs are the lanes and nothing else, via an `if:` on each publishing job rather than an inference from the `needs` chain. The cost, accepted knowingly: this workflow queues per branch and never cancels, so on two pushes in quick succession the second's lint feedback waits out the first's build. A slower red beats a fast red that ships. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR
147 lines
5.4 KiB
Python
147 lines
5.4 KiB
Python
"""Block until Postgres and Redis accept connections. Exit 0 ready, 1 timed out.
|
|
|
|
## Why the container has to do this itself
|
|
|
|
Compose has `depends_on: {condition: service_healthy}`, and **Swarm ignores
|
|
it**. `docker stack deploy` has no ordering primitive at all: every service in
|
|
the stack starts at once, so FabledCurator races Postgres on every cold
|
|
deploy and always has.
|
|
|
|
The multi-service stack hid how sharp that is. `web` ran `alembic upgrade
|
|
head`, failed against a Postgres that was still doing `initdb`, and the task
|
|
died — but Swarm restarts a failed task forever, so the service came up a few
|
|
seconds later and nobody saw a problem worth naming.
|
|
|
|
Consolidation removes that safety net. supervisord gives each program
|
|
`startretries=3`, so a web program that fails three times in the first
|
|
seconds goes FATAL and **stays** FATAL: supervisord keeps running, the
|
|
container keeps running, and the application never starts. The healthcheck
|
|
catches it — but as a container that is permanently unhealthy for a reason
|
|
that has nothing to do with the image, on a stack whose database simply took
|
|
twenty seconds to initialise.
|
|
|
|
Operator, 2026-09-23: *"it's a single container that need to connect
|
|
successfully to redis and postgres before starting work shouldn't that simply
|
|
be a check (with retries) at the start of the container."* Yes.
|
|
|
|
## A TCP connect, not a query
|
|
|
|
The same probe `build.yml`'s integration lane and the build smoke already use.
|
|
It answers the question that is actually being asked — is something listening
|
|
— and it cannot fail for a reason that retrying will never fix.
|
|
|
|
A real query would be a stronger readiness signal and a worse gate: a wrong
|
|
password or a missing database is not a transient condition, and a loop that
|
|
waits for one to heal turns a five-second misconfiguration into a two-minute
|
|
timeout with a misleading message. Those belong to alembic, which runs
|
|
seconds later and says exactly what is wrong.
|
|
|
|
The Postgres image is well behaved here: during `initdb` it serves on a unix
|
|
socket only and opens TCP when it is ready for clients, so the connect is a
|
|
good proxy for "ready" rather than merely "process exists".
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import argparse
|
|
import os
|
|
import socket
|
|
import sys
|
|
import time
|
|
from urllib.parse import urlparse
|
|
|
|
# Long enough for a first-ever `initdb` on a slow disk, which is the worst
|
|
# case this exists for and is measured in tens of seconds, not minutes. A
|
|
# deploy that is genuinely misconfigured should fail while someone is still
|
|
# watching it rather than hold the container open for a quarter of an hour.
|
|
DEFAULT_TIMEOUT = 120.0
|
|
CONNECT_TIMEOUT = 2.0
|
|
RETRY_DELAY = 1.0
|
|
# Progress every N attempts. `docker logs` on a container that is waiting must
|
|
# say what it is waiting for — silence is indistinguishable from a hang.
|
|
REPORT_EVERY = 5
|
|
|
|
|
|
def _target(url: str | None, default_port: int) -> tuple[str, int] | None:
|
|
"""(host, port) from a connection URL, or None if there is nothing to wait for."""
|
|
if not url:
|
|
return None
|
|
parsed = urlparse(url)
|
|
if not parsed.hostname:
|
|
return None
|
|
return parsed.hostname, parsed.port or default_port
|
|
|
|
|
|
def targets() -> list[tuple[str, tuple[str, int]]]:
|
|
"""What this container must reach, read from the same env the app reads.
|
|
|
|
Derived rather than passed in, so the wait cannot drift from what the
|
|
application will actually connect to — a gate that checks a different
|
|
host than the app uses is worse than no gate.
|
|
"""
|
|
out: list[tuple[str, tuple[str, int]]] = []
|
|
|
|
host = os.environ.get("DB_HOST")
|
|
if host:
|
|
out.append(("postgres", (host, int(os.environ.get("DB_PORT") or 5432))))
|
|
|
|
broker = _target(os.environ.get("CELERY_BROKER_URL"), 6379)
|
|
if broker:
|
|
out.append(("redis", broker))
|
|
|
|
return out
|
|
|
|
|
|
def _accepts(host: str, port: int) -> bool:
|
|
try:
|
|
with socket.create_connection((host, port), timeout=CONNECT_TIMEOUT):
|
|
return True
|
|
except OSError:
|
|
return False
|
|
|
|
|
|
def wait(
|
|
name: str, host: str, port: int, deadline: float, now=time.monotonic,
|
|
) -> bool:
|
|
attempt = 0
|
|
while True:
|
|
if _accepts(host, port):
|
|
print(f"[wait] {name} at {host}:{port} is accepting connections")
|
|
return True
|
|
attempt += 1
|
|
if now() >= deadline:
|
|
print(
|
|
f"[wait] TIMEOUT: {name} at {host}:{port} never accepted a "
|
|
f"connection ({attempt} attempts)",
|
|
file=sys.stderr,
|
|
)
|
|
return False
|
|
if attempt % REPORT_EVERY == 0:
|
|
left = int(deadline - now())
|
|
print(f"[wait] {name} at {host}:{port} not ready yet, {left}s left")
|
|
time.sleep(RETRY_DELAY)
|
|
|
|
|
|
def main(argv: list[str] | None = None) -> int:
|
|
ap = argparse.ArgumentParser(description="Wait for Postgres and Redis.")
|
|
ap.add_argument("--timeout", type=float, default=DEFAULT_TIMEOUT)
|
|
args = ap.parse_args(argv)
|
|
|
|
wanted = targets()
|
|
if not wanted:
|
|
# Nothing configured to wait for. Not an error: `shell` and one-off
|
|
# runs are legitimate, and refusing to start would make this gate the
|
|
# reason a debugging container will not boot.
|
|
print("[wait] no database or broker configured; nothing to wait for")
|
|
return 0
|
|
|
|
deadline = time.monotonic() + args.timeout
|
|
for name, (host, port) in wanted:
|
|
if not wait(name, host, port, deadline):
|
|
return 1
|
|
return 0
|
|
|
|
|
|
if __name__ == "__main__":
|
|
raise SystemExit(main())
|