refactor: the image carries its own healthcheck and picks it by role (4295)
CI / extension-version (push) Successful in 4s
CI / lint (push) Successful in 4s
Build images / sign-extension (push) Successful in 5s
Build images / build-agent (push) Successful in 7s
extension / lint (push) Successful in 18s
CI / frontend-build (push) Successful in 24s
CI / backend-lint-and-test (push) Successful in 33s
Build images / build-web (push) Successful in 1m42s
CI / integration (push) Successful in 2m11s
Build images / smoke-web (push) Successful in 57s
Build images / promote (push) Skipped
CI / extension-version (push) Successful in 4s
CI / lint (push) Successful in 4s
Build images / sign-extension (push) Successful in 5s
Build images / build-agent (push) Successful in 7s
extension / lint (push) Successful in 18s
CI / frontend-build (push) Successful in 24s
CI / backend-lint-and-test (push) Successful in 33s
Build images / build-web (push) Successful in 1m42s
CI / integration (push) Successful in 2m11s
Build images / smoke-web (push) Successful in 57s
Build images / promote (push) Skipped
Operator, 2026-09-23: *"why isn't the healthcheck built into the image or
base on what command runs if one is passed in. why is it manually declared in
the stack here."*
No good reason. The container is the only thing that knows what it was asked
to run, and every compose file, stack file and README had to restate it:
web -> urllib /api/health
worker -> celery inspect ping -d celery@$HOSTNAME
all -> both, for every lane
Three checks written by hand, once per service, in every file anyone ever
wrote — none of them wrong until a role changed, and all of them silently
wrong after. The same duplication the lane table exists to remove one level
down, and I built it without noticing.
`entrypoint.sh` now records the role it started. The Dockerfile declares ONE
`HEALTHCHECK` that reads it and asks the right question: HTTP for web, a
self-addressed celery ping for a worker lane, both-for-every-lane for `all`,
and nothing for shell/alembic, which are one-shot and have no liveness to
probe. `docker-compose.single.yml` and the consolidated stack declare none.
A service that wants something else can still declare its own; docker prefers
it, so the escape hatch is the default docker behaviour rather than a flag.
Two details that are load-bearing:
* The role is written ONCE, by the outermost invocation. `all` starts the
other roles through this same script under supervisord, and a child
overwriting the container's role would turn the composite check into a
web-only one — silently, and only on the consolidated path. FC_ROLE is
exported so a child sees it set and skips.
* The celery ping is addressed to THIS node, not a bare ping. A bare one is
answered by any worker on the broker, so in a stack with replicas a dead
container would report healthy for as long as a sibling lived — the check
would be measuring the cluster rather than the container it is inside.
`healthcheck_all.py` is deleted; its two probes moved into the dispatcher
rather than being a second copy beside it.
An unrecorded role PASSES. The entrypoint always writes the file, so the only
way to miss it is bypassing the entrypoint — a debugging shape, where a check
that cannot tell what it is looking at must not assert the thing is broken
(snippet #3969). Said on stdout rather than assumed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR
This commit is contained in:
@@ -76,17 +76,12 @@ services:
|
||||
# mid-backup — which is recoverable (the work is chunked and idempotent)
|
||||
# but wastes however long it had run.
|
||||
stop_grace_period: 200s
|
||||
# BOTH halves: hypercorn answers AND every configured lane is answering
|
||||
# the broker. A web-only check would report a healthy container while
|
||||
# every lane inside it had crashed — the failure mode consolidation
|
||||
# creates, since docker can no longer see the lanes as separate services.
|
||||
healthcheck:
|
||||
test: ["CMD", "python", "-m", "backend.app.scripts.healthcheck_all"]
|
||||
interval: 30s
|
||||
timeout: 15s
|
||||
retries: 3
|
||||
# Covers alembic + hypercorn boot + four celery workers registering.
|
||||
start_period: 90s
|
||||
# No healthcheck here either. The image declares one that reads the role
|
||||
# it is running, and for this one that means BOTH halves: hypercorn
|
||||
# answers AND every lane is answering the broker. A web-only check would
|
||||
# report a healthy container while every lane inside it had crashed —
|
||||
# the failure mode consolidation creates, since docker can no longer see
|
||||
# the lanes as separate services.
|
||||
environment:
|
||||
DB_USER: ${DB_USER:-curator}
|
||||
DB_PASSWORD: ${DB_PASSWORD:-postgres}
|
||||
|
||||
Reference in New Issue
Block a user