ci: the smoke gate found a real one on its first run — and had two bugs of its own
Build images / sign-extension (push) Successful in 6s
CI / lint (push) Successful in 6s
CI / extension-version (push) Successful in 4s
Build images / build-ml (push) Successful in 11s
Build images / build-agent (push) Successful in 13s
extension / lint (push) Successful in 24s
CI / frontend-build (push) Successful in 28s
CI / backend-lint-and-test (push) Successful in 35s
Build images / build-web (push) Successful in 8s
Build images / smoke-web (push) Skipped
CI / integration (push) Successful in 2m5s

Run 5296 was `smoke-web`'s first genuine execution. Checks 1 and 2 passed:
alembic built the schema from empty inside the image, all five apt binaries
resolved, and the application's own Thumbnailer produced JPEG, PNG-with-alpha,
WebP and an ffmpeg video frame against the image's libraries. Check 3 failed,
and the trap's log dump said exactly why:

  MissingCredentialKey: Fernet key file not found at
  /images/secrets/credential_key.b64. For first-time setup, set
  CURATOR_BOOTSTRAP_NEW_KEY=1.

That is the product being right. credential_crypto refuses to mint a key
unless someone opts in, because the 2026-06-02 audit found a partial restore
(DB back, /images/secrets/ lost) silently generating a fresh one and leaving a
working-looking system where every authenticated download failed AUTH_ERROR.

It is also a first-run blocker for milestone 328, filed as #3422: the variable
appears in no README, no .env.example and no compose file, so the install path
that milestone just finished writing produces a container that exits on boot.
Not fixed here — the fix trades safety against friction and is the operator's
call.

Two defects in the gate itself, both surfaced by the same run:

- A throwaway CI instance IS first-time setup, so it now passes
  CURATOR_BOOTSTRAP_NEW_KEY=1. The check was asserting a condition no fresh
  container can satisfy.

- The health loop polled a dead container for 3m35s. Docker had already
  recycled its IP, so the replies were a baffling mix of connection-refused
  and 5s timeouts from whatever took the address next. It now checks
  `.State.Running` each iteration and fails immediately with the container's
  log. The trap had the real answer the whole time; this stops burying it
  under four minutes of noise.

Also corrected a message claiming a 120s budget: 60 iterations of up to 5s
connect plus 2s sleep is nearer seven minutes.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TTjbZZ6JirCMSaJzQV1RhA
This commit is contained in:
2026-09-02 16:00:18 -04:00
co-authored by Claude Opus 5
parent 81b7b6f308
commit 4815040d74
+25 -3
View File
@@ -1213,6 +1213,14 @@ jobs:
ENVOPTS="$ENVOPTS -e DB_PORT=5432 -e DB_NAME=$DB_NAME -e SECRET_KEY=$SECRET_KEY" ENVOPTS="$ENVOPTS -e DB_PORT=5432 -e DB_NAME=$DB_NAME -e SECRET_KEY=$SECRET_KEY"
ENVOPTS="$ENVOPTS -e CELERY_BROKER_URL=redis://$RD_IP:6379/0" ENVOPTS="$ENVOPTS -e CELERY_BROKER_URL=redis://$RD_IP:6379/0"
ENVOPTS="$ENVOPTS -e CELERY_RESULT_BACKEND=redis://$RD_IP:6379/0" ENVOPTS="$ENVOPTS -e CELERY_RESULT_BACKEND=redis://$RD_IP:6379/0"
# A throwaway CI instance IS first-time setup, which is the one case
# credential_crypto allows a key to be minted in. Without it the web
# role refuses to boot — deliberately, since silently generating a
# key on a restored-DB-but-lost-secrets deployment would leave every
# Credential row undecryptable (the 2026-06-02 audit). Discovered by
# this job on its first real run; see #3422 for the fact that no
# user-facing file mentions this variable at all.
ENVOPTS="$ENVOPTS -e CURATOR_BOOTSTRAP_NEW_KEY=1"
# 1. The schema builds from empty, using the image's OWN libpq and # 1. The schema builds from empty, using the image's OWN libpq and
# psycopg. This is the same call entrypoint.sh makes before it # psycopg. This is the same call entrypoint.sh makes before it
@@ -1247,12 +1255,26 @@ jobs:
healthy=1 healthy=1
break break
fi fi
# A container that has EXITED will never answer, so stop asking.
# Without this the loop spent 3m35s polling a dead container on
# this job's first run, and — because docker recycles the IP — got
# a confusing mix of connection-refused and 5s timeouts from
# whatever took the address next. The trap's log dump had the real
# answer the whole time; this just stops burying it.
if [ "$(docker inspect -f '{{.State.Running}}' "$CID" 2>/dev/null)" != "true" ]; then
echo "smoke: FAILED — the web container exited during boot." >&2
echo "smoke: its log follows; entrypoint runs alembic BEFORE" >&2
echo "smoke: serving, so a startup exception lands here." >&2
exit 1
fi
sleep 2 sleep 2
done done
if [ -z "$healthy" ]; then if [ -z "$healthy" ]; then
echo "smoke: FAILED — web did not answer /api/health within 120s." >&2 # 60 iterations of (up to 5s connect + 2s sleep) — up to ~7min, not
echo "smoke: entrypoint runs alembic BEFORE serving, and step 1" >&2 # the 120s an earlier version of this message claimed.
echo "smoke: passed, so look at hypercorn and the python base." >&2 echo "smoke: FAILED — web is running but never answered" >&2
echo "smoke: /api/health. It is up, so look at hypercorn and the" >&2
echo "smoke: python base rather than at startup." >&2
exit 1 exit 1
fi fi
curl -fsS --max-time 5 "http://$WEB_IP:8080/api/health" curl -fsS --max-time 5 "http://$WEB_IP:8080/api/health"