ci: the smoke gate found a real one on its first run — and had two bugs of its own
Build images / sign-extension (push) Successful in 6s
CI / lint (push) Successful in 6s
CI / extension-version (push) Successful in 4s
Build images / build-ml (push) Successful in 11s
Build images / build-agent (push) Successful in 13s
extension / lint (push) Successful in 24s
CI / frontend-build (push) Successful in 28s
CI / backend-lint-and-test (push) Successful in 35s
Build images / build-web (push) Successful in 8s
Build images / smoke-web (push) Skipped
CI / integration (push) Successful in 2m5s
Build images / sign-extension (push) Successful in 6s
CI / lint (push) Successful in 6s
CI / extension-version (push) Successful in 4s
Build images / build-ml (push) Successful in 11s
Build images / build-agent (push) Successful in 13s
extension / lint (push) Successful in 24s
CI / frontend-build (push) Successful in 28s
CI / backend-lint-and-test (push) Successful in 35s
Build images / build-web (push) Successful in 8s
Build images / smoke-web (push) Skipped
CI / integration (push) Successful in 2m5s
Run 5296 was `smoke-web`'s first genuine execution. Checks 1 and 2 passed: alembic built the schema from empty inside the image, all five apt binaries resolved, and the application's own Thumbnailer produced JPEG, PNG-with-alpha, WebP and an ffmpeg video frame against the image's libraries. Check 3 failed, and the trap's log dump said exactly why: MissingCredentialKey: Fernet key file not found at /images/secrets/credential_key.b64. For first-time setup, set CURATOR_BOOTSTRAP_NEW_KEY=1. That is the product being right. credential_crypto refuses to mint a key unless someone opts in, because the 2026-06-02 audit found a partial restore (DB back, /images/secrets/ lost) silently generating a fresh one and leaving a working-looking system where every authenticated download failed AUTH_ERROR. It is also a first-run blocker for milestone 328, filed as #3422: the variable appears in no README, no .env.example and no compose file, so the install path that milestone just finished writing produces a container that exits on boot. Not fixed here — the fix trades safety against friction and is the operator's call. Two defects in the gate itself, both surfaced by the same run: - A throwaway CI instance IS first-time setup, so it now passes CURATOR_BOOTSTRAP_NEW_KEY=1. The check was asserting a condition no fresh container can satisfy. - The health loop polled a dead container for 3m35s. Docker had already recycled its IP, so the replies were a baffling mix of connection-refused and 5s timeouts from whatever took the address next. It now checks `.State.Running` each iteration and fails immediately with the container's log. The trap had the real answer the whole time; this stops burying it under four minutes of noise. Also corrected a message claiming a 120s budget: 60 iterations of up to 5s connect plus 2s sleep is nearer seven minutes. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TTjbZZ6JirCMSaJzQV1RhA
This commit is contained in:
@@ -1213,6 +1213,14 @@ jobs:
|
|||||||
ENVOPTS="$ENVOPTS -e DB_PORT=5432 -e DB_NAME=$DB_NAME -e SECRET_KEY=$SECRET_KEY"
|
ENVOPTS="$ENVOPTS -e DB_PORT=5432 -e DB_NAME=$DB_NAME -e SECRET_KEY=$SECRET_KEY"
|
||||||
ENVOPTS="$ENVOPTS -e CELERY_BROKER_URL=redis://$RD_IP:6379/0"
|
ENVOPTS="$ENVOPTS -e CELERY_BROKER_URL=redis://$RD_IP:6379/0"
|
||||||
ENVOPTS="$ENVOPTS -e CELERY_RESULT_BACKEND=redis://$RD_IP:6379/0"
|
ENVOPTS="$ENVOPTS -e CELERY_RESULT_BACKEND=redis://$RD_IP:6379/0"
|
||||||
|
# A throwaway CI instance IS first-time setup, which is the one case
|
||||||
|
# credential_crypto allows a key to be minted in. Without it the web
|
||||||
|
# role refuses to boot — deliberately, since silently generating a
|
||||||
|
# key on a restored-DB-but-lost-secrets deployment would leave every
|
||||||
|
# Credential row undecryptable (the 2026-06-02 audit). Discovered by
|
||||||
|
# this job on its first real run; see #3422 for the fact that no
|
||||||
|
# user-facing file mentions this variable at all.
|
||||||
|
ENVOPTS="$ENVOPTS -e CURATOR_BOOTSTRAP_NEW_KEY=1"
|
||||||
|
|
||||||
# 1. The schema builds from empty, using the image's OWN libpq and
|
# 1. The schema builds from empty, using the image's OWN libpq and
|
||||||
# psycopg. This is the same call entrypoint.sh makes before it
|
# psycopg. This is the same call entrypoint.sh makes before it
|
||||||
@@ -1247,12 +1255,26 @@ jobs:
|
|||||||
healthy=1
|
healthy=1
|
||||||
break
|
break
|
||||||
fi
|
fi
|
||||||
|
# A container that has EXITED will never answer, so stop asking.
|
||||||
|
# Without this the loop spent 3m35s polling a dead container on
|
||||||
|
# this job's first run, and — because docker recycles the IP — got
|
||||||
|
# a confusing mix of connection-refused and 5s timeouts from
|
||||||
|
# whatever took the address next. The trap's log dump had the real
|
||||||
|
# answer the whole time; this just stops burying it.
|
||||||
|
if [ "$(docker inspect -f '{{.State.Running}}' "$CID" 2>/dev/null)" != "true" ]; then
|
||||||
|
echo "smoke: FAILED — the web container exited during boot." >&2
|
||||||
|
echo "smoke: its log follows; entrypoint runs alembic BEFORE" >&2
|
||||||
|
echo "smoke: serving, so a startup exception lands here." >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
sleep 2
|
sleep 2
|
||||||
done
|
done
|
||||||
if [ -z "$healthy" ]; then
|
if [ -z "$healthy" ]; then
|
||||||
echo "smoke: FAILED — web did not answer /api/health within 120s." >&2
|
# 60 iterations of (up to 5s connect + 2s sleep) — up to ~7min, not
|
||||||
echo "smoke: entrypoint runs alembic BEFORE serving, and step 1" >&2
|
# the 120s an earlier version of this message claimed.
|
||||||
echo "smoke: passed, so look at hypercorn and the python base." >&2
|
echo "smoke: FAILED — web is running but never answered" >&2
|
||||||
|
echo "smoke: /api/health. It is up, so look at hypercorn and the" >&2
|
||||||
|
echo "smoke: python base rather than at startup." >&2
|
||||||
exit 1
|
exit 1
|
||||||
fi
|
fi
|
||||||
curl -fsS --max-time 5 "http://$WEB_IP:8080/api/health"
|
curl -fsS --max-time 5 "http://$WEB_IP:8080/api/health"
|
||||||
|
|||||||
Reference in New Issue
Block a user