Run 7368's smoke, on an image whose five verification lanes were all green:
WARNING Generating NEW Fernet credential key at
/images/secrets/credential_key.b64
...
ValueError: Fernet key must be 32 url-safe base64-encoded bytes.
Nothing to do with this batch's changes — it is a first-boot race that has
been there since the key file existed, and it is a RACE rather than a
certainty: the same code booted cleanly on the three runs before it.
hypercorn starts several worker processes and each one builds the app, so on a
first boot they all reach the bootstrap together. `write_bytes` creates the
file at size zero and fills it a moment later, which gives the second process
an `exists()` of True and a `read_bytes()` of `b""`.
A first boot that fails one time in five is worse than one that fails every
time, because it looks like the deployment rather than the code — and this is
the very first thing a new install does.
The key is now written to a temp file and `os.link`ed into place. `os.link` is
the atomic part: it either creates the name or raises FileExistsError, and it
cannot expose a half-written file. Deliberately NOT `os.replace`, which would
succeed — two processes that both generated a key would each think they had
won, and the loser's key would overwrite the one the winner had already handed
to Fernet. The losing branch reads the winner's key back rather than returning
its own, which is what keeps every worker on ONE key.
Tested for AGREEMENT, not for the absence of a crash: eight threads through a
barrier, and all eight must end up holding the same key. A race that left each
worker with its own would pass a "did it raise" check and produce a system
where a credential written by one worker cannot be read by the next.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR
3422 was already fixed. Commit 86abaf0 applied option 1 in full and it is on main: .env.example carries the bootstrap section with the backup warning, README explains the refusal and why it is deliberate, docker-compose forwards the variable, and the milestone-362 smoke gate that FOUND the bug now sets it (build.yml:1159) and passes. The issue's premise - "CURATOR_BOOTSTRAP_NEW_KEY appears nowhere outside backend/" - is stale.
What was left is the dependency that fix created. README.md and .env.example now both print the literal error text, the literal key path and the variable name, because a stranger greps for the string their terminal showed them. That is the right call and it means two user-facing files now depend on this module's wording with nothing connecting them - the install surface's characteristic defect, one rename away from a README that sends strangers to a path that does not exist.
Three guards, all presence checks on both sides. An absence check against prose would pass for the wrong reason the moment a sentence were reworded (snippet 3352):
- the raised message still contains the sentence README reproduces, the variable both docs say to set, and the restore-rather-than-mint alternative the whole refusal rests on;
- both docs still name _CREDENTIAL_KEY_PATH and the variable, read from the code rather than retyped, so a rename fails here;
- compose still forwards the variable - without that line the docs' "set it in .env" is silently inert and fails identically to not setting it.
Option 2 (mint when the credential table is empty) is deliberately NOT done. The issue's own guidance is "(1) now, (2) if the friction proves annoying", and the friction has not been reported. Worth recording that its predicate checks out exactly: the Fernet key protects Credential.encrypted_blob and nothing else - no other Fernet user exists - so "no credential rows means nothing can be made undecryptable" is provable rather than probable. The cost is placement: create_app() is sync and constructs the key before any engine exists, so the check cannot live where the failure is. entrypoint.sh, which already runs alembic against the DB, is the natural seam. Only the web role is affected; the Celery roles build the key lazily inside tasks.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SHQB1YukL3VyvMK8rcbmV9
Audit 2026-06-02: `_load_or_create_key` silently minted a new Fernet
key whenever the key file was missing — no log, no warning. The
failure mode the audit flagged: a partial disaster restore where the
DB was restored but `/images/secrets/` was lost would produce a
working-looking system in which every authenticated download fails
AUTH_ERROR until the operator re-uploads every credential by hand.
Two opt-ins now needed for auto-creation:
1. Explicit `bootstrap_ok=True` kwarg (tests, scripts), OR
2. `CURATOR_BOOTSTRAP_NEW_KEY=1` env var (operator first-time setup)
Otherwise the constructor raises `MissingCredentialKey` so the app
fails fast at startup and the operator can restore the key file
from backup before encrypted_blob rows go undecryptable.
Also: docstring path was wrong (said "images/data root" but actual
location is `/images/secrets/credential_key.b64`) — corrected.
Tests updated to pass `bootstrap_ok=True` explicitly, and two new
tests cover the safety behavior (missing-key-raises, env-var-bootstraps).
bvandeusenandClaude Opus 5