The operator's 500 was NOT the startup gap I attributed it to. They found it by
logging in and getting Steward's own error page -- which means create_app had
completed and the app was serving, so the gaierror came from a request handler
acquiring a connection, not from boot. Neither prior fix covers that: the
startup retry never runs, and pool_pre_ping only helps once the database is
back, since its replacement connect fails too while the container is gone.
Adds a before_request gate that acquires a pooled connection with a short
bounded retry (~1.75s over 4 tries) so a database restart is ridden out
invisibly, and renders a distinct 503 "database unavailable" page when the
budget is exhausted. The request budget is deliberately far shorter than the
startup one: nobody watches a container boot, but somebody is watching this
page load, and a page that hangs is worse than one that says what is wrong.
/health and static are exempt -- a liveness probe must stay answerable while
the database is down, or a dependency outage triggers a restart loop.
Credential and missing-database errors still propagate rather than being
reported as "unavailable", which would send the operator chasing the wrong
problem entirely.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two connection-recovery gaps, both surfaced by a Postgres restart that left
the app throwing tracebacks while the DB itself was healthy.
pool_pre_ping + pool_recycle on the app engine [#2626]: when the database
restarts, every connection already in the pool is dead at the socket level.
SQLAlchemy only discovered that by failing a real query, so the first
operation after a restart errored out on whatever triggered it. pre_ping
checks liveness on checkout and swaps the dead connection transparently;
pool_recycle caps connection age so a socket stranded by a NAT/conntrack
timeout or a Docker network rebuild is retired on a timer instead.
wait_for_database() gate at startup [#2627]: create_app touches the DB
synchronously (migrations, secret re-encryption, settings load) and assumed
it was both resolvable and accepting connections on the first try. Neither
holds after a host reboot (Docker DNS not yet serving `db` -> gaierror -2)
or an unclean shutdown (Postgres still replaying WAL -> "not yet accepting
connections"). Both are transient, so retry with capped backoff behind one
gate ahead of the first DB touch. Credential and missing-database errors
are classified by SQLSTATE and still fail immediately -- waiting cannot fix
those. Budget is bootstrap-configurable (STEWARD_DB_CONNECT_TIMEOUT /
database.connect_timeout, default 60s) since it governs reaching the DB and
so cannot live in the DB-backed settings.
Tests drive the retry loop off a fake clock, so backoff and timeout
behaviour are deterministic rather than wall-clock dependent.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Renames the Python package directory, CLI command, env var prefix,
docker-compose service/container/image, Postgres role/db, and all
visible branding. Marketing form is "Fabled Steward".
Clean break from the previous rebrand: drops the fabledscryer→roundtable
import shim in __init__.py and the FABLEDSCRYER_* env var fallback in
config.py and migrations/env.py. Env vars are now STEWARD_* only.
Heads-up for existing deployments:
- Postgres user/db renamed fabledscryer → steward in docker-compose.yml.
Existing volumes need the role/db renamed inside Postgres, or override
POSTGRES_USER/POSTGRES_DB to keep the old names.
- Host-agent systemd unit is now steward-agent.service. Existing agents
keep running under the old name; reinstall to switch.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>