fix(startup): a slow database costs seconds, not the instance (#4181)

On 2026-09-19 a host storage stall made one Postgres checkpoint of 14
buffers take 281 seconds against a 1.3-second baseline. The app restarted
into the tail of it, `get_maintenance_hour()` — the first DB read in
`before_serving` — hung with no deadline, Hypercorn killed the worker at
its 60-second lifespan timeout, and nothing retries a failed lifespan. A
five-minute disk hiccup became a three-hour outage that only a human
restart could clear. Every MCP call returned 405, which reads like a
routing fault and was nothing of the kind: nothing was serving.

Three changes, none of which prevent a stall — they stop a transient one
becoming a permanent one.

1. THE STARTUP READ IS BOUNDED (rule 156). `get_maintenance_hour` already
   answered `_DEFAULT_HOUR` for a value it could not parse; a database
   that will not answer in three seconds is the same class of "no usable
   value here". The failure is now a WARNING naming the symptom — the
   breadcrumb whose absence meant this was only diagnosable from
   Postgres's own log — and a default run-hour, instead of the app.

2. THE BACKFILL NO LONGER RACES STARTUP. Its comment said it "never
   blocks the server from accepting requests": true of requests, false of
   startup, because the task began while `before_serving` was still
   running and competed for the same pool. Both of the incident's
   cancelled statements were in flight together. It now waits on a flag
   released on the hook's way out — in a `finally`, never after the work
   (rule 157), because an undeadlined wait is only safe when the wake-up
   cannot be missed.

3. THE ENGINE CANNOT WAIT FOREVER TO CONNECT. asyncpg's default is 60s,
   the whole lifespan budget spent before a query is sent. `command_timeout`
   is deliberately NOT set alongside it and the comment says why: it would
   apply to every statement, and this app runs long ones on purpose.

tests/test_startup_survives_a_slow_database.py asserts the shape rather
than the stall: a read that never returns still yields an hour, the
warning names the symptom, a healthy read is unaffected, the backfill
does no work before release, the flag is released even when startup
raises, and the engine's connect args carry a deadline but no blanket
statement timeout.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01821k5B3Ysecp9fNYs92Kuy

src/scribe/app.py

@@ -162,6 +162,13 @@ def create_app() -> Quart:
     async def startup():
         import asyncio
 
+        # Set on the last line of this hook; the deferred backfill below waits
+        # on it. An Event rather than a bare bool so the waiter is woken
+        # instead of polling, and declared here — inside the hook, where
+        # `asyncio` is in scope and a running loop exists — rather than in
+        # `create_app`, which runs before either is true.
+        _startup_finished = asyncio.Event()
+
         from scribe.services.auth import start_auth_token_retention_loop
         from scribe.services.embeddings import (
             backfill_milestone_embeddings, backfill_note_embeddings, backfill_rule_embeddings,
@@ -174,8 +181,23 @@ def create_app() -> Quart:
         start_auth_token_retention_loop()
 
         # Backfill embeddings for any notes that don't have one. Runs in the
-        # background so it never blocks the server from accepting requests.
+        # background so it never blocks the server from accepting requests —
+        # and, since #4181, not until the rest of this hook has finished.
+        #
+        # "Background" was true of REQUESTS and false of STARTUP. A task
+        # created here begins immediately, while `before_serving` is still
+        # running, so its four passes competed with the startup hook's own
+        # database reads for the same connection pool. That is survivable on a
+        # healthy disk and fatal on a sick one: on 2026-09-19 both this
+        # backfill's first query and the hook's maintenance-hour read were
+        # cancelled together when Hypercorn killed the worker at its lifespan
+        # timeout, and the instance stayed down for three hours.
+        #
+        # The wait is on the SERVING FLAG rather than a sleep, because a sleep
+        # would be a guess about how long the rest of the hook takes and would
+        # be wrong in exactly the conditions that matter.
         async def _delayed_backfill() -> None:
+            await _startup_finished.wait()
             try:
                 await backfill_note_embeddings()
             except Exception:
@@ -202,36 +224,56 @@ def create_app() -> Quart:
             except Exception:
                 logger.warning("Snippet data backfill failed", exc_info=True)
 
+        # Created here but gated on the flag released at the END of this hook,
+        # so the task exists (nothing can forget to start it) while none of its
+        # work overlaps startup's own.
         asyncio.create_task(_delayed_backfill())
 
-        # Recurrence scheduler (recurring-task spawn every 15m)
-        from scribe.services.recurrence_scheduler import start_recurrence_scheduler
-        start_recurrence_scheduler(asyncio.get_running_loop())
+        # RELEASED IN A `finally`, never after the work (rules 156 and 157).
+        # The waiter above has no deadline of its own, and the way to make an
+        # undeadlined wait safe is to make the release unmissable: if anything
+        # below raises, the flag is still set and the task ends instead of
+        # living on as a coroutine nobody will ever wake.
+        try:
+            # Recurrence scheduler (recurring-task spawn every 15m)
+            from scribe.services.recurrence_scheduler import start_recurrence_scheduler
+            start_recurrence_scheduler(asyncio.get_running_loop())
 
-        # Version-pinning scheduler (daily auto-pin scan at 03:00 UTC)
-        from scribe.services.version_pinning_scheduler import (
-            start_version_pinning_scheduler,
-        )
-        start_version_pinning_scheduler(asyncio.get_running_loop())
+            # Version-pinning scheduler (daily auto-pin scan at 03:00 UTC)
+            from scribe.services.version_pinning_scheduler import (
+                start_version_pinning_scheduler,
+            )
+            start_version_pinning_scheduler(asyncio.get_running_loop())
 
-        # Trash retention scheduler (daily expired-trash purge at 03:30 UTC)
-        from scribe.services.trash_scheduler import start_trash_scheduler
-        start_trash_scheduler(asyncio.get_running_loop())
+            # Trash retention scheduler (daily expired-trash purge at 03:30 UTC)
+            from scribe.services.trash_scheduler import start_trash_scheduler
+            start_trash_scheduler(asyncio.get_running_loop())
 
-        # DB maintenance scheduler (daily targeted VACUUM ANALYZE, default 04:00 UTC)
-        from scribe.services.db_maintenance_scheduler import (
-            get_maintenance_hour,
-            start_db_maintenance_scheduler,
-        )
-        start_db_maintenance_scheduler(
-            asyncio.get_running_loop(), await get_maintenance_hour()
-        )
+            # DB maintenance scheduler (daily targeted VACUUM ANALYZE, default
+            # 04:00 UTC). `get_maintenance_hour` is the FIRST database read in
+            # this hook and is bounded for that reason (#4181) — see the long
+            # comment above `_STARTUP_READ_TIMEOUT`. Anything added here that
+            # touches the database needs the same treatment: a lifespan hook
+            # that does not return is a worker that never serves.
+            from scribe.services.db_maintenance_scheduler import (
+                get_maintenance_hour,
+                start_db_maintenance_scheduler,
+            )
+            start_db_maintenance_scheduler(
+                asyncio.get_running_loop(), await get_maintenance_hour()
+            )
 
-        # Diagnostic instrumentation — heartbeat, signal handlers, asyncio
-        # exception hook. Cheap (~1 log line/min), high diagnostic value when
-        # the app crashes mysteriously. See services/diagnostics.py.
-        from scribe.services.diagnostics import start_diagnostics
-        start_diagnostics(asyncio.get_running_loop())
+            # Diagnostic instrumentation — heartbeat, signal handlers, asyncio
+            # exception hook. Cheap (~1 log line/min), high diagnostic value when
+            # the app crashes mysteriously. See services/diagnostics.py.
+            from scribe.services.diagnostics import start_diagnostics
+            start_diagnostics(asyncio.get_running_loop())
+        finally:
+            # STARTUP IS OVER — release the backfill (#4181). Everything above
+            # is work the server needs done before it serves; everything the
+            # backfill does is work that can wait for a server already up.
+            logger.info("Startup complete; releasing deferred backfill")
+            _startup_finished.set()
 
     @app.after_serving
     async def shutdown():