docs(6206): app.py startup comments state the hazard, not the outage
CI & Build / Python lint (push) Successful in 4s
CI & Build / Plugin hooks (push) Successful in 26s
CI & Build / TypeScript typecheck (push) Successful in 1m11s
CI & Build / integration (push) Successful in 1m35s
CI & Build / Python tests (push) Successful in 2m23s
CI & Build / Build & push image (push) Successful in 23s

The deferred-backfill and inline-release-defaults comments describe the
lifespan-overrun hazard they guard without the dated incident; backfill
passes say what they fill rather than which milestone added the vectors.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-10-21 20:38:30 -04:00
co-authored by Claude Opus 5.5
parent cad74df190
commit 0c237172e4
+20 -23
View File
@@ -187,12 +187,12 @@ def create_app() -> Quart:
start_notification_loop()
start_auth_token_retention_loop()
# Write down any shipped default that moved in this release (#4225).
# Write down any shipped default that moved in this release.
#
# INLINE AND AWAITED, deliberately, against the instinct the block
# below argues for. What #4181 cost three hours was CONCURRENCY — a
# background task created here racing this hook for the same pool.
# Running sequentially creates no such contention, and this is twelve
# below argues for. The startup hazard is CONCURRENCY — a background
# task created here racing this hook for the same pool. Running
# sequentially creates no such contention, and this is a dozen
# single-row reads on an index against a table with a handful of rows.
#
# It has to finish before serving because the thing it protects is a
@@ -216,16 +216,14 @@ def create_app() -> Quart:
# Backfill embeddings for any notes that don't have one. Runs in the
# background so it never blocks the server from accepting requests —
# and, since #4181, not until the rest of this hook has finished.
# and not until the rest of this hook has finished.
#
# "Background" was true of REQUESTS and false of STARTUP. A task
# "Background" is true of REQUESTS and false of STARTUP. A task
# created here begins immediately, while `before_serving` is still
# running, so its four passes competed with the startup hook's own
# running, so its passes would compete with the startup hook's own
# database reads for the same connection pool. That is survivable on a
# healthy disk and fatal on a sick one: on 2026-09-19 both this
# backfill's first query and the hook's maintenance-hour read were
# cancelled together when Hypercorn killed the worker at its lifespan
# timeout, and the instance stayed down for three hours.
# healthy disk and fatal on a sick one: Hypercorn kills a worker whose
# lifespan hook overruns, and nothing retries it.
#
# The wait is on the SERVING FLAG rather than a sleep, because a sleep
# would be a guess about how long the rest of the hook takes and would
@@ -236,21 +234,20 @@ def create_app() -> Quart:
await backfill_note_embeddings()
except Exception:
logger.warning("Embedding backfill failed", exc_info=True)
# Rules got vectors in milestone 307; every rule written before it
# has none, so this is the pass that makes them findable at all.
# A rule with no vector is unfindable by meaning; this pass fills
# any that are missing.
try:
await backfill_rule_embeddings()
except Exception:
logger.warning("Rule embedding backfill failed", exc_info=True)
# Milestones got vectors in milestone 415, so a plan written before
# it is findable only after this pass.
# Likewise a milestone: a plan with no vector is findable only
# after this pass.
try:
await backfill_milestone_embeddings()
except Exception:
logger.warning("Milestone embedding backfill failed", exc_info=True)
# Systems got vectors in #4251, so before this pass every charter
# ever written is unfindable — a System could narrow a search and
# never be the answer to one.
# A System charter with no vector is unfindable — the System could
# narrow a search and never be the answer to one.
try:
await backfill_system_embeddings()
except Exception:
@@ -270,7 +267,7 @@ def create_app() -> Quart:
# work overlaps startup's own.
asyncio.create_task(_delayed_backfill())
# RELEASED IN A `finally`, never after the work (rules 156 and 157).
# RELEASED IN A `finally`, never after the work.
# The waiter above has no deadline of its own, and the way to make an
# undeadlined wait safe is to make the release unmissable: if anything
# below raises, the flag is still set and the task ends instead of
@@ -292,7 +289,7 @@ def create_app() -> Quart:
# DB maintenance scheduler (daily targeted VACUUM ANALYZE, default
# 04:00 UTC). `get_maintenance_hour` is the FIRST database read in
# this hook and is bounded for that reason (#4181) — see the long
# this hook and is bounded for that reason — see the long
# comment above `_STARTUP_READ_TIMEOUT`. Anything added here that
# touches the database needs the same treatment: a lifespan hook
# that does not return is a worker that never serves.
@@ -304,8 +301,8 @@ def create_app() -> Quart:
asyncio.get_running_loop(), await get_maintenance_hour()
)
# Scheduled backups (milestone 523). An hourly tick that reads its
# config when it fires, so starting it reads nothing here (#4181).
# Scheduled backups. An hourly tick that reads its config when it
# fires, so starting it reads nothing here.
from scribe.services.backup_job import start_backup_scheduler
start_backup_scheduler(asyncio.get_running_loop())
@@ -315,7 +312,7 @@ def create_app() -> Quart:
from scribe.services.diagnostics import start_diagnostics
start_diagnostics(asyncio.get_running_loop())
finally:
# STARTUP IS OVER — release the backfill (#4181). Everything above
# STARTUP IS OVER — release the backfill. Everything above
# is work the server needs done before it serves; everything the
# backfill does is work that can wait for a server already up.
logger.info("Startup complete; releasing deferred backfill")