docs(6206): app.py startup comments state the hazard, not the outage
CI & Build / Python lint (push) Successful in 4s
CI & Build / Plugin hooks (push) Successful in 26s
CI & Build / TypeScript typecheck (push) Successful in 1m11s
CI & Build / integration (push) Successful in 1m35s
CI & Build / Python tests (push) Successful in 2m23s
CI & Build / Build & push image (push) Successful in 23s
CI & Build / Python lint (push) Successful in 4s
CI & Build / Plugin hooks (push) Successful in 26s
CI & Build / TypeScript typecheck (push) Successful in 1m11s
CI & Build / integration (push) Successful in 1m35s
CI & Build / Python tests (push) Successful in 2m23s
CI & Build / Build & push image (push) Successful in 23s
The deferred-backfill and inline-release-defaults comments describe the lifespan-overrun hazard they guard without the dated incident; backfill passes say what they fill rather than which milestone added the vectors. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
+20
-23
@@ -187,12 +187,12 @@ def create_app() -> Quart:
|
||||
start_notification_loop()
|
||||
start_auth_token_retention_loop()
|
||||
|
||||
# Write down any shipped default that moved in this release (#4225).
|
||||
# Write down any shipped default that moved in this release.
|
||||
#
|
||||
# INLINE AND AWAITED, deliberately, against the instinct the block
|
||||
# below argues for. What #4181 cost three hours was CONCURRENCY — a
|
||||
# background task created here racing this hook for the same pool.
|
||||
# Running sequentially creates no such contention, and this is twelve
|
||||
# below argues for. The startup hazard is CONCURRENCY — a background
|
||||
# task created here racing this hook for the same pool. Running
|
||||
# sequentially creates no such contention, and this is a dozen
|
||||
# single-row reads on an index against a table with a handful of rows.
|
||||
#
|
||||
# It has to finish before serving because the thing it protects is a
|
||||
@@ -216,16 +216,14 @@ def create_app() -> Quart:
|
||||
|
||||
# Backfill embeddings for any notes that don't have one. Runs in the
|
||||
# background so it never blocks the server from accepting requests —
|
||||
# and, since #4181, not until the rest of this hook has finished.
|
||||
# and not until the rest of this hook has finished.
|
||||
#
|
||||
# "Background" was true of REQUESTS and false of STARTUP. A task
|
||||
# "Background" is true of REQUESTS and false of STARTUP. A task
|
||||
# created here begins immediately, while `before_serving` is still
|
||||
# running, so its four passes competed with the startup hook's own
|
||||
# running, so its passes would compete with the startup hook's own
|
||||
# database reads for the same connection pool. That is survivable on a
|
||||
# healthy disk and fatal on a sick one: on 2026-09-19 both this
|
||||
# backfill's first query and the hook's maintenance-hour read were
|
||||
# cancelled together when Hypercorn killed the worker at its lifespan
|
||||
# timeout, and the instance stayed down for three hours.
|
||||
# healthy disk and fatal on a sick one: Hypercorn kills a worker whose
|
||||
# lifespan hook overruns, and nothing retries it.
|
||||
#
|
||||
# The wait is on the SERVING FLAG rather than a sleep, because a sleep
|
||||
# would be a guess about how long the rest of the hook takes and would
|
||||
@@ -236,21 +234,20 @@ def create_app() -> Quart:
|
||||
await backfill_note_embeddings()
|
||||
except Exception:
|
||||
logger.warning("Embedding backfill failed", exc_info=True)
|
||||
# Rules got vectors in milestone 307; every rule written before it
|
||||
# has none, so this is the pass that makes them findable at all.
|
||||
# A rule with no vector is unfindable by meaning; this pass fills
|
||||
# any that are missing.
|
||||
try:
|
||||
await backfill_rule_embeddings()
|
||||
except Exception:
|
||||
logger.warning("Rule embedding backfill failed", exc_info=True)
|
||||
# Milestones got vectors in milestone 415, so a plan written before
|
||||
# it is findable only after this pass.
|
||||
# Likewise a milestone: a plan with no vector is findable only
|
||||
# after this pass.
|
||||
try:
|
||||
await backfill_milestone_embeddings()
|
||||
except Exception:
|
||||
logger.warning("Milestone embedding backfill failed", exc_info=True)
|
||||
# Systems got vectors in #4251, so before this pass every charter
|
||||
# ever written is unfindable — a System could narrow a search and
|
||||
# never be the answer to one.
|
||||
# A System charter with no vector is unfindable — the System could
|
||||
# narrow a search and never be the answer to one.
|
||||
try:
|
||||
await backfill_system_embeddings()
|
||||
except Exception:
|
||||
@@ -270,7 +267,7 @@ def create_app() -> Quart:
|
||||
# work overlaps startup's own.
|
||||
asyncio.create_task(_delayed_backfill())
|
||||
|
||||
# RELEASED IN A `finally`, never after the work (rules 156 and 157).
|
||||
# RELEASED IN A `finally`, never after the work.
|
||||
# The waiter above has no deadline of its own, and the way to make an
|
||||
# undeadlined wait safe is to make the release unmissable: if anything
|
||||
# below raises, the flag is still set and the task ends instead of
|
||||
@@ -292,7 +289,7 @@ def create_app() -> Quart:
|
||||
|
||||
# DB maintenance scheduler (daily targeted VACUUM ANALYZE, default
|
||||
# 04:00 UTC). `get_maintenance_hour` is the FIRST database read in
|
||||
# this hook and is bounded for that reason (#4181) — see the long
|
||||
# this hook and is bounded for that reason — see the long
|
||||
# comment above `_STARTUP_READ_TIMEOUT`. Anything added here that
|
||||
# touches the database needs the same treatment: a lifespan hook
|
||||
# that does not return is a worker that never serves.
|
||||
@@ -304,8 +301,8 @@ def create_app() -> Quart:
|
||||
asyncio.get_running_loop(), await get_maintenance_hour()
|
||||
)
|
||||
|
||||
# Scheduled backups (milestone 523). An hourly tick that reads its
|
||||
# config when it fires, so starting it reads nothing here (#4181).
|
||||
# Scheduled backups. An hourly tick that reads its config when it
|
||||
# fires, so starting it reads nothing here.
|
||||
from scribe.services.backup_job import start_backup_scheduler
|
||||
start_backup_scheduler(asyncio.get_running_loop())
|
||||
|
||||
@@ -315,7 +312,7 @@ def create_app() -> Quart:
|
||||
from scribe.services.diagnostics import start_diagnostics
|
||||
start_diagnostics(asyncio.get_running_loop())
|
||||
finally:
|
||||
# STARTUP IS OVER — release the backfill (#4181). Everything above
|
||||
# STARTUP IS OVER — release the backfill. Everything above
|
||||
# is work the server needs done before it serves; everything the
|
||||
# backfill does is work that can wait for a server already up.
|
||||
logger.info("Startup complete; releasing deferred backfill")
|
||||
|
||||
Reference in New Issue
Block a user