docs(6206): app.py startup comments state the hazard, not the outage
CI & Build / Python lint (push) Successful in 4s
CI & Build / Plugin hooks (push) Successful in 26s
CI & Build / TypeScript typecheck (push) Successful in 1m11s
CI & Build / integration (push) Successful in 1m35s
CI & Build / Python tests (push) Successful in 2m23s
CI & Build / Build & push image (push) Successful in 23s
CI & Build / Python lint (push) Successful in 4s
CI & Build / Plugin hooks (push) Successful in 26s
CI & Build / TypeScript typecheck (push) Successful in 1m11s
CI & Build / integration (push) Successful in 1m35s
CI & Build / Python tests (push) Successful in 2m23s
CI & Build / Build & push image (push) Successful in 23s
The deferred-backfill and inline-release-defaults comments describe the lifespan-overrun hazard they guard without the dated incident; backfill passes say what they fill rather than which milestone added the vectors. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
+20
-23
@@ -187,12 +187,12 @@ def create_app() -> Quart:
|
|||||||
start_notification_loop()
|
start_notification_loop()
|
||||||
start_auth_token_retention_loop()
|
start_auth_token_retention_loop()
|
||||||
|
|
||||||
# Write down any shipped default that moved in this release (#4225).
|
# Write down any shipped default that moved in this release.
|
||||||
#
|
#
|
||||||
# INLINE AND AWAITED, deliberately, against the instinct the block
|
# INLINE AND AWAITED, deliberately, against the instinct the block
|
||||||
# below argues for. What #4181 cost three hours was CONCURRENCY — a
|
# below argues for. The startup hazard is CONCURRENCY — a background
|
||||||
# background task created here racing this hook for the same pool.
|
# task created here racing this hook for the same pool. Running
|
||||||
# Running sequentially creates no such contention, and this is twelve
|
# sequentially creates no such contention, and this is a dozen
|
||||||
# single-row reads on an index against a table with a handful of rows.
|
# single-row reads on an index against a table with a handful of rows.
|
||||||
#
|
#
|
||||||
# It has to finish before serving because the thing it protects is a
|
# It has to finish before serving because the thing it protects is a
|
||||||
@@ -216,16 +216,14 @@ def create_app() -> Quart:
|
|||||||
|
|
||||||
# Backfill embeddings for any notes that don't have one. Runs in the
|
# Backfill embeddings for any notes that don't have one. Runs in the
|
||||||
# background so it never blocks the server from accepting requests —
|
# background so it never blocks the server from accepting requests —
|
||||||
# and, since #4181, not until the rest of this hook has finished.
|
# and not until the rest of this hook has finished.
|
||||||
#
|
#
|
||||||
# "Background" was true of REQUESTS and false of STARTUP. A task
|
# "Background" is true of REQUESTS and false of STARTUP. A task
|
||||||
# created here begins immediately, while `before_serving` is still
|
# created here begins immediately, while `before_serving` is still
|
||||||
# running, so its four passes competed with the startup hook's own
|
# running, so its passes would compete with the startup hook's own
|
||||||
# database reads for the same connection pool. That is survivable on a
|
# database reads for the same connection pool. That is survivable on a
|
||||||
# healthy disk and fatal on a sick one: on 2026-09-19 both this
|
# healthy disk and fatal on a sick one: Hypercorn kills a worker whose
|
||||||
# backfill's first query and the hook's maintenance-hour read were
|
# lifespan hook overruns, and nothing retries it.
|
||||||
# cancelled together when Hypercorn killed the worker at its lifespan
|
|
||||||
# timeout, and the instance stayed down for three hours.
|
|
||||||
#
|
#
|
||||||
# The wait is on the SERVING FLAG rather than a sleep, because a sleep
|
# The wait is on the SERVING FLAG rather than a sleep, because a sleep
|
||||||
# would be a guess about how long the rest of the hook takes and would
|
# would be a guess about how long the rest of the hook takes and would
|
||||||
@@ -236,21 +234,20 @@ def create_app() -> Quart:
|
|||||||
await backfill_note_embeddings()
|
await backfill_note_embeddings()
|
||||||
except Exception:
|
except Exception:
|
||||||
logger.warning("Embedding backfill failed", exc_info=True)
|
logger.warning("Embedding backfill failed", exc_info=True)
|
||||||
# Rules got vectors in milestone 307; every rule written before it
|
# A rule with no vector is unfindable by meaning; this pass fills
|
||||||
# has none, so this is the pass that makes them findable at all.
|
# any that are missing.
|
||||||
try:
|
try:
|
||||||
await backfill_rule_embeddings()
|
await backfill_rule_embeddings()
|
||||||
except Exception:
|
except Exception:
|
||||||
logger.warning("Rule embedding backfill failed", exc_info=True)
|
logger.warning("Rule embedding backfill failed", exc_info=True)
|
||||||
# Milestones got vectors in milestone 415, so a plan written before
|
# Likewise a milestone: a plan with no vector is findable only
|
||||||
# it is findable only after this pass.
|
# after this pass.
|
||||||
try:
|
try:
|
||||||
await backfill_milestone_embeddings()
|
await backfill_milestone_embeddings()
|
||||||
except Exception:
|
except Exception:
|
||||||
logger.warning("Milestone embedding backfill failed", exc_info=True)
|
logger.warning("Milestone embedding backfill failed", exc_info=True)
|
||||||
# Systems got vectors in #4251, so before this pass every charter
|
# A System charter with no vector is unfindable — the System could
|
||||||
# ever written is unfindable — a System could narrow a search and
|
# narrow a search and never be the answer to one.
|
||||||
# never be the answer to one.
|
|
||||||
try:
|
try:
|
||||||
await backfill_system_embeddings()
|
await backfill_system_embeddings()
|
||||||
except Exception:
|
except Exception:
|
||||||
@@ -270,7 +267,7 @@ def create_app() -> Quart:
|
|||||||
# work overlaps startup's own.
|
# work overlaps startup's own.
|
||||||
asyncio.create_task(_delayed_backfill())
|
asyncio.create_task(_delayed_backfill())
|
||||||
|
|
||||||
# RELEASED IN A `finally`, never after the work (rules 156 and 157).
|
# RELEASED IN A `finally`, never after the work.
|
||||||
# The waiter above has no deadline of its own, and the way to make an
|
# The waiter above has no deadline of its own, and the way to make an
|
||||||
# undeadlined wait safe is to make the release unmissable: if anything
|
# undeadlined wait safe is to make the release unmissable: if anything
|
||||||
# below raises, the flag is still set and the task ends instead of
|
# below raises, the flag is still set and the task ends instead of
|
||||||
@@ -292,7 +289,7 @@ def create_app() -> Quart:
|
|||||||
|
|
||||||
# DB maintenance scheduler (daily targeted VACUUM ANALYZE, default
|
# DB maintenance scheduler (daily targeted VACUUM ANALYZE, default
|
||||||
# 04:00 UTC). `get_maintenance_hour` is the FIRST database read in
|
# 04:00 UTC). `get_maintenance_hour` is the FIRST database read in
|
||||||
# this hook and is bounded for that reason (#4181) — see the long
|
# this hook and is bounded for that reason — see the long
|
||||||
# comment above `_STARTUP_READ_TIMEOUT`. Anything added here that
|
# comment above `_STARTUP_READ_TIMEOUT`. Anything added here that
|
||||||
# touches the database needs the same treatment: a lifespan hook
|
# touches the database needs the same treatment: a lifespan hook
|
||||||
# that does not return is a worker that never serves.
|
# that does not return is a worker that never serves.
|
||||||
@@ -304,8 +301,8 @@ def create_app() -> Quart:
|
|||||||
asyncio.get_running_loop(), await get_maintenance_hour()
|
asyncio.get_running_loop(), await get_maintenance_hour()
|
||||||
)
|
)
|
||||||
|
|
||||||
# Scheduled backups (milestone 523). An hourly tick that reads its
|
# Scheduled backups. An hourly tick that reads its config when it
|
||||||
# config when it fires, so starting it reads nothing here (#4181).
|
# fires, so starting it reads nothing here.
|
||||||
from scribe.services.backup_job import start_backup_scheduler
|
from scribe.services.backup_job import start_backup_scheduler
|
||||||
start_backup_scheduler(asyncio.get_running_loop())
|
start_backup_scheduler(asyncio.get_running_loop())
|
||||||
|
|
||||||
@@ -315,7 +312,7 @@ def create_app() -> Quart:
|
|||||||
from scribe.services.diagnostics import start_diagnostics
|
from scribe.services.diagnostics import start_diagnostics
|
||||||
start_diagnostics(asyncio.get_running_loop())
|
start_diagnostics(asyncio.get_running_loop())
|
||||||
finally:
|
finally:
|
||||||
# STARTUP IS OVER — release the backfill (#4181). Everything above
|
# STARTUP IS OVER — release the backfill. Everything above
|
||||||
# is work the server needs done before it serves; everything the
|
# is work the server needs done before it serves; everything the
|
||||||
# backfill does is work that can wait for a server already up.
|
# backfill does is work that can wait for a server already up.
|
||||||
logger.info("Startup complete; releasing deferred backfill")
|
logger.info("Startup complete; releasing deferred backfill")
|
||||||
|
|||||||
Reference in New Issue
Block a user