docs(6209): wide net, startup, stamps, slippage, miss route and parity tests
CI & Build / Python lint (push) Successful in 3s
CI & Build / Plugin hooks (push) Successful in 15s
CI & Build / TypeScript typecheck (push) Successful in 53s
CI & Build / integration (push) Successful in 1m10s
CI & Build / Build & push image (push) Canceled after 0s
CI & Build / Python tests (push) Canceled after 1m23s
CI & Build / Python lint (push) Successful in 3s
CI & Build / Plugin hooks (push) Successful in 15s
CI & Build / TypeScript typecheck (push) Successful in 53s
CI & Build / integration (push) Successful in 1m10s
CI & Build / Build & push image (push) Canceled after 0s
CI & Build / Python tests (push) Canceled after 1m23s
Eight more test files state the property they pin as a present-tense fact. Incident narratives (a dated storage stall, a week of zero tuning events, a two-project poisoned ledger, a twenty-site owner audit) become the failure shape each test defends; one-install figures that name a rule become a general worked example with the calibration numbers kept. The family-grouping regression guard is renamed for what it checks. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -1,16 +1,13 @@
|
||||
"""A sick database must not stop the app from starting — #4181, rule 156.
|
||||
"""A sick database must not stop the app from starting.
|
||||
|
||||
THE INCIDENT THESE GUARD, so a later reader knows what is being defended:
|
||||
THE FAILURE THESE GUARD. A storage stall can make Postgres answer in minutes
|
||||
instead of milliseconds. If the app starts into one, a startup read with no
|
||||
deadline hangs; Hypercorn kills the worker at its 60-second lifespan timeout;
|
||||
and nothing retries a failed lifespan. A short disk hiccup becomes an outage
|
||||
that only a human restart can clear.
|
||||
|
||||
On 2026-09-19 a host storage stall made one Postgres checkpoint of 14 buffers
|
||||
take 281 seconds against a 1.3-second baseline. The app restarted into the tail
|
||||
of it; `get_maintenance_hour()` — the first database read in `before_serving` —
|
||||
hung with no deadline; Hypercorn killed the worker at its 60-second lifespan
|
||||
timeout; and nothing retries a failed lifespan. A five-minute disk hiccup became
|
||||
a three-hour outage that only a human restart could clear.
|
||||
|
||||
Every assertion here is about the SHAPE that made that possible, not about the
|
||||
stall, which no test can reproduce and no code can prevent:
|
||||
Every assertion here is about the SHAPE that makes that possible, not about
|
||||
the stall, which no test can reproduce and no code can prevent:
|
||||
|
||||
- the startup read returns on a schedule of its own rather than the
|
||||
database's;
|
||||
@@ -48,9 +45,9 @@ async def test_a_hanging_settings_read_does_not_hang_startup():
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_the_timeout_is_logged_loudly_enough_to_find():
|
||||
"""#4181 cost hours because the app logged three scheduler lines and
|
||||
stopped; the cause was only recoverable from Postgres's own log. A WARNING
|
||||
here is the breadcrumb that would have named it in seconds."""
|
||||
"""Without a WARNING the app logs a few scheduler lines and stops, and the
|
||||
cause is only recoverable from Postgres's own log. This is the breadcrumb
|
||||
that names it."""
|
||||
from scribe.services import db_maintenance_scheduler as sched
|
||||
|
||||
async def _never_returns(*_a, **_kw):
|
||||
@@ -60,7 +57,7 @@ async def test_the_timeout_is_logged_loudly_enough_to_find():
|
||||
patch.object(sched, "_STARTUP_READ_TIMEOUT", 0.05), \
|
||||
patch.object(sched.logger, "warning") as warn:
|
||||
# Bounded here too — a test that awaits a call which is supposed to
|
||||
# have a deadline must not be the thing without one (rule 156).
|
||||
# have a deadline must not be the thing without one.
|
||||
await asyncio.wait_for(sched.get_maintenance_hour(), timeout=2)
|
||||
|
||||
assert warn.called
|
||||
@@ -105,11 +102,9 @@ def test_the_startup_deadline_leaves_room_inside_the_lifespan_budget():
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_the_deferred_backfill_waits_for_the_startup_flag():
|
||||
"""`_delayed_backfill`'s comment said it "never blocks the server from
|
||||
accepting requests" — true of requests, false of startup, because startup
|
||||
had not finished and the two competed for one connection pool. Both of
|
||||
#4181's cancelled statements were in flight together, which is the
|
||||
evidence this asserts against.
|
||||
""""Never blocks the server from accepting requests" is true of requests
|
||||
and false of startup: a backfill that starts before startup finishes
|
||||
competes with it for one connection pool.
|
||||
|
||||
Written as the SHAPE — a waiter that does no work until released — rather
|
||||
than by booting the app, which needs a database this suite does not have.
|
||||
@@ -133,7 +128,7 @@ async def test_the_deferred_backfill_waits_for_the_startup_flag():
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_the_flag_is_released_even_when_startup_fails():
|
||||
"""Rules 156 and 157 together. The waiter has no deadline of its own, so
|
||||
"""The waiter has no deadline of its own, so
|
||||
the release has to be unmissable: a `finally`, never a last line. Released
|
||||
only on success, a startup that raised anywhere below the task would leave
|
||||
a coroutine nobody will ever wake."""
|
||||
@@ -164,7 +159,7 @@ def test_startup_releases_the_flag_in_a_finally():
|
||||
finally_at = body.index("finally:")
|
||||
assert finally_at < body.index("_startup_finished.set()"), (
|
||||
"the flag is released after the work rather than in a finally — a "
|
||||
"startup that raises would strand the backfill forever (rule 157)"
|
||||
"startup that raises would strand the backfill forever"
|
||||
)
|
||||
assert "await _startup_finished.wait()" in src
|
||||
|
||||
@@ -184,7 +179,7 @@ def test_the_engine_bounds_how_long_a_connect_may_take():
|
||||
from scribe.models import _CONNECT_ARGS
|
||||
|
||||
timeout = _CONNECT_ARGS.get("timeout")
|
||||
assert timeout is not None, "a connect with no deadline (rule 156)"
|
||||
assert timeout is not None, "a connect with no deadline"
|
||||
assert 0 < timeout < 60
|
||||
|
||||
|
||||
@@ -192,7 +187,7 @@ def test_no_blanket_command_timeout_was_added_with_it():
|
||||
"""The deliberate omission, guarded so nobody adds it as an obvious
|
||||
follow-up. `command_timeout` applies to EVERY statement, and this app runs
|
||||
long ones on purpose — the embedding backfills, VACUUM ANALYZE. It would
|
||||
trade #4181's failure mode for a worse one."""
|
||||
trade a hung startup for killed maintenance."""
|
||||
from scribe.models import _CONNECT_ARGS
|
||||
|
||||
assert "command_timeout" not in _CONNECT_ARGS
|
||||
@@ -211,7 +206,7 @@ def test_the_engine_actually_uses_those_args():
|
||||
|
||||
|
||||
def test_the_startup_guards_can_fail():
|
||||
"""Rule 167: each assertion above has to be able to bite."""
|
||||
"""Each assertion above has to be able to bite."""
|
||||
from scribe.services.db_maintenance_scheduler import _STARTUP_READ_TIMEOUT
|
||||
|
||||
# An unbounded read is the thing being prevented.
|
||||
|
||||
Reference in New Issue
Block a user