docs(6209): wide net, startup, stamps, slippage, miss route and parity tests
CI & Build / Python lint (push) Successful in 3s
CI & Build / Plugin hooks (push) Successful in 15s
CI & Build / TypeScript typecheck (push) Successful in 53s
CI & Build / integration (push) Successful in 1m10s
CI & Build / Build & push image (push) Canceled after 0s
CI & Build / Python tests (push) Canceled after 1m23s

Eight more test files state the property they pin as a present-tense
fact. Incident narratives (a dated storage stall, a week of zero tuning
events, a two-project poisoned ledger, a twenty-site owner audit) become
the failure shape each test defends; one-install figures that name a rule
become a general worked example with the calibration numbers kept. The
family-grouping regression guard is renamed for what it checks.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-10-21 21:27:17 -04:00
co-authored by Claude Opus 5.5
parent d41e72d26c
commit ff08500128
8 changed files with 154 additions and 196 deletions
+19 -22
View File
@@ -1,21 +1,18 @@
"""A semantic MISS is not evidence, and the divergence check never reads one (#4208).
"""A semantic MISS is not evidence, and the divergence check never reads one.
WHAT THIS IS ABOUT. #4204 gave the divergence check a structural gate, which
silenced one of the five false prompts it was filed for. The other four are
`def` helpers beside an `async def` service canon — callables beside a
callable — and differ from #2793's acceptance case (a hand-rolled sync
`confirmDanger` where an async confirm helper is canon) only by the JOB they
do. A signature does not carry a job.
WHAT THIS IS ABOUT. The divergence check's structural gate separates a type
from a callable canon. `def` helpers beside an `async def` service canon —
callables beside a callable — differ from the readout's acceptance case (a
hand-rolled sync `confirmDanger` where an async confirm helper is canon) only
by the JOB they do. A signature does not carry a job.
#4208 tried meaning: store the proposer's semantic "compared, nothing cleared
the floor" and let it withdraw the prompt. Measured live on 2026-09-22 it
cannot discriminate. Judged instances of the service-unit canon score 0.68–
0.71 against it at best, most below 0.66; helpers score 0.66–0.75 against
snippets unrelated to them. The then-floor of 0.8 was reached by nothing, so
the "miss" fired for nearly every body — the real divergence silenced exactly
as the helper was. The gate was removed; the prompt asks and the judge
answers. The same measurement moved the arm's floor down to the write-path
hint's: 0.8 was sized for proposals nobody reads, and it proposed 0 of 150.
Meaning cannot withdraw the prompt either. Judged instances of a service-unit
canon score 0.68–0.71 against it at best, most below 0.66; helpers score
0.66–0.75 against snippets unrelated to them. No floor separates the two, so a
"compared, nothing cleared the floor" miss fires for nearly every body,
silencing the real divergence exactly as it silences the helper. So the prompt
asks and the judge answers, and the arm proposes at the write-path hint's
floor: one sized for proposals nobody reads proposes almost nothing.
These tests pin what stayed: the arm proposes on a hit and says nothing on a
miss, the divergence check does not consult it, and the capped pass reads new
@@ -32,8 +29,8 @@ from scribe.services import shape_ledger
from scribe.services.shape_ledger import _semantic_canon
# Comfortably over WRITEPATH_MIN_CODE_CHARS (48 non-whitespace characters), so
# these tests exercise the comparison rather than the substance guard. One of
# #4204's four survivors, quoted rather than invented.
# these tests exercise the comparison rather than the substance guard. A real
# helper body, quoted rather than invented.
BODY = (
"def is_registered(source: str) -> bool:\n"
" return source in _REGISTRY and _REGISTRY[source].enabled\n"
@@ -87,7 +84,7 @@ async def test_a_body_too_thin_to_embed_spends_no_embedding() -> None:
async def test_no_allowed_canon_spends_no_search() -> None:
"""The language-family gate (#2871) empties the allowed set routinely."""
"""The language-family gate empties the allowed set routinely."""
mock = _hits()
with _patch(mock):
assert await _semantic_canon(1, BODY, set()) is None
@@ -97,7 +94,7 @@ async def test_no_allowed_canon_spends_no_search() -> None:
async def test_the_arm_proposes_at_the_write_path_floor() -> None:
"""It asks the write-path hint's question of the same documents, so it
uses that surface's floor — not a stricter private one that yields no
proposals for the judge to weigh (#4208)."""
proposals for the judge to weigh."""
from scribe.services.plugin_context import WRITEPATH_DEFAULT_THRESHOLD
mock = _hits()
@@ -111,14 +108,14 @@ async def test_the_arm_proposes_at_the_write_path_floor() -> None:
def test_the_divergence_check_does_not_read_the_proposal_basis() -> None:
"""The retired gate keyed on `proposal_basis`. Asserted on the function's
structure (rule 167): any attribute read of it inside `flag_divergence` is
structure: any attribute read of it inside `flag_divergence` is
the miss being consulted again, whatever the constant is called."""
tree = ast.parse(textwrap.dedent(inspect.getsource(shape_ledger.flag_divergence)))
reads = [n for n in ast.walk(tree)
if isinstance(n, ast.Attribute) and n.attr == "proposal_basis"]
assert not reads, (
"flag_divergence reads proposal_basis — a semantic miss is 'cannot "
"tell' at the arm's floor (#4208), never 'not the canon'"
"tell' at the arm's floor, never 'not the canon'"
)
+15 -19
View File
@@ -1,15 +1,14 @@
"""One definition of what SHIPS in the plugin, and the drift guards on it.
WHAT THIS IS ABOUT (#3127 §3, milestone 334 step 2). Scribe publishes two
artifacts. `plugin/` is not in the Docker image — installs fetch it from this
WHAT THIS IS ABOUT. Scribe publishes two artifacts. `plugin/` is not in the Docker image — installs fetch it from this
repo through `.claude-plugin/marketplace.json`, so **a push IS the release**,
with no build step in between. That makes "which files reach an install?" a
question with real consequences, and it has been answered wrong twice:
question with real consequences, and it can be answered wrong two ways:
- #2198 — `plugin/**` was in no `paths:` filter, so four broken hooks
reached live installs having triggered no CI at all.
- #2209 — the fix for that shipped and still could not reach an install,
because the manifest version had not moved.
- a shipped path in no `paths:` filter reaches live installs having
triggered no CI at all;
- a fix that ships without the manifest version moving cannot reach an
install.
The set lives in `scripts/check_plugin.py`. Its second consumer is the
workflow's `paths:` trigger, which is YAML and cannot import Python — so the
@@ -66,7 +65,7 @@ def test_every_shipped_path_exists():
def test_every_shipped_path_triggers_ci():
"""#2198's exact hole, stated as an assertion.
"""A shipped path that runs no CI, stated as an assertion.
Directional on purpose: the trigger is a superset (it also fires on
`src/**`, `tests/**` and friends). What must never happen is a path that
@@ -77,7 +76,7 @@ def test_every_shipped_path_triggers_ci():
covered = any(t == path or t.startswith(f"{path}/") for t in triggers)
assert covered, (
f"{path} ships to installs but no `paths:` entry covers it — "
f"changes there would reach a live install having run no CI (#2198)"
f"changes there would reach a live install having run no CI"
)
@@ -94,10 +93,8 @@ def test_the_checker_itself_triggers_ci():
def test_no_trigger_path_names_something_that_does_not_exist():
"""The guard that catches scaffolding outliving its subsystem.
`fable-mcp/**` sat in this list for three months after the directory was
deleted (commit 91bafb6, 2026-05-27), and `assets/**` named a path that
never existed at all. Neither ever failed anything — a `paths:` entry
matching nothing simply never fires — which is precisely why a list kept
A `paths:` entry matching nothing simply never fires, so an entry for a
deleted directory, or for a path that never existed, fails nothing — which is precisely why a list kept
by hand drifts and nobody finds out.
"""
missing = [
@@ -114,7 +111,7 @@ def test_every_deriver_exists():
"""§3's table, kept honest.
The point of the table is that the next artifact is a one-line addition
(milestone 334 step 3 adds the plugin's mint script). A row pointing at a
(the plugin's mint script is one). A row pointing at a
file that has moved would make the table read as complete when it is not.
"""
for path, artifacts in DERIVERS.items():
@@ -162,7 +159,7 @@ def test_every_OTHER_manifest_field_still_demands_a_new_version(field, value):
`plugin.json` carries description, mcpServers and userConfig alongside the
version, and all of them reach an install. Excluding the file wholesale
would mean a userConfig-only edit computes an unchanged version and never
refreshes — #2209 again, with a narrower trigger and the same silence.
refreshes — a fix that cannot reach an install, in silence.
"""
assert manifest_differs_beyond_version(manifest(**{field: value}), manifest()) is True
@@ -181,7 +178,7 @@ def test_reformatting_is_not_a_content_change():
def test_unreadable_input_demands_a_new_version(bad):
"""The conservative direction, chosen deliberately.
A spurious bump costs one cache refresh. A missed one is #2209 — the fix
A spurious bump costs one cache refresh. A missed one means the fix
reaches the repo and stops there, and the only detector is a human saying
"I don't think it updated."
"""
@@ -238,9 +235,8 @@ def test_shipped_content_changed_reports_a_hook_edit_as_changed(monkeypatch):
def test_a_failed_diff_is_None_and_never_False(monkeypatch):
"""Could-not-tell and nothing-changed must not collapse into one value.
#2663 is the precedent: a read that failed inside a broad except reported
the same zero as a genuinely empty window, and every counter read zero for
weeks with nothing to distinguish the two.
A read that fails inside a broad except and reports the same zero as a
genuinely empty window leaves nothing to distinguish the two.
"""
from scripts import check_plugin
+20 -27
View File
@@ -1,18 +1,13 @@
"""A recognized retrieval miss has a route, and the route is ordered (#4133).
"""A recognized retrieval miss has a route, and the route is ordered.
WHY THIS EXISTS
Milestone 416 built the tuning loop across steps 4 and 6 — `retrieval_surfaces`
to read a dial, `tune_retrieval` to move one with its argument attached,
`migrate_retrieval_floor` to carry it across a model change — and then measured,
on 2026-09-17, that `retrieval_tuning_history` returned **zero events**. Not one
call, ever.
The cause was not a tool being wrong. The three tools appeared on NO instruction
surface: not the skills, not the hooks, not the MCP `_INSTRUCTIONS`. They were
classified in `server.py`'s read/write lists and named nowhere a session would
read them. The operator's decision for #4102 — *"the model should be the thing
handling it 9 times out of 10"* — could not begin to happen.
The tuning loop is three tools — `retrieval_surfaces` to read a dial,
`tune_retrieval` to move one with its argument attached,
`migrate_retrieval_floor` to carry it across a model change — and the model is
meant to be the one using them most of the time. A tool classified in
`server.py`'s read/write lists and named on no instruction surface is never
called: nothing a session reads tells it the tool exists.
So this file guards a ROUTE, not a behaviour: that a session which notices a
rule missed the moment it governed has somewhere written down to take it.
@@ -27,13 +22,12 @@ computed against, so a miss is evidence about that text first; a floor is one
number governing every record on the surface and cannot tell a badly-worded
trigger from a genuinely distant one.
Measured on this install, which is why the order is asserted rather than
suggested: rule 1 "`dev` is home" scored 0.6515 and ranked 5th for the moment it
governed, behind three rules that RESTRAINED the same act. Every percentile said
"lower the floor" — and at 0.60 the arm delivered those three restraints and
still not rule 1. Rewriting the trigger to lead with the symptom put the same
rule 1st at 0.7130. The dial-first path was available, statistically supported,
and wrong.
Which is why the order is asserted rather than suggested: a rule can score
0.65 and rank fifth for the moment it governs, behind rules that RESTRAIN the
same act. Every percentile says "lower the floor" — and at 0.60 the arm
delivers those restraints and still not that rule, while rewriting its trigger
to lead with the symptom puts it first at 0.71. The dial-first path can be
available, statistically supported, and wrong.
"""
from __future__ import annotations
@@ -50,8 +44,8 @@ TUNING_TOOLS = ("retrieval_telemetry", "update_rule", "retrieval_surfaces",
def _skill() -> str:
# The route lives in missed-retrieval.md, a reference file of using-scribe
# (#4398); the skill is SKILL.md and its references read together.
# The route lives in missed-retrieval.md, a reference file of using-scribe;
# the skill is SKILL.md and its references read together.
from tests.helpers import skill_text
return skill_text("using-scribe")
@@ -142,17 +136,16 @@ def test_writing_a_new_record_asks_what_already_covers_the_moment():
def test_the_route_stays_off_the_index():
"""`_INSTRUCTIONS` is the only surface every MCP client gets (decision #4027).
"""`_INSTRUCTIONS` is the only surface every MCP client gets.
It carried a one-line MISSED pointer to this route until #4389, which found
the index was being used as a rulebook: a missed rule is noticed mid-work,
not at session start, and the spec asks server instructions for the
It carries no pointer to this route, because the index is not a rulebook:
a missed rule is noticed mid-work, not at session start, and the spec asks server instructions for the
workflow across tools, not stance. using-scribe states the route (the tests
above pin it); the index keeps what_might_apply, the tool that opens it.
Whitespace-flattened before matching: the block is hard-wrapped, so a
phrase straddles a line break the moment anything before it changes length
(#4103).
phrase straddles a line break the moment anything before it changes
length.
"""
text = " ".join(_instructions().split()).lower()
assert "what_might_apply" in text
+17 -20
View File
@@ -1,4 +1,4 @@
"""Which rules fired this session, which changed an action, which did not (#4216).
"""Which rules fired this session, which changed an action, which did not.
WHY THIS CANNOT BE ASKED OF THE SERVER
@@ -7,13 +7,13 @@ a SESSION-scoped answer has to be assembled where a session is a thing that
exists. That is the plugin, from four ledgers four hooks already write:
.rules.ids an arm NAMED the rule (a teaser was shown)
.opened.ids the session called get_rule (#4100)
.acted.ids the session called rule_outcome (#4216, new here)
.checkpoint.ids the rule HELD an act (#4214)
.opened.ids the session called get_rule
.acted.ids the session called rule_outcome
.checkpoint.ids the rule HELD an act
EVERY LINE IS AN OBSERVED TOOL CALL. Nothing asks the model what it followed;
milestone 386 ruled that out because a model asked "did you apply rule 156?"
will say yes. These record what happened.
EVERY LINE IS AN OBSERVED TOOL CALL. Nothing asks the model what it followed,
because a model asked "did you apply rule 156?" will say yes. These record
what happened.
WHY THE COMPACTION SEAM
@@ -21,8 +21,8 @@ A rule read and left unresolved is invisible by construction — it looks
exactly like a rule that worked. The compaction is where that invisibility
becomes permanent: the turns holding the evidence are summarised away, and an
unjudged thing that survives as nothing is how a decision quietly becomes
nobody's. A PreCompact hook's stdout becomes the summariser's instructions
(#3680), so this does not say "you slipped" — it says which ids must be
nobody's. A PreCompact hook's stdout becomes the summariser's instructions,
so this does not say "you slipped" — it says which ids must be
carried through, which is the one thing a summary can do about it.
WHAT IS PINNED: the arithmetic, and which conditions produce which lines. NOT
@@ -63,8 +63,7 @@ def ledgers(tmp_path, *, named=(), opened=(), acted=(), held=()) -> Path:
Not hand-rolled files: `scribe_rules_append` is what creates the evidence
twin these read from, so a fixture that wrote the bytes itself would keep
passing if the twin stopped being written — which is the whole defect
#4217 found.
passing if the twin stopped being written.
"""
d = tmp_path / "scribe-priorart"
d.mkdir(parents=True, exist_ok=True)
@@ -140,7 +139,7 @@ def test_a_rule_read_with_no_outcome_is_the_headline(tmp_path):
def test_a_rule_that_held_an_act_is_reported_separately(tmp_path):
"""The strongest evidence a rule changed something: it stopped a call
before it ran (#4214). Kept apart from `read` because reading a rule and
before it ran. Kept apart from `read` because reading a rule and
having it alter what you did are different claims."""
out = readout(ledgers(tmp_path, named=[156], opened=[156], held=[156]))
assert "held an act" in out and "156" in out
@@ -158,7 +157,7 @@ def test_a_session_that_resolved_everything_makes_no_accusation(tmp_path):
def test_a_session_no_rule_touched_says_nothing_at_all(tmp_path):
"""Rule 115's reasoning: a fresh install must not be told something is
"""A fresh install must not be told something is
wrong when the truth is that nothing has happened yet."""
d = tmp_path / "scribe-priorart"
d.mkdir(parents=True)
@@ -256,14 +255,12 @@ def test_the_recorder_is_shell_valid():
subprocess.run(["bash", "-n", str(RECORDER)], check=True)
# ── Evidence outlives the compaction; exclusions do not (#4217) ────────────
# ── Evidence outlives the compaction; exclusions do not ───────────────────
#
# THE DEFECT THIS PINS, measured before it was fixed: across six sessions on
# the instance this was built on, 208 `get_rule` calls produced 3 surviving
# ledger entries. `.opened.ids` was doing two jobs with opposite lifetimes —
# "this context holds the rule" (must be forgotten at a compaction, and three
# hooks depend on that) and "this was opened" (which no compaction makes
# untrue). The sweep, correct for the first, was deleting the second.
# Two jobs with opposite lifetimes: "this context holds the rule" (must be
# forgotten at a compaction, and three hooks depend on that) and "this was
# opened" (which no compaction makes untrue). One ledger doing both loses
# nearly every opened rule to the sweep, which is correct only for the first.
def test_the_readout_is_unchanged_by_the_sweep(tmp_path):
"""The readout runs AT the compaction. If the sweep took its inputs it
+31 -36
View File
@@ -1,10 +1,9 @@
"""The review surface: rows whose evidence no longer meets the bar, and
canons whose own rows no longer agree what they are (#4204, #4208).
canons whose own rows no longer agree what they are.
WHAT THESE GUARD, and it is a property rather than an output shape. The
ledger was poisoned by a machine writing permanent classifications that
nobody read — 32 rows on one project, 334 on another. The correction is NOT
a machine that un-writes them; that is the same mistake pointed the other
WHAT THESE GUARD, and it is a property rather than an output shape. A machine
writing permanent classifications that nobody reads poisons the ledger. The
correction is NOT a machine that un-writes them; that is the same mistake pointed the other
way. It is a reader that puts the evidence in front of a judge.
So the tests below assert two things hardest:
@@ -15,8 +14,8 @@ So the tests below assert two things hardest:
* the listing changes nothing, which is checked by giving it rows and
asserting the rows come back untouched.
The round-trip test exists because the score used to live only inside a
prose sentence. Writer and parser are now one constant apart, and a change
The round-trip test exists because the score is carried inside a prose
sentence. Writer and parser are one constant apart, and a change
to one that the other does not follow is exactly the kind of silent rot the
hook's own history is made of.
"""
@@ -135,10 +134,9 @@ def test_a_canon_whose_members_agree_is_not_listed() -> None:
def test_a_canon_whose_members_are_all_different_things_is_listed() -> None:
"""Portal's #3283, in miniature: a class, a getter, a test and a binding
recorded as instances of one pattern — and in four different files, as
they really were. The paths matter now: a callable sharing a file with a
class is read as that class's method (#4220), so putting them all in one
"""A class, a getter, a test and a binding recorded as instances of one
pattern — in four different files. The paths matter: a callable sharing a
file with a class is read as that class's method, so putting them all in one
file would test the excuse rather than the disagreement."""
rows = [_member("class SessionAbsent(RuntimeError):", path="src/errors.py",
symbol="SessionAbsent"),
@@ -198,7 +196,7 @@ def test_the_review_changes_nothing_about_the_rows_it_reads() -> None:
def test_the_service_carries_no_machinery_for_bulk_withdrawal() -> None:
"""A structural guard (rule 167) on the design decision, not on output.
"""A structural guard on the design decision, not on output.
If someone adds an auto-retire path to the ledger, this fails and the
conversation about whether a machine may un-judge in bulk happens again
— deliberately, rather than in a diff nobody read."""
@@ -209,19 +207,18 @@ def test_the_service_carries_no_machinery_for_bulk_withdrawal() -> None:
assert banned not in body, banned
# ── coherence is about what a canon CANNOT account for (#4220) ────────────
# ── coherence is about what a canon CANNOT account for ────────────────────
#
# The first version of this check asked whether every member shared a form.
# On its first live day the only two canons it reported were both sound,
# while the genuinely polluted one had already been cleaned by hand — a
# surface whose whole output is noise is one that stops being read.
# Asking whether every member shares a form reports sound canons — a class and
# its methods are two forms of one shape — and a surface whose whole output is
# noise stops being read.
#
# The obvious repair — group by family, keep the majority test — is a trap,
# and `test_family_grouping_alone_would_have_gone_blind` is the guard that
# stops anyone walking back into it.
# and `test_family_grouping_alone_passes_a_polluted_canon` is the guard that
# stops anyone walking into it.
def _model(sym, path):
"""A SQLAlchemy model class, as #2844 holds one."""
"""A SQLAlchemy model class, as the model-convention canon holds one."""
return _member(f"class {sym}(Base, TimestampMixin):", path=path)
@@ -231,10 +228,9 @@ def _to_dict(path):
def test_a_class_and_the_to_dict_beside_it_are_one_shape() -> None:
"""#2844 exactly: the SQLAlchemy model convention, whose own text is
about the class AND the to_dict it must carry. Live it scored
37/62 = 0.597 and was called incoherent for containing precisely what it
says it contains."""
"""The SQLAlchemy model convention, whose own text is about the class AND
the to_dict it must carry. A form-level check scores it 37/62 = 0.597 and
calls it incoherent for containing precisely what it says it contains."""
rows = []
for i in range(6):
rows += [_model(f"M{i}", f"src/scribe/models/m{i}.py"),
@@ -244,9 +240,9 @@ def test_a_class_and_the_to_dict_beside_it_are_one_shape() -> None:
def test_a_sync_starter_and_the_async_tick_it_schedules_are_one_shape() -> None:
"""#2849: four loop-starters and three ticks. `shape_family` already
collapses fn and async-fn for the divergence gate; this surface was the
one place still asking at form level."""
"""Four loop-starters and three ticks. `shape_family` collapses fn and
async-fn for the divergence gate, and this surface asks at the same
level."""
rows = [_member("def start_notification_loop() -> None:", path="src/n.py"),
_member("def start_log_retention_loop() -> None:", path="src/l.py"),
_member("def start_auth_token_retention_loop() -> None:", path="src/a.py"),
@@ -256,10 +252,10 @@ def test_a_sync_starter_and_the_async_tick_it_schedules_are_one_shape() -> None:
assert stamps_to_review(rows)["incoherent_count"] == 0
def test_family_grouping_alone_would_have_gone_blind() -> None:
"""THE REGRESSION GUARD. Polluted #2844 as it stood this morning: 37
model classes, 25 to_dict methods beside them, and 31 rows that had no
business being there. Counted as families that is 56 callables to 37
def test_family_grouping_alone_passes_a_polluted_canon() -> None:
"""THE REGRESSION GUARD. A polluted model canon: 37 model classes, 25
to_dict methods beside them, and 31 rows that have no business being
there. Counted as families that is 56 callables to 37
types — 0.602, a clean pass under any majority test. Those 31 rows are
the ones the surface exists to find, so a change that lets this canon
read as coherent has broken the feature while keeping every other test
@@ -352,12 +348,11 @@ def test_the_coherence_verdict_reads_rows_and_writes_nothing() -> None:
for r in rows] == before
# ── a weak stamp is listed for judgment and counted by nobody (#4608) ─────
# ── a weak stamp is listed for judgment and counted by nobody ─────────────
#
# The floor guarded new writes; the rows written before it went on electing
# canons. On two projects a YAML CI snippet "dominated" a Svelte directory on
# the strength of 0.68-0.72 stamps, and every write there was told it
# diverged from it. The readers below must agree with the review surface
# The floor guards new writes; rows written below it would otherwise go on
# electing canons, so an unrelated snippet "dominates" a directory on the
# strength of 0.68-0.72 stamps and every write there is told it diverged. The readers below must agree with the review surface
# about what "weak" means — one predicate, so they cannot drift apart.
def test_one_predicate_says_what_weak_is() -> None:
+20 -25
View File
@@ -1,16 +1,13 @@
"""A sick database must not stop the app from starting — #4181, rule 156.
"""A sick database must not stop the app from starting.
THE INCIDENT THESE GUARD, so a later reader knows what is being defended:
THE FAILURE THESE GUARD. A storage stall can make Postgres answer in minutes
instead of milliseconds. If the app starts into one, a startup read with no
deadline hangs; Hypercorn kills the worker at its 60-second lifespan timeout;
and nothing retries a failed lifespan. A short disk hiccup becomes an outage
that only a human restart can clear.
On 2026-09-19 a host storage stall made one Postgres checkpoint of 14 buffers
take 281 seconds against a 1.3-second baseline. The app restarted into the tail
of it; `get_maintenance_hour()` — the first database read in `before_serving` —
hung with no deadline; Hypercorn killed the worker at its 60-second lifespan
timeout; and nothing retries a failed lifespan. A five-minute disk hiccup became
a three-hour outage that only a human restart could clear.
Every assertion here is about the SHAPE that made that possible, not about the
stall, which no test can reproduce and no code can prevent:
Every assertion here is about the SHAPE that makes that possible, not about
the stall, which no test can reproduce and no code can prevent:
- the startup read returns on a schedule of its own rather than the
database's;
@@ -48,9 +45,9 @@ async def test_a_hanging_settings_read_does_not_hang_startup():
@pytest.mark.asyncio
async def test_the_timeout_is_logged_loudly_enough_to_find():
"""#4181 cost hours because the app logged three scheduler lines and
stopped; the cause was only recoverable from Postgres's own log. A WARNING
here is the breadcrumb that would have named it in seconds."""
"""Without a WARNING the app logs a few scheduler lines and stops, and the
cause is only recoverable from Postgres's own log. This is the breadcrumb
that names it."""
from scribe.services import db_maintenance_scheduler as sched
async def _never_returns(*_a, **_kw):
@@ -60,7 +57,7 @@ async def test_the_timeout_is_logged_loudly_enough_to_find():
patch.object(sched, "_STARTUP_READ_TIMEOUT", 0.05), \
patch.object(sched.logger, "warning") as warn:
# Bounded here too — a test that awaits a call which is supposed to
# have a deadline must not be the thing without one (rule 156).
# have a deadline must not be the thing without one.
await asyncio.wait_for(sched.get_maintenance_hour(), timeout=2)
assert warn.called
@@ -105,11 +102,9 @@ def test_the_startup_deadline_leaves_room_inside_the_lifespan_budget():
@pytest.mark.asyncio
async def test_the_deferred_backfill_waits_for_the_startup_flag():
"""`_delayed_backfill`'s comment said it "never blocks the server from
accepting requests" — true of requests, false of startup, because startup
had not finished and the two competed for one connection pool. Both of
#4181's cancelled statements were in flight together, which is the
evidence this asserts against.
""""Never blocks the server from accepting requests" is true of requests
and false of startup: a backfill that starts before startup finishes
competes with it for one connection pool.
Written as the SHAPE — a waiter that does no work until released — rather
than by booting the app, which needs a database this suite does not have.
@@ -133,7 +128,7 @@ async def test_the_deferred_backfill_waits_for_the_startup_flag():
@pytest.mark.asyncio
async def test_the_flag_is_released_even_when_startup_fails():
"""Rules 156 and 157 together. The waiter has no deadline of its own, so
"""The waiter has no deadline of its own, so
the release has to be unmissable: a `finally`, never a last line. Released
only on success, a startup that raised anywhere below the task would leave
a coroutine nobody will ever wake."""
@@ -164,7 +159,7 @@ def test_startup_releases_the_flag_in_a_finally():
finally_at = body.index("finally:")
assert finally_at < body.index("_startup_finished.set()"), (
"the flag is released after the work rather than in a finally — a "
"startup that raises would strand the backfill forever (rule 157)"
"startup that raises would strand the backfill forever"
)
assert "await _startup_finished.wait()" in src
@@ -184,7 +179,7 @@ def test_the_engine_bounds_how_long_a_connect_may_take():
from scribe.models import _CONNECT_ARGS
timeout = _CONNECT_ARGS.get("timeout")
assert timeout is not None, "a connect with no deadline (rule 156)"
assert timeout is not None, "a connect with no deadline"
assert 0 < timeout < 60
@@ -192,7 +187,7 @@ def test_no_blanket_command_timeout_was_added_with_it():
"""The deliberate omission, guarded so nobody adds it as an obvious
follow-up. `command_timeout` applies to EVERY statement, and this app runs
long ones on purpose — the embedding backfills, VACUUM ANALYZE. It would
trade #4181's failure mode for a worse one."""
trade a hung startup for killed maintenance."""
from scribe.models import _CONNECT_ARGS
assert "command_timeout" not in _CONNECT_ARGS
@@ -211,7 +206,7 @@ def test_the_engine_actually_uses_those_args():
def test_the_startup_guards_can_fail():
"""Rule 167: each assertion above has to be able to bite."""
"""Each assertion above has to be able to bite."""
from scribe.services.db_maintenance_scheduler import _STARTUP_READ_TIMEOUT
# An unbounded read is the thing being prevented.
+17 -31
View File
@@ -1,34 +1,23 @@
"""Both doors can file a record under a System, at create AND at update (#4249).
"""Both doors can file a record under a System, at create AND at update.
WHAT THIS IS ABOUT. A System tag is how `list_system_records` gathers an
area's whole pile. A record that cannot be tagged is reachable by search and
by nothing else — so a missing `system_ids` parameter is not a cosmetic API
gap, it is a record that quietly never joins the area it belongs to.
THE DEFECT IS THE ASYMMETRY, NOT ANY ONE PARAMETER. When #4249 was filed it
read as "the REST door can do something the MCP door cannot", on three
instances in a row. Filling in the whole table showed that was the wrong
generalisation:
kind MCP create MCP update REST create REST update
lesson yes NO yes yes
preference NO yes - -
note yes yes NO NO
process NO NO NO NO
`preference` was the exact mirror of `lesson` — the same capability missing
from the opposite end of the same lifecycle — and `note` ran the opposite way
round from the issue's own premise. The real pattern is that **whichever door
nobody exercised for a kind is the one that never grew the parameter**, and
that is not a thing a human reviewer reliably notices, because each door is
only ever read on its own.
THE DEFECT IS THE ASYMMETRY, NOT ANY ONE PARAMETER. A missing door shows up
one kind at a time and reads like a one-off — REST lacking what MCP has for
one kind, the opposite for the next, a create without the update. The real
pattern is that **whichever door nobody exercises for a kind is the one that
never grows the parameter**, and that is not a thing a human reviewer reliably
notices, because each door is only ever read on its own.
So this file does not assert that any particular function takes the argument.
It asserts the SHAPE OF THE TABLE: every kind that can be System-tagged
anywhere can be System-tagged everywhere it is written. A fifth kind added
next month fails here rather than shipping half-wired — the same reasoning
test_derived_mirror_generic_door.py records for derived mirrors (#3734), where
testing one constant let the second kind arrive with none of the protection.
test_derived_mirror_generic_door.py records for derived mirrors, where testing
one constant lets the second kind arrive with none of the protection.
NOT A GAP — MILESTONES. `RecordSystem.note_id` is a ForeignKey to `notes.id`
and milestones live in their own table, so milestones cannot be System-tagged
@@ -94,8 +83,8 @@ def _rest_source(module: str) -> str:
def test_the_mcp_door_takes_system_ids_at_both_ends(kind, mcp_mod, create, update, _rest):
"""Create AND update. Tagging available only at create is tagging at the
moment of LEAST information — a lesson written at the end of a piece of
work is exactly when the argument gets dropped, and before #4249 there was
then no way to file it ever again."""
work is exactly when the argument gets dropped, and without the update
door there is no way to file it ever again."""
tree = _mcp(mcp_mod)
for fn_name in (create, update):
fn = _find(tree, fn_name)
@@ -138,7 +127,7 @@ def test_every_tool_module_that_tags_is_in_the_registry():
"""The guard that makes the two tests above keep working.
Without this, the registry is a list somebody has to remember to extend,
and #4249 IS the failure of somebody remembering. Any MCP tool module that
and somebody remembering is what fails. Any MCP tool module that
ACCEPTS `system_ids` must be registered — so a new kind that wires up
tagging is dragged into the parity checks by its own first use of the
parameter, rather than being half-wired quietly.
@@ -147,7 +136,7 @@ def test_every_tool_module_that_tags_is_in_the_registry():
`projects.py`, `search.py` and `systems.py`, which only name the argument
in prose while explaining it to the reader — and the fix for that is a
structural check, not a hand-kept skip list that would itself go stale
(rule 167: a guard asserts on structure).
(a guard asserts on structure).
"""
registered = {r[1] for r in REGISTRY}
tagging = set()
@@ -195,11 +184,8 @@ def test_every_tagging_write_acts_as_the_caller():
the caller's write access first (a route's `can_write_note`, or a service
that raises PermissionError). What this decides is WHOSE reach the tagging
runs with, which is exactly the kind of difference that survives review
because each call site reads fine on its own.
Four of twenty sites passed the owner before #4249 — two routes, two MCP
tools, one of them written earlier in the very session that unified them.
That is the tell that this needs a guard rather than care.
because each call site reads fine on its own — the tell that this needs a
guard rather than care.
"""
offenders = []
for path in ROOT.rglob("*.py"):
@@ -227,7 +213,7 @@ def test_every_tagging_write_acts_as_the_caller():
assert not offenders, (
"set_record_systems must be called with the acting user, not the "
"record's owner — passing the owner bypasses its own access check and "
"borrows the owner's System visibility (#47, #4249):\n "
"borrows the owner's System visibility:\n "
+ "\n ".join(offenders)
)
@@ -250,6 +236,6 @@ def test_milestones_are_absent_on_purpose():
assert 'ForeignKey("notes.id"' in seg, (
"RecordSystem no longer keys on notes.id — if it grew a generic "
"record reference, milestones may now be taggable and this test, the "
"registry above and #4249's table all need revisiting."
"registry above and the door table all need revisiting."
)
assert "milestone" not in {r[0] for r in REGISTRY}
+15 -16
View File
@@ -1,10 +1,10 @@
"""The wide net: many candidates, no bar, and a cost that stays bounded (#4103).
"""The wide net: many candidates, no bar, and a cost that stays bounded.
WHY THIS EXISTS
Milestone 416 step 5 moves "do not exclude potentially important data" off the
push, where fifty candidates would be a wall nobody reads, and onto a pull,
where they arrive only when a session asks for them.
"Do not exclude potentially important data" belongs on a pull, not the push:
fifty candidates pushed would be a wall nobody reads, and pulled they arrive
only when a session asks for them.
Three things have to hold, and each has a way of quietly failing:
@@ -12,13 +12,13 @@ Three things have to hold, and each has a way of quietly failing:
not trust a threshold to decide for them. A default floor creeping in
here would turn the wide net into the narrow one and nothing would look
wrong — the results would simply be fewer.
2. **A bounded cost.** The step was filed claiming fifty "costs nothing".
`_rule_hint_line` had already measured ~143 tokens for a line with its
trigger rendered, so fifty is ~7,000. The graduated shape (#3851) is what
keeps this affordable, and a change that renders every trigger in full
2. **A bounded cost.** Fifty candidates do not cost nothing: a line with
its trigger rendered is ~143 tokens, so fifty is ~7,000. The graduated
shape is what keeps this affordable, and a change that renders every trigger in full
would pass every other test here.
3. **A pull, logged as one.** The push arms' near-miss distributions are the
evidence #4121 argues from. A pull mixed into them moves those numbers.
evidence their floors are tuned from. A pull mixed into them moves those
numbers.
"""
from unittest.mock import AsyncMock, patch
@@ -71,7 +71,7 @@ async def test_the_limit_is_clamped_to_the_cap(asked, expected):
@pytest.mark.asyncio
async def test_the_head_carries_its_trigger_whole_and_the_tail_is_cut():
"""The graduated shape (#3851), which is what makes fifty affordable.
"""The graduated shape, which is what makes fifty affordable.
A regression that rendered every trigger in full would satisfy every other
assertion in this file, so the cut is pinned on both sides: the head is
@@ -104,8 +104,7 @@ async def test_a_short_trigger_is_never_marked_truncated():
def test_the_cut_breaks_on_a_word_and_says_it_was_cut():
"""#4036's lesson, borrowed: a raw slice ends mid-word and reads as the
whole thing."""
"""A raw slice ends mid-word and reads as the whole thing."""
text, cut = wide_net._teaser("alpha beta gamma delta " * 40)
assert cut is True
assert text.endswith("…")
@@ -152,7 +151,7 @@ async def test_the_call_is_logged_under_its_own_pull_source():
def test_the_wide_net_is_not_one_of_the_tunable_push_surfaces():
"""THE GUARD that keeps #4121's evidence clean (rule 167).
"""THE GUARD that keeps the push arms' tuning evidence clean.
The registry holds the PUSH arms — the ones with a floor and a budget the
model tunes. This source must not appear there: a pull folded into those
@@ -175,7 +174,7 @@ def test_the_wide_net_is_not_ambient_either():
@pytest.mark.asyncio
async def test_a_search_that_never_ran_is_not_reported_as_a_decline():
"""#3765: an empty query, a dead embedder and a failed query all return
"""An empty query, a dead embedder and a failed query all return
nothing, and none of them is a ranker declining."""
with _patched([], report={"searched": False, "best_available_score": None}), \
patch.object(wide_net, "record_retrieval") as rec, \
@@ -189,7 +188,7 @@ async def test_a_search_that_never_ran_is_not_reported_as_a_decline():
@pytest.mark.asyncio
async def test_what_the_bar_turned_away_is_carried_through():
"""There is no bar here, but `best_available` still answers "did the corpus
have anything at all" for a call that came back empty (#3670)."""
have anything at all" for a call that came back empty."""
with _patched([], report={"searched": True, "best_available_score": 0.31,
"best_available_id": 7}), \
patch.object(wide_net, "record_retrieval") as rec, \
@@ -230,7 +229,7 @@ def test_it_is_registered_and_readable_with_a_read_key():
def test_the_instruction_surfaces_point_at_it():
"""Rule 119: the instruction surfaces ARE the specification for product
"""The instruction surfaces ARE the specification for product
behaviour, so a tool the reflex never learns about is not shipped."""
import pathlib