fix(shapes): a semantic miss is not evidence — the divergence check stops reading it (#4208)

Measured live: true instances of the service-unit canon score 0.68-0.71
against it at best, helpers 0.66-0.75 against unrelated snippets, and
nothing reaches the 0.8 floor. The "conclusive miss" fired for nearly every
body and silenced real divergences exactly as it silenced helpers. The
stored miss basis, its flag withdrawal and the report plumbing are removed;
the floor stays at 0.8 and the new-shapes-first ordering stays. False
prompts are answered by judgment (exempt with a reason).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01821k5B3Ysecp9fNYs92Kuy

tests/test_divergence_meaning_gate.py

@@ -1,49 +1,33 @@
-"""A divergence prompt may be silenced by MEANING, never by silence (#4208).
+"""A semantic MISS is not evidence, and the divergence check never reads one (#4208).
 
-WHAT THIS IS ABOUT. #4204 gave the divergence check a structural gate: a canon
-is only urged on a shape whose form could plausibly BE it. That silenced one of
-the five false prompts it was filed for. The other four are `def` helpers in a
-directory whose canon is an `async def` service unit — callables beside a
-callable — and no refinement of `shape_form` ever separates them, because they
-differ from #2793's acceptance case (a hand-rolled sync `confirmDanger` where
-an async confirm helper is canon) only by the JOB they do. A signature does not
-carry a job.
+WHAT THIS IS ABOUT. #4204 gave the divergence check a structural gate, which
+silenced one of the five false prompts it was filed for. The other four are
+`def` helpers beside an `async def` service canon — callables beside a
+callable — and differ from #2793's acceptance case (a hand-rolled sync
+`confirmDanger` where an async confirm helper is canon) only by the JOB they
+do. A signature does not carry a job.
 
-So the lever has to be meaning, and the ledger already holds one reading of it:
-the proposer's semantic arm embeds each definition's own BODY against canon.
-What it did not do was record its misses. A hit became `proposal_basis =
-"semantic"`; a miss left the row indistinguishable from a row nobody had looked
-at yet. `flag_divergence` could therefore ask the proposer "do you agree this is
-the canon?" but never "did you check, and did you find it is not?".
+#4208 tried meaning: store the proposer's semantic "compared, nothing cleared
+the floor" and let it withdraw the prompt. Measured live on 2026-09-22 it
+cannot discriminate. Judged instances of the service-unit canon score 0.68–
+0.71 against it at best, most below 0.66; helpers score 0.66–0.75 against
+snippets unrelated to them. Nothing reaches the 0.8 floor, so the "miss"
+fires for nearly every body — the real divergence silenced exactly as the
+helper was. The gate was removed; the prompt asks and the judge answers.
 
-THE WHOLE RISK IS IN THE NEGATIVE. A miss is only evidence if the arm actually
-formed an opinion, and there are three ways for it to come back empty that look
-identical from the outside:
-
-    body too thin to embed   -> no opinion
-    no allowed canon to test -> no opinion
-    result set was truncated -> no opinion (the canon may be at N+1)
-    compared, nothing above the floor -> EVIDENCE
-
-Only the last may silence a prompt. Reading any of the others as a negative is
-how "I cannot tell" turns into "I checked" — the exact failure #4204 was opened
-on, and the one `FORM_UNKNOWN` already guards against everywhere else in this
-module: not knowing must make a check QUIETER, never more confident.
-
-These tests pin the report contract that carries that distinction. The
-end-to-end behaviour — a conclusive miss silencing a real prompt while #2793's
-acceptance case still raises — is in
-tests/test_integration_shape_classify.py, because it needs real rows.
+These tests pin what stayed: the arm proposes on a hit and says nothing on a
+miss, the divergence check does not consult it, and the capped pass reads new
+shapes first so the proposals it CAN make land where the check looks.
 """
 from __future__ import annotations
 
+import ast
+import inspect
+import textwrap
 from unittest.mock import AsyncMock, patch
 
-import pytest
-
-from scribe.services.shape_ledger import (
-    _SEMANTIC_LIMIT, BASIS_NO_SEMANTIC_MATCH, _semantic_canon,
-)
+from scribe.services import shape_ledger
+from scribe.services.shape_ledger import _semantic_canon
 
 # Comfortably over WRITEPATH_MIN_CODE_CHARS (48 non-whitespace characters), so
 # these tests exercise the comparison rather than the substance guard. One of
@@ -73,116 +57,64 @@ def _patch(mock: AsyncMock):
     return patch("scribe.services.embeddings.semantic_search_notes", mock)
 
 
-# ── the miss that IS evidence ────────────────────────────────────────────
+# ── the arm proposes on a hit ────────────────────────────────────────────
 
 
-async def test_a_short_result_set_is_a_conclusive_miss() -> None:
-    """Fewer hits than asked for means the limit was not binding: everything
-    above the floor came back, and the canon was not among it. That is a fact
-    about the corpus, not an artefact of where the list was cut."""
-    mock = _hits((0.91, OTHER))
-    report: dict = {}
-    with _patch(mock):
-        found = await _semantic_canon(1, BODY, {CANON}, report=report)
-    assert found is None
-    assert report.get("conclusive") is True
-
-
-async def test_an_empty_result_set_is_also_conclusive() -> None:
-    """Nothing cleared the floor at all — the strongest form of the miss."""
-    report: dict = {}
-    with _patch(_hits()):
-        assert await _semantic_canon(1, BODY, {CANON}, report=report) is None
-    assert report.get("conclusive") is True
-
-
-# ── the three misses that are NOT ────────────────────────────────────────
-
-
-async def test_a_full_result_set_may_have_been_truncated() -> None:
-    """The case that makes `_SEMANTIC_LIMIT` load-bearing rather than a tuning
-    knob. The search returns the top N above the floor; when it returns
-    exactly N, an allowed canon can be sitting at N+1 and this same silence
-    would mean nothing. Reading it as a negative would silence real
-    divergences in direct proportion to how many snippets the operator has —
-    a check that quietly weakens as the corpus grows, which is the worst
-    possible failure mode for a guard nobody is watching."""
-    mock = _hits(*[(0.9, OTHER + i) for i in range(_SEMANTIC_LIMIT)])
-    report: dict = {}
-    with _patch(mock):
-        assert await _semantic_canon(1, BODY, {CANON}, report=report) is None
-    assert "conclusive" not in report
-
-
-async def test_a_body_too_thin_to_embed_forms_no_opinion() -> None:
-    """And does not spend an embedding finding that out."""
-    mock = _hits()
-    report: dict = {}
-    with _patch(mock):
-        assert await _semantic_canon(1, TOO_THIN, {CANON}, report=report) is None
-    assert "conclusive" not in report
-    mock.assert_not_awaited()
-
-
-async def test_no_allowed_canon_means_nothing_was_compared() -> None:
-    """An empty allowed set is not "the canons all missed" — there were none
-    to miss. Distinct because the language-family gate (#2871) empties this
-    set routinely: a Vue body simply has no Python canon to be compared to."""
-    mock = _hits()
-    report: dict = {}
-    with _patch(mock):
-        assert await _semantic_canon(1, BODY, set(), report=report) is None
-    assert "conclusive" not in report
-    mock.assert_not_awaited()
-
-
-# ── a hit is a proposal, not a miss ──────────────────────────────────────
-
-
-async def test_a_hit_returns_the_canon_and_claims_no_miss() -> None:
-    mock = _hits((0.88, CANON))
-    report: dict = {}
-    with _patch(mock):
-        found = await _semantic_canon(1, BODY, {CANON}, report=report)
-    assert found == (CANON, 0.88)
-    assert "conclusive" not in report
+async def test_a_hit_returns_the_canon() -> None:
+    with _patch(_hits((0.88, CANON))):
+        assert await _semantic_canon(1, BODY, {CANON}) == (CANON, 0.88)
 
 
 async def test_an_allowed_canon_below_the_top_hit_still_wins() -> None:
     """The scan is over the whole result set, so a disallowed snippet ranking
-    first does not hide an allowed one behind it. Pinned because if it did,
-    the short-list case above would start reporting conclusive misses for
-    bodies that DO have a canon."""
-    mock = _hits((0.95, OTHER), (0.83, CANON))
-    report: dict = {}
-    with _patch(mock):
-        found = await _semantic_canon(1, BODY, {CANON}, report=report)
-    assert found == (CANON, 0.83)
-    assert "conclusive" not in report
+    first does not hide an allowed one behind it."""
+    with _patch(_hits((0.95, OTHER), (0.83, CANON))):
+        assert await _semantic_canon(1, BODY, {CANON}) == (CANON, 0.83)
 
 
-# ── the contract callers depend on ───────────────────────────────────────
-
-
-async def test_a_caller_that_passes_no_report_still_gets_an_answer() -> None:
-    """The existing test double is an `AsyncMock(return_value=None)` that
-    never touches a report. Absence of the key must therefore mean "cannot
-    tell" at every call site — so a stub, an older caller, or an arm that
-    threw all default to asking the question rather than excusing it."""
-    with _patch(_hits()):
+async def test_a_miss_is_none_and_nothing_more() -> None:
+    with _patch(_hits((0.91, OTHER))):
         assert await _semantic_canon(1, BODY, {CANON}) is None
 
 
-@pytest.mark.parametrize("value", ["semantic", "symbol", "reference", "derive"])
-def test_the_miss_basis_is_not_one_of_the_proposal_bases(value: str) -> None:
-    """It shares a column with them and must not collide: every reader keys
-    "is there a proposal" on `proposed_snippet_id`, but `confirm_shape_proposals`
-    filters BY basis, and a collision there would mean confirming a miss as
-    though it were a match."""
-    assert BASIS_NO_SEMANTIC_MATCH != value
+async def test_a_body_too_thin_to_embed_spends_no_embedding() -> None:
+    mock = _hits()
+    with _patch(mock):
+        assert await _semantic_canon(1, TOO_THIN, {CANON}) is None
+    mock.assert_not_awaited()
 
 
-# ── the cap spends itself on the rows its verdict can act on ─────────────
+async def test_no_allowed_canon_spends_no_search() -> None:
+    """The language-family gate (#2871) empties the allowed set routinely."""
+    mock = _hits()
+    with _patch(mock):
+        assert await _semantic_canon(1, BODY, set()) is None
+    mock.assert_not_awaited()
+
+
+# ── ...and its silence reaches nothing ───────────────────────────────────
+
+
+def test_the_divergence_check_does_not_read_the_proposal_basis() -> None:
+    """The retired gate keyed on `proposal_basis`. Asserted on the function's
+    structure (rule 167): any attribute read of it inside `flag_divergence` is
+    the miss being consulted again, whatever the constant is called."""
+    tree = ast.parse(textwrap.dedent(inspect.getsource(shape_ledger.flag_divergence)))
+    reads = [n for n in ast.walk(tree)
+             if isinstance(n, ast.Attribute) and n.attr == "proposal_basis"]
+    assert not reads, (
+        "flag_divergence reads proposal_basis — a semantic miss is 'cannot "
+        "tell' at the arm's floor (#4208), never 'not the canon'"
+    )
+
+
+def test_no_miss_basis_is_stored() -> None:
+    """A miss basis sharing the column with real bases is a thing
+    `confirm_shape_proposals(basis=…)` could filter on and confirm."""
+    assert not hasattr(shape_ledger, "BASIS_NO_SEMANTIC_MATCH")
+
+
+# ── the cap spends itself where proposals matter ─────────────────────────
 
 
 class _AgedRow:
@@ -191,11 +123,10 @@ class _AgedRow:
 
 
 def test_the_capped_pass_reads_new_shapes_first() -> None:
-    """Measured on the first live refresh after the gate shipped: a version
-    bump queued 1,533 rows, the cap read 150 of them in row order, and every
-    shape NEW since the previous refresh — the only rows `flag_divergence`
-    acts on, and the highest ids — was flagged before the arm reached it. The
-    gate silenced nothing because it never got to look."""
+    """Measured on the first live refresh after a version bump: 1,533 rows
+    queued, the cap read 150 of them in row order, and every shape NEW since
+    the previous refresh — the only rows `flag_divergence` acts on, and the
+    highest ids — was flagged before the arm could propose for it."""
     from datetime import datetime, timedelta, timezone
 
     from scribe.services.shape_ledger import _semantic_priority