fix(shapes): a semantic miss is not evidence — the divergence check stops reading it (#4208)

Measured live: true instances of the service-unit canon score 0.68-0.71
against it at best, helpers 0.66-0.75 against unrelated snippets, and
nothing reaches the 0.8 floor. The "conclusive miss" fired for nearly every
body and silenced real divergences exactly as it silenced helpers. The
stored miss basis, its flag withdrawal and the report plumbing are removed;
the floor stays at 0.8 and the new-shapes-first ordering stays. False
prompts are answered by judgment (exempt with a reason).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01821k5B3Ysecp9fNYs92Kuy

src/scribe/services/shape_ledger.py

@@ -1521,43 +1521,33 @@ _SEMANTIC_CAP = 150
 # "both are about migrations"; first live run paired every alembic
 # upgrade()/downgrade() with an unrelated canon at exactly that band.
 _SEMANTIC_FLOOR = 0.8
-# How many above-floor hits the semantic arm asks for. Named because the
-# NUMBER is load-bearing twice over: it caps the work, and a result set that
-# came back short of it is a complete picture of what cleared the floor —
-# which is what lets a miss be read as evidence rather than as a cut-off
-# (`BASIS_NO_SEMANTIC_MATCH`).
+# How many above-floor hits the semantic arm asks for.
 _SEMANTIC_LIMIT = 3
-# The proposer looked at this body, compared it against every canon in its
-# language family, and matched none of them above `_SEMANTIC_FLOOR` (#4208).
+# A MISS FROM THIS ARM IS NOT EVIDENCE, and nothing may read it as one (#4208).
 #
-# This is a NEGATIVE RESULT, and it is stored because it is the only evidence
-# in the ledger that speaks to what a shape MEANS rather than what it looks
-# like. `proposal_basis` otherwise names how a proposal was arrived at; here
-# it records that the arm ran and came back empty, with `proposed_snippet_id`
-# left NULL. Every reader keys "is there a proposal" on `proposed_snippet_id`
-# or `proposal_group`, never on the basis, so this cannot be mistaken for one:
-# `list_shapes(proposal=...)` and `confirm_shape_proposals` both filter on the
-# id, and the latter requires it non-NULL before it will confirm anything.
-#
-# It is deliberately NOT written for the two cases that merely look the same:
-# a body too thin to compare (`_substance` below the write-path minimum), and
-# a row the per-refresh cap never reached. Those are "I cannot tell", and the
-# ledger's standing discipline — the one `FORM_UNKNOWN` enforces everywhere
-# else — is that not knowing must make a check quieter, never more confident.
-BASIS_NO_SEMANTIC_MATCH = "no-semantic-match"
+# #4208 briefly stored "compared, nothing cleared the floor" and let it
+# withdraw divergence prompts. Measured live on 2026-09-22, with the arm's own
+# query shape: judged instances of the service-unit canon (#2860) score 0.68–
+# 0.71 against it at best, and most fall below 0.66; helpers beside it score
+# 0.66–0.75 against snippets they have nothing to do with. Nothing reaches
+# 0.8, so the arm answers "no canon" for almost every body, true instance or
+# not — a real divergence was silenced exactly as a helper was. A code body
+# against a prose-forward snippet document measures the wrong field (#2518),
+# and no floor separates the two bands. The arm PROPOSES on a hit; its silence
+# says nothing.
 
 
 def _semantic_priority(row) -> tuple:
-    """Order for the capped semantic pass: the rows its verdict can still act on
-    come first (#4208, measured on the first live refresh after it shipped).
+    """Order for the capped semantic pass: new shapes first (#4208, measured
+    on the first live refresh after the proposer version was bumped).
 
     The pass is capped (`_SEMANTIC_CAP`), and `flag_divergence` acts only on
-    shapes NEW since the previous refresh. In plain row order those are the
-    highest ids — the back of the queue — so whenever the backlog exceeds the
-    cap (every `_PROPOSER_VERSION` bump queues the whole todo at once), the
-    cap is spent on old rows and the new ones are flagged as "cannot tell"
-    before the arm ever reads them. That is what happened: 1,533 rows queued,
-    150 checked, and all six new shapes flagged unexamined.
+    shapes NEW since the previous refresh — and skips one the proposer matched
+    to the dominant canon. In plain row order the new shapes are the highest
+    ids, the back of the queue, so whenever the backlog exceeds the cap (every
+    `_PROPOSER_VERSION` bump queues the whole todo at once) they were flagged
+    before the arm could propose for them: 1,533 rows queued, 150 checked, and
+    every new shape left unread.
 
     So: the human todo (`unclassified`) before `scoped`, newest first within
     each. A row with no creation time sorts last in its group.
@@ -1568,8 +1558,9 @@ def _semantic_priority(row) -> tuple:
 # were examined under, so a tightened rule re-examines everything once.
 # v3: language-family gate on the sym bases, reference stoplist, semantic
 # restricted to the shape's own project (#2871).
-# v4: the semantic arm records its misses as well as its hits (#4208), so
-# every already-examined row must be looked at once more to acquire one.
+# v4: the semantic arm recorded its misses too (#4208). Retired — a miss is
+# not evidence (see _SEMANTIC_LIMIT) — but the proposals themselves did not
+# change, so no re-examination is owed; a stale miss basis is inert.
 _PROPOSER_VERSION = 4
 # Signature resemblance floor, name blanked (difflib ratio) — and a length
 # floor, because `def NAME():` resembles `def NAME(x):` at 0.95 while saying
@@ -1805,29 +1796,10 @@ def _substance(text: str) -> int:
 
 
 async def _semantic_canon(
-    user_id: int, body: str, allowed: set[int], *, report: dict | None = None
+    user_id: int, body: str, allowed: set[int],
 ) -> tuple[int, float] | None:
-    """The canon this body MEANS, or None.
-
-    `report` is an out-param in the style `semantic_search_notes` already
-    uses, and it carries the one thing the return value cannot: whether a
-    None is EVIDENCE. `report["conclusive"] = True` says the arm really
-    compared this body against the allowed canons and none cleared the floor.
-    It is left unset whenever the arm could not form an opinion — a body with
-    too little substance to embed, no allowed canon to compare against, or a
-    result set that came back full and may therefore have been truncated.
-
-    The truncation case is why `_SEMANTIC_LIMIT` is named. The search returns
-    the top N above the floor; if it returns fewer than N, N was not binding
-    and we have seen everything that cleared the floor, so "no allowed canon
-    among them" is a fact about the corpus. If it returns exactly N, an
-    allowed canon could be sitting at N+1 and the same silence means nothing.
-    Reading the second case as the first is how a cut-off becomes a finding.
-
-    Callers must treat a missing key as "cannot tell", never as "no match" —
-    which is also what makes the existing test double, an `AsyncMock` that
-    returns None and touches no report, stay correct by default.
-    """
+    """The canon this body MEANS, or None. None is "no proposal", never "not
+    the canon" — see the note above `_SEMANTIC_LIMIT`."""
     from scribe.services.embeddings import semantic_search_notes
     from scribe.services.plugin_context import (
         WRITEPATH_DEFAULT_THRESHOLD, WRITEPATH_MIN_CODE_CHARS, concept_query,
@@ -1844,8 +1816,6 @@ async def _semantic_canon(
     for score, note in hits:
         if int(note.id) in allowed:
             return int(note.id), round(float(score), 3)
-    if report is not None and len(hits) < _SEMANTIC_LIMIT:
-        report["conclusive"] = True
     return None
 
 
@@ -1939,29 +1909,16 @@ async def propose_for_repo(
                 row.proposed_sha = ""
                 continue
             checked += 1
-            verdict: dict = {}
             try:
-                found = await _semantic_canon(
-                    user_id, d[5], semantic_allowed(row.path), report=verdict,
-                )
+                found = await _semantic_canon(user_id, d[5], semantic_allowed(row.path))
             except Exception:
                 logger.warning("semantic proposal failed", exc_info=True)
                 found = None
-                # An arm that threw formed no opinion. Clearing this is not
-                # belt-and-braces: a partially-filled report would record a
-                # failure as a finding about the code.
-                verdict = {}
             if found:
                 row.proposed_snippet_id, row.proposal_score = found
                 row.proposal_basis = "semantic"
                 row.proposal_group = None
                 proposed += 1
-            elif verdict.get("conclusive"):
-                # No canon, and the arm is sure of it. Kept as the row's basis
-                # with `proposed_snippet_id` still NULL, so it reads as "asked
-                # and answered" rather than "not asked" — the distinction
-                # `flag_divergence` needs and could not previously make.
-                row.proposal_basis = BASIS_NO_SEMANTIC_MATCH
         await session.commit()
     return {"examined": examined, "proposed": proposed, "semantic_checked": checked}
 
@@ -2486,9 +2443,7 @@ async def flag_divergence(project_id: int, *, since: datetime | None) -> int:
     """Flag shapes created after ``since`` (the previous refresh) that sit
     where a canon dominates and were not proposed as that canon. With no
     previous refresh (first seed) nothing is new, nothing is flagged.
-    Standing flags persist until judged — or until the proposer's semantic
-    arm reports a conclusive miss for the row (#4208). Returns how many are
-    flagged."""
+    Standing flags persist until judged. Returns how many are flagged."""
     if since is None:
         return 0
     async with async_session() as session:
@@ -2512,17 +2467,6 @@ async def flag_divergence(project_id: int, *, since: datetime | None) -> int:
             for r in siblings:
                 if r.status not in _MECHANICAL_TODO:
                     continue
-                # A conclusive miss outranks a standing flag, and is checked
-                # before it. A flag raised on a row the semantic arm had not
-                # yet reached was raised on "cannot tell" — the arm is capped,
-                # so that is routine — and once the arm has read the body and
-                # found it is none of the canons, keeping the prompt would be
-                # asserting over a measurement we now hold. Only this evidence
-                # withdraws a flag; nothing else changes "persist until judged".
-                if r.proposal_basis == BASIS_NO_SEMANTIC_MATCH:
-                    if r.diverges_from is not None:
-                        r.diverges_from = None
-                    continue
                 if r.diverges_from is not None:
                     flagged += 1
                     continue
@@ -2530,25 +2474,14 @@ async def flag_divergence(project_id: int, *, since: datetime | None) -> int:
                     continue
                 if r.proposed_snippet_id == dom[0]:
                     continue  # the proposer already says "instance of the canon"
-                # ...and the converse, which is the only evidence here that
-                # is about MEANING rather than shape (#4208). The four false
-                # prompts #4204 left standing are callables in a directory of
-                # callables: at the signature level they are indistinguishable
-                # from #2793's acceptance case, a sync `confirmDanger` beside
-                # an async confirm canon, and no refinement of `shape_form`
-                # ever separates them — a registry accessor and a service unit
-                # differ by the JOB they do, which a signature does not carry.
-                #
-                # The proposer does read bodies, and when its semantic arm
-                # compared this one against every canon in its language family
-                # and matched none of them, that is a positive finding that
-                # this shape is not the canon's work. Urging the canon anyway
-                # would be asserting over a measurement we already hold.
-                #
-                # Only the conclusive miss is stored, so an unexamined row and
-                # a body too thin to embed still ask the question rather than
-                # being quietly excused. (Tested at the top of this loop, where
-                # it also withdraws a flag raised before the arm got there.)
+                # NOT the converse (#4208). A semantic MISS does not mean "not
+                # the canon": measured, true instances of a canon rarely clear
+                # the arm's floor either (see the note above _SEMANTIC_LIMIT).
+                # A helper beside a canon and a genuine divergence — #2793's
+                # sync `confirmDanger` beside an async confirm canon — are the
+                # same case to every signal this ledger holds, so the prompt
+                # asks, and the judge answers it (`exempt` with a reason
+                # retires a false one for good).
                 # The same structural test the write-time check applies
                 # (#4204). The sweep and the hook must agree about what counts
                 # as divergence, or an audit contradicts the line the writer