fix(shapes): a semantic miss is not evidence — the divergence check stops reading it (#4208)
CI & Build / Python lint (push) Successful in 2s
CI & Build / Plugin hooks (push) Successful in 11s
CI & Build / integration (push) Successful in 53s
CI & Build / TypeScript typecheck (push) Successful in 54s
CI & Build / Python tests (push) Successful in 1m32s
CI & Build / Build & push image (push) Successful in 24s
CI & Build / Python lint (push) Successful in 2s
CI & Build / Plugin hooks (push) Successful in 11s
CI & Build / integration (push) Successful in 53s
CI & Build / TypeScript typecheck (push) Successful in 54s
CI & Build / Python tests (push) Successful in 1m32s
CI & Build / Build & push image (push) Successful in 24s
Measured live: true instances of the service-unit canon score 0.68-0.71 against it at best, helpers 0.66-0.75 against unrelated snippets, and nothing reaches the 0.8 floor. The "conclusive miss" fired for nearly every body and silenced real divergences exactly as it silenced helpers. The stored miss basis, its flag withdrawal and the report plumbing are removed; the floor stays at 0.8 and the new-shapes-first ordering stays. False prompts are answered by judgment (exempt with a reason). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01821k5B3Ysecp9fNYs92Kuy
This commit is contained in:
@@ -890,37 +890,26 @@ async def test_a_second_confirm_dialog_is_detected_and_named(seeded):
|
||||
|
||||
|
||||
@pytest.mark.integration
|
||||
async def test_a_conclusive_meaning_miss_silences_what_the_signature_cannot(seeded):
|
||||
"""#4208: the four false prompts #4204's form gate provably cannot reach.
|
||||
async def test_a_semantic_miss_does_not_silence_the_prompt(seeded):
|
||||
"""#4208, reversed on measurement: a semantic miss is not evidence.
|
||||
|
||||
THE FIXTURE IS THE ACCEPTANCE CASE ABOVE, DELIBERATELY. That is the whole
|
||||
difficulty of this issue: a hand-rolled `confirmDanger` beside an async
|
||||
confirm canon is structurally IDENTICAL to a registry helper beside an
|
||||
async service canon — same family, same form contradiction, same directory
|
||||
density. The form gate has to keep asking about both, so nothing derived
|
||||
from a signature can separate them. The only difference is whether the
|
||||
shape does the canon's JOB, and the only reading of that the ledger holds
|
||||
is the proposer's per-symbol body comparison.
|
||||
THE FIXTURE IS THE ACCEPTANCE CASE ABOVE, DELIBERATELY. A hand-rolled
|
||||
`confirmDanger` beside an async confirm canon is structurally identical to
|
||||
a registry helper beside an async service canon, and the arm's miss was
|
||||
meant to tell them apart. Measured live, true instances of a canon rarely
|
||||
clear the arm's 0.8 floor either, so a miss fires on both — silencing on
|
||||
it silenced #2793's own case. Here the arm has read the body, found
|
||||
nothing, and the prompt is still RAISED.
|
||||
|
||||
So the two runs differ in exactly one thing. In the test above the semantic
|
||||
arm is quiet — it answers "nothing" without claiming to have looked — and
|
||||
the prompt is RAISED, which is what milestone #2793 exists to produce. Here
|
||||
it answers "I compared this body against the canons in its family and it is
|
||||
none of them", and the prompt is WITHHELD. Holding the fixture identical is
|
||||
what makes this a test of the meaning gate rather than of the setup.
|
||||
|
||||
Asserted on the stored basis as well as the outcome, so that a future
|
||||
change which silences the prompt for some other reason fails here instead
|
||||
of reading as a pass.
|
||||
Asserted on the row, not a flag count: `since` sits a second early to
|
||||
absorb clock skew, which also admits other fixture rows as "new".
|
||||
"""
|
||||
from datetime import datetime, timedelta, timezone
|
||||
from unittest.mock import AsyncMock, patch
|
||||
|
||||
from scribe.services import shape_ledger
|
||||
from scribe.services import snippets as snippets_svc
|
||||
from scribe.services.shape_ledger import (
|
||||
BASIS_NO_SEMANTIC_MATCH, flag_divergence, live_rows, propose_for_repo,
|
||||
)
|
||||
from scribe.services.shape_ledger import flag_divergence, live_rows, propose_for_repo
|
||||
|
||||
owner, pid = seeded["owner"], seeded["pid"]
|
||||
canon = await snippets_svc.create_snippet(
|
||||
@@ -950,28 +939,15 @@ async def test_a_conclusive_meaning_miss_silences_what_the_signature_cannot(seed
|
||||
)
|
||||
await sync_repo_shapes(pid, REPO, later, seen_marker="bbb222")
|
||||
|
||||
def _conclusive_miss(*_args, report=None, **_kw):
|
||||
"""The arm ran, compared, and found no canon — the one empty answer
|
||||
that is evidence. `_semantic_canon` itself decides when it may say
|
||||
this (a result set shorter than the limit); the unit tests for that
|
||||
judgment are in tests/test_divergence_meaning_gate.py."""
|
||||
if report is not None:
|
||||
report["conclusive"] = True
|
||||
return None
|
||||
|
||||
with patch.object(shape_ledger, "_semantic_canon",
|
||||
AsyncMock(side_effect=_conclusive_miss)):
|
||||
miss = AsyncMock(return_value=None)
|
||||
with patch.object(shape_ledger, "_semantic_canon", miss):
|
||||
await propose_for_repo(owner, pid, REPO, later)
|
||||
assert miss.await_count >= 1, "the arm must actually have read a body"
|
||||
|
||||
rows = await live_rows(pid)
|
||||
danger = next(r for r in rows if r.symbol == "confirmDanger")
|
||||
assert danger.proposal_basis == BASIS_NO_SEMANTIC_MATCH
|
||||
# The miss is not a proposal: nothing may read it as one.
|
||||
await flag_divergence(pid, since=previous - timedelta(seconds=1))
|
||||
danger = next(r for r in await live_rows(pid) if r.symbol == "confirmDanger")
|
||||
assert danger.proposed_snippet_id is None
|
||||
|
||||
assert await flag_divergence(pid, since=previous - timedelta(seconds=1)) == 0
|
||||
_, total = await list_project_shapes(owner, pid, flag="divergence")
|
||||
assert total == 0, "a shape the proposer measured as unrelated must not be urged"
|
||||
assert danger.diverges_from == sid, "a miss at the arm's floor is 'cannot tell'"
|
||||
|
||||
|
||||
async def _two_generations(seeded, tag: str):
|
||||
@@ -1016,12 +992,6 @@ async def _two_generations(seeded, tag: str):
|
||||
return owner, pid, later, previous
|
||||
|
||||
|
||||
def _conclusive(*_a, report=None, **_k):
|
||||
if report is not None:
|
||||
report["conclusive"] = True
|
||||
return None
|
||||
|
||||
|
||||
@pytest.mark.integration
|
||||
async def test_the_capped_semantic_pass_reaches_the_new_shape_first(seeded):
|
||||
"""Measured live: after the #4208 version bump, 1,533 rows competed for 150
|
||||
@@ -1031,64 +1001,19 @@ async def test_the_capped_semantic_pass_reaches_the_new_shape_first(seeded):
|
||||
from unittest.mock import AsyncMock, patch
|
||||
|
||||
from scribe.services import shape_ledger
|
||||
from scribe.services.shape_ledger import (
|
||||
BASIS_NO_SEMANTIC_MATCH, live_rows, propose_for_repo,
|
||||
)
|
||||
from scribe.services.shape_ledger import live_rows, propose_for_repo
|
||||
|
||||
owner, pid, later, _previous = await _two_generations(seeded, "cap")
|
||||
with patch.object(shape_ledger, "_semantic_canon",
|
||||
AsyncMock(side_effect=_conclusive)):
|
||||
with patch.object(shape_ledger, "_semantic_canon", AsyncMock(return_value=None)):
|
||||
stats = await propose_for_repo(owner, pid, REPO, later, semantic_cap=1)
|
||||
assert stats["semantic_checked"] == 1
|
||||
|
||||
rows = {r.symbol: r for r in await live_rows(pid)}
|
||||
assert rows["confirmDanger"].proposal_basis == BASIS_NO_SEMANTIC_MATCH
|
||||
assert rows["confirmDanger"].proposed_sha != "", "the new shape got the one check"
|
||||
# The old row is left for the next refresh, unexamined — never stamped.
|
||||
assert rows["oldHelper"].proposal_basis is None
|
||||
assert rows["oldHelper"].proposed_sha == ""
|
||||
|
||||
|
||||
@pytest.mark.integration
|
||||
async def test_a_later_conclusive_miss_withdraws_a_flag_raised_before_it(seeded):
|
||||
"""A flag raised while the arm had not yet read the row was raised on
|
||||
"cannot tell". When a later refresh reads the body and finds it is none of
|
||||
the canons, the flag goes — and only that evidence withdraws one."""
|
||||
from datetime import timedelta
|
||||
from unittest.mock import AsyncMock, patch
|
||||
|
||||
from scribe.services import shape_ledger
|
||||
from scribe.services.shape_ledger import (
|
||||
BASIS_NO_SEMANTIC_MATCH, flag_divergence, live_rows, propose_for_repo,
|
||||
)
|
||||
|
||||
owner, pid, later, previous = await _two_generations(seeded, "withdraw")
|
||||
since = previous - timedelta(seconds=1)
|
||||
|
||||
async def _danger():
|
||||
return next(r for r in await live_rows(pid) if r.symbol == "confirmDanger")
|
||||
|
||||
# Asserted on the ROW, not on a flag count. `since` sits a second before
|
||||
# the second sync to absorb clock skew, which also admits the fixture's
|
||||
# old helper as "new" — how many rows get flagged is a property of that
|
||||
# margin, and this test is about one row's flag coming off.
|
||||
|
||||
# First refresh: the arm is quiet (formed no opinion), so the prompt is raised.
|
||||
with patch.object(shape_ledger, "_semantic_canon", AsyncMock(return_value=None)):
|
||||
await propose_for_repo(owner, pid, REPO, later)
|
||||
await flag_divergence(pid, since=since)
|
||||
assert (await _danger()).diverges_from is not None
|
||||
|
||||
# A later refresh re-reads the body (a ruleset bump forces it) and is sure.
|
||||
with patch.object(shape_ledger, "_PROPOSER_VERSION", 10_000), \
|
||||
patch.object(shape_ledger, "_semantic_canon",
|
||||
AsyncMock(side_effect=_conclusive)):
|
||||
await propose_for_repo(owner, pid, REPO, later)
|
||||
await flag_divergence(pid, since=since)
|
||||
danger = await _danger()
|
||||
assert danger.proposal_basis == BASIS_NO_SEMANTIC_MATCH
|
||||
assert danger.diverges_from is None
|
||||
|
||||
|
||||
@pytest.mark.integration
|
||||
async def test_history_records_what_was_used_when_and_drift_asks_for_a_recheck(seeded):
|
||||
from scribe.services.shape_ledger import shape_history
|
||||
|
||||
Reference in New Issue
Block a user