Files
FabledScribe/tests/test_verification_guidance_survives.py
T
bvandeusen 1d65e98ac2
CI & Build / Python lint (push) Successful in 3s
CI & Build / Plugin hooks (push) Successful in 10s
CI & Build / TypeScript typecheck (push) Successful in 33s
CI & Build / integration (push) Successful in 43s
CI & Build / Python tests (push) Successful in 1m16s
CI & Build / Build & push image (push) Successful in 28s
docs(notes): the surfaces say WHEN a note earns a check, and a guard keeps them saying it (#3168, milestone 317 step 5)
The step that decides whether steps 1-4 were worth building. `verify_with` is
a free-text field on the highest-volume record kind in the product; described
only as "how to verify this note" it gets filled in on every note within a
week, and at that point the sweep returns the whole corpus and means nothing.
The signal is not "has a check" — it is "has a check AND almost nothing else
does".

Rule 119 puts this in the app's own instruction surfaces, never in a Scribe
rule. So:

- the using-scribe skill gains the reflex, next to "state updates in place;
  chronicles don't" — its other half, since supersession only fires once
  somebody has read a note and disagreed, which is the case where it was
  already believed;
- update_note's docstring now states the DEFAULT rather than only deferring
  to create_note for the test. Found by the new guard on its first run;
- plugin 0.1.48.

THE _INSTRUCTIONS BUDGET, decided rather than skipped. The payload is ~1980 of
the client's ~2048-char cap, so everything in it competes for the last ~68
characters. The operator declined a line for this milestone: the map's own
closing sentence says each tool's description carries the full contract, and
the sweep is a curation act rather than a session-start reflex like
enter_project or list_always_on_rules. That reasoning is now a comment beside
the constant, with the accepted cost named — an agent that never opens
create_note's docstring never learns the field exists — so the question is not
re-litigated blind.

The guard pins STRUCTURE, never wording, for the disambiguator's reason (a
test that punishes rewriting is a test that gets deleted): each write surface
must still draw the norm-vs-constraint distinction, say the empty case is
normal, and name where NOT to reach for it — plus that the skill carries the
one-question form, because the docstrings only reach a caller who already
opened the tool.

`_doc` moves to tests/helpers as `tool_doc`; it had been written twice.
2026-08-29 00:10:43 -04:00

109 lines
4.8 KiB
Python

"""The write surfaces say WHEN a note earns a check — not just that it can.
WHY THIS EXISTS
`verify_with` is a free-text field on the highest-volume record kind in the
product. A field described only as "how to verify this note" gets filled in on
every note within a week, and at that point the sweep returns the whole corpus
and means nothing. The signal is not "has a check" — it is "has a check AND
almost nothing else does".
So the guidance is not decoration on this feature, it IS the feature's
precondition, and rule 119 puts it in the tool docstrings and the skill rather
than in a Scribe rule. That makes it exactly the kind of prose a later
docstring tidy-up deletes without noticing what it was for.
WHAT THIS PINS, AND WHAT IT DOES NOT
Structure, never wording, for `test_create_tools_disambiguate`'s reason: a test
that punishes rewriting is a test that gets deleted. Each write surface must
still (a) draw the norm-vs-constraint distinction in some form, (b) say the
empty case is normal, and (c) name where NOT to reach for it.
It cannot tell whether the guidance is any good — only that the paragraph
explaining when to leave the field alone has not quietly become a parameter
list.
"""
import re
import pytest
from tests.helpers import tool_doc as _doc
# The surfaces that OFFER the field. The read surfaces (the sweep, the stamp)
# explain what a RESULT means, which is a different job — they are deliberately
# not held to this.
_WRITE_SURFACES = [
("scribe.mcp.tools.notes", "create_note"),
("scribe.mcp.tools.notes", "update_note"),
]
@pytest.mark.parametrize("module,name", _WRITE_SURFACES)
def test_the_norm_versus_constraint_distinction_is_stated(module, name):
"""The whole discipline in one line: a decision cannot go stale, a claim
about someone else's software can. Without it, "how would you check this"
reads as a chore to complete rather than a question with a usual answer of
"you wouldn't"."""
doc = _doc(module, name).lower()
assert "norm" in doc and "constraint" in doc, (
f"{name} no longer draws the norm-vs-constraint distinction. Without "
f"it the field is just a box, and a box gets filled in."
)
@pytest.mark.parametrize("module,name", _WRITE_SURFACES)
def test_the_empty_case_is_stated_as_normal(module, name):
"""Said POSITIVELY, or an empty field reads as an unfinished record. This
is the single sentence standing between the sweep and irrelevance."""
doc = _doc(module, name).lower()
assert re.search(r"leave (it|both|them|this|these)? ?empty|empty for", doc), (
f"{name} no longer says that leaving the check empty is the normal "
f"case. Most notes are decisions; the field's default must read as a "
f"deliberate state, not a gap."
)
@pytest.mark.parametrize("module,name", _WRITE_SURFACES)
def test_the_wrong_places_to_reach_for_it_are_named(module, name):
"""A task's decay is its status; a snippet has verify_snippet. The service
refuses both — this is what should mean nobody ever hits that error."""
doc = _doc(module, name).lower()
assert "task" in doc and "snippet" in doc, (
f"{name} no longer names the records that must NOT carry a check. The "
f"gate still refuses them, but a refusal the caller could have "
f"foreseen is a worse door than one that explained itself."
)
def test_expires_when_is_described_as_a_state_not_a_date():
"""The one field whose obvious reading is wrong. A date invents a staleness
schedule nobody can justify; a constraint expires when the ground moves,
which is a condition and not a time."""
doc = _doc("scribe.mcp.tools.notes", "create_note")
assert "STATE" in doc, (
"create_note no longer says expires_when is a STATE. Left to itself, "
'"expires" reads as a date, and every check would get an arbitrary one.'
)
def test_the_skill_carries_the_test_a_writer_can_actually_apply():
"""The docstrings are read by whatever is holding the tool; the skill is
read while deciding what to write. The one-question form has to be in the
second place too, or the guidance only reaches callers who already opened
the tool."""
import pathlib
skill = pathlib.Path(__file__).resolve().parents[1] / (
"plugin/skills/using-scribe/SKILL.md"
)
text = " ".join(skill.read_text().split())
assert "could this note become false without anyone editing it" in text.lower(), (
"the using-scribe skill no longer carries the one-question test. That "
"question is what makes the distinction applicable rather than merely "
"true."
)
assert "notes_due_for_verification" in text, (
"the skill names the fields but not the surface that reads them — "
"guidance for writing a check with no route to acting on one."
)