fix(drafter): a wrapped docstring line beginning "class AND the" defines a shape called AND (#4222)

The definition extractor is line-oriented and knows nothing about what a
line is INSIDE. A docstring that wraps onto a line starting with a keyword
announces a definition: `AND` reached a live session as a divergence prompt
asking it to justify a symbol that does not exist, and `is` reached it as a
repo-wide duplicate of four files that define nothing of the kind.

Measured, not assumed: running the extractor over every scannable file with
and without the scan differs by twenty-two phantoms. Two of them — `with`
and `nobody`, both out of the module docstring in check_dangling_styles.py —
are persisted `code_shapes` rows that have been judged. Those need no
migration: sync_shapes marks a row it no longer extracts as vanished.

`ast` would be the honest tool for .py and is not what this uses, because
the extractor is mirrored rule for rule by an awk program in the hook, awk
cannot parse Python, and a fix only one of the pair can run is the drift the
mirror exists to prevent. Both sides now run the same left-to-right scan and
blank comment and string spans to their own newlines before any matcher sees
a line. Three things the scan has to get right, each of which cost real
definitions while it was being written:

  - a string that HOLDS a marker is not a marker. `"red /* "` in
    test_design_stylesheet.py and a triple quote inside a single-quoted
    regex in plugin_context.py each ate every definition below them.
  - `#` is a colour in CSS and a comment in Python, and the extractor is
    handed no path. An alphanumeric straight after it settles it.
  - an unterminated opener blanks NOTHING. The scan rewinds past it and
    continues, so a stray marker costs one span rather than the rest of the
    file.

The comment claiming the two extractors agree has been the only thing
holding them together, and a comment cannot fail. The mirror test now RUNS
the hook's awk over the same vectors: with the old program it reports the
phantoms, which is what a guard that can fail looks like. Across all 631
scannable files in this repo the two now agree line for line.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01821k5B3Ysecp9fNYs92Kuy

tests/test_pattern_coverage.py

@@ -10,7 +10,10 @@ tripwire.
 """
 import io
 import json
+import shutil
+import subprocess
 import tarfile
+from pathlib import Path
 
 import pytest
 import pytest_asyncio
@@ -28,6 +31,8 @@ from scribe.services.coverage import (
 from scribe.services.shape_ledger import location_covers
 from tests.helpers import ensure_user
 
+PLUGIN = Path(__file__).resolve().parents[1] / "plugin"
+
 # --- unit: the definition extractor (shared vectors with the hook) -----------
 
 EXTRACTION_VECTORS = [
@@ -63,6 +68,41 @@ EXTRACTION_VECTORS = [
      []),
     ("dedup-within-file", "def f():\n    pass\ndef f():\n    pass\n",
      [("sym", "f")]),
+    # --- comment and string spans (#4222) ------------------------------------
+    # Prose is not code. A wrapped docstring line beginning "class AND the"
+    # announced a shape called `AND` to a live session; `with` and `nobody`
+    # out of one module docstring in scripts/check_dangling_styles.py reached
+    # persisted, judged `code_shapes` rows.
+    ("docstring-prose",
+     'def real_one():\n    """Its own text is about the\n'
+     '    class AND the to_dict, and is a def bar():\n'
+     '    class Foo: lives here too.\n    """\n    pass\n',
+     [("sym", "real_one")]),
+    # An opener with no closer blanks NOTHING: the scan rewinds past it, so a
+    # stray marker costs one span rather than the rest of the file.
+    ("unterminated-docstring",
+     'def before():\n    """oops, never closed\n\ndef after():\n    pass\n',
+     [("sym", "before"), ("sym", "after")]),
+    # The CSS half of the same defect (#2990): a wrapped comment line that
+    # happens to begin with a dotted token reads as a selector.
+    ("css-comment-selector",
+     "/* A real base rule, not just descendants: the check reads a\n"
+     "   .ghost, class that only ever appears as an ancestor */\n.check { }\n",
+     [("css", "check")]),
+    # A string that HOLDS a comment marker is not a comment — the case that
+    # made the first draft of the scan eat 70 lines of live code.
+    ("string-holds-a-marker",
+     'SAMPLE = "red /* "\ndef after_the_string():\n    pass\n',
+     [("sym", "after_the_string")]),
+    # `#` is the one marker whose meaning is the language\'s: a colour here,
+    # a comment two lines down, and the extractor is handed no path.
+    ("hash-is-a-colour-not-a-comment",
+     ".a { color: #fff; } /* .ghost,\n   class Phantom: */\n.b { }\n",
+     [("css", "a"), ("css", "b")]),
+    ("line-comment-mentioning-a-docstring",
+     'def kept():\n    pass\n# a stray """ in a comment\n'
+     'def also_kept():\n    pass\n',
+     [("sym", "kept"), ("sym", "also_kept")]),
 ]
 
 
@@ -75,6 +115,37 @@ def test_extractor_agrees_with_the_hook_on_what_defines(text, expected):
     assert extract_shapes(text) == expected
 
 
+@pytest.mark.parametrize(
+    "text",
+    [t for _i, t, _e in EXTRACTION_VECTORS],
+    ids=[i for i, _t, _e in EXTRACTION_VECTORS],
+)
+def test_the_hook_extractor_runs_and_agrees_line_for_line(text):
+    """The comment at the top of this module has been the only thing holding
+    the two extractors together, and a comment cannot fail. This RUNS the
+    hook's awk program over the same vectors.
+
+    The hook emits every definition in source order with no dedup — identity
+    there is per payload, not per file — so the comparison de-dupes its
+    output before matching, which is the one difference between the two that
+    is by design.
+    """
+    if shutil.which("awk") is None:            # pragma: no cover - env guard
+        pytest.skip("awk not available")
+    lib = PLUGIN / "hooks" / "scribe_defs.sh"
+    out = subprocess.run(
+        ["bash", "-c", f'. "{lib}"; scribe_defs'],
+        input=text, capture_output=True, text=True,
+    )
+    assert out.returncode == 0, out.stderr
+    seen: list[tuple[str, str]] = []
+    for line in out.stdout.splitlines():
+        kind, _, name = line.partition("\t")
+        if name and (kind, name) not in seen:
+            seen.append((kind, name))
+    assert seen == extract_shapes(text)
+
+
 def test_scannable_gates_prose_vendored_and_sourcemaps():
     assert scannable("src/app.py")
     assert scannable("web/button.css")