Files
FabledScribe/tests/test_design_system.py
T
bvandeusenandClaude Opus 5 d3ee24f239
CI & Build / Python lint (push) Successful in 5s
CI & Build / Plugin hooks (push) Successful in 10s
CI & Build / integration (push) Successful in 17s
CI & Build / TypeScript typecheck (push) Successful in 35s
CI & Build / Python tests (push) Successful in 44s
CI & Build / Build & push image (push) Successful in 27s
feat(design-explorer): rulebook binding + prose→claims extraction
Milestone #251 step 2 (#2259). The half of the drift panel that needed to be
testable, which is why it is Python: the frontend has no test runner, so the
fiddly extraction lives server-side and the browser only does set arithmetic
over live token values.

BINDING. A per-user setting `design_rulebook_id` names the rulebook that
describes this install's design system. A setting rather than a column: no
migration, discoverable in the Settings UI (rule #25), and honest about being a
per-install choice rather than a property of the rulebook. No rulebook
designated returns an empty set with rulebook_id: null — the NORMAL case for any
install but the one that set it up (rule #115), which the client renders as an
explanatory empty state rather than an error. The id comes back alongside the
list so "not designated" and "designated but empty" stay distinguishable.

EXTRACTION. No NLP. Rule statements are prose written for humans and should stay
that way, so this takes only what is unambiguous in any prose — the hex colours
and custom-property names a rule mentions. Anything subtler needs a rule author
to opt into a structured form, deliberately left for when someone wants it.

Three things earn their complexity:

- SENTENCE-SCOPED NEGATION. A rule routinely states what the palette requires and
  what it forbids in consecutive sentences ("Parchment #E8E4D8 …, Vellum #C2BFB4
  …. Pure white #FFFFFF is NEVER used."). Detecting negation across the whole
  statement would mark the required colours as forbidden — inverting the finding
  rather than missing it, which is worse. Per sentence, all four come out right.

- HEX NORMALISATION is load-bearing, not tidiness. The rulebook writes #FFFFFF
  and components write #fff; if those don't compare equal the largest drift
  finding in the codebase — 67 hardcoded white text colours (#2275) — reads as
  zero. Alpha forms keep their alpha, since #fff and #ffff are different colours
  and collapsing them would manufacture equality.

- SLASH SHORTHAND. Rulebooks write token families as --fs-radius-sm/md/lg/xl and
  --fs-obsidian/iron/slate/pewter. Both expand under one rule — prefix is
  everything up to and including the LAST hyphen of the first segment — which
  also handles --fs-dur-fast/base/slow. Verified against the real rule text: 18
  tokens from three different shorthand shapes.

how_to_apply is read alongside statement, because rulebooks routinely keep the
statement declarative and put the concrete values in how_to_apply; ignoring it
would miss the checkable half.

Claims dedupe on (kind, value), first source winning, so a colour named by
several rules is one expectation attributed to the rule that introduced it.
Prose with nothing checkable yields nothing — most rules are judgement, not
specification, and a panel that reported unparseable rules as problems would be
unusable.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UaYUaouG9jjhATyuxCKrQs
2026-07-30 15:25:36 -04:00

162 lines
6.6 KiB
Python

"""Rulebook prose → checkable claims (milestone #251 step 2).
This is the piece of the design explorer that most needed to be testable, which
is why it lives in Python at all: the frontend has no test runner, so the fiddly
extraction happens server-side and the browser only does set arithmetic over it.
Rule text below is representative of a real design rulebook rather than copied
from this operator's — rule #115: the product must work for an install that has
none of their data, and a test that only passes against their exact wording would
be testing the instance, not the parser.
"""
from types import SimpleNamespace
from scribe.services.design_system import (
expand_token_shorthand,
extract_expectations,
normalize_hex,
)
def _rule(rule_id, title, statement, how_to_apply=None):
return SimpleNamespace(
id=rule_id, title=title, statement=statement, how_to_apply=how_to_apply
)
# --- hex normalisation -------------------------------------------------------
def test_normalize_hex_makes_shorthand_and_case_comparable():
"""LOAD-BEARING. The rulebook writes `#FFFFFF` and components write `#fff`.
If those don't compare equal, the single largest drift finding — 67 hardcoded
white text colours (#2275) — reads as zero findings."""
assert normalize_hex("#fff") == normalize_hex("#FFFFFF") == "#ffffff"
assert normalize_hex("#E8E4D8") == "#e8e4d8"
assert normalize_hex("#14171a") == "#14171a"
def test_normalize_hex_keeps_alpha_rather_than_inventing_equality():
"""`#fff` and `#ffff` are different colours. Dropping the alpha to make them
match would manufacture agreement that isn't there."""
assert normalize_hex("#ffff") == "#ffffffff"
assert normalize_hex("#fff") != normalize_hex("#ffff")
def test_normalize_hex_rejects_non_colours():
for junk in ("", " ", "not-a-colour", "#", "#gg", "#12345"):
assert normalize_hex(junk) is None
# --- the slash shorthand -----------------------------------------------------
def test_expand_token_shorthand_handles_every_form_a_rulebook_uses():
"""One rule expands all three shapes: take everything up to and including the
LAST hyphen of the first segment as the prefix."""
assert expand_token_shorthand("--fs-radius-sm/md/lg/xl") == [
"--fs-radius-sm", "--fs-radius-md", "--fs-radius-lg", "--fs-radius-xl",
]
# Prefix is just `--fs-` here, and the same rule finds it.
assert expand_token_shorthand("--fs-obsidian/iron/slate/pewter") == [
"--fs-obsidian", "--fs-iron", "--fs-slate", "--fs-pewter",
]
assert expand_token_shorthand("--fs-dur-fast/base/slow") == [
"--fs-dur-fast", "--fs-dur-base", "--fs-dur-slow",
]
def test_expand_token_shorthand_passes_plain_names_through():
assert expand_token_shorthand("--fs-ease") == ["--fs-ease"]
# --- extraction --------------------------------------------------------------
def test_negation_is_scoped_to_the_sentence_not_the_rule():
"""THE trick that makes prohibition detection usable.
A single rule routinely states what the palette REQUIRES and what it FORBIDS
in consecutive sentences. Detecting negation across the whole statement would
mark the required colours as forbidden too — inverting the finding rather
than missing it, which is worse.
"""
rule = _rule(
52, "Text palette",
"Text tokens: Parchment #E8E4D8 (primary), Vellum #C2BFB4 (secondary), "
"Ash #9C9A92 (tertiary). Pure white #FFFFFF is NEVER used as text color.",
)
found = extract_expectations([rule])
required = {e.value for e in found if e.kind == "color"}
forbidden = {e.value for e in found if e.kind == "prohibited_color"}
assert required == {"#e8e4d8", "#c2bfb4", "#9c9a92"}
assert forbidden == {"#ffffff"}
assert not (required & forbidden)
def test_token_names_are_extracted_and_expanded():
rule = _rule(
72, "CSS custom properties",
"Expose the system as custom properties on :root — surfaces "
"(--fs-obsidian/iron/slate/pewter), radius (--fs-radius-sm/md/lg/xl), "
"and motion (--fs-ease).",
)
names = {e.value for e in extract_expectations([rule]) if e.kind == "token"}
assert "--fs-obsidian" in names and "--fs-pewter" in names
assert "--fs-radius-xl" in names
assert "--fs-ease" in names
assert len(names) == 9
def test_how_to_apply_is_read_as_well_as_the_statement():
"""Rulebooks routinely put the concrete values in how_to_apply and keep the
statement declarative, so ignoring it would miss the checkable half."""
rule = _rule(
56, "Per-app accent", "Each app owns exactly one accent.",
how_to_apply='[data-app="scribe"] #5B4A8A, [data-app="minstrel"] #4A6B5C.',
)
colours = {e.value for e in extract_expectations([rule]) if e.kind == "color"}
assert colours == {"#5b4a8a", "#4a6b5c"}
def test_claims_are_deduped_across_rules_keeping_the_first_source():
"""A colour named by several rules is one expectation, attributed to the rule
that introduced it — usually the most specific place to send a reader."""
rules = [
_rule(51, "Surfaces", "Obsidian #14171A is the page background."),
_rule(99, "Elsewhere", "Obsidian #14171A again, mentioned in passing."),
]
found = [e for e in extract_expectations(rules) if e.kind == "color"]
assert len(found) == 1
assert found[0].rule_id == 51
def test_prose_with_nothing_checkable_yields_nothing():
"""Most rules are judgement, not specification. They must contribute no
findings rather than a shrug — a panel that reports unparseable rules as
problems would be unusable."""
rule = _rule(
68, "Voice and tone",
"Voice is understated: plain language for anything functional, flavour "
"only where the user is waiting or failing. Be brief.",
)
assert extract_expectations([rule]) == []
def test_every_expectation_carries_the_sentence_it_came_from():
"""The panel has to show its working — "the rulebook says X" is only
actionable if you can see where, and in what context.
Asserts the context is the SENTENCE, not the whole statement: a rule that
states a requirement and a prohibition in consecutive sentences would
otherwise attribute both to the same undifferentiated blob of prose.
"""
rule = _rule(63, "Radius", "Radius: Small 4px. Pure white #FFFFFF is never used.")
found = extract_expectations([rule])
assert len(found) == 1
only = found[0]
assert only.kind == "prohibited_color"
assert only.rule_id == 63
assert only.rule_title == "Radius"
assert only.context == "Pure white #FFFFFF is never used."
assert "Radius: Small 4px" not in only.context