CI & Build / Python lint (push) Successful in 3s
CI & Build / Plugin hooks (push) Successful in 15s
CI & Build / TypeScript typecheck (push) Successful in 53s
CI & Build / integration (push) Successful in 56s
CI & Build / Python tests (push) Successful in 1m45s
CI & Build / Build & push image (push) Successful in 16s
Milestone 419's acceptance, and it failed the first time it was run — which
is the only reason this commit exists.
THE MEASUREMENT. Ran the step-5 readout against this session's real traffic
instead of a fixture. It reported 19 rules named by an arm and none opened.
That is false: the session had called `get_rule` 45 times. Across six real
sessions on this instance: 208 opens, 3 surviving ledger entries. 1.4%.
THE CAUSE. `.opened.ids` was doing two jobs with opposite lifetimes.
- "this context HOLDS rule 156" — false after a compaction, and three hooks
read it to decide whether to stay quiet. Clearing it is correct.
- "rule 156 WAS OPENED" — which no compaction makes untrue, and which the
session-end readout is built on.
`scribe_clear_session_ledgers` sweeps every `<sid>*.ids` on SessionStart
source=compact. Right for the first claim, and it was deleting the second.
The TTL did the same thing more quietly: `scribe_rules_live` ages an
exclusion ledger, which is right, and would have eaten the early part of any
long session's evidence too.
So the readout was reporting only the stretch since the last compaction while
reading as though it had reported the session — a statistic that cannot vary
being mistaken for a finding (#3311), which is the shape this whole milestone
exists to stop producing. It ran AT the seam it was blind to.
THE SPLIT. `scribe_rules_append` now writes both: the exclusion ledger it
always wrote, and `<kind>.keep.ids`, an evidence twin that is never aged and
never swept. The readout reads twins; the three `held` readers are untouched.
DERIVED, NOT LISTED, because a list is what broke this before — the comment
above the sweep says so about its own history. Every ledger written through
the appender gets a twin, including the next one somebody adds; a new ledger
is born on the swept side unless its name opts out. `scribe_checkpoint_allowed`
writes its twin explicitly since it bypasses the appender, and there the split
lands right on both sides: the cap counts the swept file, so a compaction
honestly restores the budget to stop an act the context can no longer justify,
while the record that a stop happened stays.
Removed `scribe_ledger_ids`, orphaned by the change — a dead helper beside a
live one is a thing the next reader trusts.
test_session_ledger_clear.py asserted every ledger dies and could not have
caught this: its fixture never created a twin, so the sweep was one glob away
from either mistake with only one of them guarded. Both sides now asserted.
The slippage tests build their ledgers through the real writers rather than
by hand, for the same reason — a fixture that writes the bytes itself keeps
passing after the writer stops.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01821k5B3Ysecp9fNYs92Kuy
162 lines
7.0 KiB
Python
162 lines
7.0 KiB
Python
"""Named is not read: the ledger's third state (#4100).
|
|
|
|
WHY THIS EXISTS
|
|
|
|
Milestone 386 made a repeat REFERENCED rather than withheld, and the line it
|
|
chose says "You saw it earlier this session". That claim was never checked.
|
|
The arms emit a TEASER — title, trigger, `get_rule(N)` — so a session can be
|
|
shown a rule twenty times and never read a word of it, and a compaction
|
|
summarises the teaser away leaving nothing at all. The server was asserting
|
|
something about the reader's context it had no way to know.
|
|
|
|
`.opened.ids` is the observable half, written by a PostToolUse hook from the
|
|
`get_rule` call itself. That is why this is not the self-report 386 rejected:
|
|
the objection was to ASKING a model about its own context, and a tool call is
|
|
an event the harness reports whether anyone asks or not.
|
|
|
|
WHAT THIS PINS
|
|
|
|
The three states and their three lines, the hook that records an open, and the
|
|
fact that BOTH ledgers die together on a compaction — a session told "you
|
|
opened it earlier" about a rule that was just summarised out of its context
|
|
would be a more confident version of the bug this removes.
|
|
"""
|
|
from __future__ import annotations
|
|
|
|
import json
|
|
import os
|
|
import re
|
|
import shutil
|
|
import subprocess
|
|
from pathlib import Path
|
|
from unittest.mock import MagicMock
|
|
|
|
import pytest
|
|
|
|
from scribe.services.plugin_context import _rule_hint_line
|
|
|
|
ROOT = Path(__file__).resolve().parents[1]
|
|
HOOKS = ROOT / "plugin" / "hooks"
|
|
RECORDER = HOOKS / "scribe_record_opened.sh"
|
|
|
|
|
|
def _rule(rid=156, kind="rule"):
|
|
return MagicMock(id=rid, kind=kind, title="A wait with no deadline is a bug",
|
|
when_to_apply="crossing a process boundary")
|
|
|
|
|
|
# ── the three lines ────────────────────────────────────────────────────────
|
|
|
|
def test_a_rule_never_surfaced_is_offered_as_new():
|
|
line = _rule_hint_line(_rule(), where="here", seen=False, held=False)
|
|
assert "not in this session's loaded set" in line
|
|
|
|
|
|
def test_a_rule_named_but_not_opened_says_so_and_still_invites():
|
|
"""The state that did not exist. It must NOT claim the reader saw it, and
|
|
must still carry the pull pointer — a session that skipped the teaser is
|
|
in nearly the position of one that was never shown it."""
|
|
line = _rule_hint_line(_rule(), where="here", seen=True, held=False)
|
|
assert "not opened" in line
|
|
assert "get_rule(156)" in line
|
|
assert "You saw it earlier" not in line, (
|
|
"the middle state is claiming the reader read something they did not"
|
|
)
|
|
|
|
|
|
def test_a_rule_the_session_opened_is_described_as_opened():
|
|
line = _rule_hint_line(_rule(), where="here", seen=True, held=True)
|
|
assert "opened it earlier" in line
|
|
assert "get_rule(156)" in line
|
|
|
|
|
|
def test_the_three_states_produce_three_different_lines():
|
|
"""Guard against a refactor collapsing two branches: each state has to be
|
|
distinguishable, or the distinction this milestone bought is gone while
|
|
every individual assertion above still passes."""
|
|
lines = {
|
|
_rule_hint_line(_rule(), where="here", seen=s, held=h)
|
|
for s, h in ((False, False), (True, False), (True, True))
|
|
}
|
|
assert len(lines) == 3
|
|
|
|
|
|
def test_held_outranks_seen_regardless_of_kind():
|
|
"""`kind` moves the head and the ledger moves the tail; #3497's history is
|
|
the two being reasoned about together and one of them being forgotten."""
|
|
for kind in ("rule", "preference"):
|
|
line = _rule_hint_line(_rule(kind=kind), where="here", seen=True, held=True)
|
|
assert "opened it earlier" in line
|
|
|
|
|
|
# ── the recorder ───────────────────────────────────────────────────────────
|
|
|
|
def _run_recorder(event: dict, tmp: Path) -> Path:
|
|
for tool in ("bash",):
|
|
if shutil.which(tool) is None:
|
|
pytest.skip(f"hook runtime tool {tool!r} not installed")
|
|
env = {"PATH": os.environ["PATH"], "HOME": str(tmp), "TMPDIR": str(tmp)}
|
|
out = subprocess.run(["bash", str(RECORDER)], input=json.dumps(event),
|
|
capture_output=True, text=True, env=env, timeout=30)
|
|
assert out.returncode == 0, out.stderr
|
|
return tmp / "scribe-priorart" / "s1.opened.ids"
|
|
|
|
|
|
def test_opening_a_rule_is_recorded(tmp_path):
|
|
led = _run_recorder(
|
|
{"session_id": "s1", "tool_name": "mcp__scribe__get_rule",
|
|
"tool_input": {"rule_id": 156}}, tmp_path)
|
|
assert led.exists()
|
|
assert led.read_text().split("\t")[0] == "156"
|
|
|
|
|
|
def test_the_entry_is_stamped_so_it_ages_like_the_naming_ledger(tmp_path):
|
|
"""Both ledgers are read by `scribe_rules_live`, which ages on that stamp.
|
|
An unstamped entry never expires — bounded, but it would mean an opened
|
|
rule stays 'opened' for a session's whole life."""
|
|
led = _run_recorder(
|
|
{"session_id": "s1", "tool_name": "mcp__scribe__get_rule",
|
|
"tool_input": {"rule_id": 9}}, tmp_path)
|
|
parts = led.read_text().strip().split("\t")
|
|
assert len(parts) == 2 and parts[1].isdigit()
|
|
|
|
|
|
@pytest.mark.parametrize("event", [
|
|
{"session_id": "s1", "tool_name": "mcp__scribe__get_rule", "tool_input": {}},
|
|
{"session_id": "s1", "tool_name": "mcp__scribe__get_rule",
|
|
"tool_input": {"rule_id": "../../etc"}},
|
|
{"tool_name": "mcp__scribe__get_rule", "tool_input": {"rule_id": 5}},
|
|
])
|
|
def test_an_event_with_nothing_usable_records_nothing_and_still_exits_zero(event, tmp_path):
|
|
"""A hook that fails a tool call over bookkeeping is worse than one that
|
|
misses an id: the cost of a miss is one extra line, in the direction that
|
|
shows more rather than less."""
|
|
led = _run_recorder(event, tmp_path)
|
|
assert not led.exists()
|
|
|
|
|
|
# What a compaction clears — including `.opened.ids`, whose claim is the one
|
|
# that would be worst to get wrong — is asserted against the running hook in
|
|
# tests/test_session_ledger_clear.py, so there is one home for it.
|
|
#
|
|
# Since #4217 the recorder writes TWO files and only this one is cleared.
|
|
# `.opened.ids` says "this context holds rule 156", which a compaction makes
|
|
# false, and three hooks read it to decide whether to stay quiet. Its twin
|
|
# `.opened.keep.ids` says "rule 156 was opened", which a compaction does not
|
|
# touch, and the session-end readout is built on that. The split is tested
|
|
# where the sweep is.
|
|
|
|
|
|
def test_the_recorder_is_registered_on_the_get_rule_tool():
|
|
hooks = json.loads((HOOKS / "hooks.json").read_text())["hooks"]
|
|
posts = hooks["PostToolUse"]
|
|
mine = [b for b in posts
|
|
if any("scribe_record_opened.sh" in h["command"] for h in b["hooks"])]
|
|
assert len(mine) == 1, "the opened-recorder is not registered exactly once"
|
|
# An MCP tool's server segment varies with how the plugin was installed, so
|
|
# the matcher must not pin one spelling of it.
|
|
matcher = mine[0]["matcher"]
|
|
assert re.fullmatch(matcher, "mcp__plugin_scribe_scribe__get_rule"), matcher
|
|
assert re.fullmatch(matcher, "mcp__scribe__get_rule"), matcher
|
|
assert not re.fullmatch(matcher, "Bash"), matcher
|