feat(plugin): the seam that erases the evidence is where the unresolved rules get named (#4216)
CI & Build / Python lint (push) Successful in 3s
CI & Build / Plugin hooks (push) Successful in 14s
CI & Build / TypeScript typecheck (push) Successful in 54s
CI & Build / integration (push) Successful in 55s
CI & Build / Python tests (push) Successful in 1m46s
CI & Build / Build & push image (push) Successful in 14s
CI & Build / Python lint (push) Successful in 3s
CI & Build / Plugin hooks (push) Successful in 14s
CI & Build / TypeScript typecheck (push) Successful in 54s
CI & Build / integration (push) Successful in 55s
CI & Build / Python tests (push) Successful in 1m46s
CI & Build / Build & push image (push) Successful in 14s
Step 5 of milestone 419. The milestone's subject is that a rule read and ignored is arithmetically identical to a rule read and followed, and the compaction is where that identity becomes permanent — the turns holding the evidence are summarised away, and the unjudged thing survives as nothing. WHY THIS IS ASSEMBLED IN THE HOOK. `rule_usage_events` has no session column; it is per user over a window. A session-scoped answer therefore cannot be asked of the server, and has to be built where a session is a thing that exists. Four ledgers four hooks already write: .rules.ids an arm NAMED the rule .opened.ids the session called get_rule (#4100) .acted.ids the session called rule_outcome (new here) .checkpoint.ids the rule HELD an act (#4214) Every one is an observed tool call. Nothing asks the model what it followed — milestone 386 ruled that out, because a model asked "did you apply rule 156?" says yes. Two subtractions: named-minus-opened is the arm talking to nobody, opened-minus-acted is the milestone's whole subject. PreCompact stdout is the compaction's custom instructions (#3680), not a message to the model, so the readout does not say "you slipped" — it says which ids must be carried through, which is the one thing a summary can do about an unjudged finding. SILENT WHEN NOTHING HAPPENED, and the accusations are conditional on having members. "0 rules unresolved" on every compaction is how a readout teaches its reader to skip it. Traffic is still reported, because the static instructions already ask for it in prose; these lines are the measured version. scribe_record_outcome.sh is the third ledger's writer, matched on mcp__.*__rule_outcome and mirroring scribe_record_opened.sh: TMPDIR only, silent, exit 0 on every path. A PostToolUse hook that spoke would put a line after every rule_outcome call and give recording an outcome a cost. Also: check_plugin.py skipped the new hook for want of a smoke event, which would have left the newest of the three ledgers as the only one the plugin lane never runs. Added, mirroring its sibling. tests/test_precompact_hook.py now isolates TMPDIR — the hook reads session ledgers from there, so without isolation a test would see whatever this real session had accumulated. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01821k5B3Ysecp9fNYs92Kuy
This commit is contained in:
@@ -1,7 +1,7 @@
|
|||||||
{
|
{
|
||||||
"name": "scribe",
|
"name": "scribe",
|
||||||
"description": "Scribe for Claude Code: connects the scribe MCP server, adds the hooks that deliver live project state and relevant records at the right moment, ships the shared client-neutral Scribe skills (using-scribe, writing-plans, reporting-back, systematic-debugging, verification, brainstorming, reusing-code, shape-accounting), and syncs your saved Scribe Processes as skills (/scribe:sync).",
|
"description": "Scribe for Claude Code: connects the scribe MCP server, adds the hooks that deliver live project state and relevant records at the right moment, ships the shared client-neutral Scribe skills (using-scribe, writing-plans, reporting-back, systematic-debugging, verification, brainstorming, reusing-code, shape-accounting), and syncs your saved Scribe Processes as skills (/scribe:sync).",
|
||||||
"version": "2026.09.21.0442",
|
"version": "2026.09.21.0452",
|
||||||
"author": {
|
"author": {
|
||||||
"name": "Bryan Van Deusen"
|
"name": "Bryan Van Deusen"
|
||||||
},
|
},
|
||||||
|
|||||||
@@ -62,6 +62,15 @@
|
|||||||
"command": "bash \"${CLAUDE_PLUGIN_ROOT}/hooks/scribe_record_opened.sh\""
|
"command": "bash \"${CLAUDE_PLUGIN_ROOT}/hooks/scribe_record_opened.sh\""
|
||||||
}
|
}
|
||||||
]
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"matcher": "mcp__.*__rule_outcome",
|
||||||
|
"hooks": [
|
||||||
|
{
|
||||||
|
"type": "command",
|
||||||
|
"command": "bash \"${CLAUDE_PLUGIN_ROOT}/hooks/scribe_record_outcome.sh\""
|
||||||
|
}
|
||||||
|
]
|
||||||
}
|
}
|
||||||
],
|
],
|
||||||
"PreCompact": [
|
"PreCompact": [
|
||||||
|
|||||||
@@ -525,6 +525,98 @@ scribe_contract_block() {
|
|||||||
return 0
|
return 0
|
||||||
}
|
}
|
||||||
|
|
||||||
|
# ── The session slippage readout (#4216, milestone 419) ───────────────────
|
||||||
|
#
|
||||||
|
# WHAT IT ANSWERS, AND WHY IT CANNOT BE ASKED OF THE SERVER. Which rules fired
|
||||||
|
# this session, which changed an action, which did not. `rule_usage_events`
|
||||||
|
# carries no session column — it is per user over a window — so a
|
||||||
|
# session-scoped answer has to be assembled where a session is a thing that
|
||||||
|
# exists. That is here, from ledgers four hooks already write:
|
||||||
|
#
|
||||||
|
# .rules.ids an arm NAMED the rule (a teaser was shown)
|
||||||
|
# .opened.ids the session called get_rule (#4100, an observed event)
|
||||||
|
# .acted.ids the session called rule_outcome (#4216)
|
||||||
|
# .checkpoint.ids the rule HELD an act (#4214, the strongest)
|
||||||
|
#
|
||||||
|
# EVERY LINE IS AN OBSERVED TOOL CALL. Nothing here asks the model what it
|
||||||
|
# followed — milestone 386 ruled that out, and rightly: a model asked "did you
|
||||||
|
# apply rule 156?" will say yes. These four files record what HAPPENED.
|
||||||
|
#
|
||||||
|
# THE SUBTRACTIONS ARE THE POINT. Named-minus-opened is the arm talking to
|
||||||
|
# nobody; opened-minus-acted is the milestone's whole subject, a rule read and
|
||||||
|
# then indistinguishable from one that worked. Neither is an accusation — a
|
||||||
|
# rule may be read and correctly judged not to apply — which is why the lines
|
||||||
|
# below ask for the leftovers to be CARRIED, not explained.
|
||||||
|
scribe_ledger_ids() {
|
||||||
|
# Live ids from one ledger as space-separated words, for set arithmetic.
|
||||||
|
# Aged through scribe_rules_live so a rule named two hours ago does not read
|
||||||
|
# as something this context still holds; a bare-id ledger (the checkpoint
|
||||||
|
# one) simply has no stamps and survives the ageing unchanged.
|
||||||
|
local f="$1"
|
||||||
|
[ -n "$f" ] && [ -f "$f" ] || return 0
|
||||||
|
scribe_rules_live "$f" | tr ',' ' '
|
||||||
|
}
|
||||||
|
|
||||||
|
scribe_slippage_lines() {
|
||||||
|
# $1 state dir, $2 sanitised session id.
|
||||||
|
#
|
||||||
|
# SILENT ONLY WHEN NO RULE TOUCHED THE SESSION AT ALL. Traffic that did
|
||||||
|
# happen is always reported, because "which rules governed this work" is
|
||||||
|
# what the static instructions above already ask the summariser to preserve
|
||||||
|
# in prose — these lines are the measured version of that, and they are
|
||||||
|
# three short lines.
|
||||||
|
#
|
||||||
|
# What is conditional is the ACCUSATION. Each subtraction prints only when
|
||||||
|
# it has members, so a session that opened everything it was shown and
|
||||||
|
# resolved everything it opened gets the traffic and no more. "0 rules
|
||||||
|
# unresolved" on every compaction is how a readout teaches its reader to
|
||||||
|
# skip it.
|
||||||
|
local dir="$1" sid="$2" named opened acted held
|
||||||
|
[ -n "$dir" ] && [ -n "$sid" ] || return 0
|
||||||
|
named=$(scribe_ledger_ids "$dir/${sid}.rules.ids")
|
||||||
|
opened=$(scribe_ledger_ids "$dir/${sid}.opened.ids")
|
||||||
|
acted=$(scribe_ledger_ids "$dir/${sid}.acted.ids")
|
||||||
|
held=$(scribe_ledger_ids "$dir/${sid}.checkpoint.ids")
|
||||||
|
[ -n "$named$opened" ] || return 0
|
||||||
|
|
||||||
|
local unread unresolved
|
||||||
|
unread=$(scribe_ids_minus "$named" "$opened")
|
||||||
|
unresolved=$(scribe_ids_minus "$opened" "$acted")
|
||||||
|
|
||||||
|
printf -- '- This session'"'"'s rule traffic, from what actually happened rather than from recollection:\n'
|
||||||
|
[ -n "$opened" ] && printf -- ' read: %s\n' "$opened"
|
||||||
|
[ -n "$held" ] && printf -- ' held an act before it ran: %s\n' "$held"
|
||||||
|
[ -n "$unread" ] && printf -- ' named by an arm and never opened: %s\n' "$unread"
|
||||||
|
if [ -n "$unresolved" ]; then
|
||||||
|
printf -- ' READ WITH NO OUTCOME RECORDED: %s. Carry these over as outstanding. A rule read and left unresolved looks exactly like one that worked, and the summary is where that difference is lost for good — say `rule_outcome(id, "applied")`, or `rule_outcome(id, "departed", why=...)` where you deliberately went another way.\n' "$unresolved"
|
||||||
|
fi
|
||||||
|
return 0
|
||||||
|
}
|
||||||
|
|
||||||
|
scribe_ids_minus() {
|
||||||
|
# Set difference over two space-separated id lists, order preserved. Written
|
||||||
|
# as one awk pass rather than a nested shell loop because the ledgers can
|
||||||
|
# hold a few dozen ids by the end of a long session and this runs inside the
|
||||||
|
# compaction path, where a slow hook delays the thing it is decorating.
|
||||||
|
local a="$1" b="$2"
|
||||||
|
[ -n "$a" ] || return 0
|
||||||
|
awk -v a="$a" -v b="$b" '
|
||||||
|
BEGIN {
|
||||||
|
n = split(b, drop, " ")
|
||||||
|
for (i = 1; i <= n; i++) if (drop[i] != "") skip[drop[i]] = 1
|
||||||
|
m = split(a, keep, " ")
|
||||||
|
out = ""
|
||||||
|
for (i = 1; i <= m; i++) {
|
||||||
|
id = keep[i]
|
||||||
|
if (id == "" || (id in skip) || (id in done)) continue
|
||||||
|
done[id] = 1
|
||||||
|
out = out (out == "" ? "" : " ") id
|
||||||
|
}
|
||||||
|
if (out != "") print out
|
||||||
|
}
|
||||||
|
' </dev/null 2>/dev/null
|
||||||
|
}
|
||||||
|
|
||||||
scribe_local_dups() {
|
scribe_local_dups() {
|
||||||
local root="$1" rel="$2" kind name pat hits count label files
|
local root="$1" rel="$2" kind name pat hits count label files
|
||||||
while IFS=$'\t' read -r kind name; do
|
while IFS=$'\t' read -r kind name; do
|
||||||
|
|||||||
@@ -46,11 +46,22 @@
|
|||||||
# text is the receipt. (Auto-compaction suppresses that notification.)
|
# text is the receipt. (Auto-compaction suppresses that notification.)
|
||||||
set -uo pipefail
|
set -uo pipefail
|
||||||
|
|
||||||
# Drain the event so the caller never sees a broken pipe. Nothing in it changes
|
# shellcheck source=plugin/hooks/scribe_defs.sh
|
||||||
# what we emit: the instruction is the same whether the operator typed
|
. "$(dirname "${BASH_SOURCE[0]}")/scribe_defs.sh"
|
||||||
# `/compact` or the session hit its limit, and it composes with any custom
|
|
||||||
# instructions they gave, which are merged ahead of ours.
|
# THE EVENT IS NOW READ, where it used to be drained. The old comment here said
|
||||||
cat >/dev/null 2>&1 || true
|
# nothing in it changes what we emit, and that was true while every line was
|
||||||
|
# static. It stopped being true at #4216: the slippage readout below is
|
||||||
|
# specific to THIS session, and `session_id` is the key to the ledgers that
|
||||||
|
# hold it. The instruction is still the same whether the operator typed
|
||||||
|
# `/compact` or the session hit its limit — what varies is the measurement
|
||||||
|
# appended to it.
|
||||||
|
event=$(cat 2>/dev/null || true)
|
||||||
|
session_id=""
|
||||||
|
if [ -n "$event" ]; then
|
||||||
|
event_flat=$(printf '%s' "$event" | scribe_json_flat 2>/dev/null || true)
|
||||||
|
session_id=$(scribe_json_pick "$event_flat" '.session_id' 2>/dev/null || true)
|
||||||
|
fi
|
||||||
|
|
||||||
cat <<'EOF'
|
cat <<'EOF'
|
||||||
Preserve the following literally in the summary — copied through, not
|
Preserve the following literally in the summary — copied through, not
|
||||||
@@ -71,4 +82,29 @@ paraphrased or counted:
|
|||||||
Everything else here can be recovered from the repository or from Scribe. These
|
Everything else here can be recovered from the repository or from Scribe. These
|
||||||
cannot: they are this session's only copy.
|
cannot: they are this session's only copy.
|
||||||
EOF
|
EOF
|
||||||
|
|
||||||
|
# ── The slippage readout (#4216, milestone 419) ───────────────────────────
|
||||||
|
#
|
||||||
|
# WHY IT BELONGS HERE RATHER THAN IN A REPLY. A rule read and left unresolved
|
||||||
|
# is invisible by construction: it looks exactly like a rule that worked. The
|
||||||
|
# compaction is where that invisibility becomes permanent — the turns holding
|
||||||
|
# the evidence are summarised away, and an unjudged thing that survives as
|
||||||
|
# nothing is how a decision quietly becomes nobody's.
|
||||||
|
#
|
||||||
|
# STDOUT HERE IS THE SUMMARISER'S INSTRUCTIONS, not a message to the model
|
||||||
|
# (the header records how that was established). So this does not say "you
|
||||||
|
# slipped"; it says WHICH ids must be carried through, which is the one thing
|
||||||
|
# the summary can do about it. The next turn then reads them in the summary
|
||||||
|
# with the work still attached.
|
||||||
|
#
|
||||||
|
# SILENT WHEN THERE IS NOTHING TO SAY. A session that opened everything it was
|
||||||
|
# shown gets no extra lines. "0 rules unresolved" on every compaction is how a
|
||||||
|
# readout teaches its reader to skip it.
|
||||||
|
if [ -n "$session_id" ]; then
|
||||||
|
safe_sid=$(printf '%s' "$session_id" | tr -c 'A-Za-z0-9._-' '_')
|
||||||
|
slippage=$(scribe_slippage_lines "${TMPDIR:-/tmp}/scribe-priorart" "$safe_sid" 2>/dev/null || true)
|
||||||
|
if [ -n "$slippage" ]; then
|
||||||
|
printf '\n%s\n' "$slippage"
|
||||||
|
fi
|
||||||
|
fi
|
||||||
exit 0
|
exit 0
|
||||||
|
|||||||
Executable
+57
@@ -0,0 +1,57 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
# Scribe — record that this session said what a rule DID, not merely that it
|
||||||
|
# read one (#4216, milestone 419).
|
||||||
|
#
|
||||||
|
# THE THIRD LEDGER, AND WHY THE TWO THAT EXIST ARE NOT ENOUGH.
|
||||||
|
#
|
||||||
|
# `.rules.ids` says a rule was NAMED. `.opened.ids` says it was READ (#4100 —
|
||||||
|
# a PostToolUse hook watches the `get_rule` call, so it is a recorded event
|
||||||
|
# rather than a model's claim about its own context). Neither can say what
|
||||||
|
# happened next, and that is the whole of what milestone 419 is about: a rule
|
||||||
|
# read and followed and a rule read and forgotten leave identical traces.
|
||||||
|
#
|
||||||
|
# `rule_outcome` (#4212) is the call that closes that gap, and the server
|
||||||
|
# records it. But the server's row carries no session — `rule_usage_events` is
|
||||||
|
# per user over a window — so a SESSION-scoped readout cannot be asked of it.
|
||||||
|
# The question "which rules changed something in THIS session" has to be
|
||||||
|
# answered where a session is a thing that exists, which is here.
|
||||||
|
#
|
||||||
|
# SAME EVIDENCE CLASS AS `.opened.ids`, deliberately. A tool call happened or
|
||||||
|
# it did not, and the harness reports it either way; nothing here asks the
|
||||||
|
# model whether it followed anything. That is the distinction milestone 386
|
||||||
|
# drew when it ruled out self-report, and this stays on the right side of it.
|
||||||
|
#
|
||||||
|
# WHAT IT CANNOT SAY: that the rule was followed WELL, or that `applied` was
|
||||||
|
# honest. It records that an outcome was declared. The value of that is not the
|
||||||
|
# claim itself — it is that the absence of one becomes visible, which is the
|
||||||
|
# state nothing could previously name.
|
||||||
|
#
|
||||||
|
# EXIT 0, ALWAYS. This decorates a ledger; a bookkeeping failure must never
|
||||||
|
# turn a successful tool call into a hook error.
|
||||||
|
set -uo pipefail
|
||||||
|
|
||||||
|
# shellcheck source=plugin/hooks/scribe_defs.sh
|
||||||
|
. "$(dirname "${BASH_SOURCE[0]}")/scribe_defs.sh"
|
||||||
|
|
||||||
|
event=$(cat 2>/dev/null || true)
|
||||||
|
[ -n "$event" ] || exit 0
|
||||||
|
|
||||||
|
event_flat=$(printf '%s' "$event" | scribe_json_flat)
|
||||||
|
session_id=$(scribe_json_pick "$event_flat" '.session_id')
|
||||||
|
[ -n "$session_id" ] || exit 0
|
||||||
|
|
||||||
|
# The matcher in hooks.json narrows to the rule_outcome tools, but the server
|
||||||
|
# segment of an MCP tool name varies with how the plugin was installed, so the
|
||||||
|
# id is read from the field rather than from an assumed tool name — the same
|
||||||
|
# reasoning scribe_record_opened.sh gives.
|
||||||
|
rule_id=$(scribe_json_pick "$event_flat" '.tool_input.rule_id')
|
||||||
|
rule_id=$(printf '%s' "$rule_id" | tr -cd '0-9')
|
||||||
|
[ -n "$rule_id" ] || exit 0
|
||||||
|
|
||||||
|
state_dir="${TMPDIR:-/tmp}/scribe-priorart"
|
||||||
|
mkdir -p "$state_dir" 2>/dev/null || true
|
||||||
|
safe_sid=$(printf '%s' "$session_id" | tr -c 'A-Za-z0-9._-' '_')
|
||||||
|
|
||||||
|
# Stamped and append-only like its two siblings, so one reader ages them all.
|
||||||
|
printf '%s\n' "$rule_id" | scribe_rules_append "$state_dir/${safe_sid}.acted.ids"
|
||||||
|
exit 0
|
||||||
@@ -375,6 +375,17 @@ SMOKE_EVENTS: dict[str, str] = {
|
|||||||
"tool_name": "mcp__scribe__get_rule", "tool_input": {"rule_id": 1},
|
"tool_name": "mcp__scribe__get_rule", "tool_input": {"rule_id": 1},
|
||||||
"tool_response": {}}
|
"tool_response": {}}
|
||||||
),
|
),
|
||||||
|
# The acted-ledger recorder (#4216). Same shape and same reasoning as its
|
||||||
|
# sibling above: TMPDIR only, and silent, because a PostToolUse hook that
|
||||||
|
# spoke would put a line after every rule_outcome call and turn recording
|
||||||
|
# an outcome into something with a cost. A skip here would have left the
|
||||||
|
# newest of the three ledgers as the only hook the lane never runs.
|
||||||
|
"scribe_record_outcome.sh": json.dumps(
|
||||||
|
{"session_id": "smoke", "cwd": ".",
|
||||||
|
"tool_name": "mcp__scribe__rule_outcome",
|
||||||
|
"tool_input": {"rule_id": 1, "outcome": "applied"},
|
||||||
|
"tool_response": {}}
|
||||||
|
),
|
||||||
# The shared library is sourced, never run; executed bare it defines
|
# The shared library is sourced, never run; executed bare it defines
|
||||||
# functions and exits — silent by construction.
|
# functions and exits — silent by construction.
|
||||||
"scribe_defs.sh": "",
|
"scribe_defs.sh": "",
|
||||||
|
|||||||
@@ -29,6 +29,8 @@ correct beside every other hook in this directory and would inject nothing.
|
|||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
import json
|
import json
|
||||||
|
import tempfile
|
||||||
|
import os
|
||||||
import shutil
|
import shutil
|
||||||
import subprocess
|
import subprocess
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
@@ -44,11 +46,22 @@ EVENT = {"session_id": "s1", "transcript_path": "/tmp/t.jsonl", "cwd": "/repo",
|
|||||||
"custom_instructions": None}
|
"custom_instructions": None}
|
||||||
|
|
||||||
|
|
||||||
def _run(event: dict) -> subprocess.CompletedProcess:
|
def _run(event: dict, tmpdir: str | None = None) -> subprocess.CompletedProcess:
|
||||||
|
"""Run the hook with its ledger directory ISOLATED.
|
||||||
|
|
||||||
|
The hook reads this session's rule ledgers since #4216, and they live under
|
||||||
|
`$TMPDIR/scribe-priorart`. Without an override these tests would read the
|
||||||
|
machine's real /tmp: on a developer box mid-session that is not empty, and
|
||||||
|
the output would depend on what some other session happened to leave
|
||||||
|
behind. None of the assertions below would fail on it today, which is
|
||||||
|
exactly why it is worth closing now rather than after it starts flaking.
|
||||||
|
"""
|
||||||
if shutil.which("bash") is None:
|
if shutil.which("bash") is None:
|
||||||
pytest.skip("bash not installed")
|
pytest.skip("bash not installed")
|
||||||
|
env = dict(os.environ)
|
||||||
|
env["TMPDIR"] = tmpdir or tempfile.mkdtemp()
|
||||||
return subprocess.run(["bash", str(HOOK)], input=json.dumps(event),
|
return subprocess.run(["bash", str(HOOK)], input=json.dumps(event),
|
||||||
capture_output=True, text=True, timeout=30)
|
capture_output=True, text=True, timeout=30, env=env)
|
||||||
|
|
||||||
|
|
||||||
def _code() -> str:
|
def _code() -> str:
|
||||||
@@ -115,9 +128,11 @@ def test_it_never_blocks_the_compaction():
|
|||||||
assert "decision" not in code, "a block decision would skip the compaction"
|
assert "decision" not in code, "a block decision would skip the compaction"
|
||||||
assert "exit 2" not in code
|
assert "exit 2" not in code
|
||||||
# A truncated or absent event must not turn into a non-zero exit either.
|
# A truncated or absent event must not turn into a non-zero exit either.
|
||||||
|
env = dict(os.environ)
|
||||||
|
env["TMPDIR"] = tempfile.mkdtemp()
|
||||||
for event in ("", "not json", "{}"):
|
for event in ("", "not json", "{}"):
|
||||||
out = subprocess.run(["bash", str(HOOK)], input=event,
|
out = subprocess.run(["bash", str(HOOK)], input=event,
|
||||||
capture_output=True, text=True, timeout=30)
|
capture_output=True, text=True, timeout=30, env=env)
|
||||||
assert out.returncode == 0, f"{event!r} → {out.returncode}: {out.stderr}"
|
assert out.returncode == 0, f"{event!r} → {out.returncode}: {out.stderr}"
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,255 @@
|
|||||||
|
"""Which rules fired this session, which changed an action, which did not (#4216).
|
||||||
|
|
||||||
|
WHY THIS CANNOT BE ASKED OF THE SERVER
|
||||||
|
|
||||||
|
`rule_usage_events` has no session column — it is per user over a window — so
|
||||||
|
a SESSION-scoped answer has to be assembled where a session is a thing that
|
||||||
|
exists. That is the plugin, from four ledgers four hooks already write:
|
||||||
|
|
||||||
|
.rules.ids an arm NAMED the rule (a teaser was shown)
|
||||||
|
.opened.ids the session called get_rule (#4100)
|
||||||
|
.acted.ids the session called rule_outcome (#4216, new here)
|
||||||
|
.checkpoint.ids the rule HELD an act (#4214)
|
||||||
|
|
||||||
|
EVERY LINE IS AN OBSERVED TOOL CALL. Nothing asks the model what it followed;
|
||||||
|
milestone 386 ruled that out because a model asked "did you apply rule 156?"
|
||||||
|
will say yes. These record what happened.
|
||||||
|
|
||||||
|
WHY THE COMPACTION SEAM
|
||||||
|
|
||||||
|
A rule read and left unresolved is invisible by construction — it looks
|
||||||
|
exactly like a rule that worked. The compaction is where that invisibility
|
||||||
|
becomes permanent: the turns holding the evidence are summarised away, and an
|
||||||
|
unjudged thing that survives as nothing is how a decision quietly becomes
|
||||||
|
nobody's. A PreCompact hook's stdout becomes the summariser's instructions
|
||||||
|
(#3680), so this does not say "you slipped" — it says which ids must be
|
||||||
|
carried through, which is the one thing a summary can do about it.
|
||||||
|
|
||||||
|
WHAT IS PINNED: the arithmetic, and which conditions produce which lines. NOT
|
||||||
|
pinned: the wording.
|
||||||
|
"""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
import shutil
|
||||||
|
import subprocess
|
||||||
|
import time
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
ROOT = Path(__file__).resolve().parents[1]
|
||||||
|
HOOKS = ROOT / "plugin" / "hooks"
|
||||||
|
DEFS = HOOKS / "scribe_defs.sh"
|
||||||
|
PRECOMPACT = HOOKS / "scribe_precompact_preserve.sh"
|
||||||
|
RECORDER = HOOKS / "scribe_record_outcome.sh"
|
||||||
|
HOOKS_JSON = HOOKS / "hooks.json"
|
||||||
|
|
||||||
|
|
||||||
|
def _need(*tools):
|
||||||
|
for t in tools:
|
||||||
|
if shutil.which(t) is None:
|
||||||
|
pytest.skip(f"hook runtime tool {t!r} not installed")
|
||||||
|
|
||||||
|
|
||||||
|
def sh(script: str) -> str:
|
||||||
|
_need("bash", "awk")
|
||||||
|
r = subprocess.run(
|
||||||
|
["bash", "-c", f'set -uo pipefail\n. "{DEFS}"\n{script}'],
|
||||||
|
capture_output=True, text=True, timeout=30,
|
||||||
|
)
|
||||||
|
assert r.returncode == 0, f"exit {r.returncode}: {r.stderr}"
|
||||||
|
return r.stdout
|
||||||
|
|
||||||
|
|
||||||
|
def ledgers(tmp_path, *, named=(), opened=(), acted=(), held=()) -> Path:
|
||||||
|
"""The four ledgers, written the way the hooks write them."""
|
||||||
|
d = tmp_path / "scribe-priorart"
|
||||||
|
d.mkdir(parents=True, exist_ok=True)
|
||||||
|
now = int(time.time())
|
||||||
|
for name, ids in (("rules", named), ("opened", opened), ("acted", acted)):
|
||||||
|
if ids:
|
||||||
|
(d / f"s.{name}.ids").write_text(
|
||||||
|
"".join(f"{i}\t{now}\n" for i in ids)
|
||||||
|
)
|
||||||
|
if held:
|
||||||
|
# The checkpoint ledger is bare ids — it records that something
|
||||||
|
# HAPPENED rather than what the context still holds, so it carries no
|
||||||
|
# stamp and never ages (#4214).
|
||||||
|
(d / "s.checkpoint.ids").write_text("".join(f"{i}\n" for i in held))
|
||||||
|
return d
|
||||||
|
|
||||||
|
|
||||||
|
def readout(d: Path) -> str:
|
||||||
|
return sh(f'scribe_slippage_lines "{d}" "s"')
|
||||||
|
|
||||||
|
|
||||||
|
# ── The arithmetic ────────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
def minus(a: str, b: str) -> str:
|
||||||
|
return sh(f'scribe_ids_minus "{a}" "{b}"').strip()
|
||||||
|
|
||||||
|
|
||||||
|
def test_the_difference_keeps_order_and_drops_members():
|
||||||
|
assert minus("1 9 34 156 173", "9 34 156") == "1 173"
|
||||||
|
|
||||||
|
|
||||||
|
def test_an_empty_difference_prints_nothing():
|
||||||
|
assert minus("9 34", "9 34") == ""
|
||||||
|
|
||||||
|
|
||||||
|
def test_an_empty_minuend_is_not_an_error():
|
||||||
|
assert minus("", "9") == ""
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_repeated_id_is_counted_once():
|
||||||
|
"""The ledgers are append-only, so the same rule can appear many times."""
|
||||||
|
assert minus("9 9 34 9", "") == "9 34"
|
||||||
|
|
||||||
|
|
||||||
|
# ── What the readout says ─────────────────────────────────────────────────
|
||||||
|
|
||||||
|
def test_the_readout_names_what_was_read(tmp_path):
|
||||||
|
out = readout(ledgers(tmp_path, named=[1, 9], opened=[9]))
|
||||||
|
assert "read: 9" in out
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_rule_named_and_never_opened_is_named_as_such(tmp_path):
|
||||||
|
"""The arm talking to nobody. Not an accusation — a teaser skimmed past
|
||||||
|
leaves nothing behind — but it is the number that says whether the arm is
|
||||||
|
earning its place."""
|
||||||
|
out = readout(ledgers(tmp_path, named=[1, 9, 173], opened=[9]))
|
||||||
|
assert "never opened" in out
|
||||||
|
line = next(ln for ln in out.splitlines() if "never opened" in ln)
|
||||||
|
assert "1" in line and "173" in line
|
||||||
|
assert " 9" not in line.split(":")[1], "an opened rule is not also unread"
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_rule_read_with_no_outcome_is_the_headline(tmp_path):
|
||||||
|
"""THE MILESTONE'S WHOLE SUBJECT. Read and unresolved is arithmetically
|
||||||
|
identical to read and followed, and this is the only place that difference
|
||||||
|
gets carried across the seam."""
|
||||||
|
out = readout(ledgers(tmp_path, named=[9, 34], opened=[9, 34], acted=[34]))
|
||||||
|
assert "READ WITH NO OUTCOME RECORDED" in out
|
||||||
|
line = next(ln for ln in out.splitlines() if "NO OUTCOME" in ln)
|
||||||
|
assert "9" in line
|
||||||
|
assert "rule_outcome" in line, "a finding with no remedy is a complaint"
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_rule_that_held_an_act_is_reported_separately(tmp_path):
|
||||||
|
"""The strongest evidence a rule changed something: it stopped a call
|
||||||
|
before it ran (#4214). Kept apart from `read` because reading a rule and
|
||||||
|
having it alter what you did are different claims."""
|
||||||
|
out = readout(ledgers(tmp_path, named=[156], opened=[156], held=[156]))
|
||||||
|
assert "held an act" in out and "156" in out
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_session_that_resolved_everything_makes_no_accusation(tmp_path):
|
||||||
|
"""Traffic is always reported — it is what the static instructions above
|
||||||
|
already ask for in prose. The SUBTRACTIONS are conditional, so a clean
|
||||||
|
session gets no scolding. '0 rules unresolved' on every compaction is how
|
||||||
|
a readout teaches its reader to skip it."""
|
||||||
|
out = readout(ledgers(tmp_path, named=[7], opened=[7], acted=[7]))
|
||||||
|
assert "read: 7" in out
|
||||||
|
assert "NO OUTCOME" not in out
|
||||||
|
assert "never opened" not in out
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_session_no_rule_touched_says_nothing_at_all(tmp_path):
|
||||||
|
"""Rule 115's reasoning: a fresh install must not be told something is
|
||||||
|
wrong when the truth is that nothing has happened yet."""
|
||||||
|
d = tmp_path / "scribe-priorart"
|
||||||
|
d.mkdir(parents=True)
|
||||||
|
assert readout(d).strip() == ""
|
||||||
|
|
||||||
|
|
||||||
|
# ── The hook that carries it ──────────────────────────────────────────────
|
||||||
|
|
||||||
|
def run_precompact(event: dict, tmpdir: Path) -> subprocess.CompletedProcess:
|
||||||
|
_need("bash")
|
||||||
|
env = dict(os.environ)
|
||||||
|
env["TMPDIR"] = str(tmpdir)
|
||||||
|
return subprocess.run(["bash", str(PRECOMPACT)], input=json.dumps(event),
|
||||||
|
capture_output=True, text=True, timeout=30, env=env)
|
||||||
|
|
||||||
|
|
||||||
|
def test_the_compaction_hook_appends_the_readout(tmp_path):
|
||||||
|
ledgers(tmp_path, named=[1, 9], opened=[9], acted=[])
|
||||||
|
r = run_precompact({"session_id": "s", "trigger": "manual"}, tmp_path)
|
||||||
|
assert r.returncode == 0
|
||||||
|
assert "Preserve the following literally" in r.stdout, "static half intact"
|
||||||
|
assert "READ WITH NO OUTCOME RECORDED" in r.stdout
|
||||||
|
|
||||||
|
|
||||||
|
def test_the_compaction_hook_still_exits_zero_with_no_session(tmp_path):
|
||||||
|
"""An event with no session_id has no ledgers to read. The static
|
||||||
|
instructions still go out — they are the part that matters most, and
|
||||||
|
losing them because a measurement was unavailable would be the worse
|
||||||
|
trade."""
|
||||||
|
r = run_precompact({"trigger": "auto"}, tmp_path)
|
||||||
|
assert r.returncode == 0
|
||||||
|
assert "Preserve the following literally" in r.stdout
|
||||||
|
|
||||||
|
|
||||||
|
def test_the_compaction_hook_never_emits_a_json_envelope(tmp_path):
|
||||||
|
"""Regression guard on the change that added the readout: for PreCompact
|
||||||
|
the envelope is pasted into the summariser's prompt rather than
|
||||||
|
unwrapped."""
|
||||||
|
ledgers(tmp_path, named=[1], opened=[1])
|
||||||
|
r = run_precompact({"session_id": "s", "trigger": "manual"}, tmp_path)
|
||||||
|
assert not r.stdout.lstrip().startswith("{")
|
||||||
|
|
||||||
|
|
||||||
|
# ── The recorder that makes `acted` mean anything ─────────────────────────
|
||||||
|
|
||||||
|
def run_recorder(event: dict, tmpdir: Path) -> subprocess.CompletedProcess:
|
||||||
|
_need("bash")
|
||||||
|
env = {"PATH": os.environ["PATH"], "HOME": str(tmpdir), "TMPDIR": str(tmpdir)}
|
||||||
|
return subprocess.run(["bash", str(RECORDER)], input=json.dumps(event),
|
||||||
|
capture_output=True, text=True, timeout=30, env=env)
|
||||||
|
|
||||||
|
|
||||||
|
def test_declaring_an_outcome_is_recorded(tmp_path):
|
||||||
|
r = run_recorder(
|
||||||
|
{"session_id": "s", "tool_input": {"rule_id": 156, "outcome": "applied"}},
|
||||||
|
tmp_path,
|
||||||
|
)
|
||||||
|
assert r.returncode == 0
|
||||||
|
assert (tmp_path / "scribe-priorart" / "s.acted.ids").read_text().startswith("156\t")
|
||||||
|
|
||||||
|
|
||||||
|
def test_the_entry_is_stamped_like_its_siblings(tmp_path):
|
||||||
|
"""One reader ages all three ledgers, so all three carry a timestamp."""
|
||||||
|
run_recorder({"session_id": "s", "tool_input": {"rule_id": 9}}, tmp_path)
|
||||||
|
line = (tmp_path / "scribe-priorart" / "s.acted.ids").read_text().strip()
|
||||||
|
assert "\t" in line and line.split("\t")[1].isdigit()
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize("event", [
|
||||||
|
{}, {"session_id": "s"}, {"tool_input": {"rule_id": 9}},
|
||||||
|
{"session_id": "s", "tool_input": {"rule_id": "not-a-number"}},
|
||||||
|
])
|
||||||
|
def test_an_unusable_event_records_nothing_and_still_exits_zero(event, tmp_path):
|
||||||
|
"""A bookkeeping failure must never turn a successful tool call into a
|
||||||
|
hook error."""
|
||||||
|
r = run_recorder(event, tmp_path)
|
||||||
|
assert r.returncode == 0
|
||||||
|
assert not (tmp_path / "scribe-priorart" / "s.acted.ids").exists()
|
||||||
|
|
||||||
|
|
||||||
|
def test_the_recorder_is_registered_on_the_rule_outcome_tool():
|
||||||
|
"""A ledger nothing writes to reads as 'nothing was acted on', which is
|
||||||
|
the exact false finding this milestone exists to avoid producing."""
|
||||||
|
cfg = json.loads(HOOKS_JSON.read_text())
|
||||||
|
entries = cfg["hooks"]["PostToolUse"]
|
||||||
|
assert any(
|
||||||
|
"rule_outcome" in e.get("matcher", "")
|
||||||
|
and any("scribe_record_outcome.sh" in h["command"] for h in e["hooks"])
|
||||||
|
for e in entries
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def test_the_recorder_is_shell_valid():
|
||||||
|
_need("bash")
|
||||||
|
subprocess.run(["bash", "-n", str(RECORDER)], check=True)
|
||||||
Reference in New Issue
Block a user