feat(rules): rules before tools — a PreToolUse arm keyed on the action (#3476)
CI & Build / Python lint (push) Successful in 4s
CI & Build / Plugin hooks (push) Successful in 8s
CI & Build / integration (push) Successful in 32s
CI & Build / TypeScript typecheck (push) Successful in 35s
CI & Build / Python tests (push) Successful in 1m9s
CI & Build / Build & push image (push) Successful in 25s
CI & Build / Python lint (push) Successful in 4s
CI & Build / Plugin hooks (push) Successful in 8s
CI & Build / integration (push) Successful in 32s
CI & Build / TypeScript typecheck (push) Successful in 35s
CI & Build / Python tests (push) Successful in 1m9s
CI & Build / Build & push image (push) Successful in 25s
The only just-in-time rule surface was registered on `Write|Edit` and queried with `code or path`, so a rule could be retrieved at the moment of a code write and nowhere else. Every rule about which tool to reach for — don't curl the forge, don't stand up a stack, don't run the suite locally, don't branch — was unreachable exactly when it mattered, and residency in the always-on preload was the only surface it had. That is the pressure that grew the resident set to 31 against #3089's ceiling of ~23; it was never a judgment anybody made. A reflex generates no query, so an instruction to check the rules cannot catch one. A mechanical trigger can: the tool call IS the query, and a reflex has to become a tool call before it can do anything. `build_tool_rule_hint` is deliberately tool-agnostic — a name and a string — so widening the matcher later is a hooks.json edit with no server change. The hook starts on Bash, which is where the action reflexes live. The two pre-tool arms share ONE session ledger of already-named rules (`<state>/<sid>.rules.ids`). Two ledgers would mean a rule named by one arm gets re-offered by the other, and the hint that fires most often is exactly the one that must not repeat itself. A test asserts both scripts build the same path, and another checks the shell hook and the Python route agree on every query-arg name (rule 33) — a rename there fails silently, looking like a surface that never finds anything rather than a broken one. Deliberately silent on outage, unlike the prior-art hook: a write is occasional, a Bash call is not, and an outage line before every command is what gets a channel muted. `tier="conditional"` matches the write arm and is the transition point — an always-on rule is already resident, so re-tier one and it starts arriving here instead of in every session's preamble. `pre_tool_rule` joins RANKED_SOURCES: this arm chose what it showed, so a pull can settle whether the choice landed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011cPyzNnegXHr5iRMzzy5KJ
This commit is contained in:
@@ -1,7 +1,7 @@
|
|||||||
{
|
{
|
||||||
"name": "scribe",
|
"name": "scribe",
|
||||||
"description": "Scribe system-of-record for Claude Code: MCP tools over your notes/tasks/projects/rules, a session-start push channel that surfaces your always-on rules + active-project context, process-skills (writing-plans, systematic-debugging, verification, brainstorming, reusing-code), and your saved Scribe Processes auto-surfaced as skills (/scribe:sync). Replaces superpowers + file-memory with one app-backed plugin.",
|
"description": "Scribe system-of-record for Claude Code: MCP tools over your notes/tasks/projects/rules, a session-start push channel that surfaces your always-on rules + active-project context, process-skills (writing-plans, systematic-debugging, verification, brainstorming, reusing-code), and your saved Scribe Processes auto-surfaced as skills (/scribe:sync). Replaces superpowers + file-memory with one app-backed plugin.",
|
||||||
"version": "2026.09.02.0438",
|
"version": "2026.09.03.0329",
|
||||||
"author": {
|
"author": {
|
||||||
"name": "Bryan Van Deusen"
|
"name": "Bryan Van Deusen"
|
||||||
},
|
},
|
||||||
|
|||||||
@@ -33,6 +33,15 @@
|
|||||||
"command": "bash \"${CLAUDE_PLUGIN_ROOT}/hooks/scribe_prior_art.sh\""
|
"command": "bash \"${CLAUDE_PLUGIN_ROOT}/hooks/scribe_prior_art.sh\""
|
||||||
}
|
}
|
||||||
]
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"matcher": "Bash",
|
||||||
|
"hooks": [
|
||||||
|
{
|
||||||
|
"type": "command",
|
||||||
|
"command": "bash \"${CLAUDE_PLUGIN_ROOT}/hooks/scribe_tool_rules.sh\""
|
||||||
|
}
|
||||||
|
]
|
||||||
}
|
}
|
||||||
],
|
],
|
||||||
"PostToolUse": [
|
"PostToolUse": [
|
||||||
|
|||||||
@@ -0,0 +1,116 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
# Scribe — PreToolUse rule arm for ACTIONS (#3476).
|
||||||
|
#
|
||||||
|
# The sibling of scribe_prior_art.sh. That hook is registered on Write|Edit and
|
||||||
|
# asks "what is recorded about the file being written". This one asks "does a
|
||||||
|
# standing rule speak to the command about to be run" — the question nothing
|
||||||
|
# could ask before, and the reason every rule about which tool to reach for had
|
||||||
|
# to live in the always-on preload instead.
|
||||||
|
#
|
||||||
|
# WHY A HOOK AND NOT AN INSTRUCTION. A reflex generates no query (note #3089):
|
||||||
|
# you reach for `curl` confidently, with no moment of doubt, so a surface that
|
||||||
|
# waits to be asked never fires. Here nothing is asked — the tool call IS the
|
||||||
|
# query, and the reflex has to become a tool call before it can do anything.
|
||||||
|
#
|
||||||
|
# SILENT ON OUTAGE, deliberately, unlike the prior-art hook. A write is
|
||||||
|
# occasional; a Bash call is not, and an "instance did not answer" line before
|
||||||
|
# every command is the noise that gets a channel muted. scribe_prior_art.sh
|
||||||
|
# still speaks for both when the instance is down.
|
||||||
|
#
|
||||||
|
# Env:
|
||||||
|
# SCRIBE_URL / SCRIBE_TOKEN override for the settings.json dogfooding path.
|
||||||
|
|
||||||
|
command -v jq >/dev/null 2>&1 || exit 0
|
||||||
|
command -v curl >/dev/null 2>&1 || exit 0
|
||||||
|
|
||||||
|
# PreToolUse delivers { session_id, cwd, tool_name, tool_input: {...}, ... }
|
||||||
|
event=$(cat 2>/dev/null || true)
|
||||||
|
tool_name=$(printf '%s' "$event" | jq -r '.tool_name // empty' 2>/dev/null) || exit 0
|
||||||
|
session_id=$(printf '%s' "$event" | jq -r '.session_id // empty' 2>/dev/null) || session_id=""
|
||||||
|
event_cwd=$(printf '%s' "$event" | jq -r '.cwd // empty' 2>/dev/null) || event_cwd=""
|
||||||
|
|
||||||
|
[ -n "$tool_name" ] || exit 0
|
||||||
|
|
||||||
|
# The action, as text. `.command` is Bash's field; the fallbacks let the matcher
|
||||||
|
# in hooks.json widen to other tools without this script changing — which is the
|
||||||
|
# whole reason the server side takes a name and a string rather than a schema.
|
||||||
|
command_text=$(printf '%s' "$event" | jq -r '
|
||||||
|
.tool_input.command //
|
||||||
|
.tool_input.url //
|
||||||
|
.tool_input.prompt //
|
||||||
|
empty' 2>/dev/null) || command_text=""
|
||||||
|
|
||||||
|
[ -n "$command_text" ] || exit 0
|
||||||
|
|
||||||
|
# shellcheck source=plugin/hooks/scribe_defs.sh
|
||||||
|
. "$(dirname "${BASH_SOURCE[0]}")/scribe_defs.sh"
|
||||||
|
|
||||||
|
# scribe_config, not a hand-rolled pair of parameter expansions: it also treats
|
||||||
|
# an UNEXPANDED `${...}` placeholder as unset, which would otherwise be sent as
|
||||||
|
# a garbage Bearer token and 401 on every call (#2198's class).
|
||||||
|
scribe_config || exit 0
|
||||||
|
|
||||||
|
# Bounded before encoding: a heredoc or a pasted script can be enormous, and
|
||||||
|
# the verb and its target — the part a rule is about — sit at the front. The
|
||||||
|
# server bounds it again; this keeps a huge payload off the wire in the first
|
||||||
|
# place. `head -c`, never `cut -c`: cut truncates each LINE and caps nothing.
|
||||||
|
command_text=$(printf '%s' "$command_text" | head -c 2000)
|
||||||
|
|
||||||
|
# -sRr, never -rR: jq -R without -s reads LINE BY LINE, so a multi-line command
|
||||||
|
# would encode per line and join with raw newlines — an invalid URL.
|
||||||
|
cmd_enc=$(printf '%s' "$command_text" | jq -sRr '@uri' 2>/dev/null) || exit 0
|
||||||
|
tool_enc=$(printf '%s' "$tool_name" | jq -sRr '@uri' 2>/dev/null) || exit 0
|
||||||
|
|
||||||
|
repo_q=""
|
||||||
|
lookup_dir=${event_cwd:-${CLAUDE_PROJECT_DIR:-$PWD}}
|
||||||
|
repo_remote=$(git -C "$lookup_dir" remote get-url origin 2>/dev/null || true)
|
||||||
|
if [ -n "$repo_remote" ]; then
|
||||||
|
repo_enc=$(printf '%s' "$repo_remote" | jq -sRr '@uri' 2>/dev/null) || repo_enc=""
|
||||||
|
[ -n "$repo_enc" ] && repo_q="&repo=${repo_enc}"
|
||||||
|
fi
|
||||||
|
|
||||||
|
# THE SHARED SESSION LEDGER, and the thing most worth getting right here.
|
||||||
|
#
|
||||||
|
# scribe_prior_art.sh keeps the rules it has already named in
|
||||||
|
# <state>/<sid>.rules.ids and passes them as exclude_rule_ids. This hook reads
|
||||||
|
# and appends to that SAME file rather than keeping its own: two ledgers would
|
||||||
|
# mean a rule named by one arm gets re-offered by the other, and the hint that
|
||||||
|
# fires most often is exactly the one that must not repeat itself.
|
||||||
|
#
|
||||||
|
# The directory keeps the prior-art name on purpose — renaming it would orphan
|
||||||
|
# every live session's state for a cosmetic gain.
|
||||||
|
state_dir="${TMPDIR:-/tmp}/scribe-priorart"
|
||||||
|
mkdir -p "$state_dir" 2>/dev/null || true
|
||||||
|
rulefile=""
|
||||||
|
rule_exclude_q=""
|
||||||
|
if [ -n "$session_id" ]; then
|
||||||
|
safe_sid=$(printf '%s' "$session_id" | tr -c 'A-Za-z0-9._-' '_')
|
||||||
|
rulefile="$state_dir/${safe_sid}.rules.ids"
|
||||||
|
if [ -f "$rulefile" ]; then
|
||||||
|
rule_seen=$(tr '\n' ',' < "$rulefile" 2>/dev/null | sed 's/,$//')
|
||||||
|
[ -n "$rule_seen" ] && rule_exclude_q="&exclude_rule_ids=${rule_seen}"
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
|
||||||
|
# `|| exit 0` here, unlike the prior-art hook: there is no local arm whose
|
||||||
|
# finding would be discarded, and an outage line before every command is worse
|
||||||
|
# than silence. See the header.
|
||||||
|
body=$(curl -fsS --max-time 5 \
|
||||||
|
-H "Authorization: Bearer ${token}" \
|
||||||
|
"${url%/}/api/plugin/tool-rules?tool=${tool_enc}&command=${cmd_enc}${repo_q}${rule_exclude_q}" 2>/dev/null) || exit 0
|
||||||
|
|
||||||
|
context=$(printf '%s' "$body" | jq -r '.context // empty' 2>/dev/null) || exit 0
|
||||||
|
[ -n "$context" ] || exit 0
|
||||||
|
|
||||||
|
# Remember what was named so it is not repeated this session.
|
||||||
|
if [ -n "$rulefile" ]; then
|
||||||
|
printf '%s' "$body" | jq -r '(.rule_ids // [])[]?' 2>/dev/null >> "$rulefile" || true
|
||||||
|
fi
|
||||||
|
|
||||||
|
jq -cn --arg ctx "$context" '{
|
||||||
|
hookSpecificOutput: {
|
||||||
|
hookEventName: "PreToolUse",
|
||||||
|
additionalContext: $ctx
|
||||||
|
}
|
||||||
|
}' 2>/dev/null || true
|
||||||
|
exit 0
|
||||||
@@ -305,6 +305,16 @@ SMOKE_EVENTS: dict[str, str] = {
|
|||||||
"tool_input": {"file_path": "src/x.py",
|
"tool_input": {"file_path": "src/x.py",
|
||||||
"new_string": f"def {_ABSENT_SYM}():\n pass\n"}}
|
"new_string": f"def {_ABSENT_SYM}():\n pass\n"}}
|
||||||
),
|
),
|
||||||
|
# The pre-tool rule arm (#3476). A real Bash call, and one whose whole
|
||||||
|
# point is that it looks harmless: reaching for curl against the forge API
|
||||||
|
# is the reflex the arm exists to catch. With no instance it must stay
|
||||||
|
# SILENT — it is deliberately not an OUTAGE_SPEAKER, because a Bash call is
|
||||||
|
# not occasional and an outage line before every command gets the channel
|
||||||
|
# muted.
|
||||||
|
"scribe_tool_rules.sh": json.dumps(
|
||||||
|
{"session_id": "smoke", "cwd": ".", "tool_name": "Bash",
|
||||||
|
"tool_input": {"command": "curl -s https://example.invalid/api/v1/runs"}}
|
||||||
|
),
|
||||||
"scribe_sync_processes.sh": json.dumps({"source": "startup"}),
|
"scribe_sync_processes.sh": json.dumps({"source": "startup"}),
|
||||||
"scribe_session_context.sh": json.dumps({"source": "startup"}),
|
"scribe_session_context.sh": json.dumps({"source": "startup"}),
|
||||||
# The after-write hook (#2901) diffs the working tree; on CI's clean
|
# The after-write hook (#2901) diffs the working tree; on CI's clean
|
||||||
|
|||||||
@@ -101,6 +101,46 @@ async def autoinject_retrieve():
|
|||||||
return jsonify(result)
|
return jsonify(result)
|
||||||
|
|
||||||
|
|
||||||
|
@plugin_bp.get("/tool-rules")
|
||||||
|
@login_required
|
||||||
|
async def pre_tool_rules():
|
||||||
|
"""Standing rules for the plugin's PreToolUse hook on ACTIONS (#3476).
|
||||||
|
|
||||||
|
Answers "does a recorded rule speak to the command about to be run?" — the
|
||||||
|
sibling of /prior-art, which can only answer that question about a code
|
||||||
|
write. Rules about which tool to reach for (don't curl the forge, don't
|
||||||
|
stand up a stack, don't run the suite locally) had no retrieval surface at
|
||||||
|
all before this, which is why they all had to live in the resident preload.
|
||||||
|
|
||||||
|
Titles + trigger only, never the statement: the hint says a rule may apply
|
||||||
|
and hands over `get_rule(id)`. One rule at most (RULEHINT_LIMIT), and empty
|
||||||
|
most of the time.
|
||||||
|
|
||||||
|
Query:
|
||||||
|
tool (str) — the tool about to run, e.g. `Bash`. Used in
|
||||||
|
the hint's wording, not in the search: a
|
||||||
|
rule is about the action, not the harness.
|
||||||
|
command (str) — the command about to run; the semantic query.
|
||||||
|
Absent or blank → empty, no search.
|
||||||
|
repo (optional) — working repo remote, resolved to the bound
|
||||||
|
project exactly as /retrieve and /prior-art.
|
||||||
|
exclude_rule_ids (opt) — comma-separated rule ids already surfaced
|
||||||
|
this session. SHARED with /prior-art's
|
||||||
|
ledger on purpose: one session keeps one
|
||||||
|
list, so a rule named by either arm is not
|
||||||
|
re-offered by the other.
|
||||||
|
"""
|
||||||
|
tool = (request.args.get("tool") or "tool").strip()
|
||||||
|
command = request.args.get("command") or ""
|
||||||
|
project_id, _repo, _unbound = await _project_scope()
|
||||||
|
exclude_rule_ids = _int_list(request.args.get("exclude_rule_ids"))
|
||||||
|
result = await plugin_ctx_svc.build_tool_rule_hint(
|
||||||
|
g.user.id, tool, command,
|
||||||
|
project_id=project_id, exclude_rule_ids=exclude_rule_ids,
|
||||||
|
)
|
||||||
|
return jsonify(result)
|
||||||
|
|
||||||
|
|
||||||
@plugin_bp.get("/prior-art")
|
@plugin_bp.get("/prior-art")
|
||||||
@login_required
|
@login_required
|
||||||
async def write_path_prior_art():
|
async def write_path_prior_art():
|
||||||
|
|||||||
@@ -140,6 +140,15 @@ RULEHINT_DEFAULT_THRESHOLD = 0.72
|
|||||||
# adds a way to misconfigure the surface (rule 25 cuts both ways).
|
# adds a way to misconfigure the surface (rule 25 cuts both ways).
|
||||||
RULEHINT_LIMIT = 1
|
RULEHINT_LIMIT = 1
|
||||||
|
|
||||||
|
# How much of a command reaches the embedding (#3476). A shell call is not a
|
||||||
|
# file: most are short, and the ones that are not are usually a heredoc or a
|
||||||
|
# pasted script whose bulk says nothing about which rule applies. The VERB AND
|
||||||
|
# ITS TARGET sit at the front — `curl https://git.fabledsword.com/api/...`,
|
||||||
|
# `docker compose up`, `git checkout -b` — and that head is the whole signal.
|
||||||
|
# Sending the tail as well would push it out of a 512-token window and let a
|
||||||
|
# heredoc's prose decide the match.
|
||||||
|
_TOOL_QUERY_CHARS = 400
|
||||||
|
|
||||||
# Minimum SUBSTANCE (non-whitespace chars) a payload must carry before the
|
# Minimum SUBSTANCE (non-whitespace chars) a payload must carry before the
|
||||||
# semantic arm will run at all — the cheap half of the operator's #89 idea
|
# semantic arm will run at all — the cheap half of the operator's #89 idea
|
||||||
# ("a sliding scale between number of characters and semantic threshold").
|
# ("a sliding scale between number of characters and semantic threshold").
|
||||||
@@ -1250,6 +1259,101 @@ async def build_write_path_hint(
|
|||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
|
async def build_tool_rule_hint(
|
||||||
|
user_id: int,
|
||||||
|
tool_name: str,
|
||||||
|
command: str,
|
||||||
|
*,
|
||||||
|
project_id: int = 0,
|
||||||
|
exclude_rule_ids: list[int] | None = None,
|
||||||
|
) -> dict:
|
||||||
|
"""Standing rules that may apply to the ACTION about to be taken (#3476).
|
||||||
|
|
||||||
|
The sibling of the write-path rule arm, and the surface that was missing.
|
||||||
|
That arm is keyed on `code or path`, so a rule can only be retrieved at the
|
||||||
|
moment of a code WRITE. Every rule about which tool to reach for — don't
|
||||||
|
curl the forge, don't stand up a stack, don't run the suite locally, don't
|
||||||
|
branch — was therefore unreachable at the moment it mattered, and residency
|
||||||
|
in the always-on preload was the only surface it had.
|
||||||
|
|
||||||
|
WHY A MECHANICAL TRIGGER AND NOT AN INSTRUCTION. Note #3089's finding is
|
||||||
|
that a reflex generates no query: you reach for `curl` confidently, with no
|
||||||
|
moment of doubt, so any surface that waits to be asked never fires. Here
|
||||||
|
nothing has to be asked — the tool call IS the query, and the reflex has to
|
||||||
|
become a tool call before it can do anything.
|
||||||
|
|
||||||
|
Deliberately TOOL-AGNOSTIC: takes a name and a string. The hook decides
|
||||||
|
which tools it watches, so widening the matcher is a `hooks.json` edit with
|
||||||
|
no change here.
|
||||||
|
|
||||||
|
CONDITIONAL ONLY, exactly as the write-path arm — an always-on rule is
|
||||||
|
already resident and repeating it is noise. That filter is also the
|
||||||
|
transition this arm exists to enable: re-tier a rule to `conditional` and
|
||||||
|
it starts arriving here instead of in every session's preamble.
|
||||||
|
|
||||||
|
Fails open and returns an empty context on any error: a recall aid may
|
||||||
|
never break the operator's action.
|
||||||
|
"""
|
||||||
|
out: dict = {"context": "", "rule_ids": []}
|
||||||
|
command = (command or "").strip()
|
||||||
|
if not command:
|
||||||
|
return out
|
||||||
|
|
||||||
|
try:
|
||||||
|
cfg = await get_writepath_config(user_id)
|
||||||
|
if not cfg.get("enabled"):
|
||||||
|
return out
|
||||||
|
|
||||||
|
# The command text is the query. A long heredoc or a pasted script
|
||||||
|
# would otherwise push the meaningful head of the command out of the
|
||||||
|
# embedding window, so it is bounded — the verb and its target sit at
|
||||||
|
# the front, which is the part a rule is about.
|
||||||
|
query = command[:_TOOL_QUERY_CHARS]
|
||||||
|
|
||||||
|
t0 = time.perf_counter()
|
||||||
|
hits = await semantic_search_rules(
|
||||||
|
user_id, query, limit=RULEHINT_LIMIT,
|
||||||
|
threshold=cfg["rule_threshold"], tier="conditional",
|
||||||
|
)
|
||||||
|
duration_ms = (time.perf_counter() - t0) * 1000.0
|
||||||
|
|
||||||
|
already = set(exclude_rule_ids or [])
|
||||||
|
fresh = [(score, rule) for score, rule in hits if rule.id not in already]
|
||||||
|
if not fresh:
|
||||||
|
return out
|
||||||
|
|
||||||
|
lines: list[str] = []
|
||||||
|
rule_ids: list[int] = []
|
||||||
|
for _score, rule in fresh:
|
||||||
|
trigger = (rule.when_to_apply or "").strip()
|
||||||
|
lines.append(
|
||||||
|
f"Standing rule that may apply to this {tool_name} call — "
|
||||||
|
f"“{rule.title}”"
|
||||||
|
+ (f" ({trigger})" if trigger else "")
|
||||||
|
+ f". Read it with get_rule({rule.id}) before deciding it "
|
||||||
|
"does not apply; it is not in this session's loaded set."
|
||||||
|
)
|
||||||
|
rule_ids.append(rule.id)
|
||||||
|
|
||||||
|
record_retrieval(
|
||||||
|
user_id=user_id, source="pre_tool_rule", query=query,
|
||||||
|
threshold=cfg["rule_threshold"], limit=RULEHINT_LIMIT,
|
||||||
|
project_id=project_id,
|
||||||
|
is_task=None, results=fresh, duration_ms=duration_ms,
|
||||||
|
)
|
||||||
|
# RANKED, not ambient: this arm chose what it showed, so a pull can
|
||||||
|
# settle whether the choice was any good. `rule_usage.RANKED_SOURCES`
|
||||||
|
# carries the same name.
|
||||||
|
record_rule_surfaced(
|
||||||
|
user_id=user_id, rule_ids=rule_ids, source="pre_tool_rule",
|
||||||
|
)
|
||||||
|
out["context"] = "\n".join(lines)
|
||||||
|
out["rule_ids"] = rule_ids
|
||||||
|
except Exception:
|
||||||
|
logger.debug("pre-tool rule arm failed", exc_info=True)
|
||||||
|
return out
|
||||||
|
|
||||||
|
|
||||||
def _derive_line(path: str, derive: list[dict]) -> str:
|
def _derive_line(path: str, derive: list[dict]) -> str:
|
||||||
"""The ledger's word on the names being written (#2900): a duplicate
|
"""The ledger's word on the names being written (#2900): a duplicate
|
||||||
family to derive, or a canon to reuse — said at the write."""
|
family to derive, or a canon to reuse — said at the write."""
|
||||||
|
|||||||
@@ -54,8 +54,11 @@ WHY THIS NAMES THE RANKED SOURCES AND THE TWIN NAMES THE AMBIENT ONES. A
|
|||||||
deliberate divergence, on the failure mode rather than on symmetry. Both shapes
|
deliberate divergence, on the failure mode rather than on symmetry. Both shapes
|
||||||
fail silently when someone adds a surface and forgets the list, so the question
|
fail silently when someone adds a surface and forgets the list, so the question
|
||||||
is which list changes more often — and here it is emphatically the ambient one:
|
is which list changes more often — and here it is emphatically the ambient one:
|
||||||
there is exactly ONE ranked rule source, and this change alone adds seven bulk
|
there are TWO ranked rule sources (the write-path arm and the pre-tool arm)
|
||||||
ones. Naming the rare, stable half means a newly-added bulk surface defaults to
|
against the seven bulk ones the preload alone contributes. Ranked sources are
|
||||||
|
added when somebody builds a ranker, which is rare and deliberate; bulk ones
|
||||||
|
appear whenever a surface hands rules over, which is most of them. Naming the
|
||||||
|
rare, slow-moving half means a newly-added bulk surface defaults to
|
||||||
`ambient`, which merely under-counts it, instead of defaulting to `ranked`,
|
`ambient`, which merely under-counts it, instead of defaulting to `ranked`,
|
||||||
which would quietly pad the pull-through denominator with surfacings nobody
|
which would quietly pad the pull-through denominator with surfacings nobody
|
||||||
chose and make the arm look imprecise. Same argument #3191 and #3430 make
|
chose and make the arm look imprecise. Same argument #3191 and #3430 make
|
||||||
@@ -82,7 +85,7 @@ logger = logging.getLogger(__name__)
|
|||||||
# surfacing is a claim ("this rule may apply to what you are doing") that a pull
|
# surfacing is a claim ("this rule may apply to what you are doing") that a pull
|
||||||
# can confirm or refute, while an ambient one is a delivery nobody decided on.
|
# can confirm or refute, while an ambient one is a delivery nobody decided on.
|
||||||
# Add a source here only when a ranker picked it.
|
# Add a source here only when a ranker picked it.
|
||||||
RANKED_SOURCES = ("write_path_rule",)
|
RANKED_SOURCES = ("write_path_rule", "pre_tool_rule")
|
||||||
|
|
||||||
|
|
||||||
def is_ambient(source: str) -> bool:
|
def is_ambient(source: str) -> bool:
|
||||||
|
|||||||
@@ -401,3 +401,179 @@ def test_the_marker_paths_stay_silent():
|
|||||||
assert "record_rule_surfaced" not in staleness, (
|
assert "record_rule_surfaced" not in staleness, (
|
||||||
"the staleness arm emits a surfacing — it compares a marker, it shows nothing"
|
"the staleness arm emits a surfacing — it compares a marker, it shows nothing"
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
|
# ── The PRE-TOOL arm: rules keyed on the action (#3476) ────────────────
|
||||||
|
#
|
||||||
|
# The write-path arm can only be reached by a code write, so every rule about
|
||||||
|
# which tool to reach for was unretrievable at the moment it mattered — which
|
||||||
|
# is why they all had to be resident. These cover the surface that changes it.
|
||||||
|
|
||||||
|
|
||||||
|
def _tool_patches(pc, hits, recorder, cfg=None):
|
||||||
|
return (
|
||||||
|
patch.object(pc, "get_writepath_config",
|
||||||
|
AsyncMock(return_value=cfg or {
|
||||||
|
"enabled": True, "threshold": 0.6,
|
||||||
|
"top_k": 3, "rule_threshold": 0.6,
|
||||||
|
})),
|
||||||
|
patch.object(pc, "semantic_search_rules", AsyncMock(return_value=hits)),
|
||||||
|
patch.object(pc, "record_retrieval", MagicMock()),
|
||||||
|
patch.object(pc, "record_rule_surfaced", recorder),
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
async def _run_tool_arm(hits, recorder, command="curl -s https://git.example/api/v1/runs",
|
||||||
|
tool="Bash", **kwargs):
|
||||||
|
from scribe.services import plugin_context as pc
|
||||||
|
with ExitStack() as stack:
|
||||||
|
for ctx in _tool_patches(pc, hits, recorder):
|
||||||
|
stack.enter_context(ctx)
|
||||||
|
return await pc.build_tool_rule_hint(1, tool, command, **kwargs)
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_the_tool_arm_names_a_rule_for_the_command_about_to_run():
|
||||||
|
"""The 2026-09-03 incident in one test: reaching for curl against the forge
|
||||||
|
API is a Bash call, and nothing watched Bash."""
|
||||||
|
rec = MagicMock()
|
||||||
|
hits = [(0.71, fake_rule(id=161,
|
||||||
|
title="Reach the forge through its MCP tools, never curl",
|
||||||
|
when_to_apply="whenever you need CI status"))]
|
||||||
|
out = await _run_tool_arm(hits, rec)
|
||||||
|
|
||||||
|
assert out["rule_ids"] == [161]
|
||||||
|
assert "Reach the forge through its MCP tools" in out["context"]
|
||||||
|
assert "get_rule(161)" in out["context"], "the hint must hand over the way to read it"
|
||||||
|
assert "Bash" in out["context"], "the hint names the tool it is about"
|
||||||
|
assert rec.call_args.kwargs["source"] == "pre_tool_rule"
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_the_tool_arm_is_a_ranked_source():
|
||||||
|
"""It CHOSE what it showed, so a pull can settle whether the choice was any
|
||||||
|
good — unlike a preload, which chose nothing. If this drifts into the
|
||||||
|
ambient class the arm becomes unjudgeable, which is the state #3311
|
||||||
|
described and M333 existed to end."""
|
||||||
|
from scribe.services.rule_usage import is_ambient
|
||||||
|
|
||||||
|
assert not is_ambient("pre_tool_rule")
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_a_rule_the_session_already_holds_is_not_re_offered():
|
||||||
|
rec = MagicMock()
|
||||||
|
hits = [(0.71, fake_rule(id=161, title="Reach the forge through its MCP tools")),
|
||||||
|
(0.70, fake_rule(id=12, title="Don't run a local stack unless asked"))]
|
||||||
|
out = await _run_tool_arm(hits, rec, exclude_rule_ids=[161])
|
||||||
|
|
||||||
|
assert out["rule_ids"] == [12]
|
||||||
|
assert "161" not in out["context"]
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_an_empty_command_asks_the_ranker_nothing():
|
||||||
|
"""Every Bash call reaches this. A blank payload must cost no embedding
|
||||||
|
query at all, not merely return nothing after paying for one."""
|
||||||
|
from scribe.services import plugin_context as pc
|
||||||
|
|
||||||
|
search = AsyncMock(return_value=[])
|
||||||
|
rec = MagicMock()
|
||||||
|
with ExitStack() as stack:
|
||||||
|
stack.enter_context(patch.object(pc, "semantic_search_rules", search))
|
||||||
|
stack.enter_context(patch.object(pc, "record_rule_surfaced", rec))
|
||||||
|
out = await pc.build_tool_rule_hint(1, "Bash", " ")
|
||||||
|
|
||||||
|
assert out == {"context": "", "rule_ids": []}
|
||||||
|
search.assert_not_called()
|
||||||
|
rec.assert_not_called()
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_the_tool_arm_fails_open():
|
||||||
|
"""A recall aid may never break the operator's action. A ranker that raises
|
||||||
|
must cost the hint, not the command."""
|
||||||
|
from scribe.services import plugin_context as pc
|
||||||
|
|
||||||
|
with ExitStack() as stack:
|
||||||
|
stack.enter_context(patch.object(pc, "get_writepath_config",
|
||||||
|
AsyncMock(side_effect=RuntimeError("boom"))))
|
||||||
|
out = await pc.build_tool_rule_hint(1, "Bash", "docker compose up -d")
|
||||||
|
|
||||||
|
assert out == {"context": "", "rule_ids": []}
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_a_long_command_is_bounded_before_it_reaches_the_ranker():
|
||||||
|
"""A heredoc or a pasted script would push the verb and its target — the
|
||||||
|
part a rule is about — out of the embedding window."""
|
||||||
|
from scribe.services import plugin_context as pc
|
||||||
|
|
||||||
|
search = AsyncMock(return_value=[])
|
||||||
|
with ExitStack() as stack:
|
||||||
|
for ctx in _tool_patches(pc, [], MagicMock()):
|
||||||
|
stack.enter_context(ctx)
|
||||||
|
stack.enter_context(patch.object(pc, "semantic_search_rules", search))
|
||||||
|
await pc.build_tool_rule_hint(1, "Bash", "git tag v1 && " + "x" * 5000)
|
||||||
|
|
||||||
|
sent = search.call_args.args[1]
|
||||||
|
assert len(sent) <= pc._TOOL_QUERY_CHARS
|
||||||
|
assert sent.startswith("git tag v1"), "the head of the command is the signal"
|
||||||
|
|
||||||
|
|
||||||
|
def test_the_two_pre_tool_arms_share_one_session_rule_ledger():
|
||||||
|
"""The integration point most worth guarding.
|
||||||
|
|
||||||
|
Two ledgers would mean a rule named by the write arm gets re-offered by the
|
||||||
|
tool arm — and the hint that fires most often is exactly the one that must
|
||||||
|
not repeat itself. Asserted on the FILENAME both scripts build, because
|
||||||
|
that is the shared thing; a copy of the path in each is how they drift.
|
||||||
|
"""
|
||||||
|
prior = Path("plugin/hooks/scribe_prior_art.sh").read_text()
|
||||||
|
tool = Path("plugin/hooks/scribe_tool_rules.sh").read_text()
|
||||||
|
|
||||||
|
for src, name in ((prior, "scribe_prior_art.sh"), (tool, "scribe_tool_rules.sh")):
|
||||||
|
assert '"${TMPDIR:-/tmp}/scribe-priorart"' in src, f"{name}: state dir moved"
|
||||||
|
assert '.rules.ids' in src, f"{name}: rules ledger filename moved"
|
||||||
|
assert "exclude_rule_ids" in src, f"{name}: does not send the exclusion"
|
||||||
|
|
||||||
|
|
||||||
|
def test_the_tool_arm_is_registered_on_bash():
|
||||||
|
"""A hook that exists and is not registered runs never — and reads exactly
|
||||||
|
like a surface nobody needed."""
|
||||||
|
import json
|
||||||
|
|
||||||
|
manifest = json.loads(Path("plugin/hooks/hooks.json").read_text())
|
||||||
|
pre = manifest["hooks"]["PreToolUse"]
|
||||||
|
entries = {
|
||||||
|
m.get("matcher"): [h["command"] for h in m["hooks"]] for m in pre
|
||||||
|
}
|
||||||
|
assert "Bash" in entries, "nothing watches Bash — the reflex surface is unguarded"
|
||||||
|
assert any("scribe_tool_rules.sh" in c for c in entries["Bash"])
|
||||||
|
# The write arm keeps its own matcher; this is an addition, not a move.
|
||||||
|
assert any("scribe_prior_art.sh" in c for c in entries["Write|Edit"])
|
||||||
|
|
||||||
|
|
||||||
|
def test_the_hook_and_the_route_agree_on_every_parameter_name():
|
||||||
|
"""Rule 33, on a brand-new integration between layers.
|
||||||
|
|
||||||
|
The hook is shell and the route is Python; nothing but this test connects
|
||||||
|
them. A renamed query arg fails SILENTLY — the route reads an absent value,
|
||||||
|
the arm quietly searches nothing, and the surface looks like one that never
|
||||||
|
finds anything rather than one that is broken.
|
||||||
|
"""
|
||||||
|
import re
|
||||||
|
|
||||||
|
hook = Path("plugin/hooks/scribe_tool_rules.sh").read_text()
|
||||||
|
route = Path("src/scribe/routes/plugin.py").read_text()
|
||||||
|
handler = route.split("async def pre_tool_rules")[1].split("\n@plugin_bp")[0]
|
||||||
|
|
||||||
|
sent = set(re.findall(r"[?&]([a-z_]+)=", hook))
|
||||||
|
assert sent == {"tool", "command", "repo", "exclude_rule_ids"}, sent
|
||||||
|
|
||||||
|
# `repo` is read by the shared _project_scope() helper, not inline.
|
||||||
|
assert "_project_scope()" in handler
|
||||||
|
for arg in ("tool", "command", "exclude_rule_ids"):
|
||||||
|
assert f'request.args.get("{arg}")' in handler, (
|
||||||
|
f"the hook sends {arg!r} and the route never reads it"
|
||||||
|
)
|
||||||
|
|||||||
Reference in New Issue
Block a user