Self-surfacing DRY — duplicate families named at the write and on arrival, tool-agnostic (milestone 299 steps 1–4) #125

Merged
bvandeusen merged 5 commits from dev into main 2026-08-22 13:41:24 -04:00
21 changed files with 1046 additions and 127 deletions
+2
View File
@@ -172,6 +172,8 @@ endpoint at `/mcp`, not these REST routes.
| GET | `/api/plugin/context` | SessionStart context payload (rules + active-project) |
| GET | `/api/plugin/retrieve` | Title-first knowledge-injection candidates |
| GET | `/api/plugin/processes` | Stored Processes for skill-stub sync |
| GET | `/api/plugin/prior-art` | Write-path hint for the plugin hooks (params: `path`, `code`, `repo`, `shapes`, `exclude_ids`, `exclude_sync_ids`, `exclude_derive`); returns `context`, `note_ids`, `sync_note_ids`, `stamped`, `divergence`, `derive`, `derive_keys` |
| GET / POST | `/api/projects/<id>/coverage`, `…/coverage/refresh` | Shape-ledger accounting (`pattern_coverage` line, counts, `derive_groups`, `derive_new`, `divergence`, `recheck`) |
| GET / PUT | `/api/plugin/marketplace-url` | Read / set the plugin marketplace URL |
## Dashboard, Export, Trash, Users
+1 -1
View File
@@ -1,7 +1,7 @@
{
"name": "scribe",
"description": "Scribe system-of-record for Claude Code: MCP tools over your notes/tasks/projects/rules, a session-start push channel that surfaces your always-on rules + active-project context, process-skills (writing-plans, systematic-debugging, verification, brainstorming, reusing-code), and your saved Scribe Processes auto-surfaced as skills (/scribe:sync). Replaces superpowers + file-memory with one app-backed plugin.",
"version": "0.1.38",
"version": "0.1.39",
"author": { "name": "Bryan Van Deusen" },
"mcpServers": {
"scribe": {
+11 -1
View File
@@ -52,8 +52,18 @@ On install you'll be asked for:
but never stop it; silent when nothing is recorded, which is most of the time.
Two framings: a REUSE menu (similar/nearby records), and a SYNC nudge when a
snippet records the exact file being edited — "updating the record is part of
the edit" — each with its own once-per-session dedup.
the edit" — each with its own once-per-session dedup. A third, ledger-fed
line names a duplicate family (no canon) or a canon recorded elsewhere for
the names being written (its own dedup channel, `exclude_derive`).
Toggle in **Settings → Knowledge auto-inject**.
- `hooks/hooks.json` → PostToolUse hook on `Bash`
(`hooks/scribe_after_write.sh`): code written through sed/heredocs/scripts
never reaches the PreToolUse hook, so this one diffs the working tree after
every Bash call (per-session path+blob snapshot; one `git status` when
nothing changed) and runs the same arms on the definitions just written,
through the same endpoint and the same dedup channels. `additionalContext`
only; silent on any failure. The extractor, the prose/data skip list and the
local by-name duplicate arm are shared in `hooks/scribe_defs.sh`.
- `skills/` → the universal process-skills, surfaced by description match.
- `hooks/scribe_sync_processes.sh` (a 2nd SessionStart hook) + the `/scribe:sync`
command → generate `~/.claude/skills/scribe-proc-*` stubs from your Scribe
+11
View File
@@ -34,6 +34,17 @@
}
]
}
],
"PostToolUse": [
{
"matcher": "Bash",
"hooks": [
{
"type": "command",
"command": "bash \"${CLAUDE_PLUGIN_ROOT}/hooks/scribe_after_write.sh\""
}
]
}
]
}
}
+221
View File
@@ -0,0 +1,221 @@
#!/usr/bin/env bash
# Scribe plugin — PostToolUse write-path trigger on Bash (#2901).
#
# scribe_prior_art.sh fires before a Write/Edit TOOL CALL. Code written any
# other way — sed, heredocs, python edit scripts, `cat > file` — never reached
# it, so a whole class of edits (the ones a long session makes most) got no
# prior-art hint, no ledger feed and no duplicate-family warning. This hook
# closes that: after EVERY Bash call it asks git what changed in the working
# tree since it last looked, and runs the same arms on the definitions that
# were just written — the local by-name duplicate arm, the recorded prior-art
# arms and the ledger's derive/divergence checks (#2900/#2793), via the same
# /api/plugin/prior-art endpoint the pre-write hook uses.
#
# Post-hoc by a few seconds, in the same moment and the same session: "the
# copy just landed; here is its family" — not "an audit found it later".
#
# Cheap when nothing changed: one `git status`. State per session, beside the
# pre-write hook's (its three dedup channels are SHARED, so a family named by
# one hook is not named again by the other):
# ${TMPDIR:-/tmp}/scribe-afterwrite/<sid>.snap path<TAB>blob-hash of every
# dirty/untracked file last seen
# ${TMPDIR:-/tmp}/scribe-priorart/<sid>.* the dedup channels
#
# NEVER BLOCKS. It returns `additionalContext` only (no decision — there is
# nothing left to decide, the write already happened). Any failure —
# unconfigured, unreachable, not a git repo, malformed — exits 0 in silence.
#
# Config (same as the other hooks):
# CLAUDE_PLUGIN_OPTION_API_ENDPOINT base URL, no trailing slash
# CLAUDE_PLUGIN_OPTION_API_TOKEN fmcp_ API key (sensitive)
# SCRIBE_URL / SCRIBE_TOKEN override for the settings.json dogfooding path.
set -uo pipefail
command -v jq >/dev/null 2>&1 || exit 0
command -v git >/dev/null 2>&1 || exit 0
# shellcheck source=plugin/hooks/scribe_defs.sh
. "$(dirname "${BASH_SOURCE[0]}")/scribe_defs.sh"
# PostToolUse delivers { session_id, cwd, tool_name, tool_input, tool_response }.
event=$(cat 2>/dev/null || true)
tool_name=$(printf '%s' "$event" | jq -r '.tool_name // empty' 2>/dev/null) || exit 0
[ "$tool_name" = "Bash" ] || exit 0
session_id=$(printf '%s' "$event" | jq -r '.session_id // empty' 2>/dev/null) || session_id=""
event_cwd=$(printf '%s' "$event" | jq -r '.cwd // empty' 2>/dev/null) || event_cwd=""
work_dir=${event_cwd:-${CLAUDE_PROJECT_DIR:-$PWD}}
repo_root=$(git -C "$work_dir" rev-parse --show-toplevel 2>/dev/null) || exit 0
[ -n "$repo_root" ] || exit 0
safe_sid=$(printf '%s' "${session_id:-nosession}" | tr -c 'A-Za-z0-9._-' '_')
snap_dir="${TMPDIR:-/tmp}/scribe-afterwrite"
mkdir -p "$snap_dir" 2>/dev/null || true
snap="$snap_dir/${safe_sid}.snap"
# What is dirty now: every modified / added / untracked path, with the blob
# hash of its working-tree content. Hash, not mtime: portable (no stat
# flags), exact (a touch is not a change), and untracked files hash the same
# way tracked ones do.
current=""
while IFS= read -r line; do
[ -n "$line" ] || continue
status=${line:0:2}
path=${line:3}
case "$status" in
D*|*D) continue ;; # a deletion defines nothing
esac
case "$path" in
*" -> "*) path=${path##* -> } ;; # rename: the new name
esac
# Porcelain quotes paths with special characters; those are skipped rather
# than unquoted badly — a filename needing quotes is not where shapes live.
case "$path" in
\"*) continue ;;
esac
[ -f "$repo_root/$path" ] || continue
sha=$(git -C "$repo_root" hash-object -- "$path" 2>/dev/null) || continue
current="${current}${path}"$'\t'"${sha}"$'\n'
done < <(git -C "$repo_root" status --porcelain --untracked-files=all 2>/dev/null)
previous=""
[ -f "$snap" ] && previous=$(cat "$snap" 2>/dev/null || true)
first_run=0
[ -f "$snap" ] || first_run=1
# Write the new snapshot NOW, before anything can fail below — the next call
# must compare against this tree, whatever happens to this one's hint.
printf '%s' "$current" > "$snap" 2>/dev/null || true
# Changed = a (path, hash) pair not in the previous snapshot. On the very
# first call of a session there is no previous snapshot; rather than report
# every pre-existing dirty file as "just written", take only files touched in
# the last minute — the Bash call that just ran is the likely author.
changed=""
while IFS=$'\t' read -r path sha; do
[ -n "${path:-}" ] || continue
if [ "$first_run" = 1 ]; then
[ -n "$(find "$repo_root/$path" -mmin -1 2>/dev/null)" ] || continue
else
case "$previous" in
*"${path}"$'\t'"${sha}"*) continue ;;
esac
fi
scribe_skip_path "$path" && continue
changed="${changed}${path}"$'\n'
done <<< "$current"
[ -n "$changed" ] || exit 0
url=${SCRIBE_URL:-${CLAUDE_PLUGIN_OPTION_API_ENDPOINT:-}}
token=${SCRIBE_TOKEN:-${CLAUDE_PLUGIN_OPTION_API_TOKEN:-}}
case "$url" in *'${'*) url="" ;; esac
case "$token" in *'${'*) token="" ;; esac
repo=$(git -C "$repo_root" remote get-url origin 2>/dev/null || true)
repo_q=""
if [ -n "$repo" ]; then
enc=$(printf '%s' "$repo" | jq -sRr '@uri' 2>/dev/null) || enc=""
[ -n "$enc" ] && repo_q="&repo=${enc}"
fi
# The dedup channels are the PRE-write hook's files, on purpose (see header).
state_dir="${TMPDIR:-/tmp}/scribe-priorart"
mkdir -p "$state_dir" 2>/dev/null || true
idfile="$state_dir/${safe_sid}.ids"
syncfile="$state_dir/${safe_sid}.sync.ids"
derivefile="$state_dir/${safe_sid}.derive.ids"
combined=""
n_files=0
while IFS= read -r rel_path; do
[ -n "${rel_path:-}" ] || continue
# A Bash call that rewrote many files is a refactor or a generator, not a
# shape being instantiated; four is enough to name what matters.
n_files=$((n_files + 1))
[ "$n_files" -le 4 ] || break
file_path="$repo_root/$rel_path"
# The code just written: the ADDED lines of the uncommitted diff for a
# tracked file (sed, not cut: this strips one marker char per line, it is
# not a payload cap), the whole file when untracked.
if git -C "$repo_root" ls-files --error-unmatch -- "$rel_path" >/dev/null 2>&1; then
code=$(git -C "$repo_root" diff -U0 -- "$rel_path" 2>/dev/null | grep '^+' | grep -v '^+++' | sed 's/^+//') || code=""
else
code=$(cat "$file_path" 2>/dev/null) || code=""
fi
[ -n "$code" ] || continue
names=$(printf '%s' "$code" | scribe_defs | sort -u | head -12) || names=""
# Nothing DEFINED in what was written (prose, data, a call-site edit) →
# nothing to say; the arms are about shapes.
[ -n "$names" ] || continue
local_lines=$(scribe_local_dups "$repo_root" "$rel_path" <<< "$names") || local_lines=""
local_context=""
if [ -n "$local_lines" ]; then
local_context="> Already defined elsewhere in this repo — \`${rel_path}\` (just written) adds another copy; check before keeping it (\`git grep\` shown; a nudge, not a gate):"$'\n'"${local_lines}"
fi
context=""
body=""
if [ -n "$url" ] && [ -n "$token" ]; then
q=$(printf '%s' "$code" | head -c 1200)
path_enc=$(printf '%s' "$rel_path" | jq -sRr '@uri' 2>/dev/null) || path_enc=""
code_enc=$(printf '%s' "$q" | jq -sRr '@uri' 2>/dev/null) || code_enc=""
shapes_q=""
enc=$(printf '%s\n' "$names" \
| awk -F'\t' 'NF>=2 {printf "%s%s:%s", (n++?",":""), $1, $2}' \
| jq -sRr '@uri' 2>/dev/null) || enc=""
[ -n "$enc" ] && shapes_q="&shapes=${enc}"
exclude_q=""; sync_exclude_q=""; derive_exclude_q=""
if [ -f "$idfile" ]; then
seen=$(tr '\n' ',' < "$idfile" 2>/dev/null | sed 's/,$//')
[ -n "$seen" ] && exclude_q="&exclude_ids=${seen}"
fi
if [ -f "$syncfile" ]; then
sync_seen=$(tr '\n' ',' < "$syncfile" 2>/dev/null | sed 's/,$//')
[ -n "$sync_seen" ] && sync_exclude_q="&exclude_sync_ids=${sync_seen}"
fi
if [ -f "$derivefile" ]; then
derive_seen=$(tr '\n' ',' < "$derivefile" 2>/dev/null | sed 's/,$//' | jq -sRr '@uri' 2>/dev/null) || derive_seen=""
[ -n "$derive_seen" ] && derive_exclude_q="&exclude_derive=${derive_seen}"
fi
if [ -n "$path_enc" ]; then
body=$(curl -fsS --max-time 4 \
-H "Authorization: Bearer ${token}" \
"${url%/}/api/plugin/prior-art?path=${path_enc}&code=${code_enc}${repo_q}${exclude_q}${sync_exclude_q}${derive_exclude_q}${shapes_q}" 2>/dev/null) || body=""
fi
if [ -n "$body" ]; then
context=$(printf '%s' "$body" | jq -r '.context // empty' 2>/dev/null) || context=""
if [ -n "$context" ]; then
printf '%s' "$body" | jq -r '((.note_ids // []) - (.sync_note_ids // []))[]?' 2>/dev/null >> "$idfile" || true
printf '%s' "$body" | jq -r '(.sync_note_ids // [])[]?' 2>/dev/null >> "$syncfile" || true
printf '%s' "$body" | jq -r '(.derive_keys // [])[]?' 2>/dev/null >> "$derivefile" || true
# Several files in one call may name the same family: keep each
# token once, so the next request's exclude list stays exact.
for f in "$idfile" "$syncfile" "$derivefile"; do
[ -s "$f" ] && { sort -u -o "$f" "$f" 2>/dev/null || true; }
done
fi
fi
fi
# The record nudge (#2664), same gate as the pre-write hook: duplication
# demonstrated locally AND nothing recorded for it.
if [ -n "$local_lines" ]; then
n_recorded=$(printf '%s' "$body" | jq -r '.note_ids | length' 2>/dev/null) || n_recorded=0
if [ "${n_recorded:-0}" = "0" ] || [ "$n_recorded" = "" ]; then
local_context="${local_context}"$'\n'"> None of those existing copies is recorded in Scribe. If the version just written is the canonical one — or this edit is consolidating the copies — record it now with create_snippet so the next session is offered it instead of writing another copy."
fi
fi
part="$local_context"
if [ -n "$context" ]; then
[ -n "$part" ] && part="${part}"$'\n'
part="${part}${context}"
fi
[ -n "$part" ] || continue
[ -n "$combined" ] && combined="${combined}"$'\n'
combined="${combined}${part}"
done <<< "$changed"
[ -n "$combined" ] || exit 0
jq -n --arg c "$combined" \
'{hookSpecificOutput: {hookEventName: "PostToolUse", additionalContext: $c}}'
exit 0
+109
View File
@@ -0,0 +1,109 @@
#!/usr/bin/env bash
# shellcheck shell=bash
# Scribe plugin — the pieces the two write-path hooks share (#2901).
#
# scribe_prior_art.sh fires BEFORE a Write/Edit tool call; scribe_after_write.sh
# fires AFTER a Bash tool call and diffs the working tree, so code written by
# sed/heredocs/scripts gets the same prior-art and ledger checks. Both need the
# same three things, kept here so they cannot drift apart:
#
# scribe_skip_path PATH formats that hold prose or data, not shapes
# scribe_defs stdin code → "kind<TAB>name" per definition
# scribe_local_dups ROOT REL "kind<TAB>name" lines on stdin → the by-name
# local-duplicate lines (ARM 1, #2280)
#
# Sourced, not executed: `. "$(dirname "${BASH_SOURCE[0]}")/scribe_defs.sh"`.
# Skip formats that hold prose or data rather than reusable code. Purely to
# avoid a pointless round-trip — the server would return nothing for these
# anyway. Config formats are NOT skipped: a CI workflow or a compose file is
# often exactly the thing worth reusing.
scribe_skip_path() {
case "$1" in
*.md|*.mdx|*.txt|*.rst|*.json|*.lock|*.log|*.csv|*.tsv|*.svg|*.png|*.jpg|*.jpeg|*.gif|*.ico|*.pdf)
return 0 ;;
esac
return 1
}
# ---------------------------------------------------------------------------
# kind<TAB>name for each thing a piece of code DEFINES, in source order. One
# program, two consumers: the local duplicate arm (every definition in the
# payload) and the ledger feed (#2791, below: the definitions being written,
# or the one enclosing an Edit). Rule-for-rule mirrored by the server's
# services/coverage.py extract_shapes — ledger rows are keyed by what THAT
# sees, so the two must agree on what counts as a definition.
scribe_defs() {
awk '
{
# CSS class definition: .name { or .name,
if (match($0, /^[[:space:]]*\.[A-Za-z][A-Za-z0-9_-]*[[:space:]]*[,{]/)) {
t = $0; sub(/^[[:space:]]*\./, "", t); sub(/[[:space:]]*[,{].*$/, "", t)
if (t != "") print "css\t" t; next
}
line = $0; sub(/^[[:space:]]+/, "", line)
# Strip leading declaration modifiers so the definition keyword is the
# first word regardless of language (export/pub/private/suspend/...).
sub(/^((pub(\([a-z]+\))?|export|default|private|internal|protected|public|static|suspend|async|open|sealed|data|abstract|final|inline|unsafe|extern|override)[[:space:]]+)*/, "", line)
# Go method with receiver: func (r *T) Name(
if (match(line, /^func[[:space:]]*\([^)]*\)[[:space:]]*[A-Za-z_]/)) {
t = line; sub(/^func[[:space:]]*\([^)]*\)[[:space:]]*/, "", t)
sub(/[^A-Za-z0-9_].*$/, "", t)
if (t != "") print "sym\t" t; next
}
# Keyword-announced definitions, functions and named types alike.
# Dunders are skipped: every class defines __init__, so "already defined
# in N other files" is guaranteed noise for them — and noise is what
# teaches sessions to skip the hint.
if (match(line, /^(function|def|class|func|fun|fn|sub|struct|trait|interface|enum|object|protocol|type)[[:space:]]+[A-Za-z_$]/)) {
t = line; sub(/^[a-z]+[[:space:]]+/, "", t)
sub(/[^A-Za-z0-9_$].*$/, "", t)
if (t != "" && t !~ /^__.*__$/) print "sym\t" t; next
}
# Arrow/expression assignment: const name = (…) / let name = async (
if (match(line, /^(const|let)[[:space:]]+[A-Za-z_$][A-Za-z0-9_$]*[[:space:]]*=[[:space:]]*(async[[:space:]]*)?[(<]/)) {
t = line; sub(/^(const|let)[[:space:]]+/, "", t)
sub(/[^A-Za-z0-9_$].*$/, "", t)
if (t != "") print "sym\t" t; next
}
}
' 2>/dev/null
}
# ---------------------------------------------------------------------------
# ARM 1 — BY NAME, LOCALLY (#2280). Does a definition of this already exist?
#
# The recorded arms ask Scribe what was RECORDED; the ledger arm (#2900) asks
# what a BOUND repo's ledger knows. A helper nobody recorded, in a repo nobody
# bound, is invisible to both — which is how `.btn-primary` came to be defined
# four times, already diverged. This arm asks the one question only the
# developer's machine can answer, inside the repo, holding the code about to
# be written: no index, no storage, no server — it runs even on an install
# that has never configured Scribe.
#
# Definition-shaped patterns only. Grepping for bare occurrences would match
# every CALL site and drown the real finding — and a hint that is mostly noise
# is one people learn to skip, which is worse than none. ALL code, not a
# language shortlist (#2682): the same keyword family scribe_defs announces.
#
# $1 repo root, $2 repo-relative path of the file being written (excluded from
# the grep — it would always match itself on an Edit). Definitions on stdin.
# Prints one "> - `name` is already defined in N other file(s): …" per hit.
scribe_local_dups() {
local root="$1" rel="$2" kind name pat hits count label files
while IFS=$'\t' read -r kind name; do
[ -n "${name:-}" ] || continue
case "$kind" in
css) pat="^[[:space:]]*\.${name}[[:space:]]*[,{]" ;;
*) pat="(function|def|class|func|fun|fn|sub|struct|trait|interface|enum|object|protocol|type)[[:space:]]+${name}[^A-Za-z0-9_]|func[[:space:]]*\([^)]*\)[[:space:]]*${name}[[:space:]]*\(|(const|let)[[:space:]]+${name}[[:space:]]*=" ;;
esac
# -I skips binaries; :(exclude) drops the file being written.
hits=$(git -C "$root" grep -I -l -E -e "$pat" -- . ":(exclude)${rel}" 2>/dev/null | head -4) || hits=""
[ -n "$hits" ] || continue
count=$(printf '%s\n' "$hits" | grep -c . 2>/dev/null || echo 0)
label=$([ "$kind" = css ] && printf '.%s' "$name" || printf '%s' "$name")
files=$(printf '%s' "$hits" | tr '\n' ' ' | sed 's/ $//')
printf '> - `%s` is already defined in %s other file(s): %s\n' "$label" "$count" "$files"
done
}
+26 -96
View File
@@ -51,14 +51,12 @@ code=$(printf '%s' "$event" | jq -r '
.tool_input.content // .tool_input.file_content //
.tool_input.new_string // .tool_input.new_str // empty' 2>/dev/null) || code=""
# Skip formats that hold prose or data rather than reusable code. Purely to
# avoid a pointless round-trip — the server would return nothing for these
# anyway. Config formats are NOT skipped: a CI workflow or a compose file is
# often exactly the thing worth reusing.
case "$file_path" in
*.md|*.mdx|*.txt|*.rst|*.json|*.lock|*.log|*.csv|*.tsv|*.svg|*.png|*.jpg|*.jpeg|*.gif|*.ico|*.pdf)
exit 0 ;;
esac
# Shared with the after-write hook (#2901): the prose/data skip list, the
# definition extractor and the local by-name duplicate arm live in
# scribe_defs.sh so the two hooks cannot drift apart.
# shellcheck source=plugin/hooks/scribe_defs.sh
. "$(dirname "${BASH_SOURCE[0]}")/scribe_defs.sh"
scribe_skip_path "$file_path" && exit 0
# Snippet locations are recorded repo-relative, so send a repo-relative path —
# an absolute one would simply match nothing. Resolved BEFORE the config gate
@@ -73,78 +71,8 @@ if [ -n "$repo_root" ]; then
esac
fi
# ---------------------------------------------------------------------------
# ARM 1 — BY NAME, LOCALLY (#2280). Does a definition of this already exist?
#
# The other two arms ask Scribe what was RECORDED. Scribe has never read a line
# of the codebase, so a helper nobody thought to record is invisible to them —
# which is how `.btn-primary` came to be defined four times, in four scoped
# stylesheets, already diverged. It was never a snippet, so no threshold and no
# query rewrite could ever have surfaced it.
#
# This arm closes that by asking the only question the record cannot answer,
# in the only place that can: the hook already runs on the developer's machine,
# inside the repo, holding the code about to be written. No index, no storage,
# no staleness, and no server — it deliberately runs even on an install that
# has never configured Scribe.
#
# Definition-shaped patterns only. Grepping for bare occurrences would match
# every CALL site and drown the real finding — and a hint that is mostly noise
# is one people learn to skip, which is worse than none.
#
# ALL code, not a language shortlist (#2682): the detector was born covering
# only the languages of the repo it was written in, which silently amputated
# this whole arm — and the record nudge gated on it — for every Go/Kotlin/Rust
# project. Definitions are announced by a small keyword family across
# languages (func/fun/fn/function/def/sub · class/struct/trait/interface/
# enum/object/protocol/type), so one modifier-strip + keyword match covers
# them all. Known out of scope: keyword-less declaration syntax (C/Java/Dart
# `ReturnType name(...)`) needs a real parser, and `impl` blocks are excluded
# because several per type is normal Rust, not duplication.
# ---------------------------------------------------------------------------
# kind<TAB>name for each thing a piece of code DEFINES, in source order. One
# program, two consumers: the local duplicate arm (every definition in the
# payload) and the ledger feed (#2791, below: the definitions being written,
# or the one enclosing an Edit). Rule-for-rule mirrored by the server's
# services/coverage.py extract_shapes — ledger rows are keyed by what THAT
# sees, so the two must agree on what counts as a definition.
scribe_defs() {
awk '
{
# CSS class definition: .name { or .name,
if (match($0, /^[[:space:]]*\.[A-Za-z][A-Za-z0-9_-]*[[:space:]]*[,{]/)) {
t = $0; sub(/^[[:space:]]*\./, "", t); sub(/[[:space:]]*[,{].*$/, "", t)
if (t != "") print "css\t" t; next
}
line = $0; sub(/^[[:space:]]+/, "", line)
# Strip leading declaration modifiers so the definition keyword is the
# first word regardless of language (export/pub/private/suspend/...).
sub(/^((pub(\([a-z]+\))?|export|default|private|internal|protected|public|static|suspend|async|open|sealed|data|abstract|final|inline|unsafe|extern|override)[[:space:]]+)*/, "", line)
# Go method with receiver: func (r *T) Name(
if (match(line, /^func[[:space:]]*\([^)]*\)[[:space:]]*[A-Za-z_]/)) {
t = line; sub(/^func[[:space:]]*\([^)]*\)[[:space:]]*/, "", t)
sub(/[^A-Za-z0-9_].*$/, "", t)
if (t != "") print "sym\t" t; next
}
# Keyword-announced definitions, functions and named types alike.
# Dunders are skipped: every class defines __init__, so "already defined
# in N other files" is guaranteed noise for them — and noise is what
# teaches sessions to skip the hint.
if (match(line, /^(function|def|class|func|fun|fn|sub|struct|trait|interface|enum|object|protocol|type)[[:space:]]+[A-Za-z_$]/)) {
t = line; sub(/^[a-z]+[[:space:]]+/, "", t)
sub(/[^A-Za-z0-9_$].*$/, "", t)
if (t != "" && t !~ /^__.*__$/) print "sym\t" t; next
}
# Arrow/expression assignment: const name = (…) / let name = async (
if (match(line, /^(const|let)[[:space:]]+[A-Za-z_$][A-Za-z0-9_$]*[[:space:]]*=[[:space:]]*(async[[:space:]]*)?[(<]/)) {
t = line; sub(/^(const|let)[[:space:]]+/, "", t)
sub(/[^A-Za-z0-9_$].*$/, "", t)
if (t != "") print "sym\t" t; next
}
}
' 2>/dev/null
}
# ARM 1 — BY NAME, LOCALLY (#2280): does a definition of this already exist
# in the repo? (scribe_local_dups in scribe_defs.sh carries the why.)
names=""
if [ -n "$code" ]; then
names=$(printf '%s' "$code" | scribe_defs | sort -u | head -12) || names=""
@@ -152,21 +80,8 @@ fi
local_lines=""
if [ -n "$repo_root" ] && [ -n "$names" ]; then
while IFS=$'\t' read -r kind name; do
[ -n "${name:-}" ] || continue
case "$kind" in
css) pat="^[[:space:]]*\.${name}[[:space:]]*[,{]" ;;
*) pat="(function|def|class|func|fun|fn|sub|struct|trait|interface|enum|object|protocol|type)[[:space:]]+${name}[^A-Za-z0-9_]|func[[:space:]]*\([^)]*\)[[:space:]]*${name}[[:space:]]*\(|(const|let)[[:space:]]+${name}[[:space:]]*=" ;;
esac
# -I skips binaries; :(exclude) drops the file being written, which would
# otherwise always match itself on an Edit.
hits=$(git -C "$repo_root" grep -I -l -E -e "$pat" -- . ":(exclude)${rel_path}" 2>/dev/null | head -4) || hits=""
[ -n "$hits" ] || continue
count=$(printf '%s\n' "$hits" | grep -c . 2>/dev/null || echo 0)
label=$([ "$kind" = css ] && printf '.%s' "$name" || printf '%s' "$name")
files=$(printf '%s' "$hits" | tr '\n' ' ' | sed 's/ $//')
local_lines="${local_lines}> - \`${label}\` is already defined in ${count} other file(s): ${files}"$'\n'
done <<< "$names"
local_lines=$(scribe_local_dups "$repo_root" "$rel_path" <<< "$names") || local_lines=""
[ -n "$local_lines" ] && local_lines="${local_lines}"$'\n'
fi
local_context=""
@@ -258,14 +173,22 @@ fi
# the sync nudge when the recorded file itself is edited later.
state_dir="${TMPDIR:-/tmp}/scribe-priorart"
mkdir -p "$state_dir" 2>/dev/null || true
#
# A THIRD channel (#2900): the ledger's derive arm names a duplicate family
# (a derive group id) or a canon elsewhere (`canon:<snippet_id>`) for the
# shapes being written. Keyed by that token, not a note id, so it dedups on
# its own file and a family is named once per session, not at every edit.
idfile=""
syncfile=""
derivefile=""
exclude_q=""
sync_exclude_q=""
derive_exclude_q=""
if [ -n "$session_id" ]; then
safe_sid=$(printf '%s' "$session_id" | tr -c 'A-Za-z0-9._-' '_')
idfile="$state_dir/${safe_sid}.ids"
syncfile="$state_dir/${safe_sid}.sync.ids"
derivefile="$state_dir/${safe_sid}.derive.ids"
if [ -f "$idfile" ]; then
seen=$(tr '\n' ',' < "$idfile" 2>/dev/null | sed 's/,$//')
[ -n "$seen" ] && exclude_q="&exclude_ids=${seen}"
@@ -274,13 +197,17 @@ if [ -n "$session_id" ]; then
sync_seen=$(tr '\n' ',' < "$syncfile" 2>/dev/null | sed 's/,$//')
[ -n "$sync_seen" ] && sync_exclude_q="&exclude_sync_ids=${sync_seen}"
fi
if [ -f "$derivefile" ]; then
derive_seen=$(tr '\n' ',' < "$derivefile" 2>/dev/null | sed 's/,$//' | jq -sRr '@uri' 2>/dev/null) || derive_seen=""
[ -n "$derive_seen" ] && derive_exclude_q="&exclude_derive=${derive_seen}"
fi
fi
# `|| true`, not `|| exit 0`: an unreachable instance must not discard a local
# finding that needed no instance to produce.
body=$(curl -fsS --max-time 5 \
-H "Authorization: Bearer ${token}" \
"${url%/}/api/plugin/prior-art?path=${path_enc}&code=${code_enc}${repo_q}${exclude_q}${sync_exclude_q}${shapes_q}" 2>/dev/null) || body=""
"${url%/}/api/plugin/prior-art?path=${path_enc}&code=${code_enc}${repo_q}${exclude_q}${sync_exclude_q}${derive_exclude_q}${shapes_q}" 2>/dev/null) || body=""
context=""
if [ -n "$body" ]; then
@@ -295,6 +222,9 @@ if [ -n "$body" ]; then
if [ -n "$syncfile" ]; then
printf '%s' "$body" | jq -r '(.sync_note_ids // [])[]?' 2>/dev/null >> "$syncfile" || true
fi
if [ -n "$derivefile" ]; then
printf '%s' "$body" | jq -r '(.derive_keys // [])[]?' 2>/dev/null >> "$derivefile" || true
fi
fi
fi
+4 -1
View File
@@ -66,7 +66,10 @@ for the operator's work, and as your own working memory across sessions.
should read as a map of every shape in it. The backstop still holds:
noticing the second copy of anything, or consolidating copies into a shared
X, means X gets recorded before that work is finished — which is how a
codebase is kept from growing four `.btn-primary` definitions.
codebase is kept from growing four `.btn-primary` definitions. The write-path
hooks (before a Write/Edit, and after any Bash call that changed the tree)
name a known duplicate family or a canon elsewhere for what was just
written — act on that line at the write, not at the next audit.
- Do **not** keep the operator's rules, plans, or project notes in local
memory / CLAUDE.md in parallel with Scribe — Scribe holds the single copy.
- **Compact at clean seams** — because you record as you go, a context
+7
View File
@@ -44,6 +44,13 @@ through recall/auto-inject; this skill is the active reflex around that.
it before you go any further. Either it's the helper you were about to
duplicate — reuse it and drop yours — or it isn't, and the record needs the new
location adding. Both are cheaper now than after the duplicate settles in.
- **A `Shape ledger at …` line is the ledger speaking, not the record.** It
names a duplicate family ("identical body in N other files, no canon") or a
canon elsewhere for a name you just wrote — for edits made through Bash
(sed, heredocs, scripts) as much as through Write/Edit. Derive the family or
reuse the canon *now*; a family that is convention rather than copies is
dismissed with `classify_shapes(..., status="exempt",
reason_code="convention-plumbing")`, never ignored.
- **A `[records this file]` hint is a duty, not a menu.** When the hint says a
snippet records the very file you're editing, the record's freshness is now
YOUR edit's responsibility: if the edit changes the recorded shape,
+28
View File
@@ -85,6 +85,34 @@ the dominant form, `create_snippet` it, migrate the outliers, then classify
the rest as instances. Canon is determined from the code; consistency comes
from the derivation, not from asking permission.
## Derive groups are drift, not audit material
The catalogue exists so the codebase is DRY **from inception**, not as DRY as
the last sweep left it. Three surfaces say so without anyone running an audit
(milestone 299):
- **At the write** — the prior-art hint (the Write/Edit hook, and since
0.1.39 the after-write hook on Bash, so sed/heredoc/script edits count too)
carries a `Shape ledger at <path>` line when a name just written is a known
**duplicate family** ("identical body in N other files, no canon") or a
**canon elsewhere** ("snippet #N at <path> — reuse, don't redefine"). Act
on it *then*: pull the canon and build from it, or derive the family now —
`create_snippet` the dominant form, repoint the copies, `classify_shapes`
them `instance`. A family is named once per session.
- **On arrival** — the coverage line's `standing:` block (shown even when
nothing is unclassified) and `derive_new` ("+N new copies since last
refresh: .x in <path>") name what drifted since the previous refresh. That
is the todo of the moment, sized to the last batch — not a backlog.
- **A family that is convention, not copies** — component-local `load` /
`toggle` / `save` that happen to share a name — is dismissed, not
consolidated: `classify_shapes(..., status="exempt",
reason_code="convention-plumbing", reason=…)` (or `classify_shapes_by_rule`
for a whole family) removes it from the queue. Dismissal is a judgment and
it is recorded; silence is not.
After the one-time pay-down the derive queue reads empty; anything in it
afterwards is drift of the moment, and the hint already said so at the write.
## The divergence readout — button B where button A is canon
Three questions the ledger answers mechanically (#2793):
+11
View File
@@ -203,6 +203,17 @@ SMOKE_EVENTS: dict[str, str] = {
),
"scribe_sync_processes.sh": json.dumps({"source": "startup"}),
"scribe_session_context.sh": json.dumps({"source": "startup"}),
# The after-write hook (#2901) diffs the working tree; on CI's clean
# checkout there is nothing to report, so silence is the right assertion.
# (On a dirty local tree with a definition just written it may speak —
# that is the hook working, not a failure of the contract.)
"scribe_after_write.sh": json.dumps(
{"session_id": "smoke", "cwd": ".", "tool_name": "Bash",
"tool_input": {"command": "true"}, "tool_response": {}}
),
# The shared library is sourced, never run; executed bare it defines
# functions and exits — silent by construction.
"scribe_defs.sh": "",
}
# The one hook that legitimately produces output with no credentials.
+3 -1
View File
@@ -295,7 +295,9 @@ async def refresh_pattern_coverage(project_id: int) -> dict:
Returns the accounting payload — total, accounted, counts by status,
unclassified, repos, largest_gaps, `proposed` (canon proposals awaiting
confirmation), `derive_groups` (the biggest repeats-with-no-canon
families), `proposer` (what this refresh examined) — plus
families), `derive_new` (copies that joined a family since the previous
refresh — the drift to act on now: derive the canon, don't queue an
audit), `proposer` (what this refresh examined) — plus
`pattern_coverage`, the same one-line summary enter_project carries.
"""
uid = current_user_id()
+8
View File
@@ -129,6 +129,10 @@ async def write_path_prior_art():
surfaced. A separate channel on purpose: a reuse
hint shown early must not suppress the record-sync
nudge when the recorded file is edited later.
exclude_derive (opt) — comma-separated derive keys (a derive group id
or `canon:<snippet_id>`) already named this
session by the ledger arm (#2900); its own
channel, like the two above.
shapes (opt) — comma-separated `kind:name` definitions the hook
found in (or enclosing) the payload, kind being
css|sym. The shape ledger's write-path feed
@@ -144,6 +148,9 @@ async def write_path_prior_art():
project_id, repo, _unbound = await _project_scope()
exclude_ids = _int_list(request.args.get("exclude_ids"))
exclude_sync_ids = _int_list(request.args.get("exclude_sync_ids"))
exclude_derive = [
p.strip() for p in (request.args.get("exclude_derive") or "").split(",") if p.strip()
]
shapes = _parse_shapes(request.args.get("shapes") or "")
api_key = getattr(g, "api_key", None)
may_stamp = api_key is None or getattr(api_key, "scope", "") == "write"
@@ -153,6 +160,7 @@ async def write_path_prior_art():
exclude_ids=exclude_ids, exclude_sync_ids=exclude_sync_ids,
stamp_shapes=shapes if may_stamp else None,
repo_key=repo_bindings_svc.normalize_repo_key(repo) if repo else "",
exclude_derive=exclude_derive,
)
return jsonify(result)
+47 -20
View File
@@ -462,15 +462,20 @@ async def compute_coverage(
await shape_ledger.apply_derive_groups(project_id)
except Exception:
logger.warning("derive-first grouping failed", exc_info=True)
# The button-B pass (#2793): shapes new since the PREVIOUS computation,
# where a canon dominates. The previous computation's stamp is the cache;
# a first seed has none, so it flags nothing (everything is new then).
# "Since the previous computation" — the cache's stamp. A first seed has
# none, so nothing is new then. Read once; two passes use it: the
# button-B flag (#2793) and the derive-new drift count (#2899).
since = None
try:
previous = await get_setting(user_id, f"{_CACHE_KEY_PREFIX}{project_id}")
since = None
if previous:
stamp = (json.loads(previous) or {}).get("computed_at")
since = datetime.fromisoformat(stamp) if stamp else None
except Exception:
logger.warning("previous coverage stamp unreadable", exc_info=True)
# The button-B pass (#2793): shapes new since the PREVIOUS computation,
# where a canon dominates.
try:
await shape_ledger.flag_divergence(project_id, since=since)
except Exception:
logger.warning("divergence pass failed", exc_info=True)
@@ -491,6 +496,7 @@ async def compute_coverage(
unclassified = counts.pop("unclassified")
proposals = shape_ledger.proposal_summary(rows)
divergence = shape_ledger.divergence_summary(rows)
derive_new = shape_ledger.derive_new_summary(rows, since=since)
return {
"total": len(rows),
"accounted": len(rows) - unclassified,
@@ -501,6 +507,10 @@ async def compute_coverage(
"proposed": proposals["proposed"],
"derive_groups": proposals["derive_groups"],
"top_canon": proposals.get("top_canon"),
# Drift since the previous refresh (#2899): copies that joined a
# duplicate family — what the arrival line names so drift is noticed
# on entering, not found by an audit.
"derive_new": derive_new,
"proposer": proposer_stats,
# The divergence readout (#2793): button B where button A is canon,
# and judged shapes whose bodies moved since they were judged.
@@ -648,29 +658,46 @@ def coverage_line(coverage: dict) -> str:
line += f"{breakdown}"
line += f" (estimate{', computed ' + day if day else ''})"
unclassified = coverage.get("unclassified", 0)
# The standing work, built whatever the todo count (#2899). Since the
# scoped bucket (#2869) a ledger can read 100% accounted and still carry
# derive groups, proposals and divergence; gating this block on
# `unclassified > 0` is how 439 derive rows went unmentioned.
standing = []
if coverage.get("proposed"):
standing.append(f"{coverage['proposed']} proposed")
n_groups = len(coverage.get("derive_groups") or [])
if n_groups:
standing.append(f"{n_groups} derive group{'s' if n_groups != 1 else ''}")
# Drift since the previous refresh: copies that joined a family, the
# first one named — the sentence the arrival moment exists to say.
new = coverage.get("derive_new") or {}
if new.get("count"):
n = new["count"]
first_new = (new.get("examples") or [{}])[0]
where = (
f": {first_new['label']} in {first_new['path']}"
if first_new.get("label") and first_new.get("path") else ""
)
standing.append(f"+{n} new cop{'y' if n == 1 else 'ies'} since last refresh{where}")
if coverage.get("divergent"):
standing.append(f"{coverage['divergent']} DIVERGENT")
# The next action, on the line (#2874): the canon with the biggest
# queue to confirm, and the widest body-identical copy to consolidate.
top = coverage.get("top_canon") or {}
if top.get("snippet_id"):
standing.append(f"top canon #{top['snippet_id']} ×{top.get('count', 0)}")
first = (coverage.get("derive_groups") or [{}])[0]
if first.get("label") and first.get("files"):
standing.append(f"top copy {first['label']} ×{first['files']} files")
if unclassified:
line += f"; {unclassified} unclassified"
standing = []
if coverage.get("proposed"):
standing.append(f"{coverage['proposed']} proposed")
n_groups = len(coverage.get("derive_groups") or [])
if n_groups:
standing.append(f"{n_groups} derive group{'s' if n_groups != 1 else ''}")
if coverage.get("divergent"):
standing.append(f"{coverage['divergent']} DIVERGENT")
# The next action, on the line (#2874): the canon with the biggest
# queue to confirm, and the widest body-identical copy to consolidate.
top = coverage.get("top_canon") or {}
if top.get("snippet_id"):
standing.append(f"top canon #{top['snippet_id']} ×{top.get('count', 0)}")
first = (coverage.get("derive_groups") or [{}])[0]
if first.get("label") and first.get("files"):
standing.append(f"top copy {first['label']} ×{first['files']} files")
if standing:
line += f" ({', '.join(standing)})"
gaps = [g["dir"] for g in coverage.get("largest_gaps") or []]
if gaps:
line += ", largest: " + ", ".join(gaps)
elif standing:
line += f"; standing: {', '.join(standing)}"
if coverage.get("recheck"):
line += f"; {coverage['recheck']} judged shape{'s' if coverage['recheck'] != 1 else ''} changed since judged — recheck"
return line
+49 -2
View File
@@ -706,6 +706,7 @@ async def build_write_path_hint(
exclude_sync_ids: list[int] | None = None,
stamp_shapes: list[tuple[str, str]] | None = None,
repo_key: str = "",
exclude_derive: list[str] | None = None,
) -> dict:
"""Prior-art hint for the plugin's PreToolUse hook on Write/Edit.
@@ -765,7 +766,7 @@ async def build_write_path_hint(
"""
cfg = await get_writepath_config(user_id)
empty = {"context": "", "note_ids": [], "sync_note_ids": [], "config": cfg,
"stamped": [], "divergence": []}
"stamped": [], "divergence": [], "derive": [], "derive_keys": []}
path = (path or "").strip()
if not cfg["enabled"] or not path:
return empty
@@ -935,7 +936,20 @@ async def build_write_path_hint(
)
except Exception:
logger.warning("write-time divergence check failed", exc_info=True)
if not synced and not menu and not stamped and not divergence:
# The in-band DERIVE check (#2900): the ledger's own knowledge of the
# names being written — a duplicate family with no canon, or a canon
# recorded elsewhere. This is the arm the by-name local grep could not
# be: it knows whether the other copies are canon or stray. Keyed per
# session (`exclude_derive`) so a family is named once, not per edit.
derive: list[dict] = []
if stamp_shapes and project_id:
try:
found = await shape_ledger_svc.write_time_derive(project_id, path, stamp_shapes)
skip = set(exclude_derive or [])
derive = [d for d in found if d.get("key") not in skip]
except Exception:
logger.warning("write-time derive check failed", exc_info=True)
if not synced and not menu and not stamped and not divergence and not derive:
return empty
owners = await owner_names_for({
@@ -1003,6 +1017,8 @@ async def build_write_path_hint(
lines.append(_stamp_line(path, stamped))
if divergence:
lines.append(_divergence_line(path, divergence))
if derive:
lines.append(_derive_line(path, derive))
# Split by arm, which is the whole reason this table exists. The place arm
# carries no score and so has no home in retrieval_logs; before #2085 a
@@ -1027,9 +1043,40 @@ async def build_write_path_hint(
"config": cfg,
"stamped": stamped,
"divergence": divergence,
"derive": derive,
"derive_keys": [d["key"] for d in derive],
}
def _derive_line(path: str, derive: list[dict]) -> str:
"""The ledger's word on the names being written (#2900): a duplicate
family to derive, or a canon to reuse — said at the write."""
parts = []
for d in derive:
if d.get("canon"):
c = d["canon"]
parts.append(
f"`{c['label']}` is canon — snippet #{c['snippet_id']} at `{c['path']}`; "
"pull it and reuse, don't redefine"
)
continue
f = d["family"]
how = "identical body" if f.get("identical") else "same name defined"
files = ", ".join(f"`{x}`" for x in f.get("files") or [])
more = f.get("file_count", 0) - len(f.get("files") or [])
if more > 0:
files += f" +{more} more"
parts.append(
f"`{f['label']}` is a duplicate family with no canon — {how} in "
f"{f.get('file_count', 0)} other file(s): {files}; derive it now: "
"record the canon (create_snippet) and make the copies instances "
"(classify_shapes) — or, if these are convention not copies, "
"`classify_shapes(..., status=\"exempt\", reason_code=\"convention-plumbing\")` "
"dismisses the family — rather than adding another copy"
)
return f"> Shape ledger at `{path}`: " + "; ".join(parts) + "."
def _divergence_line(path: str, divergence: list[dict]) -> str:
"""Button B where button A is canon — named at the write (#2793)."""
parts = [
+97
View File
@@ -1404,6 +1404,34 @@ def proposal_summary(rows: Iterable[CodeShape], *, top: int = 8) -> dict:
return {"proposed": proposed, "derive_groups": ranked[:top], "top_canon": top_canon}
def derive_new_summary(
rows: Iterable[CodeShape], *, since: datetime | None, top: int = 3
) -> dict:
"""The arrival-moment drift signal (#2899): derive-grouped rows FIRST
SEEN after ``since`` — the previous refresh's stamp, the same one
flag_divergence uses. "Since the last refresh, N more copies joined a
duplicate family" is the sentence that makes the derive queue a thing
you notice on entering, not a thing an audit finds. ``since`` None (a
first seed) means nothing is new. Judged rows never count."""
if since is None:
return {"count": 0, "examples": []}
fresh = [
r for r in rows
if r.proposal_basis == "derive" and r.proposal_group
and r.status in _MECHANICAL_TODO and r.vanished_at is None
and r.created_at is not None and r.created_at > since
]
fresh.sort(key=lambda r: r.created_at, reverse=True)
return {
"count": len(fresh),
"examples": [
{"label": ("." if r.kind == "css" else "") + r.symbol,
"path": r.path, "group": r.proposal_group}
for r in fresh[:top]
],
}
async def confirm_proposals(
user_id: int,
project_id: int,
@@ -1566,6 +1594,75 @@ async def write_time_divergence(
return out
# How many other files a family line names before "…" — enough to go look,
# not a wall.
_DERIVE_FILES_SHOWN = 4
async def write_time_derive(
project_id: int, path: str, shapes: list[tuple[str, str]]
) -> list[dict]:
"""The in-band DERIVE check (#2900): for each (kind, name) the hook
named at ``path``, what the ledger already knows about that name
elsewhere in the project —
family the name sits in a derive-first group (identical body in N
files, or the same name in ≥3): "this is a known duplicate
family with no canon — derive it now, don't add a copy";
canon a `canonical` row of that name at another path: "this is
canon #N at <path> — reuse, don't redefine".
Only for shapes not yet judged at ``path`` (a judged shape is not
re-litigated at every edit), never for the canon's own file. Returns
[{symbol, kind, key, family?|canon?}] — `key` is the dedup token the
hook keeps per session (the group id, or canon:<snippet_id>)."""
wanted = {(k, _norm_symbol(n)): n for k, n in shapes if n}
if not wanted:
return []
async with async_session() as session:
rows = (
await session.execute(
select(CodeShape).where(
CodeShape.project_id == project_id,
CodeShape.vanished_at.is_(None),
CodeShape.symbol.in_({norm for (_k, norm) in wanted}),
)
)
).scalars().all()
out: list[dict] = []
for (kind, norm), name in wanted.items():
same = [r for r in rows if r.kind == kind and _norm_symbol(r.symbol) == norm]
here = next((r for r in same if r.path == path), None)
if here is not None and here.status not in _MECHANICAL_TODO:
continue # judged here (or this IS the canon): nothing to say
others = [r for r in same if r.path != path]
label = ("." if kind == "css" else "") + name
canon = next((r for r in others if r.status == "canonical" and r.snippet_id), None)
if canon is not None:
out.append({"symbol": name, "kind": kind, "key": f"canon:{canon.snippet_id}",
"canon": {"snippet_id": canon.snippet_id, "path": canon.path,
"label": label}})
continue
grouped = [r for r in others if r.proposal_group and r.status in _MECHANICAL_TODO]
if here is not None and here.proposal_group:
grouped = [r for r in grouped if r.proposal_group == here.proposal_group] or grouped
if not grouped:
continue
group = grouped[0].proposal_group
members = [r for r in grouped if r.proposal_group == group]
files = sorted({r.path for r in members})
out.append({
"symbol": name, "kind": kind, "key": group,
"family": {
"group": group, "label": label,
"identical": not group.startswith("name:"),
"files": files[:_DERIVE_FILES_SHOWN], "file_count": len(files),
"size": len(members) + (1 if here is not None else 0),
},
})
return out
async def flag_divergence(project_id: int, *, since: datetime | None) -> int:
"""Flag shapes created after ``since`` (the previous refresh) that sit
where a canon dominates and were not proposed as that canon. With no
+145
View File
@@ -0,0 +1,145 @@
"""The PostToolUse after-write hook (#2901): code written through Bash — sed,
heredocs, scripts — gets the same prior-art / ledger checks as a Write/Edit.
Runs the real shell against a temp git repo and a throwaway HTTP sink, like
the pre-write hook's end-to-end tests. Skips where the hook's tools are
missing; asserts on content where they are present."""
from __future__ import annotations
import http.server
import json
import os
import shutil
import subprocess
import threading
import urllib.parse
from pathlib import Path
import pytest
PLUGIN = Path(__file__).resolve().parents[1] / "plugin"
HOOK = PLUGIN / "hooks" / "scribe_after_write.sh"
def _env(tmp_path, url="http://127.0.0.1:9"):
for tool in ("git", "jq", "curl", "bash"):
if shutil.which(tool) is None:
pytest.skip(f"hook runtime tool {tool!r} not installed")
return {"PATH": os.environ["PATH"], "SCRIBE_URL": url, "SCRIBE_TOKEN": "t",
"TMPDIR": str(tmp_path), "HOME": str(tmp_path),
"GIT_AUTHOR_NAME": "t", "GIT_AUTHOR_EMAIL": "t@x",
"GIT_COMMITTER_NAME": "t", "GIT_COMMITTER_EMAIL": "t@x"}
def _repo(tmp_path, env):
repo = tmp_path / "repo"
repo.mkdir()
subprocess.run(["git", "init", "-q"], cwd=repo, check=True, env=env)
(repo / "b.py").write_text("def one():\n return 1\n")
(repo / "c.py").write_text("def slug(t):\n return t.lower()\n")
subprocess.run(["git", "add", "."], cwd=repo, check=True, env=env)
subprocess.run(["git", "commit", "-q", "-m", "base"], cwd=repo, check=True, env=env)
return repo
def _run(repo, env, session="s-after-1", tool="Bash"):
out = subprocess.run(
["bash", str(HOOK)],
input=json.dumps({"session_id": session, "cwd": str(repo), "tool_name": tool,
"tool_input": {"command": "cat > x"}, "tool_response": {}}),
capture_output=True, text=True, env=env,
)
assert out.returncode == 0, out.stderr
return out.stdout
class _Sink(http.server.BaseHTTPRequestHandler):
seen: list[dict] = []
reply = b'{"context":"> family named","note_ids":[],"sync_note_ids":[],"derive_keys":["dup:483a"]}'
def do_GET(self):
type(self).seen.append(urllib.parse.parse_qs(urllib.parse.urlparse(self.path).query))
self.send_response(200)
self.send_header("Content-Type", "application/json")
self.end_headers()
self.wfile.write(type(self).reply)
def log_message(self, *a):
pass
@pytest.fixture
def sink():
_Sink.seen = []
server = http.server.HTTPServer(("127.0.0.1", 0), _Sink)
threading.Thread(target=server.serve_forever, daemon=True).start()
try:
yield server
finally:
server.shutdown()
def test_after_write_names_what_bash_just_wrote_then_stays_quiet_until_the_next_change(tmp_path, sink):
env = _env(tmp_path, url=f"http://127.0.0.1:{sink.server_port}")
repo = _repo(tmp_path, env)
# "A Bash call" wrote an untracked stylesheet and appended to a tracked file.
(repo / "a.css").write_text(".log-empty {\n color: red;\n}\n")
(repo / "b.py").write_text("def one():\n return 1\n\ndef slug(t):\n return t\n")
out = _run(repo, env)
by_path = {q["path"][0]: q for q in _Sink.seen}
assert set(by_path) == {"a.css", "b.py"} # repo-relative, like the pre hook
assert by_path["a.css"]["shapes"] == ["css:log-empty"]
assert by_path["b.py"]["shapes"] == ["sym:slug"]
# Added lines only for the tracked file — the existing def is not "just written".
assert "def slug" in by_path["b.py"]["code"][0] and "def one" not in by_path["b.py"]["code"][0]
ctx = json.loads(out)["hookSpecificOutput"]
assert ctx["hookEventName"] == "PostToolUse"
assert "> family named" in ctx["additionalContext"]
# The local by-name arm rides along: `slug` already lives in c.py.
assert "`slug` is already defined in 1 other file(s): c.py" in ctx["additionalContext"]
# Derive keys landed on the SHARED channel the pre-write hook reads.
state = tmp_path / "scribe-priorart" / "s-after-1.derive.ids"
assert "dup:483a" in state.read_text().split()
# Nothing changed → one git status, no request, no output.
_Sink.seen = []
assert _run(repo, env) == ""
assert _Sink.seen == []
# Another change → only that file, and the dedup channel goes back up.
(repo / "a.css").write_text(".log-empty {\n color: red;\n}\n.other {\n margin: 0;\n}\n")
_run(repo, env)
assert [q["path"][0] for q in _Sink.seen] == ["a.css"]
assert _Sink.seen[0]["exclude_derive"] == ["dup:483a"]
assert set(_Sink.seen[0]["shapes"][0].split(",")) == {"css:log-empty", "css:other"}
def test_after_write_is_silent_where_it_has_nothing_to_say(tmp_path):
env = _env(tmp_path)
repo = _repo(tmp_path, env)
# Not a Bash call → nothing (hooks.json matches Bash, the script re-checks).
(repo / "a.css").write_text(".x {\n color: red;\n}\n")
assert _run(repo, env, tool="Write") == ""
# Not a git repo → nothing.
loose = tmp_path / "loose"
loose.mkdir()
(loose / "a.css").write_text(".x {\n color: red;\n}\n")
assert _run(loose, env, session="s-loose") == ""
# A change that defines nothing (prose, a call-site edit) → nothing, even
# with the server unreachable (port 9 refuses): no definitions, no arms.
(repo / "README.md").write_text("# notes\n")
(repo / "b.py").write_text("def one():\n return one_more()\n")
assert _run(repo, env, session="s-quiet") == ""
def test_after_write_local_arm_and_record_nudge_work_without_a_server(tmp_path):
"""The local by-name arm needs no instance (#2280) and the record nudge
(#2664) fails open with it — a refused connection stands in for the
instance."""
env = _env(tmp_path)
repo = _repo(tmp_path, env)
(repo / "d.py").write_text("def slug(t):\n return t.lower()\n")
out = _run(repo, env, session="s-local")
ctx = json.loads(out)["hookSpecificOutput"]["additionalContext"]
assert "`slug` is already defined in 1 other file(s): c.py" in ctx
assert "create_snippet" in ctx
+76
View File
@@ -597,6 +597,82 @@ async def test_derive_groups_land_on_rows_and_in_the_summary(seeded):
assert {r.symbol for r in rows} == {"slug"} # 2 files < the name floor
@pytest.mark.integration
async def test_write_time_derive_names_the_family_or_the_canon_for_a_name(seeded):
"""#2900: against real rows — a name in a dup family → the family (other
files, count); a name whose canonical row lives elsewhere → that canon;
a judged row at the path, the canon's own file, or an unknown name →
silence."""
from scribe.services.shape_ledger import apply_derive_groups, write_time_derive
owner, pid, sid = seeded["owner"], seeded["pid"], seeded["snippet"]
defs = _defs(
("v/A.vue", "css", "log-empty", ".log-empty {", ".log-empty { color: red }"),
("v/B.vue", "css", "log-empty", ".log-empty {", ".log-empty { color: red }"),
("v/C.vue", "css", "log-empty", ".log-empty {", ".log-empty { color: red }"),
("src/factory.py", "sym", "factory", "def factory():", "def factory():\n return 1"),
)
await sync_repo_shapes(pid, REPO, defs, seen_marker="m1")
await classify_shapes(owner, pid, [
{"path": "src/factory.py", "symbol": "factory", "status": "canonical", "snippet_id": sid},
])
assert await apply_derive_groups(pid) >= 3
# A 4th copy about to be written → the family, naming the other files.
out = await write_time_derive(pid, "v/D.vue", [("css", "log-empty"), ("css", "unknown")])
assert len(out) == 1 and out[0]["symbol"] == "log-empty" and out[0]["kind"] == "css"
fam = out[0]["family"]
assert fam["identical"] is True and fam["label"] == ".log-empty"
assert fam["files"] == ["v/A.vue", "v/B.vue", "v/C.vue"] and fam["file_count"] == 3
assert out[0]["key"] == fam["group"] and fam["group"].startswith("dup:")
# Editing one existing member still names the OTHER members.
out = await write_time_derive(pid, "v/A.vue", [("css", "log-empty")])
assert out[0]["family"]["files"] == ["v/B.vue", "v/C.vue"] and out[0]["family"]["size"] == 3
# The canon's name elsewhere → the canon; in the canon's own file → silence.
out = await write_time_derive(pid, "src/other.py", [("sym", "factory")])
assert out == [{"symbol": "factory", "kind": "sym", "key": f"canon:{sid}",
"canon": {"snippet_id": sid, "path": "src/factory.py", "label": "factory"}}]
assert await write_time_derive(pid, "src/factory.py", [("sym", "factory")]) == []
# A judged row at the path is not re-litigated.
await classify_shapes(owner, pid, [
{"path": "v/B.vue", "symbol": "log-empty", "status": "exempt", "reason": "print sheet"},
])
assert await write_time_derive(pid, "v/B.vue", [("css", "log-empty")]) == []
@pytest.mark.integration
async def test_derive_new_names_the_copy_that_joined_a_family_since_the_stamp(seeded):
"""#2899: the first sync seeds one `slug`; a later sync adds an identical
copy. Against the stamp between them, derive_new counts ONLY the
newcomer — the drift since the last refresh, not the whole family."""
from datetime import datetime, timezone
from scribe.services.shape_ledger import (
apply_derive_groups, derive_new_summary, live_rows,
)
owner, pid = seeded["owner"], seeded["pid"]
first = _defs(
("a/one.py", "sym", "slug", "def slug(t):", "def slug(t):\n return t.lower()"),
)
await sync_repo_shapes(pid, REPO, first, seen_marker="m1")
stamp = datetime.now(timezone.utc)
second = _defs(
("a/one.py", "sym", "slug", "def slug(t):", "def slug(t):\n return t.lower()"),
("a/two.py", "sym", "slug", "def slug(t):", "def slug(t):\n return t.lower()"),
)
await sync_repo_shapes(pid, REPO, second, seen_marker="m2")
assert await apply_derive_groups(pid) == 2
rows = await live_rows(pid)
out = derive_new_summary(rows, since=stamp)
assert out["count"] == 1
assert out["examples"][0]["path"] == "a/two.py"
assert out["examples"][0]["label"] == "slug"
assert out["examples"][0]["group"].startswith("dup:")
assert derive_new_summary(rows, since=None)["count"] == 0
# --- #2793: the divergence readout against real rows -------------------------
+37
View File
@@ -276,6 +276,8 @@ async def test_coverage_measures_the_tree_exactly_and_caches(seeded):
assert coverage["largest_gaps"] == [
{"dir": "src", "unclassified": 2, "total": 3}
]
# #2899: a first computation has no previous stamp — nothing is "new".
assert coverage["derive_new"] == {"count": 0, "examples": []}
# The walk fed the LEDGER (#2788): every extracted shape has a row, the
# snippet reference locations are mechanically stamped canonical WITH
@@ -512,6 +514,41 @@ def test_coverage_line_names_the_proposers_standing():
assert "top canon #2844 ×78" in line and "top copy closed-msg (identical body) ×3 files" in line
def test_coverage_line_shows_standing_work_even_with_nothing_unclassified():
"""#2899: since the scoped bucket a ledger can be fully accounted and
still carry derive groups / proposals / divergence — the line names
them as `standing:` instead of hiding them behind the todo count, and
names the drift since the previous refresh first-copy-first."""
from scribe.services.coverage import coverage_line
base = {
"total": 4693, "accounted": 4693, "unclassified": 0,
"counts": {"canonical": 37, "instance": 977, "variant": 73, "exempt": 1797, "scoped": 1809},
"computed_at": "2026-08-22T00:00:00+00:00", "largest_gaps": [],
}
quiet = coverage_line(base)
assert "unclassified" not in quiet and "standing" not in quiet
line = coverage_line({
**base,
"derive_groups": [{"group": "dup:abc", "label": "log-empty (identical body)", "files": 4}],
"derive_new": {"count": 2, "examples": [
{"label": ".error-msg", "path": "frontend/src/components/InceptionCard.vue", "group": "dup:9f0"},
{"label": ".error-msg", "path": "frontend/src/components/Other.vue", "group": "dup:9f0"},
]},
"divergent": 1,
})
assert "; standing: 1 derive group, +2 new copies since last refresh: .error-msg in " \
"frontend/src/components/InceptionCard.vue, 1 DIVERGENT, top copy log-empty (identical body) ×4 files" in line
assert "unclassified" not in line
# One copy reads singular; with a todo the block keeps its old place.
one = coverage_line({**base, "derive_new": {"count": 1, "examples": []}})
assert one.endswith("; standing: +1 new copy since last refresh")
todo = coverage_line({**base, "unclassified": 3, "accounted": 4690, "proposed": 2,
"derive_new": {"count": 1, "examples": [{"label": "x", "path": "a.py"}]},
"largest_gaps": [{"dir": "src", "unclassified": 3, "total": 9}]})
assert "; 3 unclassified (2 proposed, +1 new copy since last refresh: x in a.py), largest: src" in todo
def test_coverage_line_names_divergence_and_recheck():
from scribe.services.coverage import coverage_line
+32
View File
@@ -317,6 +317,38 @@ def test_derive_groups_copy_before_name_with_floors():
assert ("i.py", "sym", "one") not in g
def test_derive_new_summary_counts_copies_first_seen_since_the_previous_refresh():
"""#2899: the arrival-moment drift signal — derive-grouped rows created
after the previous refresh's stamp, newest first, judged rows and a
first seed (since=None) never count."""
from datetime import datetime, timedelta, timezone
from scribe.models.code_shape import CodeShape
from scribe.services.shape_ledger import derive_new_summary
t0 = datetime(2026, 8, 22, 12, 0, tzinfo=timezone.utc)
def row(path, symbol, at, kind="css", status="scoped", group="dup:abc", basis="derive"):
r = CodeShape(project_id=2, repo_key="r", path=path, symbol=symbol, kind=kind,
status=status, proposal_basis=basis, proposal_group=group)
r.created_at = at
return r
rows = [
row("v/Old.vue", "error-msg", t0 - timedelta(days=3)), # before the stamp
row("v/InceptionCard.vue", "error-msg", t0 + timedelta(hours=1)), # new copy
row("v/Other.vue", "error-msg", t0 + timedelta(hours=2)), # newer copy
row("v/J.vue", "error-msg", t0 + timedelta(hours=3), status="exempt"), # judged: never
row("s/a.py", "load", t0 + timedelta(hours=1), kind="sym", group=None, basis=None), # no family
]
out = derive_new_summary(rows, since=t0)
assert out["count"] == 2
assert [e["path"] for e in out["examples"]] == ["v/Other.vue", "v/InceptionCard.vue"]
assert out["examples"][0] == {"label": ".error-msg", "path": "v/Other.vue", "group": "dup:abc"}
assert derive_new_summary(rows, since=None) == {"count": 0, "examples": []}
assert derive_new_summary(rows, since=t0, top=1)["examples"] == [
{"label": ".error-msg", "path": "v/Other.vue", "group": "dup:abc"}]
def test_proposal_summary_ranks_body_identical_groups_first_and_sees_scoped_rows():
"""#2872: dup groups (the real copies) outrank name groups (usually
convention), wider spread first; #2869: scoped rows are in the readout."""
+121 -5
View File
@@ -853,14 +853,18 @@ def test_hook_exits_silently_when_unconfigured():
def test_hook_skips_prose_and_data_files():
"""No round-trip for a markdown edit — the server would return nothing anyway."""
src = HOOK.read_text()
skip = re.search(r"case \"\$file_path\" in\n(.*?)esac", src, re.S)
assert skip, "expected an extension skip list"
"""No round-trip for a markdown edit — the server would return nothing
anyway. The list lives in the shared library (#2901) and the hook asks it."""
lib = (PLUGIN / "hooks" / "scribe_defs.sh").read_text()
skip = re.search(r"scribe_skip_path\(\) \{\n case \"\$1\" in\n(.*?)esac", lib, re.S)
assert skip, "expected an extension skip list in scribe_defs.sh"
for ext in ("*.md", "*.json", "*.lock", "*.png"):
assert ext in skip.group(1)
# Config formats are deliberately NOT skipped — a workflow file is reusable.
assert "*.yml" not in skip.group(1)
src = HOOK.read_text()
assert 'scribe_skip_path "$file_path" && exit 0' in src
assert '/scribe_defs.sh"' in src # sourced, not copied
def test_plugin_version_bumped_with_the_hook():
@@ -1150,7 +1154,9 @@ def test_hook_names_the_shapes_being_written():
that changes a body, not a signature — the definition enclosing the edit,
found by walking the target file upward from the edited lines."""
src = HOOK.read_text()
assert "scribe_defs()" in src # one extractor, two consumers
lib = (PLUGIN / "hooks" / "scribe_defs.sh").read_text()
assert "scribe_defs()" in lib # one extractor, shared (#2901)
assert "scribe_defs()" not in src # ...not a second copy here
assert ".tool_input.old_string" in src # the Edit's anchor
assert "| tac | scribe_defs | head -1" in src # nearest definition above
# The ledger feed sends NAMES, never bodies, and stays on the one GET.
@@ -1243,6 +1249,116 @@ def test_hook_sends_the_enclosing_definition_for_a_body_edit(tmp_path):
assert seen["shapes"] == ["sym:onTrash"]
@pytest.mark.asyncio
async def test_the_write_time_derive_check_names_a_family_or_a_canon_in_band():
"""#2900: the ledger's own word on the names being written — a duplicate
family with no canon, or a canon recorded elsewhere — rendered at the
write even when nothing else does; keyed so the session's exclude
channel silences a family already named."""
from scribe.services import plugin_context as pc
found = [
{"symbol": "log-empty", "kind": "css", "key": "dup:483a",
"family": {"group": "dup:483a", "label": ".log-empty", "identical": True,
"files": ["a/TaskLogSection.vue", "a/WorkspaceTaskPanel.vue"],
"file_count": 5, "size": 6}},
{"symbol": "btn-primary", "kind": "css", "key": "canon:2855",
"canon": {"snippet_id": 2855, "path": "frontend/src/assets/components.css",
"label": ".btn-primary"}},
{"symbol": "load", "kind": "sym", "key": "name:sym:load",
"family": {"group": "name:sym:load", "label": "load", "identical": False,
"files": ["a/X.vue", "a/Y.vue", "a/Z.vue"], "file_count": 3, "size": 4}},
]
check = AsyncMock(return_value=found)
patches = dict(
get_writepath_config=AsyncMock(return_value=_cfg()),
semantic_search_notes=AsyncMock(return_value=[]),
record_retrieval=MagicMock(), owner_names_for=AsyncMock(return_value={}),
)
with patch.multiple(pc, **patches), \
patch.object(pc.snippets_svc, "list_snippets", AsyncMock(return_value=([], 0))), \
patch.object(pc.shape_ledger_svc, "recent_pulls", AsyncMock(return_value={})), \
patch.object(pc.shape_ledger_svc, "write_time_divergence", AsyncMock(return_value=[])), \
patch.object(pc.shape_ledger_svc, "write_time_derive", check):
out = await pc.build_write_path_hint(
1, "frontend/src/components/New.vue", code=REAL_CODE, project_id=24,
stamp_shapes=[("css", "log-empty"), ("css", "btn-primary"), ("sym", "load")],
exclude_derive=["name:sym:load"],
)
check.assert_awaited_once_with(24, "frontend/src/components/New.vue",
[("css", "log-empty"), ("css", "btn-primary"), ("sym", "load")])
# The excluded family is gone; the other two render and are keyed.
assert [d["key"] for d in out["derive"]] == ["dup:483a", "canon:2855"]
assert out["derive_keys"] == ["dup:483a", "canon:2855"]
ctx = out["context"]
assert "Shape ledger at `frontend/src/components/New.vue`" in ctx
assert "`.log-empty` is a duplicate family with no canon — identical body in 5 other file(s): " \
"`a/TaskLogSection.vue`, `a/WorkspaceTaskPanel.vue` +3 more; derive it now" in ctx
assert "`.btn-primary` is canon — snippet #2855 at `frontend/src/assets/components.css`" in ctx
assert "convention-plumbing" in ctx
assert "`load`" not in ctx
# No project → no check; a failing check never sinks the hint.
check.reset_mock()
with patch.multiple(pc, **patches), \
patch.object(pc.snippets_svc, "list_snippets", AsyncMock(return_value=([], 0))), \
patch.object(pc.shape_ledger_svc, "recent_pulls", AsyncMock(return_value={})), \
patch.object(pc.shape_ledger_svc, "write_time_divergence", AsyncMock(return_value=[])), \
patch.object(pc.shape_ledger_svc, "write_time_derive", check):
out = await pc.build_write_path_hint(1, "x.py", code=REAL_CODE, stamp_shapes=[("sym", "f")])
check.assert_not_awaited()
assert out["derive"] == [] and out["derive_keys"] == []
boom = AsyncMock(side_effect=RuntimeError("ledger down"))
with patch.multiple(pc, **patches), \
patch.object(pc.snippets_svc, "list_snippets", AsyncMock(return_value=([], 0))), \
patch.object(pc.shape_ledger_svc, "recent_pulls", AsyncMock(return_value={})), \
patch.object(pc.shape_ledger_svc, "write_time_divergence", AsyncMock(return_value=[])), \
patch.object(pc.shape_ledger_svc, "write_time_derive", boom):
out = await pc.build_write_path_hint(1, "x.py", code=REAL_CODE, project_id=24,
stamp_shapes=[("sym", "f")])
assert out["context"] == "" and out["derive"] == []
def test_the_hook_keeps_a_derive_channel_and_sends_it_back(tmp_path):
"""#2900: derive keys the server returns land in the session's own
`.derive.ids` file and go back as `exclude_derive` on the next write —
a family is named once per session, not at every edit."""
import http.server
import threading
import urllib.parse
seen: list[dict] = []
class _Sink(http.server.BaseHTTPRequestHandler):
def do_GET(self):
seen.append(urllib.parse.parse_qs(urllib.parse.urlparse(self.path).query))
self.send_response(200)
self.send_header("Content-Type", "application/json")
self.end_headers()
self.wfile.write(b'{"context":"> family","note_ids":[],"sync_note_ids":[],'
b'"derive_keys":["dup:483a","canon:2855"]}')
def log_message(self, *a):
pass
server = http.server.HTTPServer(("127.0.0.1", 0), _Sink)
threading.Thread(target=server.serve_forever, daemon=True).start()
try:
env = dict(_hook_runtime_env(), SCRIBE_URL=f"http://127.0.0.1:{server.server_port}",
TMPDIR=str(tmp_path))
payload = {"session_id": "s-derive-1", "cwd": str(tmp_path), "tool_name": "Write",
"tool_input": {"file_path": str(tmp_path / "a.css"),
"content": ".log-empty {\n color: red;\n}\n"}}
for _ in range(2):
out = subprocess.run(["bash", str(HOOK)], input=json.dumps(payload),
capture_output=True, text=True, env=env)
assert out.returncode == 0, out.stderr
finally:
server.shutdown()
assert "exclude_derive" not in seen[0]
assert seen[1]["exclude_derive"] == ["dup:483a,canon:2855"]
state = tmp_path / "scribe-priorart" / "s-derive-1.derive.ids"
assert state.read_text().split() == ["dup:483a", "canon:2855", "dup:483a", "canon:2855"]
@pytest.mark.asyncio
async def test_the_write_time_divergence_check_is_named_in_band():
"""#2793: the hook named a shape at a path whose directory a canon