bvandeusenandClaude Opus 5
390846a3d5
CI & Build / TypeScript typecheck (push) Failing after 2s
CI & Build / Python lint (push) Successful in 3s
CI & Build / Plugin hooks (push) Successful in 8s
CI & Build / integration (push) Successful in 16s
CI & Build / Python tests (push) Successful in 42s
CI & Build / Build & push image (push) Has been skipped
Closes #2244. Retrieval matches on concept, and concepts are language-agnostic: asking about a TypeScript union-find scores 0.72-0.73 against a PYTHON snippet, comfortably over the 0.68 bar. That is useful — a different-language solution gives you the shape even when the code isn't reusable — but the menu line said nothing about it, so the reader either dismissed a good structural reference or pasted Python into a .ts file. Worth noting this predates the concept-query change: raw TS code already matched the Python snippet at 0.73, because the embedder reads identifiers and structure semantically rather than syntactically. The fail state has been shipping quietly; #2242 only made it an intended use rather than an accident. - knowledge._note_to_item projects `language` from the data mirror, same shape as the existing verification projection — a plain column read, no body parsing. - The semantic arm carries language through on the item it builds; it is the arm where these arise, since a snippet recorded AT the path you're editing is almost never in another language. - _prior_art_line folds it into the marker: [similar 0.72 · python]. Together with the score rather than after the title, because the two jointly are the judgement being offered. - One explanatory line is added to the menu, and only when something on it is actually tagged. Two deliberate calls: LABEL, DON'T FILTER. A stricter threshold for foreign-language hits would suppress exactly the shape-borrowing this exists for. They were never the problem; their being undisclosed was. ONLY CLAIM A MISMATCH YOU CAN ESTABLISH. _foreign_language returns "" when either side is unknown — unrecognised extension, or a snippet with no recorded language. A wrong "· python" is worse than no tag. Same-language hits stay unlabelled, so the common case keeps a clean line and the preamble stays off the menu entirely. Operator-typed language names fold through an alias table first (py/python3 → python, tsx → typescript, c++ → cpp); unrecognised names pass through lowercased, which still makes an unknown-but-equal pair compare equal. Trap found while building: _note() in the tests is a MagicMock, so `note.data` auto-created a truthy mock that would have rendered its repr into a menu line. Both test helpers now set data = None explicitly. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UaYUaouG9jjhATyuxCKrQs
863 lines
40 KiB
Python
863 lines
40 KiB
Python
"""Session-context rendering for the Scribe plugin's SessionStart hook.
|
|
|
|
The plugin's hook curls `GET /api/plugin/context` at session start and injects
|
|
the returned text as `additionalContext`, giving Scribe the same push channel
|
|
that superpowers and file-memory have. This module renders that text.
|
|
|
|
Design note — altitude: we inject rule *titles* grouped by topic (a compact
|
|
index), NOT every rule's full statement. The 48 always-on statements run well
|
|
past the 10k-char `additionalContext` cap, and the push channel's job is to make
|
|
Claude *aware* the rules exist and *reach* for them — not to dump them. Full
|
|
text stays one `get_rule(id)` / `list_always_on_rules()` call away. Titles are
|
|
mostly self-describing ("`dev` is home", "No GitHub — Fabled-Git only"), so the
|
|
index alone already steers behavior.
|
|
"""
|
|
from __future__ import annotations
|
|
|
|
import logging
|
|
import re
|
|
import time
|
|
|
|
from sqlalchemy import select
|
|
|
|
from scribe.models import async_session
|
|
from scribe.models.rulebook import RulebookTopic
|
|
from scribe.services import knowledge as knowledge_svc
|
|
from scribe.services import notes as notes_svc
|
|
from scribe.services import projects as projects_svc
|
|
from scribe.services import rulebooks as rulebooks_svc
|
|
from scribe.services import snippets as snippets_svc
|
|
from scribe.services.access import label_shared_items, owner_names_for
|
|
from scribe.services.embeddings import semantic_search_notes
|
|
from scribe.services.note_usage import record_surfaced
|
|
from scribe.services.retrieval_telemetry import record_retrieval
|
|
from scribe.services.settings import get_setting
|
|
|
|
logger = logging.getLogger(__name__)
|
|
|
|
# Defensive cap below Claude Code's 10k additionalContext limit.
|
|
_MAX_CHARS = 9000
|
|
|
|
# Max chars of a Process body to fold into the auto-surface description.
|
|
_PROC_PREVIEW_CHARS = 200
|
|
|
|
# --- Knowledge auto-inject (Path A: per-turn awareness push) -----------------
|
|
# Per-user settings (keys live in the generic settings table). The threshold is
|
|
# deliberately STRICTER than the pull-search default (embeddings
|
|
# DEFAULT_SIMILARITY_THRESHOLD = 0.45): an unsolicited per-turn inject must clear
|
|
# a higher bar than a search the agent chose to run. Defaults start conservative
|
|
# and are meant to be tuned from retrieval_logs (source='auto_inject') once data
|
|
# accrues — they're exposed in the Settings UI, no restart needed.
|
|
AUTOINJECT_ENABLED_KEY = "kb_autoinject_enabled"
|
|
AUTOINJECT_THRESHOLD_KEY = "kb_autoinject_threshold"
|
|
AUTOINJECT_TOP_K_KEY = "kb_autoinject_top_k"
|
|
|
|
AUTOINJECT_DEFAULT_ENABLED = True
|
|
AUTOINJECT_DEFAULT_THRESHOLD = 0.55
|
|
AUTOINJECT_DEFAULT_TOP_K = 3
|
|
|
|
# The write-path trigger (#2082) gets its own on/off switch, its own threshold,
|
|
# and shares only top-k. It originally shared the threshold too, on the argument
|
|
# that one "how loud may Scribe be" knob beats two that drift — and reserved the
|
|
# split for when telemetry showed the two surfaces wanted different values.
|
|
#
|
|
# #2223 is that evidence. Measured against the live instance, the semantic arm's
|
|
# scores for CODE sit far above what the same threshold means for PROSE:
|
|
# near-duplicate of a recorded helper 0.73-0.74 (true positive)
|
|
# unrelated colour math / Vue SFC / CSS 0.55-0.63 (false positive)
|
|
# `x = 1` 0.58 (false positive)
|
|
# Any two Python-shaped payloads share keywords, indentation and structure, so
|
|
# the floor for "some code" is ~0.55-0.63 — auto-inject's 0.55 lands INSIDE that
|
|
# noise band, and 6 of 8 probe payloads produced a nudge (4 of them noise). The
|
|
# margin gate can't rescue it either: _AUTOINJECT_BAND is relative to the top
|
|
# hit, so with a single hit it never engages.
|
|
#
|
|
# 0.68 clears every measured false positive with margin and still sits 0.05
|
|
# below both true positives. Auto-inject keeps 0.55 — it was tuned on prose and
|
|
# is not implicated. Tune from retrieval_logs (source='write_path') + note_usage
|
|
# pull-through (#2085) once a real corpus accrues; a cross-encoder rerank
|
|
# (#1038) would subsume this bump.
|
|
WRITEPATH_ENABLED_KEY = "kb_writepath_enabled"
|
|
WRITEPATH_THRESHOLD_KEY = "kb_writepath_threshold"
|
|
WRITEPATH_DEFAULT_ENABLED = True
|
|
WRITEPATH_DEFAULT_THRESHOLD = 0.68
|
|
|
|
# Minimum SUBSTANCE (non-whitespace chars) a payload must carry before the
|
|
# semantic arm will run at all — the cheap half of the operator's #89 idea
|
|
# ("a sliding scale between number of characters and semantic threshold").
|
|
#
|
|
# Deliberately NOT a settings knob and deliberately conservative. Its job is
|
|
# only to drop payloads too small to carry meaning, where an embedding is noise
|
|
# rather than signal: `x = 1`, a renamed variable, a changed string literal —
|
|
# which is what most single-line Edits look like, and the majority of Edits are
|
|
# single-line. 48 sits below the smallest plausible reusable helper (a one-line
|
|
# `def` with a body runs ~60), so it errs toward keeping recall and leaves
|
|
# precision to the threshold above, which is where the measured separation is.
|
|
# The full length↔threshold CURVE is still open in #89 — the operator flagged it
|
|
# as wanting a brainstorm, so this stays a flat floor rather than an invented
|
|
# scale. It also saves a pointless embedding round-trip on trivial edits.
|
|
WRITEPATH_MIN_CODE_CHARS = 48
|
|
|
|
# --- concept extraction for the semantic arm's query (#2242) ------------------
|
|
# A snippet's embedded text is f"{title}\n{body}", and for a snippet that body is
|
|
# composed markdown: **When to use:**, **Signature:**, **Location:**, then the
|
|
# fenced code. So `when_to_use` — the description of what the thing is FOR —
|
|
# appears twice in the vector, and the document is prose-forward.
|
|
#
|
|
# The arm used to query it with raw code and no prose at all. Measured on the
|
|
# deployed instance against snippet #2222, same corpus:
|
|
# query built from score best unrelated separation
|
|
# raw code body 0.743 0.630 0.11
|
|
# name + docstring 0.823 0.602 0.22
|
|
# hand-written concept prose 0.835 0.583 0.25
|
|
# A 12-word description beats a near-verbatim reimplementation of the function,
|
|
# and code-as-query RAISES the noise floor. It is also the cleanest explanation
|
|
# for the fragment miss recorded on #2223: a short code excerpt has almost no
|
|
# prose to match against a document that is mostly prose.
|
|
#
|
|
# So we send the concept instead — and shape it like a snippet's own title,
|
|
# "{name} — {when_to_use}", because that is the form the 0.823 measurement used.
|
|
# Undocumented code yields little, and a Vue SFC or a config file yields nothing;
|
|
# those fall back to the raw payload and behave exactly as before. This raises
|
|
# the ceiling for documented helpers rather than fixing every case.
|
|
|
|
# Declaration forms, one pattern per shape, every pattern exposing (name, params)
|
|
# so composition doesn't have to care which matched. Deliberately regex and not a
|
|
# real parser: this runs on a PreToolUse hook's critical path, the payload is
|
|
# frequently a FRAGMENT that no parser would accept (an Edit's new_string is
|
|
# rarely a valid module), and a miss costs only a fallback to today's behaviour.
|
|
_CONCEPT_DECL_PATTERNS = (
|
|
# python: def / async def, and class with optional bases
|
|
re.compile(r"^[ \t]*(?:async[ \t]+)?def[ \t]+([A-Za-z_]\w*)[ \t]*(\([^)]*\))", re.M),
|
|
re.compile(r"^[ \t]*class[ \t]+([A-Za-z_]\w*)[ \t]*(\([^)]*\))?", re.M),
|
|
# js/ts: function decl, and the const-arrow form that dominates modern code
|
|
re.compile(r"^[ \t]*(?:export[ \t]+)?(?:default[ \t]+)?(?:async[ \t]+)?function[ \t]+([A-Za-z_$][\w$]*)[ \t]*(\([^)]*\))", re.M),
|
|
re.compile(r"^[ \t]*(?:export[ \t]+)?(?:const|let|var)[ \t]+([A-Za-z_$][\w$]*)[ \t]*=[ \t]*(?:async[ \t]*)?(\([^)]*\))[ \t]*=>", re.M),
|
|
# rust / go
|
|
re.compile(r"^[ \t]*(?:pub[ \t]+)?fn[ \t]+([A-Za-z_]\w*)[ \t]*(\([^)]*\))", re.M),
|
|
re.compile(r"^[ \t]*func[ \t]+(?:\([^)]*\)[ \t]*)?([A-Za-z_]\w*)[ \t]*(\([^)]*\))", re.M),
|
|
# posix shell: name() {
|
|
re.compile(r"^[ \t]*([A-Za-z_]\w*)[ \t]*(\(\))[ \t]*\{", re.M),
|
|
)
|
|
|
|
# Doc forms, tried in order. The Python pattern also matches a triple-quoted
|
|
# string that isn't a docstring — accepted: a stray literal is still text about
|
|
# what the code does far more often than it's misleading, and the cost is a
|
|
# slightly worse query rather than a wrong answer.
|
|
_CONCEPT_PY_DOC = re.compile(r'("""|\'\'\')(.*?)\1', re.S)
|
|
_CONCEPT_JSDOC = re.compile(r"/\*\*(.*?)\*/", re.S)
|
|
_CONCEPT_LEADING_COMMENT = re.compile(r"\A(?:[ \t]*(?://|#)[^\n]*\n?)+")
|
|
# A shebang is a comment to the regex above but says nothing about what the code
|
|
# DOES, and it would otherwise open the doc with "/usr/bin/env bash".
|
|
_CONCEPT_SHEBANG = re.compile(r"\A#![^\n]*\n")
|
|
_CONCEPT_COMMENT_MARKER = re.compile(r"^[ \t]*(?://+|#+!?)[ \t]?", re.M)
|
|
_CONCEPT_JSDOC_STAR = re.compile(r"^[ \t]*\*+[ \t]?", re.M)
|
|
|
|
# Cap the doc so a long module docstring can't drown out the declaration, and cap
|
|
# declarations so a 40-function Write doesn't turn into a wall of signatures.
|
|
_CONCEPT_MAX_DOC_CHARS = 400
|
|
_CONCEPT_MAX_DECLS = 4
|
|
# Below this much substance the "concept" is too thin to be a better query than
|
|
# the code itself (e.g. all we found was `f()`), so we keep the raw payload.
|
|
_CONCEPT_MIN_CHARS = 16
|
|
|
|
# Margin gate: drop any hit more than this far below the top hit's score, so a
|
|
# single strong match doesn't drag in a wall of barely-passing neighbours.
|
|
_AUTOINJECT_BAND = 0.10
|
|
# Hard ceiling on top-k regardless of the user's setting — this is an
|
|
# awareness menu (titles only), never a content dump.
|
|
_AUTOINJECT_MAX_TOP_K = 10
|
|
|
|
|
|
def _slugify(text: str) -> str:
|
|
"""kebab-case slug for a skill directory name (a-z0-9 + single hyphens)."""
|
|
s = re.sub(r"[^a-z0-9]+", "-", (text or "").lower()).strip("-")
|
|
return s or "process"
|
|
|
|
|
|
async def build_process_manifest(user_id: int) -> dict:
|
|
"""List the user's stored Processes as auto-surfacing skill-stub specs.
|
|
|
|
The plugin's sync script (scribe_sync_processes.sh) writes one
|
|
~/.claude/skills/scribe-proc-<slug>/SKILL.md per entry — `description` is the
|
|
auto-surface trigger, and the stub body calls get_process(name) for the live
|
|
procedure (single source of truth in the DB). Reuses the list_processes query
|
|
(note_type='process'). Instance-agnostic: derived from whatever Processes the
|
|
calling install owns, no operator-specific coupling.
|
|
|
|
SCOPE: this is the most consequential passive surface Scribe has — every
|
|
entry becomes a skill file on the operator's machine that auto-surfaces and
|
|
is followed as written. It therefore uses the BROWSE scope (via the
|
|
no-query knowledge list): a Process shared directly with the operator is
|
|
never installed here, only one they own or reach through a shared project
|
|
(decision note 2094). Project-shared entries are labelled with their owner so
|
|
the stub can't pass off someone else's procedure as the operator's own.
|
|
|
|
Returns {"processes": [{id, name, slug, description, shared?, owner?}],
|
|
"total": int}. Slugs are unique within the result (collision gets -<id>).
|
|
"""
|
|
items, _ = await knowledge_svc.query_knowledge(
|
|
user_id=user_id, note_type="process", tags=[], sort="modified",
|
|
q=None, limit=100, offset=0,
|
|
)
|
|
items = await label_shared_items(user_id, items)
|
|
procs: list[dict] = []
|
|
seen: set[str] = set()
|
|
for it in items:
|
|
title = (it.get("title") or "").strip()
|
|
if not title:
|
|
continue
|
|
slug = _slugify(title)
|
|
if slug in seen:
|
|
slug = f"{slug}-{it['id']}"
|
|
seen.add(slug)
|
|
|
|
preview = " ".join((it.get("snippet") or "").split())
|
|
if len(preview) > _PROC_PREVIEW_CHARS:
|
|
preview = preview[:_PROC_PREVIEW_CHARS].rstrip() + "…"
|
|
if it.get("shared"):
|
|
owner = it.get("owner") or "another user"
|
|
description = (
|
|
f'A shared Scribe process "{title}", authored by {owner} — NOT the'
|
|
f" operator's own."
|
|
+ (f" {preview}" if preview else "")
|
|
+ f' Use when {title}-type work is requested, or when asked to run'
|
|
f' the "{title}" process — but summarise it and get the operator\'s'
|
|
f" go-ahead before following it, since it reflects {owner}'s"
|
|
f" judgement rather than theirs."
|
|
)
|
|
else:
|
|
description = (
|
|
f'Run the operator\'s saved Scribe process "{title}".'
|
|
+ (f" {preview}" if preview else "")
|
|
+ f' Use when {title}-type work is requested, or when asked to run'
|
|
f' the "{title}" process.'
|
|
)
|
|
entry = {
|
|
"id": it["id"], "name": title, "slug": slug,
|
|
"description": description,
|
|
}
|
|
if it.get("shared"):
|
|
entry["shared"] = True
|
|
entry["owner"] = it.get("owner")
|
|
procs.append(entry)
|
|
return {"processes": procs, "total": len(procs)}
|
|
|
|
|
|
async def get_autoinject_config(user_id: int) -> dict:
|
|
"""Resolve a user's auto-inject settings, falling back to the defaults.
|
|
|
|
Returns {"enabled": bool, "threshold": float, "top_k": int}, clamped to
|
|
sane ranges (threshold to [0,1]; top_k to [1, _AUTOINJECT_MAX_TOP_K]).
|
|
"""
|
|
enabled_raw = await get_setting(
|
|
user_id, AUTOINJECT_ENABLED_KEY,
|
|
"true" if AUTOINJECT_DEFAULT_ENABLED else "false",
|
|
)
|
|
enabled = enabled_raw.strip().lower() in ("true", "1", "yes", "on")
|
|
|
|
try:
|
|
threshold = float(await get_setting(
|
|
user_id, AUTOINJECT_THRESHOLD_KEY, str(AUTOINJECT_DEFAULT_THRESHOLD)))
|
|
except (TypeError, ValueError):
|
|
threshold = AUTOINJECT_DEFAULT_THRESHOLD
|
|
threshold = min(1.0, max(0.0, threshold))
|
|
|
|
try:
|
|
top_k = int(float(await get_setting(
|
|
user_id, AUTOINJECT_TOP_K_KEY, str(AUTOINJECT_DEFAULT_TOP_K))))
|
|
except (TypeError, ValueError):
|
|
top_k = AUTOINJECT_DEFAULT_TOP_K
|
|
top_k = min(_AUTOINJECT_MAX_TOP_K, max(1, top_k))
|
|
|
|
return {"enabled": enabled, "threshold": threshold, "top_k": top_k}
|
|
|
|
|
|
def _record_kind(note) -> str:
|
|
"""The one-word kind marker for an injected menu line.
|
|
|
|
The menu is drawn from every record that carries an embedding, so a snippet,
|
|
a stored process, an issue and a stray dev-log all arrive looking identical.
|
|
Recorded prior art only stands out if the line says what it is — and the kind
|
|
is also what tells the reader which tool opens it.
|
|
|
|
Task-ness wins over `note_type` because it's the more useful distinction at a
|
|
glance: "there's an open issue about this" beats "there's a note about this".
|
|
"""
|
|
if note.is_task:
|
|
return "issue" if note.task_kind == "issue" else "task"
|
|
return note.note_type or "note"
|
|
|
|
|
|
async def build_autoinject_hint(
|
|
user_id: int,
|
|
query: str,
|
|
project_id: int = 0,
|
|
exclude_ids: list[int] | None = None,
|
|
) -> dict:
|
|
"""Title-first awareness hint for the plugin's UserPromptSubmit hook.
|
|
|
|
The four anti-bloat gates (see the module + milestone-93 design):
|
|
1. high-confidence threshold (stricter than pull) — set per-user;
|
|
2. margin gate — keep only hits within _AUTOINJECT_BAND of the top score;
|
|
3. session dedup — caller passes already-injected ids as `exclude_ids`;
|
|
4. title-first payload — id + kind + title + score only, never bodies.
|
|
Disabled, blank-query, or nothing-clears-the-gates all return empty context,
|
|
so most turns inject nothing.
|
|
|
|
Returns {"context": str, "note_ids": list[int], "config": dict}. Every
|
|
retrieval (even empty) is logged to retrieval_logs as source='auto_inject'
|
|
so the threshold can be tuned from data.
|
|
"""
|
|
cfg = await get_autoinject_config(user_id)
|
|
empty = {"context": "", "note_ids": [], "config": cfg}
|
|
q = (query or "").strip()
|
|
if not cfg["enabled"] or not q:
|
|
return empty
|
|
|
|
t0 = time.perf_counter()
|
|
hits = await semantic_search_notes(
|
|
user_id, q,
|
|
limit=cfg["top_k"],
|
|
threshold=cfg["threshold"],
|
|
project_id=(project_id or None),
|
|
exclude_ids=set(exclude_ids or []),
|
|
# Injection is the one retrieval nobody asked for, so it takes the BROWSE
|
|
# scope: never a record shared one-to-one with the operator. What can
|
|
# still appear is a collaborator's note inside a shared project — legible
|
|
# only because the line below names its owner.
|
|
scope="browse",
|
|
)
|
|
record_retrieval(
|
|
user_id=user_id, source="auto_inject", query=q,
|
|
threshold=cfg["threshold"], limit=cfg["top_k"],
|
|
project_id=(project_id or None), is_task=None, results=hits,
|
|
duration_ms=(time.perf_counter() - t0) * 1000.0,
|
|
)
|
|
if not hits:
|
|
return empty
|
|
|
|
# Margin gate: keep only hits close to the strongest one.
|
|
top_score = hits[0][0]
|
|
kept = [(s, n) for s, n in hits if s >= top_score - _AUTOINJECT_BAND]
|
|
|
|
# A collaborator's note can reach this menu via a shared project, and the
|
|
# operator never asked for it — so say whose it is. Unattributed, it reads as
|
|
# something they wrote and settled.
|
|
owners = await owner_names_for({
|
|
int(n.user_id) for _s, n in kept if n.user_id != user_id
|
|
})
|
|
|
|
# "records", not "notes" — the menu can hold snippets, processes and tasks
|
|
# too, and the kind marker on each line is only legible if the header doesn't
|
|
# already claim they're all one thing.
|
|
lines = [
|
|
"> Possibly relevant from your Scribe records — open any in full with "
|
|
"`get_note(id)`, or `get_snippet` / `get_process` for those kinds "
|
|
"(titles only; injected once per session):",
|
|
]
|
|
note_ids: list[int] = []
|
|
for score, note in kept:
|
|
note_ids.append(int(note.id))
|
|
title = (note.title or "(untitled)").replace("\n", " ").strip()
|
|
line = f"> - #{note.id} [{_record_kind(note)}] \"{title}\" ({score:.2f})"
|
|
if note.user_id != user_id:
|
|
who = owners.get(int(note.user_id)) or "another user"
|
|
line += f" — shared by {who}, treat as a suggestion"
|
|
lines.append(line)
|
|
|
|
# Records what SURVIVED the margin gate, not what the ranker returned — the
|
|
# menu the agent actually saw. retrieval_logs already holds the full
|
|
# candidate set for threshold tuning; conflating the two would make
|
|
# "surfaced" mean two different things depending on the surface (#2085).
|
|
record_surfaced(user_id=user_id, note_ids=note_ids, source="auto_inject")
|
|
|
|
return {"context": "\n".join(lines), "note_ids": note_ids, "config": cfg}
|
|
|
|
|
|
# --- Write-path trigger (#2082): prior art at the moment code is written ------
|
|
# Auto-inject above fires on the operator's prompt. The moment reuse is actually
|
|
# lost is later — when the AGENT decides mid-task to write a helper — and nothing
|
|
# fired there. This is that trigger: the plugin's PreToolUse hook on Write/Edit
|
|
# asks what prior art is already recorded for the file being written.
|
|
#
|
|
# Two arms, deliberately different in kind:
|
|
# - BY PLACE — a snippet recorded at this path (or in its directory) is prior
|
|
# art by definition, not by resemblance, so it isn't scored or thresholded.
|
|
# This is what the reverse lookup (#2083) was built to answer.
|
|
# - BY MEANING — semantic search over snippets only, using the code about to be
|
|
# written, under the same gates as auto-inject.
|
|
# Place beats meaning in the menu because "there is already a canonical helper in
|
|
# this exact file" is a stronger claim than "this resembles something".
|
|
|
|
|
|
def _prior_art_line(item: dict, marker: str, owner: str | None, foreign_lang: str = "") -> str:
|
|
"""One menu line: `- #12 [here] "title"`, attributed when it isn't yours.
|
|
|
|
A foreign language is folded into the marker (`[similar 0.72 · python]`)
|
|
rather than appended after the title, so the reader sees it while still
|
|
reading the score — the two together are the judgement being offered.
|
|
"""
|
|
title = (item.get("title") or "(untitled)").replace("\n", " ").strip()
|
|
mark = f"{marker} · {foreign_lang}" if foreign_lang else marker
|
|
line = f"> - #{item['id']} [{mark}] \"{title}\""
|
|
if owner:
|
|
line += f" — shared by {owner}, treat as a suggestion"
|
|
return line
|
|
|
|
|
|
# --- cross-language prior art (#2244) ----------------------------------------
|
|
# Retrieval is concept-shaped now, and concepts are language-agnostic: a query
|
|
# about a TypeScript union-find matches a PYTHON snippet at 0.72-0.73, comfortably
|
|
# over the bar. That is a feature — the operator's framing is "borrow the shape of
|
|
# the solution even when the code isn't directly reusable" — but only if the line
|
|
# SAYS so. An unlabelled Python hit offered while writing TypeScript either gets
|
|
# dismissed as irrelevant or, worse, pasted into a .ts file. Measured note: this
|
|
# cross-language matching predates concept queries; it was always happening, just
|
|
# never disclosed.
|
|
#
|
|
# Deliberately NOT gated behind a stricter threshold for foreign-language hits: a
|
|
# higher bar would suppress exactly the shape-borrowing this is for. Label, don't
|
|
# filter.
|
|
_LANG_BY_EXT = {
|
|
"py": "python", "pyi": "python",
|
|
"ts": "typescript", "tsx": "typescript", "mts": "typescript", "cts": "typescript",
|
|
"js": "javascript", "jsx": "javascript", "mjs": "javascript", "cjs": "javascript",
|
|
"vue": "vue", "svelte": "svelte",
|
|
"go": "go", "rs": "rust", "rb": "ruby", "php": "php",
|
|
"java": "java", "kt": "kotlin", "kts": "kotlin", "scala": "scala",
|
|
"c": "c", "h": "c", "cc": "cpp", "cpp": "cpp", "cxx": "cpp", "hpp": "cpp",
|
|
"cs": "csharp", "swift": "swift", "m": "objectivec", "mm": "objectivec",
|
|
"sh": "shell", "bash": "shell", "zsh": "shell", "fish": "shell",
|
|
"sql": "sql", "css": "css", "scss": "scss", "less": "less",
|
|
"html": "html", "htm": "html", "yml": "yaml", "yaml": "yaml",
|
|
"toml": "toml", "ini": "ini", "dockerfile": "dockerfile",
|
|
"ex": "elixir", "exs": "elixir", "erl": "erlang", "hs": "haskell",
|
|
"lua": "lua", "pl": "perl", "r": "r", "dart": "dart", "zig": "zig",
|
|
}
|
|
|
|
# `language` on a snippet is operator-typed free text, so fold the spellings that
|
|
# mean the same thing before comparing. Anything unrecognised passes through
|
|
# lowercased — an unknown-but-equal pair still compares equal, which is the only
|
|
# thing this needs to get right.
|
|
_LANG_ALIASES = {
|
|
"py": "python", "python3": "python",
|
|
"ts": "typescript", "tsx": "typescript",
|
|
"js": "javascript", "jsx": "javascript", "node": "javascript",
|
|
"sh": "shell", "bash": "shell", "zsh": "shell", "shell-script": "shell",
|
|
"c++": "cpp", "cplusplus": "cpp", "c#": "csharp", "objective-c": "objectivec",
|
|
"golang": "go", "rs": "rust", "rb": "ruby", "yml": "yaml",
|
|
"postgres": "sql", "postgresql": "sql", "psql": "sql",
|
|
"vuejs": "vue", "vue3": "vue",
|
|
}
|
|
|
|
|
|
def _canonical_language(name: str) -> str:
|
|
"""Fold a free-text language name to a comparable token ("" if absent)."""
|
|
token = (name or "").strip().lower()
|
|
return _LANG_ALIASES.get(token, token)
|
|
|
|
|
|
def _language_for_path(path: str) -> str:
|
|
"""The language implied by a file path's extension ("" when unknown)."""
|
|
tail = (path or "").rsplit("/", 1)[-1].lower()
|
|
if tail.startswith("dockerfile"):
|
|
return "dockerfile"
|
|
if "." not in tail:
|
|
return ""
|
|
return _LANG_BY_EXT.get(tail.rsplit(".", 1)[-1], "")
|
|
|
|
|
|
def _foreign_language(item: dict, target: str) -> str:
|
|
"""The item's language when it DIFFERS from the target file's, else "".
|
|
|
|
Returns "" whenever either side is unknown: we can only claim a mismatch we
|
|
can actually establish, and a wrong "· python" tag is worse than no tag.
|
|
Same-language hits stay unlabelled so the common case keeps a clean line.
|
|
"""
|
|
if not target:
|
|
return ""
|
|
theirs = _canonical_language(item.get("language") or "")
|
|
if not theirs or theirs == target:
|
|
return ""
|
|
return theirs
|
|
|
|
|
|
def _concept_doc(code: str) -> str:
|
|
"""The first doc-ish prose in `code`: docstring, else JSDoc, else leading comments."""
|
|
m = _CONCEPT_PY_DOC.search(code)
|
|
if m:
|
|
return _collapse(m.group(2))
|
|
|
|
m = _CONCEPT_JSDOC.search(code)
|
|
if m:
|
|
return _collapse(_CONCEPT_JSDOC_STAR.sub("", m.group(1)))
|
|
|
|
# Only a comment block at the very TOP counts. A comment further down is
|
|
# usually about one line of the implementation, not about the whole thing.
|
|
m = _CONCEPT_LEADING_COMMENT.match(_CONCEPT_SHEBANG.sub("", code))
|
|
if m:
|
|
return _collapse(_CONCEPT_COMMENT_MARKER.sub("", m.group(0)))
|
|
|
|
return ""
|
|
|
|
|
|
def _collapse(text: str) -> str:
|
|
"""One line, single-spaced, length-capped — embedder input, not display text."""
|
|
return " ".join((text or "").split())[:_CONCEPT_MAX_DOC_CHARS].strip()
|
|
|
|
|
|
def concept_query(code: str) -> str:
|
|
"""Rewrite a write payload as a CONCEPT query, or "" to keep the raw payload.
|
|
|
|
Returns something shaped like a snippet's own title — "name(params) — what it
|
|
does" — because that is the form that measured best against the prose-forward
|
|
snippet documents (#2242; see the table at _CONCEPT_DECL_PATTERNS).
|
|
|
|
Returns "" rather than raising or guessing whenever there's nothing worth
|
|
sending: no declarations and no doc, or a result too thin to beat the code it
|
|
would replace. The caller treats "" as "use the payload as-is", so every
|
|
unhandled language degrades to exactly the previous behaviour.
|
|
"""
|
|
if not code or not code.strip():
|
|
return ""
|
|
|
|
decls: list[str] = []
|
|
for pattern in _CONCEPT_DECL_PATTERNS:
|
|
for match in pattern.finditer(code):
|
|
name, params = match.group(1), match.group(2) or ""
|
|
label = f"{name}{params}".strip()
|
|
if label and label not in decls:
|
|
decls.append(label)
|
|
if len(decls) >= _CONCEPT_MAX_DECLS:
|
|
break
|
|
if len(decls) >= _CONCEPT_MAX_DECLS:
|
|
break
|
|
|
|
doc = _concept_doc(code)
|
|
|
|
# NO DOC, NO REWRITE. An identifier alone is not a concept, and it measured
|
|
# WORSE than the code it would replace: `collapse_into_clusters(edges)` scored
|
|
# 0.671 against #2222 where the full code body scored 0.743. Separation from
|
|
# the noise floor is identical (0.113 either way), but the absolute value
|
|
# drops below the 0.68 bar — so preferring a bare name would convert a
|
|
# comfortable hit into a miss. Undocumented code keeps the raw payload.
|
|
if not doc:
|
|
return ""
|
|
|
|
head = ", ".join(decls)
|
|
query = f"{head} — {doc}" if head else doc
|
|
|
|
# Guard against a doc so terse it says nothing ("# TODO", "/** x */").
|
|
if len("".join(query.split())) < _CONCEPT_MIN_CHARS:
|
|
return ""
|
|
return query
|
|
|
|
|
|
async def get_writepath_config(user_id: int) -> dict:
|
|
"""Write-path trigger settings: its own `enabled` and `threshold`, auto-inject's top_k.
|
|
|
|
The threshold OVERRIDES the inherited auto-inject value — code embeddings
|
|
have a much higher similarity floor than prose, so the two surfaces need
|
|
different bars. See WRITEPATH_DEFAULT_THRESHOLD for the measurements (#2223).
|
|
top_k is still shared: "how many titles at once" means the same thing on
|
|
both surfaces, and nothing suggests they want different ceilings.
|
|
"""
|
|
cfg = await get_autoinject_config(user_id)
|
|
enabled_raw = await get_setting(
|
|
user_id, WRITEPATH_ENABLED_KEY,
|
|
"true" if WRITEPATH_DEFAULT_ENABLED else "false",
|
|
)
|
|
|
|
try:
|
|
threshold = float(await get_setting(
|
|
user_id, WRITEPATH_THRESHOLD_KEY, str(WRITEPATH_DEFAULT_THRESHOLD)))
|
|
except (TypeError, ValueError):
|
|
threshold = WRITEPATH_DEFAULT_THRESHOLD
|
|
threshold = min(1.0, max(0.0, threshold))
|
|
|
|
return {
|
|
**cfg,
|
|
"enabled": enabled_raw.strip().lower() in ("true", "1", "yes", "on"),
|
|
"threshold": threshold,
|
|
}
|
|
|
|
|
|
async def build_write_path_hint(
|
|
user_id: int,
|
|
path: str,
|
|
code: str = "",
|
|
project_id: int = 0,
|
|
exclude_ids: list[int] | None = None,
|
|
) -> dict:
|
|
"""Prior-art hint for the plugin's PreToolUse hook on Write/Edit.
|
|
|
|
`path` is the file about to be written, REPO-RELATIVE — matching the
|
|
convention snippet locations are recorded in. `code` is what's about to be
|
|
written, used only as the semantic query.
|
|
|
|
Carries auto-inject's anti-bloat gates (margin, session dedup via
|
|
`exclude_ids`, titles-never-bodies) plus the shared top-k cap across BOTH
|
|
arms — so a file with a lot of recorded history can't turn one edit into a
|
|
wall of text. Two gates are its OWN, because code is not prose: a stricter
|
|
similarity threshold, and a minimum-substance floor on `code` below which the
|
|
semantic arm doesn't run at all (#2223 — see WRITEPATH_DEFAULT_THRESHOLD and
|
|
WRITEPATH_MIN_CODE_CHARS). Returns empty context when disabled, when there's
|
|
no path, or when nothing is recorded — which is the common case, and the point.
|
|
|
|
Note the repo↔project mapping is deliberately one-way: the hook sends a git
|
|
remote, which the ROUTE resolves to `project_id` through the repo bindings.
|
|
It is never used as the location `repo` filter — a snippet's `repo` is a
|
|
free-text label the operator typed ("Scribe"), not a remote URL, and matching
|
|
one against the other would silently return nothing.
|
|
|
|
Returns {"context": str, "note_ids": list[int], "config": dict}. The semantic
|
|
arm is logged to retrieval_logs as source='write_path' — its own source, so
|
|
its precision is tunable separately from auto-inject's.
|
|
|
|
Location hits still carry no score and so stay out of retrieval_logs, whose
|
|
score distribution they would corrupt. What closed the gap (#2085) is that
|
|
un-scored surfacing now has its own home: BOTH arms emit note_usage_events,
|
|
tagged 'write_path_place' vs 'write_path_semantic', so the place arm is
|
|
finally measurable — and the two arms' pull-through rates are comparable,
|
|
which is the number that says whether place really does beat meaning here.
|
|
"""
|
|
cfg = await get_writepath_config(user_id)
|
|
empty = {"context": "", "note_ids": [], "config": cfg}
|
|
path = (path or "").strip()
|
|
if not cfg["enabled"] or not path:
|
|
return empty
|
|
|
|
top_k = cfg["top_k"]
|
|
excluded = set(exclude_ids or [])
|
|
scope_project = project_id or None
|
|
|
|
# --- arm 1: by place ---
|
|
here: list[dict] = []
|
|
nearby: list[dict] = []
|
|
try:
|
|
here, _ = await snippets_svc.list_snippets(
|
|
user_id, path=path, limit=top_k, project_id=scope_project,
|
|
)
|
|
directory = path.rsplit("/", 1)[0] if "/" in path else ""
|
|
if directory and len(here) < top_k:
|
|
nearby, _ = await snippets_svc.list_snippets(
|
|
user_id, path=directory, limit=top_k, project_id=scope_project,
|
|
)
|
|
except Exception:
|
|
logger.warning("Write-path location lookup failed", exc_info=True)
|
|
|
|
seen: set[int] = set(excluded)
|
|
placed: list[tuple[str, dict]] = []
|
|
for marker, items in (("here", here), ("nearby", nearby)):
|
|
for item in items:
|
|
nid = int(item["id"])
|
|
if nid in seen:
|
|
continue
|
|
seen.add(nid)
|
|
placed.append((marker, item))
|
|
|
|
# --- arm 2: by meaning ---
|
|
scored: list[tuple[str, dict]] = []
|
|
remaining = top_k - len(placed)
|
|
query = (code or "").strip()
|
|
# Drop payloads too small to carry meaning before spending an embedding on
|
|
# them — a one-line Edit is not a helper being rewritten, and its embedding
|
|
# scores off the corpus floor rather than off any real resemblance (#2223).
|
|
# Whitespace doesn't count: code is indentation-heavy, so raw length would
|
|
# let a deeply-nested one-liner through on padding alone.
|
|
if len("".join(query.split())) < WRITEPATH_MIN_CODE_CHARS:
|
|
query = ""
|
|
# ORDER MATTERS: the floor above judges the RAW payload, this rewrites it.
|
|
# Snippet documents are prose-forward, so a concept query out-scores the code
|
|
# itself by a wide margin (#2242). The rewritten query is allowed to be
|
|
# short — "slugify(t) — turn text into a url slug" is a fine query at 38
|
|
# chars, and it only exists because the raw payload already cleared the
|
|
# floor. Applying the floor after this would throw away the best queries.
|
|
if query:
|
|
query = concept_query(query) or query
|
|
if remaining > 0 and query:
|
|
t0 = time.perf_counter()
|
|
hits = await semantic_search_notes(
|
|
user_id, query,
|
|
limit=remaining,
|
|
threshold=cfg["threshold"],
|
|
project_id=scope_project,
|
|
exclude_ids=seen,
|
|
note_type="snippet",
|
|
# Same reasoning as auto-inject: nobody asked for this, so it takes
|
|
# the browse scope and never surfaces a one-to-one direct share.
|
|
scope="browse",
|
|
)
|
|
record_retrieval(
|
|
user_id=user_id, source="write_path", query=query,
|
|
threshold=cfg["threshold"], limit=remaining,
|
|
project_id=scope_project, is_task=False, results=hits,
|
|
duration_ms=(time.perf_counter() - t0) * 1000.0,
|
|
)
|
|
if hits:
|
|
top_score = hits[0][0]
|
|
for score, note in hits:
|
|
if score < top_score - _AUTOINJECT_BAND:
|
|
continue
|
|
scored.append((
|
|
f"similar {score:.2f}",
|
|
{
|
|
"id": int(note.id), "title": note.title, "user_id": note.user_id,
|
|
# Carried so the line can disclose a cross-language hit
|
|
# (#2244). The semantic arm is where these actually arise —
|
|
# a snippet recorded at the path you're editing is almost
|
|
# never in another language, but a concept match easily is.
|
|
"language": (note.data or {}).get("language") if note.data else None,
|
|
},
|
|
))
|
|
|
|
menu = (placed + scored)[:top_k]
|
|
if not menu:
|
|
return empty
|
|
|
|
owners = await owner_names_for({
|
|
int(it["user_id"]) for _m, it in menu
|
|
if it.get("user_id") is not None and int(it["user_id"]) != user_id
|
|
})
|
|
|
|
target_lang = _language_for_path(path)
|
|
rendered: list[tuple[dict, str, str | None, str]] = []
|
|
for marker, item in menu:
|
|
owner_id = item.get("user_id")
|
|
owner = None
|
|
if owner_id is not None and int(owner_id) != user_id:
|
|
owner = owners.get(int(owner_id)) or "another user"
|
|
rendered.append((item, marker, owner, _foreign_language(item, target_lang)))
|
|
|
|
lines = [
|
|
f"> Prior art already recorded in Scribe for `{path}` — open one with "
|
|
"`get_snippet(id)` and reuse it rather than writing a fresh one-off "
|
|
"(titles only; shown once per session):",
|
|
]
|
|
# Say what a language tag MEANS, and only when one is actually on the menu.
|
|
# Without this the reader has to infer why "· python" is attached to a hit on
|
|
# a .ts file, and the two ways of guessing wrong are both bad: dismiss it as
|
|
# irrelevant, or paste Python into TypeScript. Retrieval matches on concept,
|
|
# so these are genuinely useful — as the SHAPE of a solution, not as code.
|
|
if any(lang for _i, _m, _o, lang in rendered):
|
|
lines.append(
|
|
"> A tagged language means that snippet is in a DIFFERENT language "
|
|
"than this file — it matched on what it does, so treat it as the "
|
|
"shape of a solution to adapt, not code to copy."
|
|
)
|
|
|
|
note_ids: list[int] = []
|
|
for item, marker, owner, foreign_lang in rendered:
|
|
note_ids.append(int(item["id"]))
|
|
lines.append(_prior_art_line(item, marker, owner, foreign_lang))
|
|
|
|
# Split by arm, which is the whole reason this table exists. The place arm
|
|
# carries no score and so has no home in retrieval_logs; before #2085 a
|
|
# snippet surfaced BY PLACE left no trace anywhere, making the arm that
|
|
# fires on the strongest possible claim ("there is already a canonical
|
|
# helper in this exact file") the one arm nobody could measure.
|
|
by_arm: dict[str, list[int]] = {}
|
|
for marker, item in menu:
|
|
arm = "write_path_place" if marker in ("here", "nearby") else "write_path_semantic"
|
|
by_arm.setdefault(arm, []).append(int(item["id"]))
|
|
for arm, ids in by_arm.items():
|
|
record_surfaced(user_id=user_id, note_ids=ids, source=arm)
|
|
|
|
return {"context": "\n".join(lines), "note_ids": note_ids, "config": cfg}
|
|
|
|
|
|
async def _topic_titles(topic_ids: set[int]) -> dict[int, str]:
|
|
"""Map topic_id -> title for the given ids (live topics only)."""
|
|
if not topic_ids:
|
|
return {}
|
|
async with async_session() as session:
|
|
rows = await session.execute(
|
|
select(RulebookTopic.id, RulebookTopic.title).where(
|
|
RulebookTopic.id.in_(topic_ids),
|
|
RulebookTopic.deleted_at.is_(None),
|
|
)
|
|
)
|
|
return {tid: title for tid, title in rows.all()}
|
|
|
|
|
|
async def build_session_context(
|
|
user_id: int, project_id: int = 0, unbound_repo: str = ""
|
|
) -> dict:
|
|
"""Render the SessionStart context for a user, optionally project-scoped.
|
|
|
|
Args:
|
|
user_id: the operator.
|
|
project_id: the resolved active project (0 = none). The endpoint
|
|
resolves this from the working repo's remote, not from config.
|
|
unbound_repo: when the hook sent a repo remote that maps to no project,
|
|
its normalized key — triggers a one-line "bind this repo" hint so
|
|
the binding is self-healing.
|
|
|
|
Returns {"context": str, "rule_count": int, "project": dict | None}.
|
|
`context` is markdown ready to drop into `additionalContext`; it is capped
|
|
at _MAX_CHARS with an explicit truncation note so the hook can pass it
|
|
through verbatim.
|
|
"""
|
|
rules = await rulebooks_svc.list_always_on_rules(user_id)
|
|
topic_map = await _topic_titles({r.topic_id for r in rules if r.topic_id})
|
|
|
|
lines: list[str] = [
|
|
"# Scribe — standing session context (auto-injected by the Scribe plugin)",
|
|
"",
|
|
"You are working with Scribe, the operator's self-hosted second brain. "
|
|
"The always-on rules below are BINDING this session. Titles only — full "
|
|
"text via `list_always_on_rules()` or `get_rule(id)`.",
|
|
"",
|
|
"## Always-on rules (by topic)",
|
|
]
|
|
|
|
# rules already arrive ordered by rulebook/topic/order, so grouping by
|
|
# consecutive topic_id preserves the intended sequence.
|
|
current_topic: int | None = object() # sentinel distinct from any id/None
|
|
for r in rules:
|
|
if r.topic_id != current_topic:
|
|
current_topic = r.topic_id
|
|
heading = topic_map.get(r.topic_id, "ungrouped") if r.topic_id else "ungrouped"
|
|
lines.append(f"### {heading}")
|
|
lines.append(f"- [{r.id}] {r.title}")
|
|
|
|
project_dict: dict | None = None
|
|
if project_id:
|
|
project = await projects_svc.get_project(user_id, project_id)
|
|
if project is not None:
|
|
_, open_count = await notes_svc.list_notes(
|
|
user_id, is_task=True, status="todo", project_id=project_id, limit=1,
|
|
)
|
|
goal = (getattr(project, "goal", "") or "").strip()
|
|
project_dict = {"id": project.id, "title": project.title}
|
|
lines += [
|
|
"",
|
|
f"## Active project: {project.title} (id {project.id})",
|
|
f"Goal: {goal[:200]}" if goal else "",
|
|
f"Open todo tasks: {open_count}",
|
|
]
|
|
elif unbound_repo:
|
|
lines += [
|
|
"",
|
|
"## Repository not yet bound",
|
|
f"This repo (`{unbound_repo}`) isn't mapped to a Scribe project, so "
|
|
"no project context was loaded. Bind it once with "
|
|
f'`bind_repo(repo_url="{unbound_repo}", project_id=<id>)` '
|
|
"(call `list_projects` to find the id) and future sessions here will "
|
|
"auto-load that project's context.",
|
|
]
|
|
|
|
lines += [
|
|
"",
|
|
"Reflex: search Scribe (search / list_tasks / list_notes, scoped to the "
|
|
"active project) before answering or starting work; prefer UPDATING an "
|
|
"existing note/rule over creating a new one.",
|
|
]
|
|
|
|
context = "\n".join(line for line in lines if line is not None)
|
|
if len(context) > _MAX_CHARS:
|
|
context = context[:_MAX_CHARS].rstrip() + "\n\n…(truncated — call list_always_on_rules())"
|
|
|
|
return {"context": context, "rule_count": len(rules), "project": project_dict}
|