feat(ledger): mechanical proposer — every refresh proposes instances against canon and groups derive-first candidates; agents confirm in batches (#2792, milestone 294 step 6)
CI & Build / Python lint (push) Successful in 5s
CI & Build / Plugin hooks (push) Successful in 13s
CI & Build / integration (push) Failing after 34s
CI & Build / TypeScript typecheck (push) Successful in 35s
CI & Build / Python tests (push) Successful in 1m17s
CI & Build / Build & push image (push) Successful in 48s

Shapes now carry a content fingerprint (signature + whitespace/comment-
insensitive body_sha; migration 0080) and the proposer runs inside the
coverage refresh, the one moment bodies exist: symbol elsewhere → textual
containment → body references the canon → signature resemblance → semantic
(capped per refresh, unreached rows stay unexamined for the next). A hit is
a proposal on the row (proposed_snippet_id/basis/score), never a
classification; rows with no canon hit group by the derive-first rule
(identical body in ≥2 places, same name in ≥3 files) as proposal_basis=
derive + a group key. list_shapes(proposal=any|canon|derive|<basis>) is the
queue; confirm_shape_proposals(project_id, snippet_id|path|basis) confirms
in batches as agent instances; any classify_shapes/hook stamp retires the
proposal. Readout carries proposed + derive_groups (line, payload, card).
Plugin 0.1.35 (skill: the machine proposes, judgment classifies).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
2026-08-20 21:30:35 -04:00
co-authored by Claude Fable 5
parent a9e1cddba7
commit ba0030e51d
12 changed files with 1190 additions and 43 deletions
+65 -7
View File
@@ -62,6 +62,7 @@ async def list_shapes(
include_vanished: bool = False,
limit: int = 100,
offset: int = 0,
proposal: str = "",
) -> dict:
"""Read a project's shape ledger — `status="unclassified"` IS the todo.
@@ -75,6 +76,11 @@ async def list_shapes(
snippet_id: rows classified against this snippet — a consumer map.
include_vanished: include shapes no longer in the tree (history).
limit/offset: page through big ledgers (limit caps at 500).
proposal: the proposer's queue (#2792) — "any", "canon" (rows the
machine thinks are an instance of a snippet: `proposal` carries
snippet_id, basis, score), "derive" (rows that repeat with NO
canon: `proposal.group` names the family), or one basis
(symbol/text/reference/signature/semantic).
Returns {"shapes": [...], "total": N} — total counts every match, not
just this page. Each row's `classified_by` says who judged: agent /
@@ -84,20 +90,61 @@ async def list_shapes(
was stamped an instance with the evidence in `reason`. A hook row is
overridable by any classify_shapes call; it never overrides yours.
Classify what you can judge with classify_shapes; a repeating shape
with NO recorded canon is a derive-one-first moment (consolidate onto a
reference, create_snippet it, then classify the rest against it), never
N loose classifications.
THE FAST PATH through a big todo is the proposer's queue: every coverage
refresh matches unclassified shapes against canon (strongest basis
first: same symbol elsewhere → textual containment → body references
the canon → signature resemblance → semantic) and attaches a
`proposal` to each row it can speak for. Review `proposal="canon"` by
snippet or directory, then confirm_shape_proposals the ones that hold —
hundreds at a time — and classify_shapes the rest (variant/exempt, or
instance of a different snippet). `proposal="derive"` lists the
derive-first candidates: a repeating shape with NO recorded canon is
never N loose classifications — consolidate onto a reference,
create_snippet it, then classify the group against it.
"""
uid = current_user_id()
rows, total = await shape_ledger_svc.list_project_shapes(
uid, project_id,
status=status, path=path, snippet_id=snippet_id,
include_vanished=include_vanished, limit=limit, offset=offset,
proposal=proposal,
)
return {"shapes": [r.to_dict() for r in rows], "total": total}
async def confirm_shape_proposals(
project_id: int,
snippet_id: int = 0,
path: str = "",
basis: str = "",
min_score: float = 0.0,
) -> dict:
"""Confirm the proposer's canon proposals you have reviewed, in batch.
The machine proposes, judgment classifies (#2792): each matching row —
live, unclassified, carrying a `proposal` with a snippet_id — becomes
`instance` of that snippet, classified_by="agent", reason naming the
basis and score. Narrow to what you actually looked at: at least one of
snippet_id (confirm one canon's whole queue after reading its
`list_shapes(proposal="canon", ...)` page), path (a directory you
audited), or basis (e.g. "symbol" and "reference" are near-certain;
"semantic" deserves a look first) is required — a bare confirm-all is
not a judgment. min_score trims a basis's tail.
Proposals you do NOT confirm are judged with classify_shapes (variant,
exempt, or instance of a different snippet) — any judgment retires the
proposal. Requires write access. Returns {"confirmed": N}.
"""
uid = current_user_id()
try:
return await shape_ledger_svc.confirm_proposals(
uid, project_id, snippet_id=snippet_id, path=path, basis=basis,
min_score=min_score,
)
except ValueError as exc:
return {"error": str(exc)}
async def refresh_pattern_coverage(project_id: int) -> dict:
"""Seed or refresh the project's shape ledger NOW, and return the readout.
@@ -114,9 +161,17 @@ async def refresh_pattern_coverage(project_id: int) -> dict:
owner adds one (Settings → Integrations → Git Forges); no served repo →
bind_repo on a host a connection serves.
The refresh is also when the mechanical proposer runs (#2792): with the
repo bodies in hand it matches every changed unclassified shape against
canon and records proposals (see list_shapes proposal=), then regroups
the derive-first candidates. Semantic matching is capped per refresh, so
a large ledger's queue grows across refreshes rather than in one.
Returns the accounting payload — total, accounted, counts by status,
unclassified, repos, largest_gaps — plus `pattern_coverage`, the same
one-line summary enter_project carries.
unclassified, repos, largest_gaps, `proposed` (canon proposals awaiting
confirmation), `derive_groups` (the biggest repeats-with-no-canon
families), `proposer` (what this refresh examined) — plus
`pattern_coverage`, the same one-line summary enter_project carries.
"""
uid = current_user_id()
coverage = await coverage_svc.refresh_for_caller(uid, project_id)
@@ -127,5 +182,8 @@ async def refresh_pattern_coverage(project_id: int) -> dict:
def register(mcp) -> None:
for fn in (classify_shapes, list_shapes, refresh_pattern_coverage):
for fn in (
classify_shapes, list_shapes, refresh_pattern_coverage,
confirm_shape_proposals,
):
mcp.tool(name=fn.__name__)(fn)
+50
View File
@@ -3,6 +3,7 @@ from datetime import datetime
from sqlalchemy import (
BigInteger,
DateTime,
Float,
ForeignKey,
Index,
Integer,
@@ -19,6 +20,12 @@ from scribe.models.base import TimestampMixin
SHAPE_STATUSES = ("canonical", "instance", "variant", "exempt", "unclassified")
SHAPE_CLASSIFIERS = ("agent", "audit", "hook", "mechanical", "import")
# How the mechanical proposer (#2792) arrived at a proposal, strongest first.
# `derive` is the odd one out: not "this is an instance of #N" but "this
# shape repeats with NO canon — derive one first" (note 2786's derive-first
# rule), so it carries a group key instead of a snippet.
PROPOSAL_BASES = ("symbol", "text", "reference", "signature", "semantic", "derive")
class CodeShape(Base, TimestampMixin):
"""One extracted code shape and its classification against canon (#2787).
@@ -42,6 +49,20 @@ class CodeShape(Base, TimestampMixin):
snippet_id is SET NULL on snippet deletion: the classification's target
is gone but the judgment happened; the sync pass (step 2) re-files such
rows as unclassified so they rejoin the todo instead of dangling.
`signature` / `body_sha` (#2792) are the shape's content fingerprint —
its definition line and a whitespace/comment-insensitive hash of its
block — refreshed by every sync. They are what the mechanical proposer
matches on and what a later drift recheck compares against; the ledger
still never stores code bodies.
The proposal columns hold the proposer's standing suggestion for an
UNCLASSIFIED row: `proposed_snippet_id` + `proposal_basis` + score for
"looks like an instance of #N", or `proposal_basis="derive"` +
`proposal_group` for "repeats with no canon". `proposed_sha` is the
body_sha the row was last examined at, so a refresh re-examines only
what changed. A judgment clears the proposal — the machine proposes,
judgment classifies.
"""
__tablename__ = "code_shapes"
@@ -52,6 +73,7 @@ class CodeShape(Base, TimestampMixin):
),
Index("ix_code_shapes_project_status", "project_id", "status"),
Index("ix_code_shapes_snippet", "snippet_id"),
Index("ix_code_shapes_proposed", "project_id", "proposed_snippet_id"),
)
id: Mapped[int] = mapped_column(primary_key=True)
@@ -76,6 +98,31 @@ class CodeShape(Base, TimestampMixin):
vanished_at: Mapped[datetime | None] = mapped_column(
DateTime(timezone=True), nullable=True
)
signature: Mapped[str] = mapped_column(Text, default="")
body_sha: Mapped[str] = mapped_column(Text, default="")
proposed_snippet_id: Mapped[int | None] = mapped_column(
BigInteger, ForeignKey("notes.id", ondelete="SET NULL"), nullable=True
)
proposal_basis: Mapped[str | None] = mapped_column(Text, nullable=True)
proposal_score: Mapped[float | None] = mapped_column(Float, nullable=True)
proposal_group: Mapped[str | None] = mapped_column(Text, nullable=True)
proposed_at: Mapped[datetime | None] = mapped_column(
DateTime(timezone=True), nullable=True
)
proposed_sha: Mapped[str] = mapped_column(Text, default="")
@property
def proposal(self) -> dict | None:
"""The standing proposal as one object, or None when the proposer
has nothing to say about this row."""
if self.proposed_snippet_id is None and not self.proposal_group:
return None
out: dict = {"basis": self.proposal_basis, "score": self.proposal_score}
if self.proposed_snippet_id is not None:
out["snippet_id"] = self.proposed_snippet_id
if self.proposal_group:
out["group"] = self.proposal_group
return out
def to_dict(self) -> dict:
return {
@@ -93,6 +140,9 @@ class CodeShape(Base, TimestampMixin):
"first_seen_commit": self.first_seen_commit,
"last_seen_commit": self.last_seen_commit,
"vanished_at": self.vanished_at.isoformat() if self.vanished_at else None,
"signature": self.signature,
"body_sha": self.body_sha,
"proposal": self.proposal,
"created_at": self.created_at.isoformat(),
"updated_at": self.updated_at.isoformat(),
}
+5
View File
@@ -1163,6 +1163,11 @@ async def _restore_v2(data: dict) -> dict:
first_seen_commit=cs_data.get("first_seen_commit", ""),
last_seen_commit=cs_data.get("last_seen_commit", ""),
vanished_at=_dt(cs_data["vanished_at"]) if cs_data.get("vanished_at") else None,
# Fingerprints restore; proposals (#2792) deliberately do not —
# they are mechanical, and the next refresh recomputes them
# against the restored snippet ids.
signature=cs_data.get("signature", ""),
body_sha=cs_data.get("body_sha", ""),
created_at=_dt(cs_data.get("created_at")),
updated_at=_dt(cs_data.get("updated_at")),
))
+164 -28
View File
@@ -25,12 +25,14 @@ only ever reads the cache.
"""
from __future__ import annotations
import hashlib
import io
import json
import logging
import posixpath
import re
import tarfile
from typing import NamedTuple
from datetime import datetime, timedelta, timezone
from scribe.services.forge import ForgeSelector, get_forges
@@ -91,6 +93,104 @@ _ARROW_RE = re.compile(
)
class Definition(NamedTuple):
"""One extracted definition with its content fingerprint (#2792).
`signature` is the definition line itself; `body_sha` hashes the block
whitespace- and comment-insensitively; `body` is the block's text, held
only for the duration of a refresh (the proposer matches on it) and
never stored.
"""
kind: str
name: str
signature: str
body_sha: str
body: str
def _definition_on(raw: str) -> tuple[str, str] | None:
"""The (kind, name) this one line defines, or None. First match wins —
the same order the hook's awk program tries."""
m = _CSS_RE.match(raw)
if m:
return ("css", m.group(1))
line = _MODIFIERS_RE.sub("", raw.lstrip())
if m := _GO_METHOD_RE.match(line):
return ("sym", m.group(1))
if m := _KEYWORD_RE.match(line):
name = m.group(1)
if name.startswith("__") and name.endswith("__"):
return None
return ("sym", name)
if m := _ARROW_RE.match(line):
return ("sym", m.group(1))
return None
# A definition's block runs from its line until the next non-blank line at
# its own indentation or shallower that is not a closer — so a Python def ends
# at the next top-level statement, a braces block keeps its `}`, a CSS rule
# keeps its `}`. Capped so a generated monolith can't make one shape's
# fingerprint cover the file.
_BLOCK_CAP = 120
_CLOSERS = ("}", ")", "]", "end", "};", "});", ");", "})", "]);")
# Lines that don't change what a shape IS: comments and decorators. Dropped
# from the fingerprint so touching a comment above the next function doesn't
# read as this one's body changing.
_NOISE_PREFIXES = ("#", "//", "/*", "*", "*/", "@", "<!--", "-->")
_SIGNATURE_CAP = 300
def _indent(line: str) -> int:
return len(line) - len(line.lstrip())
def _block_sha(lines: list[str]) -> str:
kept = [
" ".join(ln.split())
for ln in lines
if ln.strip() and not ln.lstrip().startswith(_NOISE_PREFIXES)
]
return hashlib.sha1("\n".join(kept).encode("utf-8")).hexdigest()[:16]
def extract_definitions(text: str) -> list[Definition]:
"""Every definition this text makes, with signature + fingerprint.
Duplicate (kind, name) within one text collapse to the first — the
ledger's identity is per file, so a second definition of the same name
(an overload, a re-declaration) is the same shape to it.
"""
lines = text.splitlines()
starts: list[tuple[int, str, str]] = []
for i, raw in enumerate(lines):
hit = _definition_on(raw)
if hit:
starts.append((i, hit[0], hit[1]))
seen: set[tuple[str, str]] = set()
out: list[Definition] = []
for i, kind, name in starts:
if (kind, name) in seen:
continue
seen.add((kind, name))
base = _indent(lines[i])
end = min(len(lines), i + _BLOCK_CAP)
for j in range(i + 1, min(len(lines), i + _BLOCK_CAP)):
ln = lines[j]
if not ln.strip():
continue
if _indent(ln) <= base and ln.strip() not in _CLOSERS:
end = j
break
block = lines[i:end]
out.append(Definition(
kind, name, lines[i].strip()[:_SIGNATURE_CAP], _block_sha(block),
"\n".join(block),
))
return out
def extract_shapes(text: str) -> list[tuple[str, str]]:
"""Every (kind, name) this text DEFINES — kind is "css" or "sym".
@@ -98,29 +198,7 @@ def extract_shapes(text: str) -> list[tuple[str, str]]:
line, dunders are skipped (every class defines __init__ — guaranteed
noise), duplicates within one text count once.
"""
seen: set[tuple[str, str]] = set()
out: list[tuple[str, str]] = []
for raw in text.splitlines():
m = _CSS_RE.match(raw)
if m:
shape = ("css", m.group(1))
else:
line = _MODIFIERS_RE.sub("", raw.lstrip())
if m := _GO_METHOD_RE.match(line):
shape = ("sym", m.group(1))
elif m := _KEYWORD_RE.match(line):
name = m.group(1)
if name.startswith("__") and name.endswith("__"):
continue
shape = ("sym", name)
elif m := _ARROW_RE.match(line):
shape = ("sym", m.group(1))
else:
continue
if shape not in seen:
seen.add(shape)
out.append(shape)
return out
return [(d.kind, d.name) for d in extract_definitions(text)]
def scannable(path: str) -> bool:
@@ -131,14 +209,33 @@ def scannable(path: str) -> bool:
return not path.lower().endswith(_SKIP_SUFFIXES)
class ArchiveShape(NamedTuple):
"""A definition located in a repo archive — what the sync upserts and
the proposer matches. The leading (path, kind, name) triple is the
ledger identity; the rest is the fingerprint and the transient body."""
path: str
kind: str
name: str
signature: str
body_sha: str
body: str
def shapes_from_archive(blob: bytes) -> list[tuple[str, str, str]]:
"""(path, kind, name) for every definition in a repo tarball.
"""(path, kind, name) for every definition in a repo tarball — the
identity view of definitions_from_archive."""
return [(d.path, d.kind, d.name) for d in definitions_from_archive(blob)]
def definitions_from_archive(blob: bytes) -> list[ArchiveShape]:
"""Every definition in a repo tarball, with its fingerprint and body.
Forge archives wrap content in a single top-level directory (repo-ref/);
that component is stripped so paths match recorded snippet locations,
which are repo-relative. Non-UTF-8 files are binaries and skipped.
"""
shapes: list[tuple[str, str, str]] = []
shapes: list[ArchiveShape] = []
with tarfile.open(fileobj=io.BytesIO(blob), mode="r:gz") as tar:
for member in tar:
if not member.isfile() or "/" not in member.name:
@@ -153,7 +250,10 @@ def shapes_from_archive(blob: bytes) -> list[tuple[str, str, str]]:
text = handle.read().decode("utf-8")
except UnicodeDecodeError:
continue
shapes.extend((path, kind, name) for kind, name in extract_shapes(text))
shapes.extend(
ArchiveShape(path, d.kind, d.name, d.signature, d.body_sha, d.body)
for d in extract_definitions(text)
)
return shapes
@@ -253,13 +353,17 @@ async def compute_coverage(
served: list[tuple[str, str]] = []
recorded = await _recorded_locations(user_id, project_id)
# The proposer's canon catalog, read once per refresh and shared across
# the project's repos (#2792).
canons = None
proposer_stats = {"examined": 0, "proposed": 0, "semantic_checked": 0}
for key in await keys_for_project(user_id, project_id):
hit = selector.resolve(key)
if hit is None:
continue # bound to a host no connection serves
forge, api_repo = hit
ref = await forge.default_branch(api_repo)
shapes = shapes_from_archive(await forge.archive(api_repo, ref))
definitions = definitions_from_archive(await forge.archive(api_repo, ref))
# The head commit is provenance sugar on the ledger rows; failing to
# learn it must not fail the sync — the ref names the point well
# enough and the row timestamps carry the when.
@@ -268,13 +372,31 @@ async def compute_coverage(
except ForgeError:
marker = ref
await shape_ledger.sync_repo_shapes(
project_id, key, shapes, seen_marker=marker
project_id, key, definitions, seen_marker=marker
)
served.append((key, ref))
# Propose while the bodies are in hand — the one moment they exist.
# Canonical marking below only touches rows the proposer leaves
# alone (a canon's own location never gets a proposal), so the order
# is immaterial; the proposer must not be able to fail the refresh.
try:
if canons is None:
canons = await shape_ledger.canon_catalog(user_id)
stats = await shape_ledger.propose_for_repo(
user_id, project_id, key, definitions, canons=canons
)
for k in proposer_stats:
proposer_stats[k] += stats.get(k, 0)
except Exception:
logger.warning("shape proposer failed for %s", key, exc_info=True)
if not served:
return None
await shape_ledger.mark_canonicals(project_id, recorded)
try:
await shape_ledger.apply_derive_groups(project_id)
except Exception:
logger.warning("derive-first grouping failed", exc_info=True)
# Project-wide readout, deliberately wider than this walk: a second bound
# repo that was unreachable today still has live rows, and they count.
@@ -290,11 +412,17 @@ async def compute_coverage(
agg["accounted"] += row.status != "unclassified"
unclassified = counts.pop("unclassified")
proposals = shape_ledger.proposal_summary(rows)
return {
"total": len(rows),
"accounted": len(rows) - unclassified,
"unclassified": unclassified,
"counts": counts,
# The proposer's standing (#2792): canon proposals awaiting a
# confirm, the largest derive-first groups, and what this refresh did.
"proposed": proposals["proposed"],
"derive_groups": proposals["derive_groups"],
"proposer": proposer_stats,
# Honesty flag, not decoration: every surface that shows the number
# is expected to carry it through.
"estimate": True,
@@ -438,6 +566,14 @@ def coverage_line(coverage: dict) -> str:
unclassified = coverage.get("unclassified", 0)
if unclassified:
line += f"; {unclassified} unclassified"
standing = []
if coverage.get("proposed"):
standing.append(f"{coverage['proposed']} proposed")
n_groups = len(coverage.get("derive_groups") or [])
if n_groups:
standing.append(f"{n_groups} derive group{'s' if n_groups != 1 else ''}")
if standing:
line += f" ({', '.join(standing)})"
gaps = [g["dir"] for g in coverage.get("largest_gaps") or []]
if gaps:
line += ", largest: " + ", ".join(gaps)
+467 -7
View File
@@ -21,9 +21,11 @@ an extracted shape.
"""
from __future__ import annotations
import difflib
import logging
import re
from datetime import datetime, timedelta, timezone
from typing import Iterable, NamedTuple
from sqlalchemy import select
@@ -65,15 +67,17 @@ def location_covers(loc_path: str, loc_symbol: str, path: str, name: str) -> boo
async def sync_repo_shapes(
project_id: int,
repo_key: str,
shapes: list[tuple[str, str, str]],
shapes: list,
*,
seen_marker: str,
) -> None:
"""Upsert one repo's extracted (path, kind, name) shapes into the ledger.
"""Upsert one repo's extracted shapes into the ledger.
``seen_marker`` is the commit the archive was read at when the forge can
say, else the ref name — provenance sugar; the row timestamps carry the
when.
``shapes`` are (path, kind, name) triples, or the richer ArchiveShape
records (#2792) whose 4th/5th fields — signature, body_sha — refresh the
row's content fingerprint. ``seen_marker`` is the commit the archive was
read at when the forge can say, else the ref name — provenance sugar;
the row timestamps carry the when.
"""
now = datetime.now(timezone.utc)
async with async_session() as session:
@@ -87,7 +91,10 @@ async def sync_repo_shapes(
).scalars().all()
by_key = {(r.path, r.symbol, r.kind): r for r in rows}
seen: set[tuple[str, str, str]] = set()
for path, kind, name in shapes:
for shape in shapes:
path, kind, name = shape[0], shape[1], shape[2]
signature = shape[3] if len(shape) > 3 else ""
body_sha = shape[4] if len(shape) > 4 else ""
key = (path, name, kind)
if key in seen:
continue
@@ -98,9 +105,14 @@ async def sync_repo_shapes(
project_id=project_id, repo_key=repo_key,
path=path, symbol=name, kind=kind,
first_seen_commit=seen_marker, last_seen_commit=seen_marker,
signature=signature, body_sha=body_sha,
))
continue
row.last_seen_commit = seen_marker
if signature:
row.signature = signature
if body_sha:
row.body_sha = body_sha
# A shape that vanished and came back is live again — the vanish
# stays visible in history via updated_at, not as a dead flag.
row.vanished_at = None
@@ -297,6 +309,10 @@ async def classify_shapes(
status = item["status"]
for row in matches:
row.status = status
# The machine proposes, judgment classifies: any judgment
# retires the standing proposal; a withdrawal also forgets
# the examination so the next refresh proposes afresh.
_clear_proposal(row, reexamine=(status == "unclassified"))
if status == "unclassified":
row.snippet_id = None
row.reason = None
@@ -324,12 +340,16 @@ async def list_project_shapes(
include_vanished: bool = False,
limit: int = 100,
offset: int = 0,
proposal: str = "",
) -> tuple[list[CodeShape], int]:
"""A filtered page of a project's ledger, with the unfiltered-match total.
([], 0) when the caller can't read the project — the same silence every
other project list gives. ``path`` matches the exact file or anything
beneath it, mirroring recorded-location semantics.
beneath it, mirroring recorded-location semantics. ``proposal`` narrows
to rows the proposer has spoken about: "any", "canon" (an instance-of-#N
suggestion), "derive" (a repeats-with-no-canon group), or one basis
name (symbol/reference/text/signature/semantic).
"""
from sqlalchemy import func, or_
@@ -349,6 +369,17 @@ async def list_project_shapes(
))
if snippet_id:
conds.append(CodeShape.snippet_id == snippet_id)
if proposal == "any":
conds.append(or_(
CodeShape.proposed_snippet_id.isnot(None),
CodeShape.proposal_group.isnot(None),
))
elif proposal == "canon":
conds.append(CodeShape.proposed_snippet_id.isnot(None))
elif proposal == "derive":
conds.append(CodeShape.proposal_group.isnot(None))
elif proposal:
conds.append(CodeShape.proposal_basis == proposal)
async with async_session() as session:
total = (
await session.execute(
@@ -589,6 +620,7 @@ async def stamp_write_path_instances(
row.reason = why
row.classified_by = "hook"
row.classified_at = now
_clear_proposal(row)
stamped.append({
"path": path, "symbol": name, "kind": kind,
"snippet_id": sid, "reason": why,
@@ -596,3 +628,431 @@ async def stamp_write_path_instances(
if stamped:
await session.commit()
return stamped
# --- the mechanical proposer (#2792): the machine proposes, judgment classifies
#
# Runs inside the coverage refresh — the one moment the shape BODIES exist
# (the archive is in memory; the ledger stores fingerprints, never code).
# Over every live unclassified row whose content changed since it was last
# examined, it tries, strongest first:
#
# symbol the shape bears a recorded canon's symbol at another location
# (a second definition of the canon's name — an instance or a
# duplicate to consolidate; either way, it answers to #N);
# text whitespace-insensitive containment either way between the
# body and the canon's recorded code (git-grep, in effect —
# a verbatim copy, or the canon's own call-site example);
# reference the body calls/uses a canon's symbol — the call-site shape
# the P7 backfill classified by hand (#2790);
# signature a sym whose definition line, name blanked, resembles the
# canon's (the move_*/resequence family shape);
# semantic the widest net: the body (concept-queried, as the write-path
# arm does) scores above the write-path threshold against a
# canon — capped per refresh, so the cost is bounded and a big
# ledger is worked through across refreshes.
#
# A hit is a PROPOSAL on the row (proposed_snippet_id/basis/score), never a
# classification: an agent confirms in batches (confirm_proposals) or judges
# otherwise (classify_shapes clears it). Rows with no canon hit are grouped
# by the derive-first rule (note 2786): the same body fingerprint in ≥2
# places, or the same name defined in ≥3 files, is "a repeating shape with
# no canon — derive one first", carried as proposal_basis="derive" + a group
# key so sessions see the consolidation candidates as one thing.
# Derive-first floors. Identical bodies twice is already a copy; a bare name
# needs more repetition before it reads as a family (setup/handler/register
# recur by convention, not by duplication).
_DERIVE_MIN_DUP = 2
_DERIVE_MIN_NAME = 3
# Semantic checks per repo per refresh — an embedding each (local fastembed),
# bounded so a 4,000-row ledger is worked through over refreshes, not in one.
_SEMANTIC_CAP = 150
# Signature resemblance floor, name blanked (difflib ratio) — and a length
# floor, because `def NAME():` resembles `def NAME(x):` at 0.95 while saying
# nothing; a family shape has parameters to resemble.
_SIGNATURE_FLOOR = 0.8
_SIGNATURE_MIN_LEN = 30
# Textual containment needs enough substance to mean anything.
_TEXT_FLOOR = 40
_BASIS_ORDER = ("symbol", "text", "reference", "signature")
class Canon(NamedTuple):
snippet_id: int
kind: str
symbol: str
locations: tuple[tuple[str, str], ...]
signature: str
code_norm: str
def _norm_text(text: str) -> str:
return " ".join((text or "").split())
def signature_similarity(sig_a: str, name_a: str, sig_b: str, name_b: str) -> float:
"""How alike two definition lines are once their own names are blanked —
`def move_event(project_id, event_id, after_id)` against
`def move_beat(project_id, beat_id, after_id)` reads high."""
a = _norm_text(sig_a)
b = _norm_text(sig_b)
if name_a:
a = a.replace(name_a, "NAME")
if name_b:
b = b.replace(name_b, "NAME")
if len(a) < _SIGNATURE_MIN_LEN or len(b) < _SIGNATURE_MIN_LEN:
return 0.0
return difflib.SequenceMatcher(None, a, b).ratio()
def text_contains(body: str, code: str) -> bool:
"""Whitespace-insensitive containment, either way, above a substance
floor — the proposer's git-grep."""
a = _norm_text(body)
b = _norm_text(code)
if len(a) < _TEXT_FLOOR or len(b) < _TEXT_FLOOR:
return False
return a in b or b in a
def match_canon(
kind: str, path: str, symbol: str, signature: str, body: str,
canons: Iterable[Canon],
) -> tuple[int, str, float] | None:
"""The strongest (snippet_id, basis, score) a shape earns against the
canon catalog, by the non-semantic bases — or None. Strongest by
basis order, then by score within the basis."""
best: dict[str, tuple[float, int]] = {}
def offer(basis: str, score: float, sid: int) -> None:
cur = best.get(basis)
if cur is None or score > cur[0]:
best[basis] = (score, sid)
norm_sym = _norm_symbol(symbol)
for c in canons:
if c.kind != kind:
continue
if c.symbol and _norm_symbol(c.symbol) == norm_sym:
if not any(location_covers(lp, ls, path, symbol) for lp, ls in c.locations):
offer("symbol", 1.0, c.snippet_id)
continue # its own location is canonical territory, not a proposal
if c.symbol and references_symbol(body, c.symbol, kind):
offer("reference", 0.9, c.snippet_id)
if c.code_norm and text_contains(body, c.code_norm):
offer("text", 0.95, c.snippet_id)
if kind == "sym" and c.signature:
ratio = signature_similarity(signature, symbol, c.signature, c.symbol)
if ratio >= _SIGNATURE_FLOOR:
offer("signature", round(ratio, 3), c.snippet_id)
for basis in _BASIS_ORDER:
if basis in best:
score, sid = best[basis]
return (sid, basis, score)
return None
async def canon_catalog(user_id: int) -> list[Canon]:
"""Every snippet this user can browse, as matchable canon — own projects
and shared ones alike, because family canon counts (note 2786)."""
from scribe.models.note import Note
from scribe.services.access import browsable_notes_clause
from scribe.services.coverage import extract_definitions
from scribe.services.snippets import SNIPPET_NOTE_TYPE, snippet_fields
async with async_session() as session:
notes = (
await session.execute(
select(Note).where(
browsable_notes_clause(user_id),
Note.note_type == SNIPPET_NOTE_TYPE,
Note.deleted_at.is_(None),
)
)
).scalars().all()
out: list[Canon] = []
for note in notes:
fields = snippet_fields(note)
symbol = (fields.get("symbol") or "").strip()
kind = snippet_kind(symbol, fields.get("language") or "")
code = fields.get("code") or ""
defs = extract_definitions(code)
signature = ""
if defs:
own = next(
(d for d in defs if _norm_symbol(d.name) == _norm_symbol(symbol)), defs[0]
)
signature = own.signature
out.append(Canon(
int(note.id), kind, symbol,
tuple(
((loc.get("path") or ""), (loc.get("symbol") or ""))
for loc in fields.get("locations") or []
),
signature, _norm_text(code),
))
return out
def _clear_proposal(row: CodeShape, *, reexamine: bool = False) -> None:
row.proposed_snippet_id = None
row.proposal_basis = None
row.proposal_score = None
row.proposal_group = None
if reexamine:
row.proposed_at = None
row.proposed_sha = ""
def _substance(text: str) -> int:
return len("".join((text or "").split()))
async def _semantic_canon(
user_id: int, body: str, allowed: set[int]
) -> tuple[int, float] | None:
from scribe.services.embeddings import semantic_search_notes
from scribe.services.plugin_context import (
WRITEPATH_DEFAULT_THRESHOLD, WRITEPATH_MIN_CODE_CHARS, concept_query,
)
if _substance(body) < WRITEPATH_MIN_CODE_CHARS or not allowed:
return None
query = concept_query(body) or body
hits = await semantic_search_notes(
user_id, query, limit=3, threshold=WRITEPATH_DEFAULT_THRESHOLD,
note_type="snippet", scope="browse",
)
for score, note in hits:
if int(note.id) in allowed:
return int(note.id), round(float(score), 3)
return None
async def propose_for_repo(
user_id: int,
project_id: int,
repo_key: str,
definitions: list,
*,
canons: list[Canon] | None = None,
semantic_cap: int = _SEMANTIC_CAP,
) -> dict:
"""Examine one repo's live unclassified rows against canon and record
proposals. ``definitions`` are the ArchiveShape records the sync just
upserted (their bodies are the matching material). Rows whose fingerprint
is unchanged since their last examination are skipped; rows that only
the capped semantic pass could not reach stay unexamined, so the next
refresh reaches the next slice. Returns counts."""
if canons is None:
canons = await canon_catalog(user_id)
by_key = {(d[0], d[1], d[2]): d for d in definitions}
sym_canon_ids = {c.snippet_id for c in canons if c.kind == "sym"}
now = datetime.now(timezone.utc)
examined = proposed = checked = 0
async with async_session() as session:
rows = (
await session.execute(
select(CodeShape).where(
CodeShape.project_id == project_id,
CodeShape.repo_key == repo_key,
CodeShape.status == "unclassified",
CodeShape.vanished_at.is_(None),
)
)
).scalars().all()
semantic_todo: list[tuple[CodeShape, object]] = []
for row in rows:
d = by_key.get((row.path, row.kind, row.symbol))
if d is None:
continue
signature, body_sha, body = d[3], d[4], d[5]
if row.proposed_at is not None and row.proposed_sha == body_sha:
continue
examined += 1
group = row.proposal_group # derive grouping is reassigned below
hit = match_canon(row.kind, row.path, row.symbol, signature, body, canons)
_clear_proposal(row)
row.proposal_group = group
row.proposed_at = now
row.proposed_sha = body_sha
if hit:
row.proposed_snippet_id, row.proposal_basis, row.proposal_score = hit
row.proposal_group = None
proposed += 1
elif row.kind == "sym":
semantic_todo.append((row, d))
for i, (row, d) in enumerate(semantic_todo):
if i >= semantic_cap:
# Not reached this refresh: leave it unexamined so the next
# refresh picks it up, rather than stamping a false "nothing".
row.proposed_at = None
row.proposed_sha = ""
continue
checked += 1
try:
found = await _semantic_canon(user_id, d[5], sym_canon_ids)
except Exception:
logger.warning("semantic proposal failed", exc_info=True)
found = None
if found:
row.proposed_snippet_id, row.proposal_score = found
row.proposal_basis = "semantic"
row.proposal_group = None
proposed += 1
await session.commit()
return {"examined": examined, "proposed": proposed, "semantic_checked": checked}
def derive_groups(
rows: Iterable[tuple[str, str, str, str]]
) -> dict[tuple[str, str, str], str]:
"""The derive-first grouping over (path, kind, symbol, body_sha) rows
that matched no canon: {(path, kind, symbol): group_key}. Identical
bodies in ≥2 places group as `dup:<sha>`; the same name defined in ≥3
files groups as `name:<kind>:<symbol>`; a row joins at most one group,
the copy before the name."""
by_sha: dict[str, list[tuple[str, str, str]]] = {}
by_name: dict[tuple[str, str], list[tuple[str, str, str]]] = {}
for path, kind, symbol, sha in rows:
key = (path, kind, symbol)
if sha:
by_sha.setdefault(sha, []).append(key)
by_name.setdefault((kind, _norm_symbol(symbol)), []).append(key)
out: dict[tuple[str, str, str], str] = {}
for sha, keys in by_sha.items():
if len(set(keys)) >= _DERIVE_MIN_DUP:
for key in keys:
out.setdefault(key, f"dup:{sha}")
for (kind, symbol), keys in by_name.items():
if len({k[0] for k in keys}) >= _DERIVE_MIN_NAME:
for key in keys:
out.setdefault(key, f"name:{kind}:{symbol}")
return out
async def apply_derive_groups(project_id: int) -> int:
"""Recompute derive-first groups over the project's live unclassified
rows that carry no canon proposal; returns how many rows are grouped."""
now = datetime.now(timezone.utc)
grouped = 0
async with async_session() as session:
rows = (
await session.execute(
select(CodeShape).where(
CodeShape.project_id == project_id,
CodeShape.status == "unclassified",
CodeShape.vanished_at.is_(None),
CodeShape.proposed_snippet_id.is_(None),
)
)
).scalars().all()
groups = derive_groups(
(r.path, r.kind, r.symbol, r.body_sha or "") for r in rows
)
sizes: dict[str, int] = {}
for g in groups.values():
sizes[g] = sizes.get(g, 0) + 1
for row in rows:
key = groups.get((row.path, row.kind, row.symbol))
if key:
row.proposal_basis = "derive"
row.proposal_group = key
row.proposal_score = float(sizes[key])
if row.proposed_at is None:
row.proposed_at = now
grouped += 1
elif row.proposal_group:
row.proposal_basis = None
row.proposal_group = None
row.proposal_score = None
await session.commit()
return grouped
def proposal_summary(rows: Iterable[CodeShape], *, top: int = 8) -> dict:
"""The readout's view of the proposer's standing: how many canon
proposals await confirmation, and the largest derive-first groups."""
proposed = 0
groups: dict[str, dict] = {}
for row in rows:
if row.status != "unclassified":
continue
if row.proposed_snippet_id is not None:
proposed += 1
elif row.proposal_group:
g = groups.setdefault(row.proposal_group, {
"group": row.proposal_group, "kind": row.kind,
"label": (
("." if row.kind == "css" else "") + row.symbol
if row.proposal_group.startswith("name:")
else f"{row.symbol} (identical body)"
),
"size": 0, "paths": [],
})
g["size"] += 1
if len(g["paths"]) < 3:
g["paths"].append(row.path)
ranked = sorted(groups.values(), key=lambda g: (-g["size"], g["group"]))
return {"proposed": proposed, "derive_groups": ranked[:top]}
async def confirm_proposals(
user_id: int,
project_id: int,
*,
snippet_id: int = 0,
path: str = "",
basis: str = "",
min_score: float = 0.0,
) -> dict:
"""Turn reviewed canon proposals into `instance` rows, in one batch.
At least one of snippet_id / path / basis must narrow the batch — "confirm
everything proposed" without having looked is not a judgment. Each row
becomes instance-of-its-proposed-snippet, classified_by="agent", reason
naming the basis and score; the proposal is retired. Returns
{"confirmed": N}."""
from scribe.services import access
if not (snippet_id or path.strip() or basis.strip()):
raise ValueError(
"name what you reviewed: confirm by snippet_id, path, and/or basis"
)
if not await access.can_write_project(user_id, project_id):
raise ValueError(f"project {project_id} not found or no write access")
from sqlalchemy import or_
conds = [
CodeShape.project_id == project_id,
CodeShape.status == "unclassified",
CodeShape.vanished_at.is_(None),
CodeShape.proposed_snippet_id.isnot(None),
]
if snippet_id:
conds.append(CodeShape.proposed_snippet_id == snippet_id)
if path.strip():
clean = path.strip().strip("/")
conds.append(or_(CodeShape.path == clean, CodeShape.path.like(clean + "/%")))
if basis.strip():
conds.append(CodeShape.proposal_basis == basis.strip())
now = datetime.now(timezone.utc)
confirmed = 0
async with async_session() as session:
rows = (await session.execute(select(CodeShape).where(*conds))).scalars().all()
for row in rows:
if (row.proposal_score or 0.0) < min_score:
continue
row.status = "instance"
row.snippet_id = row.proposed_snippet_id
row.reason = (
f"confirmed {row.proposal_basis} proposal"
f" ({(row.proposal_score or 0.0):.2f})"
)
row.classified_by = "agent"
row.classified_at = now
_clear_proposal(row)
confirmed += 1
await session.commit()
return {"confirmed": confirmed}