bvandeusenandClaude Opus 5
91cde6c3e4
CI & Build / Python lint (push) Successful in 2s
CI & Build / Plugin hooks (push) Successful in 9s
CI & Build / TypeScript typecheck (push) Successful in 52s
CI & Build / integration (push) Successful in 52s
CI & Build / Python tests (push) Successful in 1m33s
CI & Build / Build & push image (push) Successful in 28s
#4204 gave the divergence check a structural gate, which silenced one of the five false prompts it was filed for. The other four are `def` helpers in a directory whose canon is an `async def` service unit. No refinement of `shape_form` reaches them: they differ from #2793's acceptance case — a hand-rolled sync `confirmDanger` where an async confirm helper is canon — only by the JOB they do, and a signature does not carry a job. The proposer's semantic arm already reads bodies per symbol, which is the comparison option 2 asked for and was thought to be missing. What it did not do was record its MISSES: a hit became `proposal_basis = "semantic"`, a miss left the row indistinguishable from one nobody had looked at. So `flag_divergence` could ask the proposer "do you agree this is the canon?" but never "did you check, and is it not?". `_semantic_canon` now reports whether an empty answer is evidence, and `flag_divergence` withholds the prompt when it is. The whole risk is in the negative, so only a conclusive miss is stored. A body too thin to embed, a row the per-refresh cap never reached, an arm that threw, and a result set that came back full — and may therefore have hidden the canon behind the limit — all stay "cannot tell" and still ask the question. That is the discipline `FORM_UNKNOWN` already enforces here: not knowing must make a check quieter, never more confident. `_SEMANTIC_LIMIT` is named for that reason; the number is load-bearing, not a tuning knob. No migration: `proposal_basis` is nullable Text with no CHECK constraint (verified in the model and across alembic/versions), so rule 36 does not bite. Nothing can mistake the miss for a proposal either — every reader keys on `proposed_snippet_id` or `proposal_group`, and `confirm_shape_proposals` requires the id non-NULL before it will confirm anything. `_PROPOSER_VERSION` 3 -> 4, per its own contract: rows remember the ruleset they were examined under, and without the bump no already-examined row would ever acquire a miss. Option 1 (widening `kind`) stays closed, on the merits rather than on cost: bucketing density by exact form takes the async canon out of a sync candidate's denominator and silences #2793's acceptance case by the identical mechanism, one layer down. The reasoning is on #4208. Tests: tests/test_divergence_meaning_gate.py pins the report contract, with the truncation case tested hardest — reading a cut-off as a negative would weaken the guard in proportion to how many snippets the operator has. The end-to-end discrimination is in test_integration_shape_classify.py on deliberately the SAME fixture as #2793's acceptance case, so the two runs differ in exactly one thing: whether the arm claims to have looked. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01821k5B3Ysecp9fNYs92Kuy
2712 lines
119 KiB
Python
2712 lines
119 KiB
Python
"""The shape ledger's write side — sync and mechanical marking (#2788).
|
||
|
||
The coverage walk (services/coverage.py) is the only feed that sees every
|
||
shape, so it is the ledger's sync point: each refresh upserts one repo's
|
||
extracted shapes — new shapes arrive `unclassified` (THE todo state, note
|
||
2786), surviving shapes bump their last-seen marker, vanished shapes get
|
||
stamped rather than deleted (history is the point). Classifications survive
|
||
recompute by construction: the upsert never touches a judgment, with two
|
||
deliberate exceptions —
|
||
|
||
- a judgment whose snippet target is gone (SET NULL on snippet deletion)
|
||
is re-filed as unclassified so it rejoins the todo instead of dangling;
|
||
- a MECHANICALLY-stamped canonical row whose snippet location no longer
|
||
covers it falls back to unclassified. Only mechanical stamps self-heal;
|
||
an agent's judgment is never unwound by machinery.
|
||
|
||
`location_covers` is the one covering predicate — the same doctrine the
|
||
recorded-location drift check uses — shared by the sync's canonical marking
|
||
and by anything else that must decide whether a recorded location speaks for
|
||
an extracted shape.
|
||
"""
|
||
from __future__ import annotations
|
||
|
||
import difflib
|
||
import logging
|
||
import re
|
||
from datetime import datetime, timedelta, timezone
|
||
from typing import Iterable, NamedTuple
|
||
|
||
from sqlalchemy import select
|
||
|
||
from scribe.models import async_session
|
||
from scribe.models.code_shape import (
|
||
REASON_CODES, CodeShape, CodeShapeConsumer, CodeShapeEvent, CodeShapeUse,
|
||
)
|
||
from scribe.models.base import iso
|
||
|
||
logger = logging.getLogger(__name__)
|
||
|
||
# Statuses whose meaning requires a snippet target.
|
||
_NEEDS_TARGET = ("canonical", "instance", "variant")
|
||
|
||
|
||
def _norm_symbol(kind_or_symbol: str) -> str:
|
||
# CSS shapes and recorded CSS symbols may or may not carry the leading
|
||
# dot; compare without it so ".btn-primary" and "btn-primary" agree.
|
||
return kind_or_symbol.lstrip(".").strip()
|
||
|
||
|
||
def location_covers(loc_path: str, loc_symbol: str, path: str, name: str) -> bool:
|
||
"""Does a recorded (path, symbol) location speak for this shape?
|
||
|
||
Symbol-less locations never cover a shape — a whole-file record makes no
|
||
claim about any particular definition inside it. Path semantics are the
|
||
drift check's own: exact file, or the recorded path is a directory the
|
||
file lives under.
|
||
"""
|
||
if not (loc_symbol or "").strip():
|
||
return False
|
||
if _norm_symbol(loc_symbol) != _norm_symbol(name):
|
||
return False
|
||
if not loc_path:
|
||
# Symbol-only record: the symbol match is all the claim there is.
|
||
return True
|
||
from scribe.services.snippets import _path_touches
|
||
|
||
return _path_touches(loc_path, path)
|
||
|
||
|
||
# The rows the machine may still speak about: nobody's judgment stands on
|
||
# them. `scoped` (#2869) is the sync's own by-construction stamp — the
|
||
# proposer, derive grouping, divergence, hook evidence and sweeps treat it
|
||
# like the todo; only the human todo (`unclassified`) excludes it.
|
||
_MECHANICAL_TODO = ("unclassified", "scoped")
|
||
_SCOPED_REASON = (
|
||
"by construction: a Vue component's scoped <style> rule / <script setup> "
|
||
"function — unreachable from any other file (stamped by the coverage sync)"
|
||
)
|
||
|
||
|
||
async def sync_repo_shapes(
|
||
project_id: int,
|
||
repo_key: str,
|
||
shapes: list,
|
||
*,
|
||
seen_marker: str,
|
||
) -> None:
|
||
"""Upsert one repo's extracted shapes into the ledger.
|
||
|
||
``shapes`` are (path, kind, name) triples, or the richer ArchiveShape
|
||
records (#2792) whose 4th/5th fields — signature, body_sha — refresh the
|
||
row's content fingerprint, and whose 7th (#2869) says the shape is a
|
||
one-off by construction: such rows are stamped `scoped` (mechanical)
|
||
while unjudged, and un-stamped if a later tree makes them reachable. ``seen_marker`` is the commit the archive was
|
||
read at when the forge can say, else the ref name — provenance sugar;
|
||
the row timestamps carry the when.
|
||
"""
|
||
now = datetime.now(timezone.utc)
|
||
async with async_session() as session:
|
||
rows = (
|
||
await session.execute(
|
||
select(CodeShape).where(
|
||
CodeShape.project_id == project_id,
|
||
CodeShape.repo_key == repo_key,
|
||
)
|
||
)
|
||
).scalars().all()
|
||
by_key = {(r.path, r.symbol, r.kind): r for r in rows}
|
||
seen: set[tuple[str, str, str]] = set()
|
||
for shape in shapes:
|
||
path, kind, name = shape[0], shape[1], shape[2]
|
||
signature = shape[3] if len(shape) > 3 else ""
|
||
body_sha = shape[4] if len(shape) > 4 else ""
|
||
scoped = bool(shape[6]) if len(shape) > 6 else False
|
||
key = (path, name, kind)
|
||
if key in seen:
|
||
continue
|
||
seen.add(key)
|
||
row = by_key.get(key)
|
||
if row is None:
|
||
row = CodeShape(
|
||
project_id=project_id, repo_key=repo_key,
|
||
path=path, symbol=name, kind=kind,
|
||
first_seen_commit=seen_marker, last_seen_commit=seen_marker,
|
||
signature=signature, body_sha=body_sha,
|
||
)
|
||
if scoped:
|
||
await _judge(session, row, status="scoped", snippet_id=None,
|
||
by="mechanical", reason=_SCOPED_REASON, at=now)
|
||
else:
|
||
session.add(row)
|
||
continue
|
||
row.last_seen_commit = seen_marker
|
||
if scoped and row.status == "unclassified":
|
||
await _judge(session, row, status="scoped", snippet_id=None,
|
||
by="mechanical", reason=_SCOPED_REASON, at=now)
|
||
elif not scoped and row.status == "scoped":
|
||
await _judge(session, row, status="unclassified", snippet_id=None,
|
||
by=None, reason=None, at=now)
|
||
if signature:
|
||
row.signature = signature
|
||
if body_sha and body_sha != row.body_sha:
|
||
judged = row.status in ("instance", "variant")
|
||
if judged and not row.classified_sha:
|
||
# Judged before fingerprints existed: the first sync
|
||
# that sees a body adopts it as the judged content.
|
||
row.classified_sha = body_sha
|
||
elif judged and row.classified_sha != body_sha and row.recheck_at is None:
|
||
# The body moved under a standing judgment: the judgment
|
||
# stands, but asks to be confirmed again (#2793).
|
||
row.recheck_at = now
|
||
session.add(_event(row, "drifted", now, commit=seen_marker))
|
||
row.body_sha = body_sha
|
||
elif row.status in ("instance", "variant") and not row.classified_sha:
|
||
row.classified_sha = row.body_sha
|
||
# A shape that vanished and came back is live again — the vanish
|
||
# stays visible in history via updated_at, not as a dead flag.
|
||
if row.vanished_at is not None:
|
||
row.vanished_at = None
|
||
session.add(_event(row, "reappeared", now, commit=seen_marker))
|
||
if row.status in _NEEDS_TARGET and row.snippet_id is None:
|
||
row.status = "unclassified"
|
||
row.classified_by = None
|
||
row.classified_at = None
|
||
row.reason = None
|
||
for key, row in by_key.items():
|
||
if key not in seen and row.vanished_at is None:
|
||
row.vanished_at = now
|
||
session.add(_event(row, "vanished", now, commit=row.last_seen_commit))
|
||
await session.commit()
|
||
|
||
|
||
def _event(row: CodeShape, event: str, at: datetime, *, commit: str = "") -> CodeShapeEvent:
|
||
"""A history row for a state change on ``row`` — status/snippet/by/reason
|
||
are the row's CURRENT values, which for `classified` is the judgment
|
||
just made and for presence events is the standing one."""
|
||
return CodeShapeEvent(
|
||
shape_id=row.id, project_id=row.project_id,
|
||
path=row.path, symbol=row.symbol, kind=row.kind,
|
||
event=event, status=row.status, snippet_id=row.snippet_id,
|
||
classified_by=row.classified_by, reason=row.reason,
|
||
commit=commit or row.last_seen_commit or "", at=at,
|
||
)
|
||
|
||
|
||
async def _judge(
|
||
session, row: CodeShape, *, status: str, snippet_id: int | None,
|
||
by: str | None, reason: str | None, at: datetime, reason_code: str | None = None,
|
||
) -> None:
|
||
"""Apply a judgment to a row — the ONE place a status is set — and write
|
||
its history. Clears what a judgment settles: the standing proposal, the
|
||
recheck ask, the divergence flag; remembers the fingerprint judged.
|
||
`unclassified` is the withdrawal: fields clear, the examination is
|
||
forgotten so the proposer looks again, and history records the
|
||
withdrawal too."""
|
||
row.status = status
|
||
row.snippet_id = snippet_id if status in _NEEDS_TARGET else None
|
||
row.reason = (reason or "").strip() or None
|
||
row.reason_code = (reason_code or "").strip() or None
|
||
row.classified_by = by if status != "unclassified" else None
|
||
row.classified_at = at if status != "unclassified" else None
|
||
row.classified_sha = row.body_sha if status != "unclassified" else ""
|
||
row.recheck_at = None
|
||
row.diverges_from = None
|
||
_clear_proposal(row, reexamine=(status == "unclassified"))
|
||
if row.id is None:
|
||
# A provisional row (hook stamp on a shape not yet synced): flush so
|
||
# the event can point at it.
|
||
session.add(row)
|
||
await session.flush()
|
||
session.add(_event(row, "classified", at))
|
||
|
||
|
||
async def record_uses(
|
||
session, row: CodeShape, snippet_ids, *, basis: str, evidence: str | None = None,
|
||
) -> int:
|
||
"""Upsert consumption edges shape → snippet (#2870). A judgment-grade
|
||
basis (agent/audit/import) overwrites a mechanical one (reference/hook)
|
||
on the same edge; mechanical never overwrites a judgment. Returns the
|
||
number of edges written or refreshed. The row must be persisted (flushed)
|
||
so it has an id."""
|
||
wanted = {int(x) for x in (snippet_ids or []) if x}
|
||
if not wanted:
|
||
return 0
|
||
if row.id is None:
|
||
session.add(row)
|
||
await session.flush()
|
||
existing = {
|
||
e.snippet_id: e
|
||
for e in (
|
||
await session.execute(
|
||
select(CodeShapeUse).where(CodeShapeUse.shape_id == row.id)
|
||
)
|
||
).scalars().all()
|
||
}
|
||
judged = basis in _CALLER_VIAS
|
||
n = 0
|
||
for sid in wanted:
|
||
edge = existing.get(sid)
|
||
if edge is None:
|
||
session.add(CodeShapeUse(shape_id=row.id, snippet_id=sid, basis=basis, evidence=evidence))
|
||
n += 1
|
||
elif judged or edge.basis not in _CALLER_VIAS:
|
||
edge.basis, edge.evidence = basis, evidence
|
||
n += 1
|
||
return n
|
||
|
||
|
||
async def uses_of(shape_ids) -> dict[int, list[CodeShapeUse]]:
|
||
"""{shape_id: [edges]} for a set of rows — the read side of record_uses."""
|
||
ids = [int(x) for x in shape_ids if x]
|
||
if not ids:
|
||
return {}
|
||
async with async_session() as session:
|
||
edges = (
|
||
await session.execute(
|
||
select(CodeShapeUse).where(CodeShapeUse.shape_id.in_(ids))
|
||
.order_by(CodeShapeUse.shape_id, CodeShapeUse.snippet_id)
|
||
)
|
||
).scalars().all()
|
||
out: dict[int, list[CodeShapeUse]] = {}
|
||
for e in edges:
|
||
out.setdefault(e.shape_id, []).append(e)
|
||
return out
|
||
|
||
|
||
# --- the CSS consumer map (milestone 302) ------------------------------------
|
||
|
||
|
||
def resolve_consumers(
|
||
css_rows: Iterable[tuple[int, str, str]],
|
||
references: dict[str, dict[str, int]],
|
||
) -> dict[tuple[int, str], int]:
|
||
"""{(shape_id, consumer_path): count} — which CSS rows each file's markup
|
||
consumes. ``css_rows`` are (id, path, symbol) of the repo's live css rows;
|
||
``references`` is scan_archive's path → class token → count.
|
||
|
||
Resolution (note 2917): a class named in file F resolves to F's OWN row
|
||
of that name when F defines it (a scoped rule is consumed by its own
|
||
template); otherwise to every other file's row of that name — a shared
|
||
sheet, or, when several files define it, all of them: the map says
|
||
"ambiguous" by fanning out rather than guessing one.
|
||
|
||
A token ending in ``PREFIX_MARK`` is a PREFIX reference (#2970) — the
|
||
static head of a name the template concatenates, `status-*` from
|
||
`` `status-${s}` ``. It stands for every row whose symbol starts with
|
||
that head, each resolved by the same own-file-else-fan-out rule. The
|
||
template cannot tell us WHICH of them it built, so the map credits all
|
||
of them rather than calling live rules unused."""
|
||
# Lazy, like the extract_definitions import below: coverage reaches into
|
||
# this module during a refresh, so neither may import the other at load.
|
||
from scribe.services.coverage import PREFIX_MARK
|
||
|
||
by_symbol: dict[str, list[tuple[int, str]]] = {}
|
||
for sid, path, symbol in css_rows:
|
||
by_symbol.setdefault(symbol, []).append((sid, path))
|
||
out: dict[tuple[int, str], int] = {}
|
||
|
||
def credit(rows: list[tuple[int, str]], consumer: str, count: int) -> None:
|
||
own = [sid for sid, path in rows if path == consumer]
|
||
for sid in own or [sid for sid, _path in rows]:
|
||
out[(sid, consumer)] = out.get((sid, consumer), 0) + int(count)
|
||
|
||
for consumer, tokens in references.items():
|
||
for token, count in tokens.items():
|
||
if token.endswith(PREFIX_MARK):
|
||
head = token[: -len(PREFIX_MARK)]
|
||
for symbol, rows in by_symbol.items():
|
||
if symbol.startswith(head):
|
||
credit(rows, consumer, count)
|
||
continue
|
||
rows = by_symbol.get(token)
|
||
if rows:
|
||
credit(rows, consumer, count)
|
||
return out
|
||
|
||
|
||
async def sync_repo_consumers(
|
||
project_id: int, repo_key: str, references: dict[str, dict[str, int]]
|
||
) -> int:
|
||
"""Rebuild one repo's consumer edges from its archive's class references:
|
||
insert the new, refresh changed counts, delete what the tree no longer
|
||
says (a template rewritten, a class renamed, a file gone). Edges hang on
|
||
live rows only; a vanished row's edges go with this pass. Returns how
|
||
many edges stand afterwards."""
|
||
async with async_session() as session:
|
||
rows = (
|
||
await session.execute(
|
||
select(CodeShape.id, CodeShape.path, CodeShape.symbol, CodeShape.vanished_at).where(
|
||
CodeShape.project_id == project_id,
|
||
CodeShape.repo_key == repo_key,
|
||
CodeShape.kind == "css",
|
||
)
|
||
)
|
||
).all()
|
||
live = [(r[0], r[1], r[2]) for r in rows if r[3] is None]
|
||
all_ids = [r[0] for r in rows]
|
||
wanted = resolve_consumers(live, references)
|
||
existing = (
|
||
await session.execute(
|
||
select(CodeShapeConsumer).where(CodeShapeConsumer.shape_id.in_(all_ids))
|
||
)
|
||
).scalars().all() if all_ids else []
|
||
have = {(e.shape_id, e.path): e for e in existing}
|
||
for key, edge in have.items():
|
||
if key not in wanted:
|
||
await session.delete(edge)
|
||
elif edge.count != wanted[key]:
|
||
edge.count = wanted[key]
|
||
for (sid, path), count in wanted.items():
|
||
if (sid, path) not in have:
|
||
session.add(CodeShapeConsumer(shape_id=sid, path=path, count=count, basis="template"))
|
||
await session.commit()
|
||
return len(wanted)
|
||
|
||
|
||
async def consumers_of(shape_ids) -> dict[int, list[CodeShapeConsumer]]:
|
||
"""{shape_id: [edges]} for a set of rows — the read side of the map,
|
||
ordered by path so a readout is stable."""
|
||
ids = [int(x) for x in shape_ids if x]
|
||
if not ids:
|
||
return {}
|
||
async with async_session() as session:
|
||
edges = (
|
||
await session.execute(
|
||
select(CodeShapeConsumer).where(CodeShapeConsumer.shape_id.in_(ids))
|
||
.order_by(CodeShapeConsumer.shape_id, CodeShapeConsumer.path)
|
||
)
|
||
).scalars().all()
|
||
out: dict[int, list[CodeShapeConsumer]] = {}
|
||
for e in edges:
|
||
out.setdefault(e.shape_id, []).append(e)
|
||
return out
|
||
|
||
|
||
# How many consumer files a readout names before "+N more".
|
||
_CONSUMERS_SHOWN = 4
|
||
|
||
|
||
def consumer_summary(paths: Iterable[str]) -> dict:
|
||
"""{"count", "paths"} — distinct consumer files, sorted, the first few
|
||
named. The one shape every surface uses for "used by N template(s)"."""
|
||
files = sorted(set(paths))
|
||
return {"count": len(files), "paths": files[:_CONSUMERS_SHOWN]}
|
||
|
||
|
||
async def used_by_map(rows: Iterable[CodeShape]) -> dict[int, dict]:
|
||
"""{shape_id: consumer_summary} for every css row given — a row with no
|
||
consumer gets {"count": 0, "paths": []}: "no template names it" is a
|
||
finding, not an absence."""
|
||
css = [r for r in rows if r.kind == "css"]
|
||
if not css:
|
||
return {}
|
||
edges = await consumers_of([r.id for r in css])
|
||
return {r.id: consumer_summary(e.path for e in edges.get(r.id, [])) for r in css}
|
||
|
||
|
||
async def mark_canonicals(
|
||
project_id: int, recorded: list[tuple[int, str, str]]
|
||
) -> None:
|
||
"""Stamp snippet reference locations as `canonical` — the one mechanical
|
||
rule that is always safe (the judgment happened when the snippet was
|
||
minted; this row just makes it queryable).
|
||
|
||
``recorded`` is (snippet_note_id, path, symbol) for every live snippet
|
||
location in the project. Touches only rows machinery owns: unclassified
|
||
rows gain the stamp; mechanically-stamped canonicals no longer covered
|
||
fall back to unclassified. Agent judgments are never overwritten.
|
||
"""
|
||
usable = [(nid, p, s) for nid, p, s in recorded if (s or "").strip()]
|
||
now = datetime.now(timezone.utc)
|
||
async with async_session() as session:
|
||
rows = (
|
||
await session.execute(
|
||
select(CodeShape).where(
|
||
CodeShape.project_id == project_id,
|
||
CodeShape.vanished_at.is_(None),
|
||
)
|
||
)
|
||
).scalars().all()
|
||
for row in rows:
|
||
covering = next(
|
||
(
|
||
nid for nid, lp, ls in usable
|
||
if location_covers(lp, ls, row.path, row.symbol)
|
||
),
|
||
None,
|
||
)
|
||
if covering is not None and row.status in _MECHANICAL_TODO:
|
||
await _judge(session, row, status="canonical", snippet_id=covering,
|
||
by="mechanical", reason=None, at=now)
|
||
elif (
|
||
covering is None
|
||
and row.status == "canonical"
|
||
and row.classified_by == "mechanical"
|
||
):
|
||
await _judge(session, row, status="unclassified", snippet_id=None,
|
||
by=None, reason=None, at=now)
|
||
await session.commit()
|
||
|
||
|
||
async def live_rows_for(user_id: int, project_id: int) -> list[CodeShape]:
|
||
"""`live_rows` behind the project read gate — for callers that arrive from
|
||
outside (an MCP tool, a route) rather than from a job that already
|
||
established who is asking. Empty for a project the caller cannot read,
|
||
never a partial answer."""
|
||
# Deferred, like every other access import in this module: services/access
|
||
# reaches back here, and a module-level import closes the cycle.
|
||
from scribe.services import access
|
||
|
||
if not await access.can_read_project(user_id, project_id):
|
||
return []
|
||
return await live_rows(project_id)
|
||
|
||
|
||
async def live_rows(project_id: int) -> list[CodeShape]:
|
||
"""Every un-vanished ledger row for a project — the accounting readout's
|
||
input, across ALL its repos (a repo unreachable this refresh still counts;
|
||
accounting is project-wide)."""
|
||
async with async_session() as session:
|
||
return list(
|
||
(
|
||
await session.execute(
|
||
select(CodeShape).where(
|
||
CodeShape.project_id == project_id,
|
||
CodeShape.vanished_at.is_(None),
|
||
)
|
||
)
|
||
).scalars().all()
|
||
)
|
||
|
||
|
||
# --- classification (#2789): the judgment write path --------------------------
|
||
|
||
# Statuses an explicit classification may set. All five: setting a row back to
|
||
# `unclassified` is how a judgment is deliberately withdrawn.
|
||
_SETTABLE = ("canonical", "instance", "variant", "exempt", "unclassified")
|
||
|
||
# Who may appear as the classifier on this path. `hook` and `mechanical` are
|
||
# server-internal feeds (steps 5-6) — a caller claiming them would launder a
|
||
# judgment as machinery.
|
||
_CALLER_VIAS = ("agent", "audit", "import")
|
||
|
||
|
||
def validate_classifications(items: list[dict]) -> str | None:
|
||
"""The structural error a classification batch would earn, or None.
|
||
|
||
Pure and checked BEFORE anything is touched: a batch either applies or
|
||
errors whole — the StrictArgs lesson (#2709), a caller must never learn
|
||
later that half a batch silently happened.
|
||
"""
|
||
if not items:
|
||
return "classifications is empty — nothing to apply"
|
||
for i, item in enumerate(items):
|
||
if not isinstance(item, dict):
|
||
return f"classifications[{i}] is not an object"
|
||
path = (item.get("path") or "").strip()
|
||
symbol = (item.get("symbol") or "").strip()
|
||
if not path or not symbol:
|
||
return f"classifications[{i}] needs both path and symbol"
|
||
status = item.get("status") or ""
|
||
if status not in _SETTABLE:
|
||
return (
|
||
f"classifications[{i}] has unknown status {status!r} "
|
||
f"(one of: {', '.join(_SETTABLE)})"
|
||
)
|
||
snippet_id = item.get("snippet_id") or 0
|
||
if status in _NEEDS_TARGET and not snippet_id:
|
||
return (
|
||
f"classifications[{i}]: status {status!r} needs snippet_id — "
|
||
"the snippet this shape is (or departs from)"
|
||
)
|
||
if status in ("variant", "exempt") and not (item.get("reason") or "").strip():
|
||
return (
|
||
f"classifications[{i}]: status {status!r} needs a reason — "
|
||
"the WHY is the record (note 2786)"
|
||
)
|
||
code = (item.get("reason_code") or "").strip()
|
||
if code and code not in REASON_CODES:
|
||
return (
|
||
f"classifications[{i}]: unknown reason_code {code!r} "
|
||
f"(one of: {', '.join(REASON_CODES)})"
|
||
)
|
||
uses = item.get("uses")
|
||
if uses is not None and (
|
||
not isinstance(uses, list) or not all(isinstance(u, int) and u > 0 for u in uses)
|
||
):
|
||
return f"classifications[{i}]: uses must be a list of snippet ids"
|
||
return None
|
||
|
||
|
||
async def classify_shapes(
|
||
user_id: int,
|
||
project_id: int,
|
||
classifications: list[dict],
|
||
*,
|
||
via: str = "agent",
|
||
) -> dict:
|
||
"""Apply a batch of judgments to a project's live ledger rows.
|
||
|
||
All-or-nothing on errors: the whole batch is validated (structure, write
|
||
access, every snippet target readable by the caller) before any row is
|
||
touched. Rows are matched by exact (path, symbol) — plus kind when the
|
||
item carries one — and a target no live row matches is reported in
|
||
``unmatched``, not an error: the tree may simply have moved since the
|
||
caller listed. Idempotent by construction.
|
||
"""
|
||
from scribe.services import access
|
||
from scribe.services import snippets as snippets_svc
|
||
|
||
if via not in _CALLER_VIAS:
|
||
raise ValueError(f"via must be one of: {', '.join(_CALLER_VIAS)}")
|
||
error = validate_classifications(classifications)
|
||
if error:
|
||
raise ValueError(error)
|
||
if not await access.can_write_project(user_id, project_id):
|
||
raise ValueError(f"project {project_id} not found or no write access")
|
||
|
||
# Snippet targets resolve through the caller's own read access — a
|
||
# family-canon snippet in another project counts (note 2786), a snippet
|
||
# the caller cannot read does not exist for them.
|
||
target_ids = {
|
||
int(item["snippet_id"])
|
||
for item in classifications
|
||
if item.get("status") in _NEEDS_TARGET
|
||
}
|
||
for item in classifications:
|
||
target_ids.update(int(u) for u in (item.get("uses") or []))
|
||
for sid in sorted(target_ids):
|
||
if await snippets_svc.get_snippet(user_id, sid) is None:
|
||
raise ValueError(f"snippet {sid} not found (or not readable)")
|
||
|
||
now = datetime.now(timezone.utc)
|
||
classified = 0
|
||
unmatched: list[dict] = []
|
||
async with async_session() as session:
|
||
rows = (
|
||
await session.execute(
|
||
select(CodeShape).where(
|
||
CodeShape.project_id == project_id,
|
||
CodeShape.vanished_at.is_(None),
|
||
)
|
||
)
|
||
).scalars().all()
|
||
by_key: dict[tuple[str, str], list[CodeShape]] = {}
|
||
for row in rows:
|
||
by_key.setdefault((row.path, row.symbol), []).append(row)
|
||
for item in classifications:
|
||
matches = by_key.get(
|
||
((item.get("path") or "").strip(), (item.get("symbol") or "").strip())
|
||
) or []
|
||
kind = (item.get("kind") or "").strip()
|
||
if kind:
|
||
matches = [r for r in matches if r.kind == kind]
|
||
if not matches:
|
||
unmatched.append({
|
||
"path": item.get("path"), "symbol": item.get("symbol"),
|
||
})
|
||
continue
|
||
status = item["status"]
|
||
for row in matches:
|
||
await _judge(
|
||
session, row, status=status,
|
||
snippet_id=int(item["snippet_id"]) if status in _NEEDS_TARGET else None,
|
||
by=via, reason=item.get("reason"), at=now,
|
||
reason_code=item.get("reason_code"),
|
||
)
|
||
if item.get("uses"):
|
||
await record_uses(session, row, item["uses"], basis=via,
|
||
evidence=item.get("reason"))
|
||
classified += 1
|
||
await session.commit()
|
||
return {"classified": classified, "unmatched": unmatched}
|
||
|
||
|
||
def rule_matches(row: CodeShape, *, path: str, pattern: str, kind: str) -> bool:
|
||
"""Does a ledger row fall under a rule-form classification (#2868)?
|
||
``path`` is a file or a directory (everything beneath it), ``pattern``
|
||
a shell glob on the symbol (``""`` = every symbol), ``kind`` narrows to
|
||
sym/css. Pure, so the sweep's reach can be tested without a database."""
|
||
import fnmatch
|
||
|
||
clean = (path or "").strip().strip("/")
|
||
if clean and not (row.path == clean or row.path.startswith(clean + "/")):
|
||
return False
|
||
if kind and row.kind != kind:
|
||
return False
|
||
if pattern and not fnmatch.fnmatchcase(_norm_symbol(row.symbol), pattern):
|
||
return False
|
||
return True
|
||
|
||
|
||
async def classify_shapes_where(
|
||
user_id: int,
|
||
project_id: int,
|
||
*,
|
||
path: str,
|
||
status: str,
|
||
pattern: str = "",
|
||
kind: str = "",
|
||
snippet_id: int | None = None,
|
||
reason: str | None = None,
|
||
via: str = "agent",
|
||
include_judged: bool = False,
|
||
reason_code: str | None = None,
|
||
uses: list[int] | None = None,
|
||
) -> dict:
|
||
"""The sweep form of classify_shapes (#2868): one judgment applied to
|
||
every live row under ``path`` whose symbol matches ``pattern`` (and
|
||
``kind``). By default only unjudged rows are touched — `unclassified`
|
||
and the sync's mechanical `scoped` stamp — a sweep must never silently
|
||
overwrite a judgment; ``include_judged`` opts in.
|
||
Same gates as the row form (status vocabulary, snippet target, reason
|
||
for variant/exempt, write access); one transaction, so it applies whole
|
||
or not at all. Returns the count and a sample of what it judged."""
|
||
from scribe.services import access
|
||
from scribe.services import snippets as snippets_svc
|
||
|
||
if via not in _CALLER_VIAS:
|
||
raise ValueError(f"via must be one of: {', '.join(_CALLER_VIAS)}")
|
||
if not (path or "").strip():
|
||
raise ValueError("path is required — a sweep names the directory it judges")
|
||
if status == "canonical":
|
||
raise ValueError("canonical is the sync's stamp on a snippet's own location — a sweep cannot set it")
|
||
probe = {"path": path, "symbol": "*", "status": status,
|
||
"snippet_id": snippet_id or 0, "reason": reason or "",
|
||
"reason_code": reason_code or "", "uses": uses}
|
||
error = validate_classifications([probe])
|
||
if error:
|
||
raise ValueError(error.replace("classifications[0]", "rule"))
|
||
if not await access.can_write_project(user_id, project_id):
|
||
raise ValueError(f"project {project_id} not found or no write access")
|
||
if status in _NEEDS_TARGET and await snippets_svc.get_snippet(user_id, int(snippet_id)) is None:
|
||
raise ValueError(f"snippet {snippet_id} not found (or not readable)")
|
||
for sid in sorted({int(u) for u in (uses or [])}):
|
||
if await snippets_svc.get_snippet(user_id, sid) is None:
|
||
raise ValueError(f"snippet {sid} not found (or not readable)")
|
||
|
||
now = datetime.now(timezone.utc)
|
||
judged: list[str] = []
|
||
async with async_session() as session:
|
||
conds = [CodeShape.project_id == project_id, CodeShape.vanished_at.is_(None)]
|
||
if not include_judged:
|
||
conds.append(CodeShape.status.in_(_MECHANICAL_TODO))
|
||
rows = (await session.execute(select(CodeShape).where(*conds))).scalars().all()
|
||
for row in rows:
|
||
if not rule_matches(row, path=path, pattern=pattern, kind=kind):
|
||
continue
|
||
await _judge(
|
||
session, row, status=status,
|
||
snippet_id=int(snippet_id) if status in _NEEDS_TARGET else None,
|
||
by=via, reason=reason, at=now, reason_code=reason_code,
|
||
)
|
||
if uses:
|
||
await record_uses(session, row, uses, basis=via, evidence=reason)
|
||
judged.append(f"{row.path}::{row.symbol}")
|
||
await session.commit()
|
||
return {"classified": len(judged), "sample": sorted(judged)[:12]}
|
||
|
||
|
||
async def list_project_shapes(
|
||
user_id: int,
|
||
project_id: int,
|
||
*,
|
||
status: str = "",
|
||
path: str = "",
|
||
snippet_id: int = 0,
|
||
include_vanished: bool = False,
|
||
limit: int = 100,
|
||
offset: int = 0,
|
||
proposal: str = "",
|
||
flag: str = "",
|
||
uses: int = 0,
|
||
) -> tuple[list[CodeShape], int]:
|
||
"""A filtered page of a project's ledger, with the unfiltered-match total.
|
||
|
||
([], 0) when the caller can't read the project — the same silence every
|
||
other project list gives. ``path`` matches the exact file or anything
|
||
beneath it, mirroring recorded-location semantics. ``proposal`` narrows
|
||
to rows the proposer has spoken about: "any", "canon" (an instance-of-#N
|
||
suggestion), "derive" (a repeats-with-no-canon group), or one basis
|
||
name (symbol/reference/text/signature/semantic). ``flag`` narrows to
|
||
the readout's asks (#2793): "divergence" (new where a canon dominates,
|
||
`diverges_from` names it), "recheck" (a judged shape whose body moved),
|
||
or "unused-css" (milestone 302: a css rule no template names).
|
||
"""
|
||
from sqlalchemy import func, or_
|
||
|
||
from scribe.services import access
|
||
|
||
if not await access.can_read_project(user_id, project_id):
|
||
return [], 0
|
||
conds = [CodeShape.project_id == project_id]
|
||
if not include_vanished:
|
||
conds.append(CodeShape.vanished_at.is_(None))
|
||
if status:
|
||
conds.append(CodeShape.status == status)
|
||
if path:
|
||
clean = path.strip().strip("/")
|
||
conds.append(or_(
|
||
CodeShape.path == clean, CodeShape.path.like(clean + "/%")
|
||
))
|
||
if snippet_id:
|
||
conds.append(CodeShape.snippet_id == snippet_id)
|
||
if proposal == "any":
|
||
conds.append(or_(
|
||
CodeShape.proposed_snippet_id.isnot(None),
|
||
CodeShape.proposal_group.isnot(None),
|
||
))
|
||
elif proposal == "canon":
|
||
conds.append(CodeShape.proposed_snippet_id.isnot(None))
|
||
elif proposal == "derive":
|
||
conds.append(CodeShape.proposal_group.isnot(None))
|
||
elif proposal:
|
||
conds.append(CodeShape.proposal_basis == proposal)
|
||
if flag == "divergence":
|
||
conds.append(CodeShape.diverges_from.isnot(None))
|
||
elif flag == "recheck":
|
||
conds.append(CodeShape.recheck_at.isnot(None))
|
||
elif flag == "unused-css":
|
||
# The consumer map's negative space (milestone 302): a live css rule
|
||
# no file's markup names. A candidate for deletion, surfaced — never
|
||
# deleted — because the map reads templates only (a class built at
|
||
# runtime, or used from a script, is invisible to it).
|
||
conds.append(CodeShape.kind == "css")
|
||
conds.append(~CodeShape.id.in_(select(CodeShapeConsumer.shape_id)))
|
||
if uses:
|
||
# Consumers of a canon (#2870): rows with a uses edge to it, whatever
|
||
# shape they themselves are.
|
||
conds.append(CodeShape.id.in_(
|
||
select(CodeShapeUse.shape_id).where(CodeShapeUse.snippet_id == uses)
|
||
))
|
||
async with async_session() as session:
|
||
total = (
|
||
await session.execute(
|
||
select(func.count()).select_from(CodeShape).where(*conds)
|
||
)
|
||
).scalar_one()
|
||
rows = (
|
||
await session.execute(
|
||
select(CodeShape).where(*conds)
|
||
.order_by(CodeShape.path, CodeShape.symbol, CodeShape.kind)
|
||
.limit(max(1, min(limit, 500))).offset(max(0, offset))
|
||
)
|
||
).scalars().all()
|
||
return list(rows), int(total)
|
||
|
||
|
||
def _consumer_dict(row: CodeShape) -> dict:
|
||
"""The compact shape a snippet's consumer map carries — enough to open
|
||
the file, none of the ledger bookkeeping."""
|
||
out = {
|
||
"project_id": row.project_id,
|
||
"repo": row.repo_key,
|
||
"path": row.path,
|
||
"symbol": row.symbol,
|
||
"kind": row.kind,
|
||
"classified_by": row.classified_by,
|
||
}
|
||
if row.reason:
|
||
out["reason"] = row.reason
|
||
return out
|
||
|
||
|
||
async def snippet_consumers(user_id: int, note_id: int) -> dict:
|
||
"""The structured consumer map for one snippet (#2789): its `instances`
|
||
(rows judged to conform) and `variants` (named departures, each carrying
|
||
its why). Rows are filtered to projects the CALLER can read — a shared
|
||
snippet must not become a side channel into someone else's project
|
||
layout. Empty lists mean "attach nothing" (#2483)."""
|
||
from scribe.services import access
|
||
|
||
async with async_session() as session:
|
||
rows = (
|
||
await session.execute(
|
||
select(CodeShape).where(
|
||
CodeShape.snippet_id == note_id,
|
||
CodeShape.status.in_(("instance", "variant")),
|
||
CodeShape.vanished_at.is_(None),
|
||
)
|
||
)
|
||
).scalars().all()
|
||
# The consumption edges (#2870): rows that USE this canon, whatever
|
||
# shape they are themselves — the call-site map.
|
||
using = (
|
||
await session.execute(
|
||
select(CodeShape, CodeShapeUse.basis, CodeShapeUse.evidence)
|
||
.join(CodeShapeUse, CodeShapeUse.shape_id == CodeShape.id)
|
||
.where(CodeShapeUse.snippet_id == note_id, CodeShape.vanished_at.is_(None))
|
||
)
|
||
).all()
|
||
readable: dict[int, bool] = {}
|
||
|
||
async def can_read(pid: int) -> bool:
|
||
if pid not in readable:
|
||
readable[pid] = await access.can_read_project(user_id, pid)
|
||
return readable[pid]
|
||
|
||
out: dict[str, list[dict]] = {"instances": [], "variants": [], "uses": []}
|
||
for row in rows:
|
||
if not await can_read(row.project_id):
|
||
continue
|
||
out["instances" if row.status == "instance" else "variants"].append(
|
||
_consumer_dict(row)
|
||
)
|
||
for row, basis, evidence in using:
|
||
if not await can_read(row.project_id):
|
||
continue
|
||
d = _consumer_dict(row)
|
||
d["basis"] = basis
|
||
if evidence:
|
||
d["evidence"] = evidence
|
||
d.pop("reason", None)
|
||
out["uses"].append(d)
|
||
return out
|
||
|
||
|
||
# --- write-path stamping (#2791): hook evidence lands as rows ----------------
|
||
#
|
||
# The write-path hook (plugin/hooks/scribe_prior_art.sh) fires on every
|
||
# Write/Edit and already carries the two halves of a consumer-map row: the
|
||
# file being written and the definitions in (or enclosing) the payload. What
|
||
# it could not say on its own is WHICH canon the session is instantiating.
|
||
# The pull stream answers that: a snippet the session opened in full
|
||
# (get_snippet) and is now writing code that references or resembles is being
|
||
# reused — and a reused canon's call site is an `instance` (note 2786).
|
||
#
|
||
# The rule, deliberately two-sided so it cannot fire on noise:
|
||
# PULLED — a PULLED usage event by this user inside PULL_WINDOW. Offered-
|
||
# but-ignored (surfaced, never opened) stamps nothing.
|
||
# IN PLAY — the payload references the snippet's symbol by name, or the
|
||
# semantic arm scored it above the write-path threshold for this
|
||
# very payload. Either is evidence; the pull alone is not.
|
||
# Both hold → every shape the hook named at that path, of the snippet's kind,
|
||
# becomes instance-of-N with classified_by="hook" and the evidence as reason.
|
||
#
|
||
# A hook row is EVIDENCE, not judgment: it only ever lands on rows nobody has
|
||
# judged (unclassified) or rows an earlier hook stamped, never on a canonical
|
||
# row or an agent/audit/import judgment. Re-judge with classify_shapes.
|
||
|
||
# "The write path actually pulled it": a working session's reach. The
|
||
# precision comes from the in-play test above, not from this window.
|
||
PULL_WINDOW = timedelta(hours=6)
|
||
|
||
# ── WHAT KIND OF THING a shape is, past `css | sym` (#4204) ──────────────
|
||
#
|
||
# THE PROBLEM THIS EXISTS FOR. `kind` has exactly two values, so a frozen
|
||
# dataclass, a module constant, a sync predicate, a class and an async service
|
||
# function are all siblings of one another. Nothing downstream could possibly
|
||
# discriminate on a distinction the column does not carry — which is how a
|
||
# directory base rate ended up standing alone as the entire divergence
|
||
# argument. It was not chosen over a better signal; it was the only signal
|
||
# there was. Measured consequence: writing a registry module of pure helpers
|
||
# produced five prompts to build them from the `async_session` service canon,
|
||
# and on another project one snippet had been recorded as the canon for a
|
||
# `class`, a plain `def`, an `async def` and a dozen test functions at once.
|
||
#
|
||
# DERIVED FROM THE SIGNATURE ON READ, not stored. `kind` is part of the row
|
||
# identity — ("project_id", "repo_key", "path", "symbol", "kind") — so
|
||
# widening that column rewrites every row's key and needs a migration plus a
|
||
# re-extract. Reading the form off the signature costs nothing, needs no
|
||
# migration, and is reversible. Widening `kind` remains the principled fix and
|
||
# this does not foreclose it: every caller below asks `shape_form`, so the day
|
||
# the column carries the answer, this function returns it instead.
|
||
#
|
||
# DELIBERATELY COARSE, and it must stay that way. It answers "could these
|
||
# plausibly be the same kind of thing", never "what language construct is
|
||
# this". A finer taxonomy would need per-language parsing and would start
|
||
# disagreeing with itself across the four languages this ledger already holds
|
||
# (Python, Go, TypeScript, Vue SFC) — and a classifier that is wrong in a new
|
||
# way is worse than the coarse one it replaced.
|
||
FORM_UNKNOWN = ""
|
||
|
||
# How much a canon's own instances must agree before it may assert a form.
|
||
# A canon whose rows disagree about what they are cannot tell anyone else what
|
||
# to be — and that disagreement is the SYMPTOM of the bad stamping this same
|
||
# change fixes, so reading it as "no opinion" makes the two halves cooperate:
|
||
# a canon poisoned by loose stamping falls silent instead of flagging.
|
||
_FORM_SHARE = 0.6
|
||
|
||
# How much of a canon's membership may be shapes it cannot account for
|
||
# before the canon stops meaning anything. Not a majority test — see
|
||
# `canon_coherence` for why a majority test goes blind exactly when the
|
||
# ledger is worst.
|
||
_STRANGER_SHARE = 0.2
|
||
|
||
# Leading words that say nothing about the form of the thing being declared.
|
||
_FORM_NOISE = ("export ", "default ", "public ", "private ", "static ", "final ")
|
||
|
||
# A declared TYPE, across the languages in play. Coarse on purpose: a Go
|
||
# struct, a TS interface and a Python class are one bucket because the
|
||
# question is only ever "is the other thing also a type".
|
||
_FORM_TYPE_WORDS = ("class ", "interface ", "type ", "struct ", "enum ")
|
||
_FORM_FN_WORDS = ("def ", "function ", "func ", "fn ", "sub ")
|
||
|
||
# THE DECLARING KEYWORD IS OPTIONAL, because Python has none. A module
|
||
# constant is written `MAX_BUDGET = 10` — no `const`, no `let` — and a
|
||
# pattern that required one classified every Python constant as unreadable,
|
||
# which then read downstream as "do not assert" and quietly excluded a whole
|
||
# form from both checks. Caught by the payload test, not by review.
|
||
_FORM_BINDING = re.compile(r"^(?:(?:const|let|var)\s+)?[\w$]+\s*(?::[^=]+)?=")
|
||
_FORM_ARROW = re.compile(
|
||
r"^(?:(?:const|let|var)\s+)?[\w$]+\s*(?::[^=]+)?=\s*(async\s*)?\("
|
||
)
|
||
|
||
|
||
def shape_form(signature: str, kind: str = "sym") -> str:
|
||
"""The structural form of a shape: css / type / async-fn / fn / binding.
|
||
|
||
Returns FORM_UNKNOWN when the signature does not say, and every caller
|
||
treats that as "do not assert", never as "no match". An unreadable
|
||
signature must make this quieter, not more confident.
|
||
"""
|
||
if kind == "css":
|
||
return "css"
|
||
sig = (signature or "").strip()
|
||
if not sig:
|
||
return FORM_UNKNOWN
|
||
changed = True
|
||
while changed:
|
||
changed = False
|
||
for lead in _FORM_NOISE:
|
||
if sig.startswith(lead):
|
||
sig, changed = sig[len(lead):].lstrip(), True
|
||
if sig.startswith(_FORM_TYPE_WORDS):
|
||
return "type"
|
||
if sig.startswith("async "):
|
||
return "async-fn"
|
||
if sig.startswith(_FORM_FN_WORDS):
|
||
return "fn"
|
||
m = _FORM_ARROW.match(sig)
|
||
if m:
|
||
return "async-fn" if m.group(1) else "fn"
|
||
if _FORM_BINDING.match(sig):
|
||
return "binding"
|
||
return FORM_UNKNOWN
|
||
|
||
|
||
def forms_agree(a: str, b: str) -> bool:
|
||
"""Do two forms match well enough to assert a relationship?
|
||
|
||
BOTH must be known. "Unknown equals unknown" would make two shapes nobody
|
||
can read into a confident pair, which is the failure this whole change is
|
||
about — a guess dressed as a finding.
|
||
"""
|
||
return bool(a) and bool(b) and a == b
|
||
|
||
|
||
def forms_conflict(a: str, b: str) -> bool:
|
||
"""Do two KNOWN forms rule each other out?
|
||
|
||
The weaker sibling of `forms_agree`, and the pair exists because the two
|
||
kinds of evidence this ledger acts on deserve different burdens.
|
||
|
||
An explicit by-name reference — the payload literally names the canon's
|
||
symbol — is strong, so it needs only the absence of a contradiction: stamp
|
||
unless the forms are both readable and different. A payload-level
|
||
RESEMBLANCE SCORE is weak, computed against the whole file, so one score
|
||
speaks for every symbol in it; that needs positive agreement before
|
||
asserting anything, which is `forms_agree`.
|
||
|
||
Demanding agreement everywhere was the first version of this and it was
|
||
wrong: it silenced the by-name path whenever a shape's definition was not
|
||
in the payload — an Edit rather than a Write — turning strong evidence
|
||
into no evidence for a reason that has nothing to do with the code.
|
||
"""
|
||
return bool(a) and bool(b) and a != b
|
||
|
||
|
||
def shape_family(form: str) -> str:
|
||
"""The coarse category a form belongs to, or "" when the form is unknown.
|
||
|
||
WHY THIS IS SEPARATE FROM `shape_form`, and it is the correction to the
|
||
first version of this change. The two checks in this module ask opposite
|
||
questions, so they cannot share a burden:
|
||
|
||
STAMPING asserts "this IS that canon". Agreement in form is evidence
|
||
FOR the claim, so `forms_agree` / `forms_conflict` — the precise level,
|
||
where `fn` and `async-fn` are different — is right.
|
||
|
||
DIVERGENCE asserts "this is NOT the canon that dominates here — did you
|
||
mean to?" A form MISMATCH is the premise of that prompt, not an
|
||
objection to it. Gating it on `forms_agree` inverted the check: it went
|
||
silent on exactly the mismatches it exists to catch, and the acceptance
|
||
case of #2793 — a hand-rolled sync `confirmDanger` in a directory where
|
||
an async confirm helper is canon — stopped being flagged.
|
||
|
||
So divergence gates at FAMILY level and only on contradiction. A sync
|
||
function beside an async one is still a fair question. A frozen dataclass
|
||
told to build from an async service function is not a question at all,
|
||
and that is the #4204 prompt this removes.
|
||
|
||
This does NOT remove every false prompt #4204 recorded. Of the five, it
|
||
silences `Point` (a type, against a callable canon); `_p`, `get_point`,
|
||
`is_registered` and `sources_expected_to_emit` are callables like the
|
||
canon and still ask. At the signature level they are indistinguishable
|
||
from the #2793 case above, so nothing readable here can separate them —
|
||
only widening `kind` past `css | sym` (#4204 option 2) or comparing
|
||
meaning rather than form can. Stated here so the next reader does not
|
||
assume the gap is an oversight.
|
||
"""
|
||
if form in ("fn", "async-fn"):
|
||
return "callable"
|
||
if form == "type":
|
||
return "type"
|
||
if form == "binding":
|
||
return "value"
|
||
if form == "css":
|
||
return "css"
|
||
return ""
|
||
|
||
|
||
def families_conflict(a: str, b: str) -> bool:
|
||
"""Do two forms belong to categorically different KNOWN families?
|
||
|
||
Like `forms_conflict`, both sides must be readable: an unknown form makes
|
||
this quieter, never more confident.
|
||
"""
|
||
fa, fb = shape_family(a), shape_family(b)
|
||
return bool(fa) and bool(fb) and fa != fb
|
||
|
||
|
||
def canon_form(rows: Iterable, snippet_id: int) -> str:
|
||
"""The form a canon's own judged rows agree on, or FORM_UNKNOWN."""
|
||
forms: dict[str, int] = {}
|
||
for r in rows:
|
||
if r.snippet_id != snippet_id or r.status not in ("canonical", "instance"):
|
||
continue
|
||
f = shape_form(getattr(r, "signature", "") or "", r.kind)
|
||
if f:
|
||
forms[f] = forms.get(f, 0) + 1
|
||
if not forms:
|
||
return FORM_UNKNOWN
|
||
top, n = max(forms.items(), key=lambda kv: kv[1])
|
||
return top if n / sum(forms.values()) >= _FORM_SHARE else FORM_UNKNOWN
|
||
|
||
|
||
# A judgment nobody read before it was written. "hook" is the write-path
|
||
# stamp — a similarity score and no reader. Every other value ("agent",
|
||
# "audit", "import", "mechanical") came from something that examined the
|
||
# shape and said so, which is a different kind of evidence, not a stronger
|
||
# score.
|
||
_UNATTENDED_BY = ("hook",)
|
||
|
||
|
||
def canon_coherence(members: Iterable) -> dict:
|
||
"""Does a canon's membership still agree on what the canon IS?
|
||
|
||
The verdict is NOT "do all the rows share a form". That was the first
|
||
version, and on this surface's first live day both canons it reported
|
||
were sound (#4220) while the one genuinely polluted canon had already
|
||
been cleaned by hand. Two things were wrong with it, and the fix for the
|
||
first nearly broke the second:
|
||
|
||
1. FAMILY, NOT FORM. A sync loop-starter and the async tick it schedules
|
||
are one shape written two ways; `shape_family` already collapses `fn`
|
||
and `async-fn` into `callable` for exactly this reason on the
|
||
divergence side, and this side never asked. Snippet #2849 — four
|
||
starters, three ticks, every row judged by an audit — scored 4/7 and
|
||
was called incoherent when nothing about it is.
|
||
|
||
2. A METHOD OF A MEMBER IS NOT A STRANGER. Snippet #2844 is the
|
||
SQLAlchemy model convention and its own text covers the class AND the
|
||
`to_dict` the class must carry. Its 37 classes and 25 serialisers
|
||
scored 37/62 = 0.597, missing the bar by three thousandths for
|
||
containing exactly what it says it contains.
|
||
|
||
THE TRAP, and it is why this is not simply a family histogram: grouping
|
||
by family and keeping a majority test makes the check BLIND. Before #2844
|
||
was cleaned it held 37 classes and 56 callables; as families that is
|
||
56/93 = 0.602, a clean pass, and the 31 rows that had no business being
|
||
there — Vue functions, route handlers, a dozen tests — would never have
|
||
been reported. A looser bar in the same shape is not a fix.
|
||
|
||
So the verdict is inverted. Instead of asking whether most rows agree, it
|
||
asks how many rows the canon CANNOT ACCOUNT FOR:
|
||
|
||
* a row in the majority family is accounted for;
|
||
* a callable defined in a FILE that also holds a majority-family `type`
|
||
row is accounted for — it is a method of a member;
|
||
* everything else is a STRANGER, and strangers above `_STRANGER_SHARE`
|
||
of the readable rows make the canon incoherent.
|
||
|
||
The majority vote abstains those methods, so a class's own serialisers
|
||
cannot outvote the classes and turn the members into the strangers. On
|
||
the real ledger the three populations separate completely: clean #2844
|
||
has 0 strangers in 62, #2849 has 0 in 7, and polluted #2844 had 31 in 93.
|
||
|
||
Scoped to the REVIEW surface on purpose. `canon_form` still answers at
|
||
the precise form level for stamping and divergence, where a sync helper
|
||
beside an async canon is a fair question; nothing here changes what the
|
||
ledger writes.
|
||
|
||
Returns a dict, deliberately. Widening a tuple return is the #4204 break
|
||
exactly — a 4-tuple grew a fifth field and one consumer went on unpacking
|
||
four, and Python said nothing until that line ran.
|
||
"""
|
||
forms: dict[str, int] = {}
|
||
families: dict[str, int] = {}
|
||
per_row: list[tuple[object, str]] = []
|
||
has_type: set[str] = set()
|
||
for r in members:
|
||
f = shape_form(getattr(r, "signature", "") or "", r.kind)
|
||
if not f:
|
||
continue # unreadable: never counts either way
|
||
fam = shape_family(f)
|
||
forms[f] = forms.get(f, 0) + 1
|
||
families[fam] = families.get(fam, 0) + 1
|
||
per_row.append((r, fam))
|
||
if fam == "type":
|
||
has_type.add(r.path)
|
||
|
||
out = {
|
||
"readable": len(per_row),
|
||
"forms": dict(sorted(forms.items(), key=lambda kv: -kv[1])),
|
||
"families": dict(sorted(families.items(), key=lambda kv: -kv[1])),
|
||
"family": FORM_UNKNOWN,
|
||
"attached": 0,
|
||
"strangers": [],
|
||
"coheres": True,
|
||
}
|
||
if not per_row:
|
||
return out # nothing legible: say nothing, not "broken"
|
||
|
||
# The vote, with methods abstaining. A callable sitting in a file that
|
||
# defines a class is presumed to belong to it and does not get to argue
|
||
# that the canon is really about callables.
|
||
def _attachable(row, fam: str) -> bool:
|
||
return fam == "callable" and row.path in has_type
|
||
|
||
votes: dict[str, int] = {}
|
||
for r, fam in per_row:
|
||
if _attachable(r, fam):
|
||
continue
|
||
votes[fam] = votes.get(fam, 0) + 1
|
||
if not votes: # every readable row is a method
|
||
votes = dict(families)
|
||
# Ties go to `type`: a canon that defines a class is about the class.
|
||
family = max(votes, key=lambda k: (votes[k], k == "type"))
|
||
|
||
attached, strangers = 0, []
|
||
for r, fam in per_row:
|
||
if fam == family:
|
||
continue
|
||
if family == "type" and _attachable(r, fam):
|
||
attached += 1
|
||
else:
|
||
strangers.append(r)
|
||
out["family"] = family
|
||
out["attached"] = attached
|
||
out["strangers"] = strangers
|
||
out["coheres"] = len(strangers) / len(per_row) <= _STRANGER_SHARE
|
||
return out
|
||
|
||
|
||
def signature_in(code: str, symbol: str, kind: str) -> str:
|
||
"""The line in ``code`` that DEFINES ``symbol``, or "" if none does.
|
||
|
||
The write-time checks are asked about a shape that may have no ledger row
|
||
yet — it is being written right now — so the payload is the only place its
|
||
signature exists. Finding nothing returns "", which reads downstream as
|
||
FORM_UNKNOWN and therefore as silence.
|
||
"""
|
||
if not code or not symbol:
|
||
return ""
|
||
name = re.escape(symbol.lstrip("."))
|
||
if kind == "css":
|
||
pat = re.compile(r"^\s*\." + name + r"\b")
|
||
else:
|
||
pat = re.compile(
|
||
r"^\s*(?:(?:export|default|public|private|static|final)\s+)*"
|
||
r"(?:"
|
||
r"(?:(?:async\s+)?(?:def|function|func|fn|sub)|class|interface|type"
|
||
r"|struct|enum|const|let|var)\s+" + name + r"\b"
|
||
# A bare binding — `MAX = 10`, `Handler = ...` — which is how
|
||
# Python (and plain JS assignment) declares one.
|
||
r"|" + name + r"\s*(?::[^=\n]+)?=(?!=)"
|
||
r")"
|
||
)
|
||
for line in code.splitlines():
|
||
if pat.match(line):
|
||
return line.strip()
|
||
return ""
|
||
|
||
|
||
def snippet_kind(symbol: str, language: str) -> str:
|
||
"""The ledger kind a snippet's reference belongs to — "css" when its
|
||
symbol is a class selector (or it is a stylesheet with no symbol),
|
||
else "sym"."""
|
||
sym = (symbol or "").strip()
|
||
if sym.startswith("."):
|
||
return "css"
|
||
if not sym and (language or "").strip().lower() in ("css", "scss", "sass", "less"):
|
||
return "css"
|
||
return "sym"
|
||
|
||
|
||
def references_symbol(code: str, symbol: str, kind: str) -> bool:
|
||
"""Does this payload name the snippet's symbol? Word-bounded so `confirm`
|
||
never claims `confirmed`; a CSS class matches as `.btn` or inside a class
|
||
attribute (`btn btn-primary`), dashes counting as part of the name."""
|
||
sym = _norm_symbol(symbol or "")
|
||
if not sym or not code:
|
||
return False
|
||
if kind == "css":
|
||
pattern = rf"(?<![\w-]){re.escape(sym)}(?![\w-])"
|
||
else:
|
||
pattern = rf"(?<![\w$]){re.escape(sym)}(?![\w$])"
|
||
return re.search(pattern, code) is not None
|
||
|
||
|
||
async def recent_pulls(user_id: int, *, window: timedelta = PULL_WINDOW) -> dict[int, datetime]:
|
||
"""{note_id: last pulled at} for every note this user opened in full
|
||
inside ``window`` — the "actually pulled it" half of the stamping rule.
|
||
Reads the usage telemetry table; an unreadable table means no evidence."""
|
||
from sqlalchemy import func
|
||
|
||
from scribe.models.note_usage import PULLED, NoteUsageEvent
|
||
|
||
since = datetime.now(timezone.utc) - window
|
||
try:
|
||
async with async_session() as session:
|
||
rows = await session.execute(
|
||
select(NoteUsageEvent.note_id, func.max(NoteUsageEvent.created_at))
|
||
.where(
|
||
NoteUsageEvent.user_id == user_id,
|
||
NoteUsageEvent.event == PULLED,
|
||
NoteUsageEvent.created_at >= since,
|
||
)
|
||
.group_by(NoteUsageEvent.note_id)
|
||
)
|
||
return {int(nid): ts for nid, ts in rows.all()}
|
||
except Exception:
|
||
logger.warning("recent_pulls read failed — no hook stamping this write", exc_info=True)
|
||
return {}
|
||
|
||
|
||
# THE HOOK'S EVIDENCE, WRITTEN ONCE AND READ BACK BY ONE PARSER.
|
||
#
|
||
# The stamp's score used to exist only inside a prose sentence, which meant
|
||
# nothing could ask "how strong was the evidence for this row?" without
|
||
# re-deriving it. That is how 32 rows on one project and 334 on another sat
|
||
# unexamined: the number was printed, never stored, never queried. Format and
|
||
# parser live beside each other so a change to one that breaks the other is a
|
||
# visible edit rather than a silent drift (rule 33's reasoning, one module
|
||
# down).
|
||
_REFERENCE_REASON = "hook: pulled #{sid}; payload references `{symbol}`"
|
||
_RESEMBLE_REASON = "hook: pulled #{sid}; payload resembles it ({score:.2f})"
|
||
_RESEMBLE_READ = re.compile(
|
||
r"^hook: pulled #(\d+); payload resembles it \(([0-9]*\.?[0-9]+)\)$"
|
||
)
|
||
|
||
|
||
def stamp_score(reason: str | None) -> float | None:
|
||
"""The resemblance a hook stamp recorded, or None if it was not one.
|
||
|
||
None means "this row was not written on a similarity score" — a by-name
|
||
reference, an agent judgment, an audit — NOT "the score was zero". Every
|
||
caller treats the two differently, because a row a person judged is not
|
||
weak evidence, it is a different kind of evidence entirely.
|
||
"""
|
||
m = _RESEMBLE_READ.match((reason or "").strip())
|
||
return float(m.group(2)) if m else None
|
||
|
||
|
||
def _stamp_allowed(rank: int, mine: str, canon: str) -> bool:
|
||
"""May this evidence assert that a shape of form ``mine`` IS ``canon``?
|
||
|
||
Named rather than inlined because the asymmetry is the decision, not an
|
||
implementation detail: rank 2 (the payload names the canon's symbol) has
|
||
only to avoid contradicting, rank 1 (a whole-file similarity score) has to
|
||
positively agree. See `forms_conflict` for why both exist.
|
||
"""
|
||
if rank >= 2:
|
||
return not forms_conflict(mine, canon)
|
||
return forms_agree(mine, canon)
|
||
|
||
|
||
async def stamp_write_path_instances(
|
||
user_id: int,
|
||
project_id: int,
|
||
*,
|
||
path: str,
|
||
shapes: list[tuple[str, str]],
|
||
code: str,
|
||
pulled: dict[int, datetime],
|
||
resembles: dict[int, float] | None = None,
|
||
repo_key: str = "",
|
||
) -> list[dict]:
|
||
"""Land hook evidence as `instance` rows for the shapes being written.
|
||
|
||
``shapes`` is the hook's (kind, name) list for ``path``; ``pulled`` is
|
||
recent_pulls(); ``resembles`` maps snippet ids the semantic arm scored
|
||
for this payload to their score. Returns the rows stamped, each
|
||
{path, symbol, kind, snippet_id, reason} — empty in the common case.
|
||
|
||
A shape the ledger has no live row for yet (it is being written right
|
||
now) gets a PROVISIONAL row under ``repo_key`` — first/last-seen empty —
|
||
so the stamp is not lost to the next sync, which either confirms the
|
||
shape (sets its seen marker) or stamps it vanished. No repo key → only
|
||
existing rows are stamped.
|
||
|
||
When more than one pulled snippet is in play for a shape, a by-name
|
||
reference beats resemblance and the most recent pull breaks ties: a row
|
||
holds one canon (the known model limit logged on #2790).
|
||
"""
|
||
from scribe.services import access
|
||
from scribe.services import snippets as snippets_svc
|
||
from scribe.services.snippets import snippet_fields
|
||
|
||
resembles = resembles or {}
|
||
path = (path or "").strip()
|
||
wanted = [(k, n.strip()) for k, n in shapes if k in ("css", "sym") and n.strip()]
|
||
if not project_id or not path or not wanted or not pulled:
|
||
return []
|
||
if not await access.can_write_project(user_id, project_id):
|
||
return []
|
||
|
||
# Which pulled canons are in play for this payload, by kind, ranked.
|
||
#
|
||
# THE FORM OF THE CANON TRAVELS WITH IT (#4204), because what gets written
|
||
# here is an ASSERTION — "this shape IS that canon" — and it is permanent
|
||
# until something re-judges it. The evidence below is payload-level: a
|
||
# resemblance score is computed against the whole file, so without a
|
||
# per-shape test every symbol in that file inherits one verdict. Measured
|
||
# on another project: a single write stamped `class SessionAbsent`,
|
||
# `async def attach` and `_run_control_client` as instances of one snippet.
|
||
in_play: dict[str, list[tuple[int, datetime, int, str, str]]] = {}
|
||
for sid, pulled_at in pulled.items():
|
||
note = await snippets_svc.get_snippet(user_id, sid)
|
||
if note is None:
|
||
continue
|
||
fields = snippet_fields(note)
|
||
symbol = fields.get("symbol") or ""
|
||
kind = snippet_kind(symbol, fields.get("language") or "")
|
||
if references_symbol(code, symbol, kind):
|
||
rank, why = 2, _REFERENCE_REASON.format(
|
||
sid=sid, symbol=_norm_symbol(symbol)
|
||
)
|
||
elif resembles.get(sid, 0.0) >= _RESEMBLE_MIN:
|
||
rank, why = 1, _RESEMBLE_REASON.format(sid=sid, score=resembles[sid])
|
||
else:
|
||
# A SCORE BELOW THE FLOOR IS NOT WEAK EVIDENCE, IT IS NONE. This
|
||
# branch used to be `elif sid in resembles`, which took any score
|
||
# at all — 0.69 counted exactly as much as 0.95, and the number was
|
||
# printed into the reason line while never being tested against
|
||
# anything. A row written on that basis is indistinguishable
|
||
# afterwards from one written on real evidence.
|
||
continue
|
||
in_play.setdefault(kind, []).append(
|
||
(rank, pulled_at, sid, why, shape_form(fields.get("signature") or "", kind))
|
||
)
|
||
if not in_play:
|
||
return []
|
||
for bucket in in_play.values():
|
||
bucket.sort(key=lambda t: (t[0], t[1]), reverse=True)
|
||
|
||
now = datetime.now(timezone.utc)
|
||
stamped: list[dict] = []
|
||
async with async_session() as session:
|
||
rows = (
|
||
await session.execute(
|
||
select(CodeShape).where(
|
||
CodeShape.project_id == project_id,
|
||
CodeShape.path == path,
|
||
CodeShape.vanished_at.is_(None),
|
||
)
|
||
)
|
||
).scalars().all()
|
||
by_key = {(r.symbol, r.kind): r for r in rows}
|
||
for kind, name in wanted:
|
||
bucket = in_play.get(kind)
|
||
if not bucket:
|
||
continue
|
||
row = by_key.get((name, kind))
|
||
# PER SHAPE, NOT PER FILE (#4204). The best candidate whose FORM
|
||
# matches this shape's — not simply the best candidate. A class and
|
||
# a function in one file can no longer be handed the same canon
|
||
# because the file as a whole resembled it.
|
||
#
|
||
# The signature comes from the payload first: a shape being written
|
||
# right now may have no ledger row yet, and then the code is the
|
||
# only place it exists. No readable signature on either side means
|
||
# no stamp — `forms_agree` requires both to be known, so an
|
||
# unreadable shape falls silent instead of matching everything.
|
||
mine = shape_form(
|
||
signature_in(code, name, kind) or getattr(row, "signature", "") or "", kind
|
||
)
|
||
# THE BURDEN SCALES WITH THE EVIDENCE. Rank 2 is an explicit
|
||
# by-name reference to the canon in this very payload; it stands
|
||
# unless the forms actively contradict. Rank 1 is a similarity
|
||
# score over the whole file — one number that would otherwise
|
||
# speak for every symbol in it — so it must positively agree.
|
||
cand = next(
|
||
(t for t in bucket if _stamp_allowed(t[0], mine, t[4])), None
|
||
)
|
||
if cand is None:
|
||
continue
|
||
_rank, _at, sid, why, _cform = cand
|
||
if row is None:
|
||
if not repo_key:
|
||
continue
|
||
row = CodeShape(
|
||
project_id=project_id, repo_key=repo_key,
|
||
path=path, symbol=name, kind=kind,
|
||
)
|
||
session.add(row)
|
||
by_key[(name, kind)] = row
|
||
elif not (row.status in _MECHANICAL_TODO or row.classified_by == "hook"):
|
||
continue # a judgment — or the canon itself — stands
|
||
await _judge(session, row, status="instance", snippet_id=sid, by="hook",
|
||
reason=why, at=now)
|
||
stamped.append({
|
||
"path": path, "symbol": name, "kind": kind,
|
||
"snippet_id": sid, "reason": why,
|
||
})
|
||
# Every pulled canon the payload NAMES is a uses edge (#2870) — the
|
||
# call-site fact, independent of which one the row is judged to be.
|
||
await record_uses(
|
||
session, row,
|
||
# Indexed, not destructured: this list widened from 4 to 5 when
|
||
# the canon's form joined it (#4204) and a positional unpack
|
||
# here went on reading four. The unit lane never touches
|
||
# `record_uses`, so it stayed green and the integration lane
|
||
# was the only thing that said so.
|
||
[t[2] for t in bucket if t[0] == 2],
|
||
basis="hook", evidence="write path: pulled the snippet, payload names its symbol",
|
||
)
|
||
if stamped:
|
||
await session.commit()
|
||
return stamped
|
||
|
||
|
||
# --- the mechanical proposer (#2792): the machine proposes, judgment classifies
|
||
#
|
||
# Runs inside the coverage refresh — the one moment the shape BODIES exist
|
||
# (the archive is in memory; the ledger stores fingerprints, never code).
|
||
# Over every live unclassified row whose content changed since it was last
|
||
# examined, it tries, strongest first:
|
||
#
|
||
# symbol the shape bears a recorded canon's symbol at another location
|
||
# (a second definition of the canon's name — an instance or a
|
||
# duplicate to consolidate; either way, it answers to #N);
|
||
# text whitespace-insensitive containment either way between the
|
||
# body and the canon's recorded code (git-grep, in effect —
|
||
# a verbatim copy, or the canon's own call-site example);
|
||
# reference the body calls/uses a canon's symbol — the call-site shape
|
||
# the P7 backfill classified by hand (#2790);
|
||
# signature a sym whose definition line, name blanked, resembles the
|
||
# canon's (the move_*/resequence family shape);
|
||
# semantic the widest net: the body (concept-queried, as the write-path
|
||
# arm does) scores above the write-path threshold against a
|
||
# canon — capped per refresh, so the cost is bounded and a big
|
||
# ledger is worked through across refreshes.
|
||
#
|
||
# A hit is a PROPOSAL on the row (proposed_snippet_id/basis/score), never a
|
||
# classification: an agent confirms in batches (confirm_proposals) or judges
|
||
# otherwise (classify_shapes clears it). Rows with no canon hit are grouped
|
||
# by the derive-first rule (note 2786): the same body fingerprint in ≥2
|
||
# places, or the same name defined in ≥3 files, is "a repeating shape with
|
||
# no canon — derive one first", carried as proposal_basis="derive" + a group
|
||
# key so sessions see the consolidation candidates as one thing.
|
||
|
||
# Derive-first floors. Identical bodies twice is already a copy; a bare name
|
||
# needs more repetition before it reads as a family (setup/handler/register
|
||
# recur by convention, not by duplication).
|
||
_DERIVE_MIN_DUP = 2
|
||
_DERIVE_MIN_NAME = 3
|
||
# CSS is never grouped by body (note 2917): classes for different purposes
|
||
# share declarations because the style system makes them alike — `.text-muted`
|
||
# and `.pin-badge-auto` carrying the same `color: var(--fs-text-tertiary)` are
|
||
# two meanings, not two copies. A CSS family is a NAME defined in more than
|
||
# one file: that is a recipe living in several places, and two is already
|
||
# the signal (a class name is deliberate in a way `setup`/`load` are not).
|
||
_DERIVE_MIN_NAME_CSS = 2
|
||
# Semantic checks per repo per refresh — an embedding each (local fastembed),
|
||
# bounded so a 4,000-row ledger is worked through over refreshes, not in one.
|
||
_SEMANTIC_CAP = 150
|
||
# The semantic basis's own floor — stricter than the write-path arm's, because
|
||
# a code BODY against a prose-forward snippet document scores 0.68–0.75 for
|
||
# "both are about migrations"; first live run paired every alembic
|
||
# upgrade()/downgrade() with an unrelated canon at exactly that band.
|
||
_SEMANTIC_FLOOR = 0.8
|
||
# How many above-floor hits the semantic arm asks for. Named because the
|
||
# NUMBER is load-bearing twice over: it caps the work, and a result set that
|
||
# came back short of it is a complete picture of what cleared the floor —
|
||
# which is what lets a miss be read as evidence rather than as a cut-off
|
||
# (`BASIS_NO_SEMANTIC_MATCH`).
|
||
_SEMANTIC_LIMIT = 3
|
||
# The proposer looked at this body, compared it against every canon in its
|
||
# language family, and matched none of them above `_SEMANTIC_FLOOR` (#4208).
|
||
#
|
||
# This is a NEGATIVE RESULT, and it is stored because it is the only evidence
|
||
# in the ledger that speaks to what a shape MEANS rather than what it looks
|
||
# like. `proposal_basis` otherwise names how a proposal was arrived at; here
|
||
# it records that the arm ran and came back empty, with `proposed_snippet_id`
|
||
# left NULL. Every reader keys "is there a proposal" on `proposed_snippet_id`
|
||
# or `proposal_group`, never on the basis, so this cannot be mistaken for one:
|
||
# `list_shapes(proposal=...)` and `confirm_shape_proposals` both filter on the
|
||
# id, and the latter requires it non-NULL before it will confirm anything.
|
||
#
|
||
# It is deliberately NOT written for the two cases that merely look the same:
|
||
# a body too thin to compare (`_substance` below the write-path minimum), and
|
||
# a row the per-refresh cap never reached. Those are "I cannot tell", and the
|
||
# ledger's standing discipline — the one `FORM_UNKNOWN` enforces everywhere
|
||
# else — is that not knowing must make a check quieter, never more confident.
|
||
BASIS_NO_SEMANTIC_MATCH = "no-semantic-match"
|
||
# Bump when a basis's rule changes: rows remember the (body, ruleset) they
|
||
# were examined under, so a tightened rule re-examines everything once.
|
||
# v3: language-family gate on the sym bases, reference stoplist, semantic
|
||
# restricted to the shape's own project (#2871).
|
||
# v4: the semantic arm records its misses as well as its hits (#4208), so
|
||
# every already-examined row must be looked at once more to acquire one.
|
||
_PROPOSER_VERSION = 4
|
||
# Signature resemblance floor, name blanked (difflib ratio) — and a length
|
||
# floor, because `def NAME():` resembles `def NAME(x):` at 0.95 while saying
|
||
# nothing; a family shape has parameters to resemble.
|
||
_SIGNATURE_FLOOR = 0.8
|
||
_SIGNATURE_MIN_LEN = 30
|
||
# Textual containment needs enough substance to mean anything.
|
||
_TEXT_FLOOR = 40
|
||
_BASIS_ORDER = ("symbol", "text", "reference", "signature")
|
||
|
||
|
||
class Canon(NamedTuple):
|
||
snippet_id: int
|
||
kind: str
|
||
symbol: str
|
||
locations: tuple[tuple[str, str], ...]
|
||
signature: str
|
||
code_norm: str
|
||
project_id: int = 0
|
||
language: str = "" # the snippet's recorded language; "" = unknown, no gate
|
||
|
||
|
||
# Language families: the sym bases only propose within one. The 2026-08
|
||
# audit (#2871) found every cross-language hit wrong — a Python tool-module
|
||
# canon named `register` proposed for Vue `handleSubmit`s that call
|
||
# `authStore.register()`, and a TS store's `register` matched it by symbol;
|
||
# Minstrel/Forge TS canon proposed for Python bodies by resemblance. CSS is
|
||
# its own kind and is not gated here.
|
||
_FAMILY_BY_LANG = {
|
||
"python": "py", "py": "py",
|
||
"typescript": "js", "ts": "js", "tsx": "js", "javascript": "js", "js": "js",
|
||
"jsx": "js", "vue": "js", "mjs": "js", "cjs": "js",
|
||
"css": "css", "scss": "css", "sass": "css", "less": "css",
|
||
"bash": "sh", "sh": "sh", "shell": "sh", "zsh": "sh",
|
||
"sql": "sql",
|
||
}
|
||
_FAMILY_BY_EXT = {
|
||
".py": "py", ".pyi": "py",
|
||
".ts": "js", ".tsx": "js", ".js": "js", ".jsx": "js", ".vue": "js", ".mjs": "js", ".cjs": "js",
|
||
".css": "css", ".scss": "css", ".sass": "css", ".less": "css",
|
||
".sh": "sh", ".bash": "sh", ".zsh": "sh",
|
||
".sql": "sql",
|
||
}
|
||
|
||
|
||
def language_family(language: str) -> str:
|
||
"""The family a recorded snippet language belongs to ("" when unknown)."""
|
||
return _FAMILY_BY_LANG.get((language or "").strip().lower(), "")
|
||
|
||
|
||
def path_family(path: str) -> str:
|
||
"""The family a file path belongs to, by extension ("" when unknown)."""
|
||
p = (path or "").lower()
|
||
for ext, fam in _FAMILY_BY_EXT.items():
|
||
if p.endswith(ext):
|
||
return fam
|
||
return ""
|
||
|
||
|
||
def same_family(path: str, canon_language: str) -> bool:
|
||
"""A sym basis may propose this canon for this path: both families known
|
||
and equal, or either unknown (no evidence either way → no gate)."""
|
||
a = path_family(path)
|
||
b = language_family(canon_language)
|
||
return not a or not b or a == b
|
||
|
||
|
||
# Reference basis: generic verbs name too many unrelated things to count a
|
||
# bare mention as a call site of THIS canon (`register`, `load`, `save` …).
|
||
# The symbol basis still catches a second definition of such a name; the
|
||
# call-site relation for these becomes a `uses` edge once #2870 lands.
|
||
_REFERENCE_STOPLIST = frozenset({
|
||
"get", "set", "put", "post", "load", "save", "run", "main", "init", "setup",
|
||
"register", "restore", "reset", "toggle", "close", "open", "submit", "handler",
|
||
"update", "create", "delete", "remove", "add", "start", "stop", "send",
|
||
"receive", "render", "mount", "dispatch", "call", "apply", "execute",
|
||
})
|
||
|
||
|
||
def _norm_text(text: str) -> str:
|
||
return " ".join((text or "").split())
|
||
|
||
|
||
def signature_similarity(sig_a: str, name_a: str, sig_b: str, name_b: str) -> float:
|
||
"""How alike two definition lines are once their own names are blanked —
|
||
`def move_event(project_id, event_id, after_id)` against
|
||
`def move_beat(project_id, beat_id, after_id)` reads high."""
|
||
a = _norm_text(sig_a)
|
||
b = _norm_text(sig_b)
|
||
if name_a:
|
||
a = a.replace(name_a, "NAME")
|
||
if name_b:
|
||
b = b.replace(name_b, "NAME")
|
||
if len(a) < _SIGNATURE_MIN_LEN or len(b) < _SIGNATURE_MIN_LEN:
|
||
return 0.0
|
||
return difflib.SequenceMatcher(None, a, b).ratio()
|
||
|
||
|
||
def text_contains(body: str, code: str) -> bool:
|
||
"""Whitespace-insensitive containment, either way, above a substance
|
||
floor — the proposer's git-grep."""
|
||
a = _norm_text(body)
|
||
b = _norm_text(code)
|
||
if len(a) < _TEXT_FLOOR or len(b) < _TEXT_FLOOR:
|
||
return False
|
||
return a in b or b in a
|
||
|
||
|
||
def reference_canons(kind: str, path: str, symbol: str, body: str, canons: Iterable[Canon]) -> list[int]:
|
||
"""Every canon this body NAMES (#2870) — the uses edges the proposer can
|
||
write mechanically: same kind, same language family, symbol not in the
|
||
generic-verb stoplist, and not the shape's own name."""
|
||
norm_sym = _norm_symbol(symbol)
|
||
out: list[int] = []
|
||
for c in canons:
|
||
if c.kind != kind or not c.symbol:
|
||
continue
|
||
if kind == "sym" and not same_family(path, c.language):
|
||
continue
|
||
if _norm_symbol(c.symbol) == norm_sym:
|
||
continue
|
||
if _norm_symbol(c.symbol).lower() in _REFERENCE_STOPLIST:
|
||
continue
|
||
if references_symbol(body, c.symbol, kind):
|
||
out.append(c.snippet_id)
|
||
return out
|
||
|
||
|
||
def match_canon(
|
||
kind: str, path: str, symbol: str, signature: str, body: str,
|
||
canons: Iterable[Canon], *, project_id: int = 0,
|
||
) -> tuple[int, str, float] | None:
|
||
"""The strongest (snippet_id, basis, score) a shape earns against the
|
||
canon catalog, by the non-semantic bases — or None. Strongest by
|
||
basis order, then by score within the basis; on a tie, a canon recorded
|
||
in the shape's own project beats family canon from another (the same
|
||
helper recorded in two projects is the shape's own first)."""
|
||
best: dict[str, tuple[float, bool, int]] = {}
|
||
|
||
def offer(basis: str, score: float, canon: Canon) -> None:
|
||
cur = best.get(basis)
|
||
same = bool(project_id) and canon.project_id == project_id
|
||
if cur is None or (score, same) > (cur[0], cur[1]):
|
||
best[basis] = (score, same, canon.snippet_id)
|
||
|
||
norm_sym = _norm_symbol(symbol)
|
||
for c in canons:
|
||
if c.kind != kind:
|
||
continue
|
||
if kind == "sym" and not same_family(path, c.language):
|
||
continue # a Python canon says nothing about a Vue body, and vice versa
|
||
if c.symbol and _norm_symbol(c.symbol) == norm_sym:
|
||
if not any(location_covers(lp, ls, path, symbol) for lp, ls in c.locations):
|
||
offer("symbol", 1.0, c)
|
||
continue # its own location is canonical territory, not a proposal
|
||
if (
|
||
c.symbol
|
||
and _norm_symbol(c.symbol).lower() not in _REFERENCE_STOPLIST
|
||
and references_symbol(body, c.symbol, kind)
|
||
):
|
||
offer("reference", 0.9, c)
|
||
if c.code_norm and text_contains(body, c.code_norm):
|
||
offer("text", 0.95, c)
|
||
if kind == "sym" and c.signature:
|
||
ratio = signature_similarity(signature, symbol, c.signature, c.symbol)
|
||
if ratio >= _SIGNATURE_FLOOR:
|
||
offer("signature", round(ratio, 3), c)
|
||
for basis in _BASIS_ORDER:
|
||
if basis in best:
|
||
score, _same, sid = best[basis]
|
||
return (sid, basis, score)
|
||
return None
|
||
|
||
|
||
async def canon_catalog(user_id: int) -> list[Canon]:
|
||
"""Every snippet this user can browse, as matchable canon — own projects
|
||
and shared ones alike, because family canon counts (note 2786)."""
|
||
from scribe.models.note import Note
|
||
from scribe.services.access import browsable_notes_clause
|
||
from scribe.services.coverage import extract_definitions
|
||
from scribe.services.snippets import SNIPPET_NOTE_TYPE, snippet_fields
|
||
|
||
async with async_session() as session:
|
||
notes = (
|
||
await session.execute(
|
||
select(Note).where(
|
||
browsable_notes_clause(user_id),
|
||
Note.note_type == SNIPPET_NOTE_TYPE,
|
||
Note.deleted_at.is_(None),
|
||
)
|
||
)
|
||
).scalars().all()
|
||
out: list[Canon] = []
|
||
for note in notes:
|
||
fields = snippet_fields(note)
|
||
symbol = (fields.get("symbol") or "").strip()
|
||
kind = snippet_kind(symbol, fields.get("language") or "")
|
||
code = fields.get("code") or ""
|
||
# The signature basis matches against the canon's OWN definition
|
||
# line — never against an example in its code. A snippet whose code
|
||
# opens with a call-site example (confirmed()'s does) would otherwise
|
||
# make every `async function x(): Promise<void>` resemble it.
|
||
own = next(
|
||
(d for d in extract_definitions(code)
|
||
if symbol and _norm_symbol(d.name) == _norm_symbol(symbol)),
|
||
None,
|
||
)
|
||
signature = own.signature if own else ""
|
||
out.append(Canon(
|
||
int(note.id), kind, symbol,
|
||
tuple(
|
||
((loc.get("path") or ""), (loc.get("symbol") or ""))
|
||
for loc in fields.get("locations") or []
|
||
),
|
||
signature, _norm_text(code), int(note.project_id or 0),
|
||
(fields.get("language") or "").strip().lower(),
|
||
))
|
||
return out
|
||
|
||
|
||
def _clear_proposal(row: CodeShape, *, reexamine: bool = False) -> None:
|
||
row.proposed_snippet_id = None
|
||
row.proposal_basis = None
|
||
row.proposal_score = None
|
||
row.proposal_group = None
|
||
if reexamine:
|
||
row.proposed_at = None
|
||
row.proposed_sha = ""
|
||
|
||
|
||
def _substance(text: str) -> int:
|
||
return len("".join((text or "").split()))
|
||
|
||
|
||
async def _semantic_canon(
|
||
user_id: int, body: str, allowed: set[int], *, report: dict | None = None
|
||
) -> tuple[int, float] | None:
|
||
"""The canon this body MEANS, or None.
|
||
|
||
`report` is an out-param in the style `semantic_search_notes` already
|
||
uses, and it carries the one thing the return value cannot: whether a
|
||
None is EVIDENCE. `report["conclusive"] = True` says the arm really
|
||
compared this body against the allowed canons and none cleared the floor.
|
||
It is left unset whenever the arm could not form an opinion — a body with
|
||
too little substance to embed, no allowed canon to compare against, or a
|
||
result set that came back full and may therefore have been truncated.
|
||
|
||
The truncation case is why `_SEMANTIC_LIMIT` is named. The search returns
|
||
the top N above the floor; if it returns fewer than N, N was not binding
|
||
and we have seen everything that cleared the floor, so "no allowed canon
|
||
among them" is a fact about the corpus. If it returns exactly N, an
|
||
allowed canon could be sitting at N+1 and the same silence means nothing.
|
||
Reading the second case as the first is how a cut-off becomes a finding.
|
||
|
||
Callers must treat a missing key as "cannot tell", never as "no match" —
|
||
which is also what makes the existing test double, an `AsyncMock` that
|
||
returns None and touches no report, stay correct by default.
|
||
"""
|
||
from scribe.services.embeddings import semantic_search_notes
|
||
from scribe.services.plugin_context import (
|
||
WRITEPATH_DEFAULT_THRESHOLD, WRITEPATH_MIN_CODE_CHARS, concept_query,
|
||
)
|
||
|
||
if _substance(body) < WRITEPATH_MIN_CODE_CHARS or not allowed:
|
||
return None
|
||
query = concept_query(body) or body
|
||
hits = await semantic_search_notes(
|
||
user_id, query, limit=_SEMANTIC_LIMIT,
|
||
threshold=max(WRITEPATH_DEFAULT_THRESHOLD, _SEMANTIC_FLOOR),
|
||
note_type="snippet", scope="browse",
|
||
)
|
||
for score, note in hits:
|
||
if int(note.id) in allowed:
|
||
return int(note.id), round(float(score), 3)
|
||
if report is not None and len(hits) < _SEMANTIC_LIMIT:
|
||
report["conclusive"] = True
|
||
return None
|
||
|
||
|
||
async def propose_for_repo(
|
||
user_id: int,
|
||
project_id: int,
|
||
repo_key: str,
|
||
definitions: list,
|
||
*,
|
||
canons: list[Canon] | None = None,
|
||
semantic_cap: int = _SEMANTIC_CAP,
|
||
) -> dict:
|
||
"""Examine one repo's live unclassified rows against canon and record
|
||
proposals. ``definitions`` are the ArchiveShape records the sync just
|
||
upserted (their bodies are the matching material). Rows whose fingerprint
|
||
is unchanged since their last examination are skipped; rows that only
|
||
the capped semantic pass could not reach stay unexamined, so the next
|
||
refresh reaches the next slice. Returns counts."""
|
||
if canons is None:
|
||
canons = await canon_catalog(user_id)
|
||
by_key = {(d[0], d[1], d[2]): d for d in definitions}
|
||
# The semantic arm is the widest net and, across projects, was pure noise
|
||
# in the 2026-08 audit (#2871): it is held to the shape's own project and
|
||
# language family. The precise bases (symbol/text) still reach family
|
||
# canon in other projects (note 2786).
|
||
sym_canons = [c for c in canons if c.kind == "sym" and c.project_id == project_id]
|
||
|
||
def semantic_allowed(path: str) -> set[int]:
|
||
return {c.snippet_id for c in sym_canons if same_family(path, c.language)}
|
||
now = datetime.now(timezone.utc)
|
||
examined = proposed = checked = 0
|
||
async with async_session() as session:
|
||
rows = (
|
||
await session.execute(
|
||
select(CodeShape).where(
|
||
CodeShape.project_id == project_id,
|
||
CodeShape.repo_key == repo_key,
|
||
CodeShape.vanished_at.is_(None),
|
||
)
|
||
)
|
||
).scalars().all()
|
||
semantic_todo: list[tuple[CodeShape, object]] = []
|
||
for row in rows:
|
||
d = by_key.get((row.path, row.kind, row.symbol))
|
||
if d is None:
|
||
continue
|
||
signature, body_sha, body = d[3], d[4], d[5]
|
||
examined_as = f"{body_sha}@{_PROPOSER_VERSION}"
|
||
if row.proposed_at is not None and row.proposed_sha == examined_as:
|
||
continue
|
||
if row.status not in _MECHANICAL_TODO:
|
||
# A judged row gets no proposal — but its uses edges (#2870)
|
||
# are a fact about the body, judged or not: the consumer map
|
||
# of a canon must include the call sites someone already
|
||
# classified. Mark it examined so the scan runs once per body.
|
||
used = reference_canons(row.kind, row.path, row.symbol, body, canons)
|
||
if used:
|
||
await record_uses(session, row, used, basis="reference",
|
||
evidence="proposer: body names the canon's symbol")
|
||
row.proposed_at = now
|
||
row.proposed_sha = examined_as
|
||
continue
|
||
examined += 1
|
||
group = row.proposal_group # derive grouping is reassigned below
|
||
hit = match_canon(
|
||
row.kind, row.path, row.symbol, signature, body, canons,
|
||
project_id=project_id,
|
||
)
|
||
_clear_proposal(row)
|
||
row.proposal_group = group
|
||
row.proposed_at = now
|
||
row.proposed_sha = examined_as
|
||
# Consumption is recorded for every canon the body names (#2870),
|
||
# whatever the row is then judged to be.
|
||
used = reference_canons(row.kind, row.path, row.symbol, body, canons)
|
||
if used:
|
||
await record_uses(session, row, used, basis="reference",
|
||
evidence="proposer: body names the canon's symbol")
|
||
if hit:
|
||
row.proposed_snippet_id, row.proposal_basis, row.proposal_score = hit
|
||
row.proposal_group = None
|
||
proposed += 1
|
||
elif row.kind == "sym":
|
||
semantic_todo.append((row, d))
|
||
for i, (row, d) in enumerate(semantic_todo):
|
||
if i >= semantic_cap:
|
||
# Not reached this refresh: leave it unexamined so the next
|
||
# refresh picks it up, rather than stamping a false "nothing".
|
||
row.proposed_at = None
|
||
row.proposed_sha = ""
|
||
continue
|
||
checked += 1
|
||
verdict: dict = {}
|
||
try:
|
||
found = await _semantic_canon(
|
||
user_id, d[5], semantic_allowed(row.path), report=verdict,
|
||
)
|
||
except Exception:
|
||
logger.warning("semantic proposal failed", exc_info=True)
|
||
found = None
|
||
# An arm that threw formed no opinion. Clearing this is not
|
||
# belt-and-braces: a partially-filled report would record a
|
||
# failure as a finding about the code.
|
||
verdict = {}
|
||
if found:
|
||
row.proposed_snippet_id, row.proposal_score = found
|
||
row.proposal_basis = "semantic"
|
||
row.proposal_group = None
|
||
proposed += 1
|
||
elif verdict.get("conclusive"):
|
||
# No canon, and the arm is sure of it. Kept as the row's basis
|
||
# with `proposed_snippet_id` still NULL, so it reads as "asked
|
||
# and answered" rather than "not asked" — the distinction
|
||
# `flag_divergence` needs and could not previously make.
|
||
row.proposal_basis = BASIS_NO_SEMANTIC_MATCH
|
||
await session.commit()
|
||
return {"examined": examined, "proposed": proposed, "semantic_checked": checked}
|
||
|
||
|
||
def derive_groups(
|
||
rows: Iterable[tuple[str, str, str, str]]
|
||
) -> dict[tuple[str, str, str], str]:
|
||
"""The derive-first grouping over (path, kind, symbol, body_sha) rows
|
||
that matched no canon: {(path, kind, symbol): group_key}. For code
|
||
(kind `sym`) identical bodies in ≥2 places group as `dup:<sha>` and the
|
||
same name defined in ≥3 files groups as `name:sym:<symbol>`, the copy
|
||
before the name. CSS groups by name only — the same class defined in
|
||
≥2 files is `name:css:<symbol>`; its body never groups it (note 2917)."""
|
||
by_sha: dict[str, list[tuple[str, str, str]]] = {}
|
||
by_name: dict[tuple[str, str], list[tuple[str, str, str]]] = {}
|
||
for path, kind, symbol, sha in rows:
|
||
key = (path, kind, symbol)
|
||
if sha and kind != "css":
|
||
by_sha.setdefault(sha, []).append(key)
|
||
by_name.setdefault((kind, _norm_symbol(symbol)), []).append(key)
|
||
out: dict[tuple[str, str, str], str] = {}
|
||
for sha, keys in by_sha.items():
|
||
if len(set(keys)) >= _DERIVE_MIN_DUP:
|
||
for key in keys:
|
||
out.setdefault(key, f"dup:{sha}")
|
||
for (kind, symbol), keys in by_name.items():
|
||
floor = _DERIVE_MIN_NAME_CSS if kind == "css" else _DERIVE_MIN_NAME
|
||
if len({k[0] for k in keys}) >= floor:
|
||
for key in keys:
|
||
out.setdefault(key, f"name:{kind}:{symbol}")
|
||
return out
|
||
|
||
|
||
async def apply_derive_groups(project_id: int) -> int:
|
||
"""Recompute derive-first groups over the project's live unclassified
|
||
rows that carry no canon proposal; returns how many rows are grouped."""
|
||
now = datetime.now(timezone.utc)
|
||
grouped = 0
|
||
async with async_session() as session:
|
||
rows = (
|
||
await session.execute(
|
||
select(CodeShape).where(
|
||
CodeShape.project_id == project_id,
|
||
CodeShape.status.in_(_MECHANICAL_TODO),
|
||
CodeShape.vanished_at.is_(None),
|
||
CodeShape.proposed_snippet_id.is_(None),
|
||
)
|
||
)
|
||
).scalars().all()
|
||
groups = derive_groups(
|
||
(r.path, r.kind, r.symbol, r.body_sha or "") for r in rows
|
||
)
|
||
sizes: dict[str, int] = {}
|
||
for g in groups.values():
|
||
sizes[g] = sizes.get(g, 0) + 1
|
||
for row in rows:
|
||
key = groups.get((row.path, row.kind, row.symbol))
|
||
if key:
|
||
row.proposal_basis = "derive"
|
||
row.proposal_group = key
|
||
row.proposal_score = float(sizes[key])
|
||
if row.proposed_at is None:
|
||
row.proposed_at = now
|
||
grouped += 1
|
||
elif row.proposal_group:
|
||
row.proposal_basis = None
|
||
row.proposal_group = None
|
||
row.proposal_score = None
|
||
await session.commit()
|
||
return grouped
|
||
|
||
|
||
def proposal_summary(
|
||
rows: Iterable[CodeShape], *, top: int = 8,
|
||
consumer_paths: dict[int, list[str]] | None = None,
|
||
) -> dict:
|
||
"""The readout's view of the proposer's standing: how many canon
|
||
proposals await confirmation, and the largest derive-first groups.
|
||
``consumer_paths`` (shape_id → files whose markup names it, milestone
|
||
302) puts `consumers` on each group — the family's distinct consumer
|
||
files across its members, the datum that separates a shared recipe
|
||
from a scoped convention."""
|
||
proposed = 0
|
||
by_canon: dict[int, int] = {}
|
||
groups: dict[str, dict] = {}
|
||
files: dict[str, set[str]] = {}
|
||
consumers: dict[str, set[str]] = {}
|
||
for row in rows:
|
||
if row.status not in _MECHANICAL_TODO:
|
||
continue
|
||
if row.proposed_snippet_id is not None:
|
||
proposed += 1
|
||
by_canon[row.proposed_snippet_id] = by_canon.get(row.proposed_snippet_id, 0) + 1
|
||
elif row.proposal_group:
|
||
dup = not row.proposal_group.startswith("name:")
|
||
g = groups.setdefault(row.proposal_group, {
|
||
"group": row.proposal_group, "kind": row.kind,
|
||
"label": (
|
||
f"{row.symbol} (identical body)" if dup
|
||
else ("." if row.kind == "css" else "") + row.symbol
|
||
),
|
||
"size": 0, "files": 0, "paths": [],
|
||
})
|
||
g["size"] += 1
|
||
files.setdefault(row.proposal_group, set()).add(row.path)
|
||
if len(g["paths"]) < 3:
|
||
g["paths"].append(row.path)
|
||
if consumer_paths is not None and row.kind == "css":
|
||
consumers.setdefault(row.proposal_group, set()).update(
|
||
consumer_paths.get(row.id) or ()
|
||
)
|
||
for key, g in groups.items():
|
||
g["files"] = len(files[key])
|
||
if key in consumers:
|
||
g["consumers"] = consumer_summary(consumers[key])
|
||
# Body-identical groups first (#2872): the things an audit actually
|
||
# consolidated were identical bodies under different names/files; a
|
||
# name repeated across modules is usually convention. Within a tier,
|
||
# the group spread over more files is the bigger copy.
|
||
ranked = sorted(
|
||
groups.values(),
|
||
key=lambda g: (g["group"].startswith("name:"), -g["files"], -g["size"], g["group"]),
|
||
)
|
||
top_canon = None
|
||
if by_canon:
|
||
sid, n = max(by_canon.items(), key=lambda kv: (kv[1], -kv[0]))
|
||
top_canon = {"snippet_id": sid, "count": n}
|
||
return {"proposed": proposed, "derive_groups": ranked[:top], "top_canon": top_canon}
|
||
|
||
|
||
def derive_new_summary(
|
||
rows: Iterable[CodeShape], *, since: datetime | None, top: int = 3
|
||
) -> dict:
|
||
"""The arrival-moment drift signal (#2899): derive-grouped rows FIRST
|
||
SEEN after ``since`` — the previous refresh's stamp, the same one
|
||
flag_divergence uses. "Since the last refresh, N more copies joined a
|
||
duplicate family" is the sentence that makes the derive queue a thing
|
||
you notice on entering, not a thing an audit finds. ``since`` None (a
|
||
first seed) means nothing is new. Judged rows never count."""
|
||
if since is None:
|
||
return {"count": 0, "examples": []}
|
||
fresh = [
|
||
r for r in rows
|
||
if r.proposal_basis == "derive" and r.proposal_group
|
||
and r.status in _MECHANICAL_TODO and r.vanished_at is None
|
||
and r.created_at is not None and r.created_at > since
|
||
]
|
||
fresh.sort(key=lambda r: r.created_at, reverse=True)
|
||
return {
|
||
"count": len(fresh),
|
||
"examples": [
|
||
{"label": ("." if r.kind == "css" else "") + r.symbol,
|
||
"path": r.path, "group": r.proposal_group}
|
||
for r in fresh[:top]
|
||
],
|
||
}
|
||
|
||
|
||
async def confirm_proposals(
|
||
user_id: int,
|
||
project_id: int,
|
||
*,
|
||
snippet_id: int = 0,
|
||
path: str = "",
|
||
basis: str = "",
|
||
min_score: float = 0.0,
|
||
) -> dict:
|
||
"""Turn reviewed canon proposals into `instance` rows, in one batch.
|
||
|
||
At least one of snippet_id / path / basis must narrow the batch — "confirm
|
||
everything proposed" without having looked is not a judgment. Each row
|
||
becomes instance-of-its-proposed-snippet, classified_by="agent", reason
|
||
naming the basis and score; the proposal is retired. Returns
|
||
{"confirmed": N}."""
|
||
from scribe.services import access
|
||
|
||
if not (snippet_id or path.strip() or basis.strip()):
|
||
raise ValueError(
|
||
"name what you reviewed: confirm by snippet_id, path, and/or basis"
|
||
)
|
||
if not await access.can_write_project(user_id, project_id):
|
||
raise ValueError(f"project {project_id} not found or no write access")
|
||
from sqlalchemy import or_
|
||
|
||
conds = [
|
||
CodeShape.project_id == project_id,
|
||
CodeShape.status.in_(_MECHANICAL_TODO),
|
||
CodeShape.vanished_at.is_(None),
|
||
CodeShape.proposed_snippet_id.isnot(None),
|
||
]
|
||
if snippet_id:
|
||
conds.append(CodeShape.proposed_snippet_id == snippet_id)
|
||
if path.strip():
|
||
clean = path.strip().strip("/")
|
||
conds.append(or_(CodeShape.path == clean, CodeShape.path.like(clean + "/%")))
|
||
if basis.strip():
|
||
conds.append(CodeShape.proposal_basis == basis.strip())
|
||
now = datetime.now(timezone.utc)
|
||
confirmed = 0
|
||
async with async_session() as session:
|
||
rows = (await session.execute(select(CodeShape).where(*conds))).scalars().all()
|
||
for row in rows:
|
||
if (row.proposal_score or 0.0) < min_score:
|
||
continue
|
||
await _judge(
|
||
session, row, status="instance", snippet_id=row.proposed_snippet_id,
|
||
by="agent", at=now,
|
||
reason=(
|
||
f"confirmed {row.proposal_basis} proposal"
|
||
f" ({(row.proposal_score or 0.0):.2f})"
|
||
),
|
||
)
|
||
confirmed += 1
|
||
await session.commit()
|
||
return {"confirmed": confirmed}
|
||
|
||
|
||
# --- the divergence readout (#2793): button B where button A is canon -------
|
||
#
|
||
# Three answers the ledger can now give mechanically:
|
||
# DIVERGENCE a shape NEW since the previous refresh, in a directory+kind
|
||
# where one canon dominates the judged siblings, that the
|
||
# proposer did not match to that canon → `diverges_from=#N`.
|
||
# Read: "button B appeared where button A is canon — divergence
|
||
# or variant? classify it." Surfaced in the coverage readout and
|
||
# in-band at write time (the prior-art hook names the shapes).
|
||
# HISTORY every judgment / vanish / reappearance / drift is an event;
|
||
# shape_history answers "what was used here, when, and why".
|
||
# RECHECK an instance/variant whose body moved since it was judged is
|
||
# flagged recheck_at (sync) — the judgment stands, re-confirm it.
|
||
|
||
# A canon dominates a directory+kind when at least this many siblings are
|
||
# judged (canonical/instance) and this share of them answer to one snippet.
|
||
# How much a payload must resemble a canon before the hook may assert, with
|
||
# nobody watching, that a shape IS an instance of it.
|
||
#
|
||
# WHY A FLOOR AT ALL. There was none: any score the semantic arm produced
|
||
# counted, so 0.69 asserted as confidently as 0.95, and the score went into
|
||
# the reason line without ever being compared to anything. Those rows are
|
||
# permanent — they feed `dominant_canon`, which then tells the next writer in
|
||
# that directory what to build from — so a thin stamp does not stay thin, it
|
||
# compounds.
|
||
#
|
||
# 0.80 rather than the retrieval floors near 0.70, deliberately. Those bars
|
||
# decide whether to SHOW someone a record, where being wrong costs a glance.
|
||
# This one decides whether to RECORD a claim about the codebase unattended,
|
||
# where being wrong costs a wrong instruction to everyone who writes in that
|
||
# directory afterwards. An unattended write should need more evidence than a
|
||
# suggestion, not the same.
|
||
_RESEMBLE_MIN = 0.80
|
||
|
||
_DENSITY_MIN_JUDGED = 3
|
||
_DENSITY_SHARE = 0.6
|
||
|
||
|
||
def comparable_siblings(rows: Iterable, form: str) -> list:
|
||
"""The siblings a candidate of ``form`` can honestly be counted against.
|
||
|
||
Excludes only a KNOWN family contradiction, via `families_conflict` — the
|
||
same predicate the divergence gate uses, so the denominator and the gate
|
||
cannot drift into disagreeing about what "comparable" means.
|
||
|
||
`form` is a FORM (`fn`, `async-fn`, `type`, …), never a family:
|
||
`families_conflict` coarsens both sides itself, and handing it a family
|
||
makes `shape_family("callable")` return "" so nothing is excluded — a
|
||
narrowing that silently becomes a no-op while still reading as applied.
|
||
|
||
AN UNREADABLE SIBLING STAYS COUNTED, and that direction is the point.
|
||
Dropping it would shrink `judged`, raise the dominant canon's share, and
|
||
make the check fire MORE on the directories it can read least. Every
|
||
unknown-form decision in this module goes the same way: quieter, never
|
||
more confident.
|
||
"""
|
||
if not form:
|
||
return list(rows)
|
||
return [
|
||
r for r in rows
|
||
if not families_conflict(
|
||
shape_form(getattr(r, "signature", "") or "", getattr(r, "kind", "sym")),
|
||
form,
|
||
)
|
||
]
|
||
|
||
|
||
def dominant_canon(rows: Iterable[CodeShape]) -> tuple[int, int, int] | None:
|
||
"""(snippet_id, its_count, judged_count) when one canon dominates these
|
||
sibling rows (same directory + kind), else None."""
|
||
counts: dict[int, int] = {}
|
||
judged = 0
|
||
for r in rows:
|
||
if r.status in ("canonical", "instance") and r.snippet_id is not None:
|
||
judged += 1
|
||
counts[r.snippet_id] = counts.get(r.snippet_id, 0) + 1
|
||
if judged < _DENSITY_MIN_JUDGED or not counts:
|
||
return None
|
||
sid, n = max(counts.items(), key=lambda kv: (kv[1], -kv[0]))
|
||
if n / judged < _DENSITY_SHARE:
|
||
return None
|
||
return sid, n, judged
|
||
|
||
|
||
def _dir_of(path: str) -> str:
|
||
return path.rsplit("/", 1)[0] if "/" in path else ""
|
||
|
||
|
||
async def canon_density(
|
||
project_id: int, path: str, kind: str, form: str = ""
|
||
) -> tuple[int, int, int, str] | None:
|
||
"""The dominant canon for the directory ``path`` sits in, for ``kind`` —
|
||
the write-time question "is this a canon-dense place?".
|
||
|
||
``form`` is the candidate's own FORM — `fn`, `async-fn`, `type`, … as
|
||
`shape_form` returns it, NOT a family. `families_conflict` coarsens both
|
||
sides itself, and handing it a family makes `shape_family("callable")`
|
||
return "" so nothing is ever excluded: the narrowing silently becomes a
|
||
no-op that still reads as applied. It narrows the DENOMINATOR
|
||
(#4208). Without it the base rate was computed over every code symbol in
|
||
the directory as one bucket: "372 judged siblings" counted a dataclass, a
|
||
CSS-less constant and an async service unit as three comparable things,
|
||
and the share that came out of that was a statement about a population
|
||
nobody had asked a question about.
|
||
|
||
EXCLUDES ONLY A KNOWN CONFLICT, using `families_conflict` — the same
|
||
predicate the divergence gate uses, so the two cannot drift apart. A
|
||
sibling whose form is unreadable STAYS COUNTED. That direction is
|
||
deliberate and it is the one that matters: dropping unknown rows would
|
||
shrink `judged`, raise the share, and make the check fire MORE on exactly
|
||
the directories it can read least. Every other unknown-form decision in
|
||
this module goes the same way — quieter, never more confident.
|
||
|
||
NOT narrowed to the candidate's exact form, for the reason `shape_family`
|
||
gives at length: `fn` beside `async-fn` is the acceptance case of
|
||
milestone #2793, not noise. Bucketing the denominator by form would take
|
||
the async canon out of a sync candidate's count and silence that flag —
|
||
the same inversion the first form gate made, one layer down.
|
||
"""
|
||
directory = _dir_of(path)
|
||
async with async_session() as session:
|
||
rows = (
|
||
await session.execute(
|
||
select(CodeShape).where(
|
||
CodeShape.project_id == project_id,
|
||
CodeShape.kind == kind,
|
||
CodeShape.vanished_at.is_(None),
|
||
CodeShape.path.like(directory + "/%") if directory
|
||
else CodeShape.path.notlike("%/%"),
|
||
)
|
||
)
|
||
).scalars().all()
|
||
siblings = [r for r in rows if _dir_of(r.path) == directory]
|
||
siblings = comparable_siblings(siblings, form)
|
||
dom = dominant_canon(siblings)
|
||
if dom is None:
|
||
return None
|
||
# The canon's form travels with the count, read from its OWN judged rows.
|
||
# `dominant_canon` keeps its three-value shape: it answers "what dominates
|
||
# here", which is still a true and separately useful question, and its
|
||
# test pins that arithmetic. What changed is that nobody acts on the count
|
||
# alone any more.
|
||
return dom[0], dom[1], dom[2], canon_form(siblings, dom[0])
|
||
|
||
|
||
async def write_time_divergence(
|
||
project_id: int, path: str, shapes: list[tuple[str, str]], stamped: list[dict],
|
||
code: str = "",
|
||
) -> list[dict]:
|
||
"""The in-band check for the shapes the hook named at ``path``: for each
|
||
kind whose directory has a dominant canon, the named shapes that are
|
||
not (already or just now) that canon's instance/canonical — new or
|
||
unclassified rows only; a judged shape is not re-litigated at every
|
||
edit. Returns [{symbol, kind, canon_snippet_id, instances, judged}]."""
|
||
just_stamped = {(s["symbol"], s["kind"]): s["snippet_id"] for s in stamped}
|
||
out: list[dict] = []
|
||
if not shapes:
|
||
return out
|
||
# DENSITY IS NOW PER CANDIDATE, not per kind (#4208): the denominator
|
||
# excludes siblings whose family contradicts what is being written, so it
|
||
# cannot be computed until the candidate's own form is known. Cached on
|
||
# (kind, family) — a write names a handful of shapes and they collapse to
|
||
# one or two buckets, so this is the same one-or-two queries as before.
|
||
#
|
||
# The cost is that the row load below no longer sits behind an early exit
|
||
# on "nothing is dense here". That is one indexed lookup on
|
||
# (project_id, path), and it has to happen first regardless: the
|
||
# candidate's signature comes from its stored row when the payload does
|
||
# not carry one.
|
||
_density: dict[tuple[str, str], tuple[int, int, int, str] | None] = {}
|
||
|
||
async def density_for(kind: str, form: str):
|
||
# Keyed and passed as a FORM, not a family — see `canon_density`.
|
||
key = (kind, form)
|
||
if key not in _density:
|
||
_density[key] = await canon_density(project_id, path, kind, form)
|
||
return _density[key]
|
||
|
||
async with async_session() as session:
|
||
rows = (
|
||
await session.execute(
|
||
select(CodeShape).where(
|
||
CodeShape.project_id == project_id,
|
||
CodeShape.path == path,
|
||
CodeShape.vanished_at.is_(None),
|
||
)
|
||
)
|
||
).scalars().all()
|
||
by_key = {(r.symbol, r.kind): r for r in rows}
|
||
for kind, name in shapes:
|
||
row = by_key.get((name, kind))
|
||
# The candidate's own form, read before anything is counted. Signature
|
||
# from the payload first — a shape being written now may have no row
|
||
# yet — falling back to the stored row's.
|
||
mine = shape_form(
|
||
signature_in(code, name, kind) or getattr(row, "signature", "") or "", kind
|
||
)
|
||
dom = await density_for(kind, mine)
|
||
if not dom:
|
||
continue
|
||
sid, n, judged, cform = dom
|
||
if just_stamped.get((name, kind)) == sid:
|
||
continue
|
||
if row is not None and (
|
||
row.status != "unclassified" or row.proposed_snippet_id == sid
|
||
):
|
||
continue
|
||
# IS THIS EVEN THE SAME CATEGORY OF THING AS THE CANON? (#4204)
|
||
# Without this the line was a pure base rate: "most things here are X,
|
||
# so be X", with the candidate never examined at all. It told a frozen
|
||
# dataclass to build from the `async_session` service canon.
|
||
#
|
||
# FAMILY, not form, and only on contradiction — see `shape_family`. A
|
||
# divergence prompt is ABOUT a mismatch, so requiring the candidate to
|
||
# match would silence the check precisely where it belongs.
|
||
#
|
||
# `mine` was read above, before the denominator was counted — the
|
||
# same value serves both, and they must not be able to disagree.
|
||
# An unreadable signature produces a fair question rather than a
|
||
# guess, because `families_conflict` needs both sides.
|
||
if families_conflict(mine, cform):
|
||
continue
|
||
out.append({"symbol": name, "kind": kind, "canon_snippet_id": sid,
|
||
"instances": n, "judged": judged})
|
||
return out
|
||
|
||
|
||
# How many other files a family line names before "…" — enough to go look,
|
||
# not a wall.
|
||
_DERIVE_FILES_SHOWN = 4
|
||
|
||
|
||
async def write_time_derive(
|
||
project_id: int, path: str, shapes: list[tuple[str, str]]
|
||
) -> list[dict]:
|
||
"""The in-band DERIVE check (#2900): for each (kind, name) the hook
|
||
named at ``path``, what the ledger already knows about that name
|
||
elsewhere in the project —
|
||
|
||
family the name sits in a derive-first group (code: identical body
|
||
in N files or the same name in ≥3; CSS: the same class in
|
||
≥2 files, note 2917): "this is a known family with no canon
|
||
— derive it now, don't add a copy";
|
||
canon a `canonical` row of that name at another path: "this is
|
||
canon #N at <path> — reuse, don't redefine".
|
||
|
||
Only for shapes not yet judged at ``path`` (a judged shape is not
|
||
re-litigated at every edit), never for the canon's own file. Returns
|
||
[{symbol, kind, key, family?|canon?}] — `key` is the dedup token the
|
||
hook keeps per session (the group id, or canon:<snippet_id>)."""
|
||
wanted = {(k, _norm_symbol(n)): n for k, n in shapes if n}
|
||
if not wanted:
|
||
return []
|
||
async with async_session() as session:
|
||
rows = (
|
||
await session.execute(
|
||
select(CodeShape).where(
|
||
CodeShape.project_id == project_id,
|
||
CodeShape.vanished_at.is_(None),
|
||
CodeShape.symbol.in_({norm for (_k, norm) in wanted}),
|
||
)
|
||
)
|
||
).scalars().all()
|
||
out: list[dict] = []
|
||
for (kind, norm), name in wanted.items():
|
||
same = [r for r in rows if r.kind == kind and _norm_symbol(r.symbol) == norm]
|
||
here = next((r for r in same if r.path == path), None)
|
||
if here is not None and here.status not in _MECHANICAL_TODO:
|
||
continue # judged here (or this IS the canon): nothing to say
|
||
others = [r for r in same if r.path != path]
|
||
label = ("." if kind == "css" else "") + name
|
||
canon = next((r for r in others if r.status == "canonical" and r.snippet_id), None)
|
||
if canon is not None:
|
||
out.append({"symbol": name, "kind": kind, "key": f"canon:{canon.snippet_id}",
|
||
"canon": {"snippet_id": canon.snippet_id, "path": canon.path,
|
||
"label": label}})
|
||
continue
|
||
grouped = [r for r in others if r.proposal_group and r.status in _MECHANICAL_TODO]
|
||
if here is not None and here.proposal_group:
|
||
grouped = [r for r in grouped if r.proposal_group == here.proposal_group] or grouped
|
||
if not grouped:
|
||
continue
|
||
group = grouped[0].proposal_group
|
||
members = [r for r in grouped if r.proposal_group == group]
|
||
files = sorted({r.path for r in members})
|
||
family = {
|
||
"group": group, "label": label,
|
||
"identical": not group.startswith("name:"),
|
||
"files": files[:_DERIVE_FILES_SHOWN], "file_count": len(files),
|
||
"size": len(members) + (1 if here is not None else 0),
|
||
}
|
||
if kind == "css":
|
||
# What renders the family (milestone 302): the members' consumer
|
||
# files, the row at `path` included when it already exists.
|
||
ids = [r.id for r in members] + ([here.id] if here is not None else [])
|
||
edges = await consumers_of(ids)
|
||
family["consumers"] = consumer_summary(
|
||
e.path for es in edges.values() for e in es
|
||
)
|
||
out.append({"symbol": name, "kind": kind, "key": group, "family": family})
|
||
return out
|
||
|
||
|
||
async def flag_divergence(project_id: int, *, since: datetime | None) -> int:
|
||
"""Flag shapes created after ``since`` (the previous refresh) that sit
|
||
where a canon dominates and were not proposed as that canon. With no
|
||
previous refresh (first seed) nothing is new, nothing is flagged.
|
||
Standing flags persist until judged. Returns how many are flagged."""
|
||
if since is None:
|
||
return 0
|
||
async with async_session() as session:
|
||
rows = (
|
||
await session.execute(
|
||
select(CodeShape).where(
|
||
CodeShape.project_id == project_id,
|
||
CodeShape.vanished_at.is_(None),
|
||
)
|
||
)
|
||
).scalars().all()
|
||
by_dir: dict[tuple[str, str], list[CodeShape]] = {}
|
||
for r in rows:
|
||
by_dir.setdefault((_dir_of(r.path), r.kind), []).append(r)
|
||
flagged = 0
|
||
for siblings in by_dir.values():
|
||
dom = dominant_canon(siblings)
|
||
# Computed once per directory rather than per row: the canon's
|
||
# form is a property of the canon, not of who is being judged.
|
||
cform = canon_form(siblings, dom[0]) if dom else FORM_UNKNOWN
|
||
for r in siblings:
|
||
if r.status not in _MECHANICAL_TODO:
|
||
continue
|
||
if r.diverges_from is not None:
|
||
flagged += 1
|
||
continue
|
||
if dom is None or r.created_at is None or r.created_at <= since:
|
||
continue
|
||
if r.proposed_snippet_id == dom[0]:
|
||
continue # the proposer already says "instance of the canon"
|
||
# ...and the converse, which is the only evidence here that
|
||
# is about MEANING rather than shape (#4208). The four false
|
||
# prompts #4204 left standing are callables in a directory of
|
||
# callables: at the signature level they are indistinguishable
|
||
# from #2793's acceptance case, a sync `confirmDanger` beside
|
||
# an async confirm canon, and no refinement of `shape_form`
|
||
# ever separates them — a registry accessor and a service unit
|
||
# differ by the JOB they do, which a signature does not carry.
|
||
#
|
||
# The proposer does read bodies, and when its semantic arm
|
||
# compared this one against every canon in its language family
|
||
# and matched none of them, that is a positive finding that
|
||
# this shape is not the canon's work. Urging the canon anyway
|
||
# would be asserting over a measurement we already hold.
|
||
#
|
||
# Only the conclusive miss is stored, so an unexamined row and
|
||
# a body too thin to embed still ask the question rather than
|
||
# being quietly excused.
|
||
if r.proposal_basis == BASIS_NO_SEMANTIC_MATCH:
|
||
continue
|
||
# The same structural test the write-time check applies
|
||
# (#4204). The sweep and the hook must agree about what counts
|
||
# as divergence, or an audit contradicts the line the writer
|
||
# was shown at the keyboard.
|
||
if families_conflict(shape_form(r.signature or "", r.kind), cform):
|
||
continue
|
||
r.diverges_from = dom[0]
|
||
flagged += 1
|
||
await session.commit()
|
||
return flagged
|
||
|
||
|
||
# How many of a canon's rows a review listing shows before "…" — enough to
|
||
# judge from, not the whole table.
|
||
_REVIEW_ROWS_SHOWN = 12
|
||
|
||
|
||
def stamps_to_review(rows: Iterable[CodeShape], *, top: int = 10) -> dict:
|
||
"""Judged rows whose EVIDENCE no longer meets the bar the ledger now
|
||
holds, and canons whose own rows no longer agree what they are.
|
||
|
||
THIS CHANGES NOTHING, AND THAT IS THE POINT. Every row here was written
|
||
unattended by the write-path hook on a similarity score, and the damage
|
||
that caused was not the score being wrong once — it was a machine
|
||
asserting a permanent classification with nobody reading it. A sweep that
|
||
silently un-asserted them would be the identical mistake with a larger
|
||
blast radius and a cleaner conscience. So this reports; an agent reads the
|
||
evidence, decides, and records the decision through `classify_shapes`
|
||
under its own name. The ledger should be able to say "these look wrong"
|
||
without being able to act on it alone.
|
||
|
||
Two lists, because they are different questions:
|
||
|
||
`weak` — rows stamped on a resemblance below `_RESEMBLE_MIN`. The floor
|
||
arrived after they did (#4204), and a guard at the point of classification
|
||
does not undo the classifications already stored (lesson #4202). Rows a
|
||
person or an audit judged are never listed, however old: a human judgment
|
||
is not weak evidence, it is a different kind of evidence.
|
||
|
||
`incoherent` — canons whose membership no longer agrees what the canon
|
||
IS. A canon is a claim that some set of shapes are the same sort of
|
||
thing; when its own members are a class, three getters and a dozen tests,
|
||
that claim has stopped being true, and every base-rate reading built on
|
||
it is reading noise. `canon_form` already makes such a canon fall silent
|
||
— this is what makes it VISIBLE, which is the half that was missing.
|
||
|
||
What counts as disagreement is `canon_coherence`, and it is deliberately
|
||
looser than "one form": it compares FAMILIES, and it does not hold a
|
||
method against the class it is defined beside. Both corrections came from
|
||
this surface's first live day, when the only two canons it reported were
|
||
both sound (#4220). A review surface whose output is noise is one that
|
||
stops being read, which costs more than the check was ever worth.
|
||
|
||
`strangers` names the rows that do not fit, not the first dozen members:
|
||
the reader's question is which ones are wrong. `unattended` counts rows
|
||
written by the hook with nobody reading — an incoherence made entirely of
|
||
judged rows is far more likely to be this check being too strict than a
|
||
ledger full of junk, and the reader should be able to see that without
|
||
opening the canon.
|
||
"""
|
||
live = [r for r in rows if r.vanished_at is None]
|
||
weak = []
|
||
for r in live:
|
||
if r.status not in _NEEDS_TARGET or r.classified_by != "hook":
|
||
continue
|
||
score = stamp_score(r.reason)
|
||
if score is None or score >= _RESEMBLE_MIN:
|
||
continue
|
||
weak.append({
|
||
"path": r.path, "symbol": r.symbol, "kind": r.kind,
|
||
"status": r.status, "snippet_id": r.snippet_id,
|
||
"score": score, "signature": r.signature or "",
|
||
"form": shape_form(r.signature or "", r.kind),
|
||
"classified_at": r.classified_at.isoformat() if r.classified_at else None,
|
||
})
|
||
weak.sort(key=lambda d: (d["score"], d["path"], d["symbol"]))
|
||
|
||
by_canon: dict[int, list[CodeShape]] = {}
|
||
for r in live:
|
||
if r.status in _NEEDS_TARGET and r.snippet_id:
|
||
by_canon.setdefault(int(r.snippet_id), []).append(r)
|
||
incoherent = []
|
||
for sid, members in by_canon.items():
|
||
coh = canon_coherence(members)
|
||
if coh["readable"] < _DENSITY_MIN_JUDGED:
|
||
continue # too few to say anything either way
|
||
if coh["coheres"]:
|
||
continue
|
||
unattended = sum(1 for r in members if r.classified_by in _UNATTENDED_BY)
|
||
incoherent.append({
|
||
"snippet_id": sid,
|
||
"judged": len(members),
|
||
"forms": coh["forms"],
|
||
"families": coh["families"],
|
||
"family": coh["family"],
|
||
"attached": coh["attached"],
|
||
"unattended": unattended,
|
||
# The listing below is capped; without this you cannot tell a
|
||
# canon with twelve strangers from one with three hundred.
|
||
"stranger_count": len(coh["strangers"]),
|
||
"weak_rows": sum(
|
||
1 for r in members
|
||
if r.classified_by in _UNATTENDED_BY
|
||
and (stamp_score(r.reason) or 1.0) < _RESEMBLE_MIN
|
||
),
|
||
# The rows that do NOT fit — not the first dozen members. The
|
||
# reader's question is "which ones are wrong", and a sample of
|
||
# the majority cannot answer it.
|
||
"strangers": [
|
||
{"path": r.path, "symbol": r.symbol,
|
||
"signature": r.signature or "", "by": r.classified_by,
|
||
"form": shape_form(r.signature or "", r.kind)}
|
||
for r in coh["strangers"][:_REVIEW_ROWS_SHOWN]
|
||
],
|
||
})
|
||
incoherent.sort(key=lambda d: (-d["weak_rows"], -d["unattended"], -d["judged"]))
|
||
return {
|
||
"weak_count": len(weak),
|
||
"weak": weak[:top],
|
||
"incoherent_count": len(incoherent),
|
||
"incoherent": incoherent[:top],
|
||
"floor": _RESEMBLE_MIN,
|
||
}
|
||
|
||
|
||
def divergence_summary(rows: Iterable[CodeShape], *, top: int = 10) -> dict:
|
||
"""Readout view: flagged shapes (newest first) and the recheck count."""
|
||
flagged = [r for r in rows if r.diverges_from is not None and r.status in _MECHANICAL_TODO]
|
||
flagged.sort(key=lambda r: (r.created_at or datetime.min.replace(tzinfo=timezone.utc)), reverse=True)
|
||
recheck = sum(1 for r in rows if r.recheck_at is not None and r.vanished_at is None)
|
||
return {
|
||
"divergent": len(flagged),
|
||
"divergence": [
|
||
{"path": r.path, "symbol": r.symbol, "kind": r.kind,
|
||
"canon_snippet_id": r.diverges_from}
|
||
for r in flagged[:top]
|
||
],
|
||
"recheck": recheck,
|
||
}
|
||
|
||
|
||
async def shape_history(
|
||
user_id: int, project_id: int, path: str, *, symbol: str = "", limit: int = 200
|
||
) -> dict:
|
||
"""What was used at ``path`` (a file or directory), when, and why: the
|
||
current rows plus their events, oldest first. Read-gated like every
|
||
other ledger read; {} when the caller cannot read the project."""
|
||
from sqlalchemy import or_
|
||
|
||
from scribe.services import access
|
||
|
||
if not await access.can_read_project(user_id, project_id):
|
||
return {}
|
||
clean = (path or "").strip().strip("/")
|
||
conds = [CodeShape.project_id == project_id]
|
||
if clean:
|
||
conds.append(or_(CodeShape.path == clean, CodeShape.path.like(clean + "/%")))
|
||
if symbol.strip():
|
||
conds.append(CodeShape.symbol == symbol.strip())
|
||
async with async_session() as session:
|
||
rows = (
|
||
await session.execute(
|
||
select(CodeShape).where(*conds)
|
||
.order_by(CodeShape.path, CodeShape.symbol, CodeShape.kind)
|
||
.limit(500)
|
||
)
|
||
).scalars().all()
|
||
ids = [r.id for r in rows]
|
||
events = (
|
||
await session.execute(
|
||
select(CodeShapeEvent).where(CodeShapeEvent.shape_id.in_(ids))
|
||
.order_by(CodeShapeEvent.at.asc(), CodeShapeEvent.id.asc())
|
||
.limit(max(1, min(limit, 1000)))
|
||
)
|
||
).scalars().all() if ids else []
|
||
return {
|
||
"shapes": [
|
||
{
|
||
"path": r.path, "symbol": r.symbol, "kind": r.kind,
|
||
"status": r.status, "snippet_id": r.snippet_id,
|
||
"classified_by": r.classified_by, "reason": r.reason,
|
||
"first_seen_commit": r.first_seen_commit,
|
||
"last_seen_commit": r.last_seen_commit,
|
||
"first_seen_at": iso(r.created_at),
|
||
"vanished_at": iso(r.vanished_at),
|
||
"recheck_at": iso(r.recheck_at),
|
||
"diverges_from": r.diverges_from,
|
||
}
|
||
for r in rows
|
||
],
|
||
"events": [e.to_dict() for e in events],
|
||
}
|