bvandeusenandClaude Opus 5.5
37ca2ee958
CI & Build / Python lint (push) Successful in 2s
CI & Build / Plugin hooks (push) Successful in 14s
CI & Build / TypeScript typecheck (push) Successful in 57s
CI & Build / integration (push) Successful in 2m1s
CI & Build / Python tests (push) Successful in 2m23s
CI & Build / Build & push image (push) Successful in 52s
Lessons share one register, so any two score well above unrelated text and a fixed similarity bar sat inside that band: every no-rule lesson "resembled" most of the others and the named group grew with the pool. A neighbour now counts only when it stands above the lesson's own background (median and MAD of its similarity to every lesson it can reach, in that register's spread), and a group is a clique: every pair clears the bar from both sides, so one broad lesson cannot join unrelated ones. The bar, the minimum background and the fetch sizes are stated as defaults. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
922 lines
39 KiB
Python
922 lines
39 KiB
Python
"""Lessons point at rules (milestone 440, #4196).
|
|
|
|
A lesson is a non-binding record of one situation; a rule is the binding
|
|
choice for a class of them. This service owns the link between the two — which
|
|
rule a lesson is an instance of — and both directions of reading it.
|
|
|
|
THE STATES (models/lesson_rule_link.py says why each exists):
|
|
`suggested` while evidence accumulates, `confirmed` or `rejected` once a
|
|
judgment is made. Every write here is a JUDGMENT, so every write lands as
|
|
confirmed or rejected; `suggested` rows come from the co-surfacing recorder
|
|
(#4637), never from a caller naming a rule.
|
|
|
|
THE THIRD ANSWER (#4631). "Which rule is this an instance of?" has three
|
|
answers, and only two of them are links: a rule named (confirmed), or NO RULE
|
|
FITS — a `lesson_no_rule` row carrying the reason. A lesson with neither is
|
|
UNJUDGED, and `list_unjudged` lists those. The three are kept apart because a
|
|
lesson that stands alone is the raw material for a rule nobody has written yet
|
|
(#4634), and that is unreadable if "nobody looked" means the same thing.
|
|
|
|
THE SOFT LINK (#4637). When one hook request puts a lesson and a rule in
|
|
front of the reader together, `co_surfaced` records the pair as evidence on a
|
|
SUGGESTED link, and once that evidence crosses a bar the next co-arrival asks
|
|
the reader to judge it. The evidence counts distinct SITUATIONS, not
|
|
arrivals: two records that resemble each other arrive together on every
|
|
similar prompt, and counting each time would measure their similarity over
|
|
and over rather than any relation between them.
|
|
|
|
ACL (rule 78). Linking changes what a lesson says about itself, so it needs
|
|
WRITE on the lesson (`access.can_write_note`, share-aware). It names a rule, so
|
|
it needs the rule to be one the caller may read — rules are owner-scoped, and
|
|
`rulebooks._fetch_owned_rule` / `_owned_rules_clause` are that check's two
|
|
forms. A read shows only the rules the READER owns: a lesson shared with
|
|
someone must not hand them the titles of its owner's private rules.
|
|
"""
|
|
from __future__ import annotations
|
|
|
|
import hashlib
|
|
import logging
|
|
import re
|
|
import statistics
|
|
from datetime import datetime, timedelta, timezone
|
|
|
|
from sqlalchemy import and_, delete, exists, func, not_, or_, select
|
|
from sqlalchemy.exc import IntegrityError
|
|
|
|
from scribe.models import async_session
|
|
from scribe.models.lesson_rule_link import (
|
|
CONFIRMED, REJECTED, SUGGESTED, LessonNoRule, LessonRuleLink,
|
|
)
|
|
from scribe.models.note import Note
|
|
from scribe.models.rulebook import Rule
|
|
|
|
logger = logging.getLogger(__name__)
|
|
|
|
# What a caller says to judge one pair. Words rather than the stored states,
|
|
# because "confirm" and "reject" are acts and the states are their results.
|
|
VERDICTS = {"confirm": CONFIRMED, "reject": REJECTED}
|
|
|
|
# Recorded on a link that set-semantics removed, so a reader of the rejected
|
|
# row can tell an explicit "not this rule" from a link dropped by a rewrite.
|
|
_REMOVED_NOTE = "removed from the lesson's rules by an update"
|
|
|
|
# Recorded on a confirmed link that a "no rule fits" answer overturned, ahead
|
|
# of that answer's reason.
|
|
_NO_RULE_NOTE = "no rule fits: "
|
|
|
|
# What `rule_judgment` reads on a lesson payload — the three answers.
|
|
LINKED, NO_RULE, UNJUDGED = "linked", "no_rule", "unjudged"
|
|
|
|
# THE SOFT-LINK EVIDENCE BAR (#4637) — defaults, stated as defaults (rules 32,
|
|
# 115): nothing about this install chose them.
|
|
#
|
|
# Three distinct situations before a pair is put to the reader. One is
|
|
# coincidence and two is a pattern only by courtesy; three different prompts
|
|
# or files that each brought the same lesson and rule up together is the
|
|
# smallest count that says the pairing is not a single resemblance repeated.
|
|
PROPOSE_SITUATIONS = 3
|
|
# A proposal the reader passed over is asked again only after this long, so
|
|
# one session's run of related prompts does not repeat the same question on
|
|
# every turn.
|
|
PROPOSE_COOLDOWN = timedelta(hours=6)
|
|
# Fingerprints kept per link. The bar needs three; the rest is for the reader
|
|
# judging later, and a list that grew with every situation would bloat a row
|
|
# whose answer stopped changing long ago.
|
|
_SITUATION_CAP = 50
|
|
# A token shorter than this is grammar, not situation — "a", "is", "to".
|
|
_MIN_TOKEN = 3
|
|
|
|
# How many rules a new lesson is offered to judge against. A handful, not the
|
|
# fifty `what_might_apply` returns: this is read in the create response, at
|
|
# the moment of writing, and a long tail there buries the one that fits.
|
|
CANDIDATE_LIMIT = 5
|
|
|
|
|
|
def _ids(values) -> list[int]:
|
|
"""Positive ints, de-duplicated, in the order given."""
|
|
out: list[int] = []
|
|
for v in values or []:
|
|
try:
|
|
i = int(v)
|
|
except (TypeError, ValueError):
|
|
raise ValueError(f"rule id {v!r} is not an integer")
|
|
if i > 0 and i not in out:
|
|
out.append(i)
|
|
return out
|
|
|
|
|
|
async def _require_lesson_writable(user_id: int, lesson_id: int) -> None:
|
|
from scribe.services import access
|
|
from scribe.services import lessons as lessons_svc
|
|
|
|
note = await lessons_svc.get_lesson(user_id, lesson_id)
|
|
if note is None:
|
|
raise ValueError(f"lesson {lesson_id} not found")
|
|
if not await access.can_write_note(user_id, lesson_id):
|
|
raise PermissionError(f"lesson {lesson_id} is not yours to change")
|
|
|
|
|
|
async def require_rules(user_id: int, rule_ids) -> list[int]:
|
|
"""The ids, validated as rules the caller owns — all of them or none.
|
|
|
|
Checked BEFORE anything is written, by every caller, so a lesson create
|
|
that names a rule it cannot see fails without leaving a half-linked
|
|
lesson behind.
|
|
"""
|
|
from scribe.services import rulebooks as rulebooks_svc
|
|
|
|
wanted = _ids(rule_ids)
|
|
missing = [rid for rid in wanted if await rulebooks_svc.get_rule(rid, user_id) is None]
|
|
if missing:
|
|
raise ValueError(
|
|
f"rule(s) {missing} not found — a lesson can point only at a rule "
|
|
"you can read. Nothing was linked."
|
|
)
|
|
return wanted
|
|
|
|
|
|
def require_one_answer(linked, no_rule: str) -> None:
|
|
"""Refuse "these rules fit" and "no rule fits" together, before any write
|
|
— they contradict, and either order of applying them loses one."""
|
|
if linked and (no_rule or "").strip():
|
|
raise ValueError(
|
|
"rule_ids and no_rule are the two answers to \"which rule is this "
|
|
"an instance of?\" — give the one that holds. Nothing was written."
|
|
)
|
|
|
|
|
|
async def _upsert(session, lesson_id: int, rule_id: int, state: str, note: str) -> None:
|
|
now = datetime.now(timezone.utc)
|
|
row = (await session.execute(
|
|
select(LessonRuleLink).where(
|
|
LessonRuleLink.lesson_id == lesson_id,
|
|
LessonRuleLink.rule_id == rule_id,
|
|
)
|
|
)).scalar_one_or_none()
|
|
if row is None:
|
|
session.add(LessonRuleLink(
|
|
lesson_id=lesson_id, rule_id=rule_id, state=state,
|
|
note=note or None, judged_at=now,
|
|
))
|
|
return
|
|
# Evidence is kept across a judgment: a confirmation reads best beside
|
|
# what it rested on.
|
|
row.state = state
|
|
row.note = note or row.note
|
|
row.judged_at = now
|
|
|
|
|
|
async def _clear_no_rule(session, lesson_id: int) -> None:
|
|
"""A confirmed rule overturns "no rule fits" — both cannot be the answer."""
|
|
await session.execute(delete(LessonNoRule).where(LessonNoRule.lesson_id == lesson_id))
|
|
|
|
|
|
async def set_lesson_rules(
|
|
user_id: int, lesson_id: int, rule_ids, *, note: str = "",
|
|
) -> None:
|
|
"""Make the lesson's CONFIRMED rules exactly `rule_ids` (set-semantics).
|
|
|
|
A rule named here is confirmed, whatever state it was in: the writer of a
|
|
lesson saying "this is an instance of rule N" is the judgment the
|
|
suggested state waits for, so it needs no evidence bar.
|
|
|
|
A rule that WAS confirmed and is no longer named becomes `rejected`, not
|
|
deleted. Dropping it is a judgment that the lesson is not an instance of
|
|
that rule, and a deleted row would let the co-surfacing recorder propose
|
|
the same pair again.
|
|
"""
|
|
await _require_lesson_writable(user_id, lesson_id)
|
|
wanted = await require_rules(user_id, rule_ids)
|
|
async with async_session() as session:
|
|
current = (await session.execute(
|
|
select(LessonRuleLink).where(
|
|
LessonRuleLink.lesson_id == lesson_id,
|
|
LessonRuleLink.state == CONFIRMED,
|
|
)
|
|
)).scalars().all()
|
|
for row in current:
|
|
if row.rule_id not in wanted:
|
|
row.state = REJECTED
|
|
row.note = _REMOVED_NOTE
|
|
row.judged_at = datetime.now(timezone.utc)
|
|
for rid in wanted:
|
|
await _upsert(session, lesson_id, rid, CONFIRMED, note)
|
|
if wanted:
|
|
await _clear_no_rule(session, lesson_id)
|
|
await session.commit()
|
|
|
|
|
|
async def set_no_rule(user_id: int, lesson_id: int, why: str) -> dict:
|
|
"""Record that no rule governs this lesson's situation, and why.
|
|
|
|
A confirmed link the lesson still carries becomes `rejected`, with the
|
|
reason: saying none fits is saying the linked one does not either. A
|
|
second answer replaces the first, so the reason on file is the latest.
|
|
"""
|
|
why = (why or "").strip()
|
|
if not why:
|
|
raise ValueError(
|
|
"no_rule takes the reason no rule fits, in a line — it is what "
|
|
"lets the answer be re-judged when a rule is later written for "
|
|
"this situation."
|
|
)
|
|
await _require_lesson_writable(user_id, lesson_id)
|
|
now = datetime.now(timezone.utc)
|
|
async with async_session() as session:
|
|
confirmed = (await session.execute(
|
|
select(LessonRuleLink).where(
|
|
LessonRuleLink.lesson_id == lesson_id,
|
|
LessonRuleLink.state == CONFIRMED,
|
|
)
|
|
)).scalars().all()
|
|
for row in confirmed:
|
|
row.state = REJECTED
|
|
row.note = _NO_RULE_NOTE + why
|
|
row.judged_at = now
|
|
answer = await session.get(LessonNoRule, lesson_id)
|
|
if answer is None:
|
|
session.add(LessonNoRule(lesson_id=lesson_id, why=why, judged_at=now))
|
|
else:
|
|
answer.why = why
|
|
answer.judged_at = now
|
|
await session.commit()
|
|
return {"why": why, "judged_at": now.isoformat()}
|
|
|
|
|
|
async def judge_link(
|
|
user_id: int, lesson_id: int, rule_id: int, verdict: str, note: str = "",
|
|
) -> dict:
|
|
"""Confirm or reject one (lesson, rule) pair, suggested or not."""
|
|
state = VERDICTS.get((verdict or "").strip().lower())
|
|
if state is None:
|
|
raise ValueError(f"verdict must be one of {sorted(VERDICTS)}, got {verdict!r}")
|
|
await _require_lesson_writable(user_id, lesson_id)
|
|
[rid] = await require_rules(user_id, [rule_id])
|
|
async with async_session() as session:
|
|
await _upsert(session, lesson_id, rid, state, note)
|
|
if state == CONFIRMED:
|
|
await _clear_no_rule(session, lesson_id)
|
|
await session.commit()
|
|
row = (await session.execute(
|
|
select(LessonRuleLink).where(
|
|
LessonRuleLink.lesson_id == lesson_id,
|
|
LessonRuleLink.rule_id == rid,
|
|
)
|
|
)).scalar_one()
|
|
return row.to_dict()
|
|
|
|
|
|
async def rules_for_lessons(user_id: int, lesson_ids) -> dict[int, list[dict]]:
|
|
"""{lesson_id: [{id, title, kind, state, note}]}, for rules the READER owns.
|
|
|
|
One query for the whole set — a lesson list would otherwise be N+1.
|
|
Confirmed first, then suggested, then rejected: the order a reader cares
|
|
about them in.
|
|
"""
|
|
from scribe.services.rulebooks import _owned_rules_clause
|
|
|
|
ids = [int(i) for i in lesson_ids or []]
|
|
out: dict[int, list[dict]] = {i: [] for i in ids}
|
|
if not ids:
|
|
return out
|
|
order = {s: n for n, s in enumerate((CONFIRMED, SUGGESTED, REJECTED))}
|
|
async with async_session() as session:
|
|
rows = (await session.execute(
|
|
select(LessonRuleLink, Rule)
|
|
.join(Rule, Rule.id == LessonRuleLink.rule_id)
|
|
.where(LessonRuleLink.lesson_id.in_(ids))
|
|
.where(_owned_rules_clause(user_id))
|
|
)).all()
|
|
for link, rule in sorted(rows, key=lambda r: (order.get(r[0].state, 9), r[1].id)):
|
|
item = {
|
|
"id": rule.id, "title": rule.title, "kind": rule.kind,
|
|
"state": link.state, "note": link.note or "",
|
|
}
|
|
if link.state == SUGGESTED:
|
|
# What the suggestion rests on, so a reader can judge it from
|
|
# here rather than having to go and count.
|
|
item["evidence"] = evidence_summary(link.evidence)
|
|
out[link.lesson_id].append(item)
|
|
return out
|
|
|
|
|
|
async def lessons_for_rule(user_id: int, rule_id: int) -> list[dict]:
|
|
"""The lessons that point at one rule, readable by the caller (share-aware).
|
|
|
|
The reverse direction, and the one a rule's page needs: the concrete
|
|
situations that have been judged instances of it.
|
|
"""
|
|
from scribe.services.access import readable_notes_clause
|
|
|
|
async with async_session() as session:
|
|
rows = (await session.execute(
|
|
select(LessonRuleLink, Note)
|
|
.join(Note, Note.id == LessonRuleLink.lesson_id)
|
|
.where(LessonRuleLink.rule_id == int(rule_id))
|
|
.where(Note.deleted_at.is_(None))
|
|
.where(readable_notes_clause(user_id))
|
|
)).all()
|
|
order = {s: n for n, s in enumerate((CONFIRMED, SUGGESTED, REJECTED))}
|
|
return [
|
|
{"id": note.id, "title": note.title, "state": link.state, "note": link.note or ""}
|
|
for link, note in sorted(rows, key=lambda r: (order.get(r[0].state, 9), r[1].id))
|
|
]
|
|
|
|
|
|
async def judgments_for_lessons(lesson_ids) -> dict[int, dict]:
|
|
"""{lesson_id: {"rule_judgment": linked|no_rule|unjudged, "no_rule"?}}.
|
|
|
|
Read from the link and answer tables directly, NOT from the reader's view
|
|
of the rules: a lesson shared with someone who cannot see its owner's rule
|
|
is still a judged lesson, and calling it unjudged would invite them to
|
|
judge it again.
|
|
"""
|
|
ids = [int(i) for i in lesson_ids or []]
|
|
if not ids:
|
|
return {}
|
|
async with async_session() as session:
|
|
linked = set((await session.execute(
|
|
select(LessonRuleLink.lesson_id).where(
|
|
LessonRuleLink.lesson_id.in_(ids),
|
|
LessonRuleLink.state == CONFIRMED,
|
|
)
|
|
)).scalars().all())
|
|
answers = {
|
|
a.lesson_id: a for a in (await session.execute(
|
|
select(LessonNoRule).where(LessonNoRule.lesson_id.in_(ids))
|
|
)).scalars().all()
|
|
}
|
|
out: dict[int, dict] = {}
|
|
for i in ids:
|
|
if i in linked:
|
|
out[i] = {"rule_judgment": LINKED}
|
|
elif i in answers:
|
|
out[i] = {"rule_judgment": NO_RULE, "no_rule": answers[i].to_dict()}
|
|
else:
|
|
out[i] = {"rule_judgment": UNJUDGED}
|
|
return out
|
|
|
|
|
|
async def attach_lesson_rules(user_id: int, rows: list[dict], *, key: str = "id") -> None:
|
|
"""Add `rules` and `rule_judgment` (plus `no_rule` when that is the answer)
|
|
to each lesson payload row, in place — a fixed number of queries per page.
|
|
|
|
Fail-open (snippet #4286): this decorates a lesson the caller already has,
|
|
so a failed lookup leaves the keys off and logs, rather than refusing the
|
|
lesson. An absent key reads as "not attached", never as "no rule".
|
|
"""
|
|
ids = [int(r[key]) for r in rows if isinstance(r.get(key), int)]
|
|
try:
|
|
found = await rules_for_lessons(user_id, ids)
|
|
judged = await judgments_for_lessons(ids)
|
|
except Exception:
|
|
logger.warning("lesson→rule links could not be read", exc_info=True)
|
|
return
|
|
for r in rows:
|
|
if isinstance(r.get(key), int):
|
|
r["rules"] = found.get(r[key], [])
|
|
r.update(judged.get(r[key], {}))
|
|
|
|
|
|
def unjudged_clause():
|
|
"""SQL: a lesson (a `notes` row) with no confirmed rule and no "no rule
|
|
fits" answer. A rejected link alone leaves it unjudged — "not that rule"
|
|
does not say whether another one fits."""
|
|
from scribe.models.note import Note
|
|
|
|
return and_(
|
|
not_(exists().where(and_(
|
|
LessonRuleLink.lesson_id == Note.id,
|
|
LessonRuleLink.state == CONFIRMED,
|
|
))),
|
|
not_(exists().where(LessonNoRule.lesson_id == Note.id)),
|
|
)
|
|
|
|
|
|
async def list_unjudged(
|
|
user_id: int, *, tag: str = "", project_id: int | None = None,
|
|
limit: int = 50, offset: int = 0,
|
|
) -> tuple[list[dict], int]:
|
|
"""The lessons nobody has answered "which rule?" for, newest edit first.
|
|
|
|
The browse scope (`browsable_notes_clause`), the list it sits beside uses:
|
|
a lesson shared directly with the caller is search-only and stays out of
|
|
an ambient list. Rows come back in the knowledge list's item shape.
|
|
"""
|
|
from scribe.models.note import Note
|
|
from scribe.services.access import browsable_notes_clause
|
|
from scribe.services.knowledge import _note_to_item
|
|
from scribe.services.lessons import LESSON_NOTE_TYPE
|
|
|
|
base = (
|
|
select(Note)
|
|
.where(browsable_notes_clause(user_id))
|
|
.where(Note.note_type == LESSON_NOTE_TYPE)
|
|
.where(Note.deleted_at.is_(None))
|
|
.where(unjudged_clause())
|
|
)
|
|
if tag:
|
|
base = base.where(Note.tags.contains([tag]))
|
|
if project_id is not None:
|
|
base = base.where(Note.project_id == project_id)
|
|
async with async_session() as session:
|
|
total = (await session.execute(
|
|
select(func.count()).select_from(base.subquery())
|
|
)).scalar_one()
|
|
rows = (await session.execute(
|
|
base.order_by(Note.updated_at.desc()).limit(limit).offset(offset)
|
|
)).scalars().all()
|
|
return [_note_to_item(n) for n in rows], int(total)
|
|
|
|
|
|
async def rule_candidates(
|
|
user_id: int, what: str, when_to_apply: str, project_id: int | None = None,
|
|
) -> list[dict] | None:
|
|
"""The rules a lesson most resembles, for the writer to judge against.
|
|
|
|
Searched with the lesson's claim and its trigger — the same two halves a
|
|
rule's own document leads with (`trigger_title`), so like is
|
|
compared with like; the insight is the story and would dilute the vector.
|
|
Scoped as any project read is: global rules plus the lesson's project's
|
|
own. The bar is the explicit rule search's, since this is an explicit
|
|
question rather than an injection.
|
|
|
|
None when the search could not run (no embedder, a failed query), so a
|
|
caller can say "candidates unavailable" rather than "nothing resembles".
|
|
"""
|
|
from scribe.services.embeddings import (
|
|
DEFAULT_SIMILARITY_THRESHOLD, semantic_search_rules, trigger_title,
|
|
)
|
|
|
|
# The rule document's own title join, so like is compared with like.
|
|
query = trigger_title(what, when_to_apply)
|
|
report: dict = {}
|
|
found = await semantic_search_rules(
|
|
user_id, query, limit=CANDIDATE_LIMIT,
|
|
threshold=DEFAULT_SIMILARITY_THRESHOLD, report=report,
|
|
project_id=project_id or None,
|
|
)
|
|
if not report.get("searched"):
|
|
return None
|
|
return [
|
|
{
|
|
"id": rule.id, "title": rule.title, "kind": rule.kind,
|
|
"when_to_apply": rule.when_to_apply or "", "score": round(score, 3),
|
|
}
|
|
for score, rule in found
|
|
]
|
|
|
|
|
|
async def attach_rule_lessons(user_id: int, data: dict, rule_id: int) -> None:
|
|
"""Add `lessons` to a rule payload, in place, when there are any.
|
|
|
|
Present-only, as a rule's `systems` and `relations` are (#2483), and
|
|
fail-open for the reason `attach_lesson_rules` gives.
|
|
"""
|
|
try:
|
|
lessons = await lessons_for_rule(user_id, rule_id)
|
|
except Exception:
|
|
logger.warning("rule→lesson links could not be read", exc_info=True)
|
|
return
|
|
if lessons:
|
|
data["lessons"] = lessons
|
|
|
|
|
|
|
|
def situation_key(arm: str, text: str) -> str:
|
|
"""A fingerprint of one situation, so a repeat counts once.
|
|
|
|
`arm` is part of the key — "p" for a prompt, "w" for a file being written,
|
|
"s" for a session (the rule↔moment signal, milestone 458 step 7), "d"
|
|
for a day (the misfire signal, step 7b, whose door carries no session) —
|
|
because each names a different kind of situation and must not collide.
|
|
On the prompt arm the text is the prompt: lowercased, split into word
|
|
tokens of _MIN_TOKEN or more, de-duplicated and sorted, so the same ask
|
|
re-sent with different spacing, punctuation, case or word order is one
|
|
situation. On the write arm the text is the PATH, not the code: every
|
|
edit to one file is one situation, however much the code differs between
|
|
them. The session and day arms key on the id or date as given — a date's
|
|
"10" and "05" are shorter than _MIN_TOKEN, so tokenising it would make
|
|
every day of a year one situation. Empty when there is nothing to key on.
|
|
"""
|
|
if arm in ("w", "s", "d"):
|
|
basis = (text or "").strip()
|
|
else:
|
|
tokens = sorted({t for t in re.findall(r"[a-z0-9]+", (text or "").lower())
|
|
if len(t) >= _MIN_TOKEN})
|
|
basis = " ".join(tokens)
|
|
if not basis:
|
|
return ""
|
|
return f"{arm}:" + hashlib.sha1(basis.encode()).hexdigest()[:16]
|
|
|
|
|
|
def evidence_summary(evidence) -> dict:
|
|
"""The counts a reader judges a suggested link by."""
|
|
ev = evidence if isinstance(evidence, dict) else {}
|
|
return {
|
|
"situations": len(ev.get("situations") or []),
|
|
"projects": len(ev.get("projects") or []),
|
|
"co_surfaced": int(ev.get("co_surfaced") or 0),
|
|
}
|
|
|
|
|
|
def add_evidence(evidence, key: str, project_id: int | None, now: datetime) -> dict:
|
|
"""A NEW evidence dict with this co-arrival counted. Pure, so the counting
|
|
rules are testable without a database; a new dict, so SQLAlchemy sees the
|
|
JSONB column change."""
|
|
ev = dict(evidence) if isinstance(evidence, dict) else {}
|
|
ev["co_surfaced"] = int(ev.get("co_surfaced") or 0) + 1
|
|
situations = list(ev.get("situations") or [])
|
|
if key and key not in situations and len(situations) < _SITUATION_CAP:
|
|
situations.append(key)
|
|
ev["situations"] = situations
|
|
projects = list(ev.get("projects") or [])
|
|
if project_id and int(project_id) not in projects:
|
|
projects.append(int(project_id))
|
|
ev["projects"] = projects
|
|
ev.setdefault("first_at", now.isoformat())
|
|
ev["last_at"] = now.isoformat()
|
|
return ev
|
|
|
|
|
|
def proposal_due(evidence, now: datetime) -> bool:
|
|
"""At the bar, and not asked within the cooldown."""
|
|
ev = evidence if isinstance(evidence, dict) else {}
|
|
if len(ev.get("situations") or []) < PROPOSE_SITUATIONS:
|
|
return False
|
|
last = ev.get("proposed_at")
|
|
if not last:
|
|
return True
|
|
try:
|
|
return now - datetime.fromisoformat(last) >= PROPOSE_COOLDOWN
|
|
except (TypeError, ValueError):
|
|
return True
|
|
|
|
|
|
def _proposal_line(lesson_id: int, lesson_title: str, rule_id: int, rule_title: str,
|
|
kind: str, evidence: dict) -> str:
|
|
counts = evidence_summary(evidence)
|
|
where = (f" across {counts['projects']} projects" if counts["projects"] > 1 else "")
|
|
return (
|
|
f"> Lesson #{lesson_id} \"{lesson_title}\" and {kind} #{rule_id} "
|
|
f"\"{rule_title}\" keep arriving together — {counts['situations']} "
|
|
f"distinct situations{where}. If the lesson is an instance of the "
|
|
f"{kind}, `judge_lesson_link({lesson_id}, {rule_id}, \"confirm\", "
|
|
f"note=\"why\")` links them and the {kind} becomes reachable through "
|
|
f"the lesson's situation; if it is not, `\"reject\"` with the why "
|
|
f"stops the pair being asked about again."
|
|
)
|
|
|
|
|
|
async def co_surfaced(
|
|
user_id: int, lesson_ids, rule_ids, *, arm: str, situation: str,
|
|
project_id: int | None = None,
|
|
) -> str:
|
|
"""Record that these lessons and rules arrived together in one request,
|
|
and return a proposal line when a pair has earned one ("" otherwise).
|
|
|
|
Called by the hook doors that put both kinds in front of the reader in
|
|
the SAME response — `/retrieve` on a prompt, `/prior-art` on a write — so
|
|
the pairing is exact: nothing is joined across requests, and no session
|
|
identity is needed.
|
|
|
|
Only pairs the reader could confirm are recorded: a lesson they may write
|
|
and a rule they own, the two checks `judge_link` makes. A pair already
|
|
judged (confirmed or rejected) gathers nothing more — a confirmation needs
|
|
no evidence, and a rejection is what stops the asking.
|
|
|
|
One proposal per request at most, so a menu never becomes a queue of
|
|
questions. Fails open: this rides a hook, and a recall aid must never
|
|
break the prompt or the write it decorates.
|
|
"""
|
|
try:
|
|
return await _co_surfaced(user_id, lesson_ids, rule_ids, arm, situation, project_id)
|
|
except Exception:
|
|
logger.warning("lesson/rule co-surfacing could not be recorded", exc_info=True)
|
|
return ""
|
|
|
|
|
|
async def _co_surfaced(user_id, lesson_ids, rule_ids, arm, situation, project_id) -> str:
|
|
from scribe.services.access import can_write_note
|
|
from scribe.services.rulebooks import _owned_rules_clause
|
|
|
|
lessons = _ids(lesson_ids)
|
|
rules = _ids(rule_ids)
|
|
key = situation_key(arm, situation)
|
|
if not lessons or not rules or not key:
|
|
return ""
|
|
lessons = [lid for lid in lessons if await can_write_note(user_id, lid)]
|
|
if not lessons:
|
|
return ""
|
|
now = datetime.now(timezone.utc)
|
|
proposal = None
|
|
async with async_session() as session:
|
|
owned = {
|
|
r.id: r for r in (await session.execute(
|
|
select(Rule).where(Rule.id.in_(rules)).where(_owned_rules_clause(user_id))
|
|
)).scalars().all()
|
|
}
|
|
if not owned:
|
|
return ""
|
|
existing = {
|
|
(row.lesson_id, row.rule_id): row for row in (await session.execute(
|
|
select(LessonRuleLink).where(
|
|
LessonRuleLink.lesson_id.in_(lessons),
|
|
LessonRuleLink.rule_id.in_(list(owned)),
|
|
)
|
|
)).scalars().all()
|
|
}
|
|
for lid in lessons:
|
|
for rid in owned:
|
|
row = existing.get((lid, rid))
|
|
if row is not None and row.state != SUGGESTED:
|
|
continue
|
|
if row is None:
|
|
row = LessonRuleLink(lesson_id=lid, rule_id=rid, state=SUGGESTED)
|
|
session.add(row)
|
|
ev = add_evidence(row.evidence, key, project_id, now)
|
|
if proposal is None and proposal_due(ev, now):
|
|
ev["proposed_at"] = now.isoformat()
|
|
ev["proposed_count"] = int(ev.get("proposed_count") or 0) + 1
|
|
proposal = (lid, rid, ev)
|
|
row.evidence = ev
|
|
try:
|
|
await session.commit()
|
|
except IntegrityError:
|
|
# Two hook requests created the same pair at once; the other
|
|
# one's row stands, and this co-arrival is simply not counted.
|
|
await session.rollback()
|
|
return ""
|
|
if proposal is None:
|
|
return ""
|
|
lid, rid, ev = proposal
|
|
lesson_title = (await session.execute(
|
|
select(Note.title).where(Note.id == lid)
|
|
)).scalar_one_or_none() or ""
|
|
rule = owned[rid]
|
|
return _proposal_line(lid, lesson_title, rid, rule.title, rule.kind or "rule", ev)
|
|
|
|
|
|
# ── Reaching a rule through its lessons (#4633) ──────────────────────────────
|
|
|
|
|
|
async def confirmed_lessons(user_id: int) -> set[int]:
|
|
"""The lessons, readable by the caller, that carry a CONFIRMED link to a
|
|
rule the caller owns — the only lessons that can bring a rule along.
|
|
|
|
CONFIRMED ONLY, and the reason is the soft link's: a suggested link that
|
|
carried its rule would put the two side by side on every prompt that
|
|
found the lesson, which is exactly the co-arrival #4637 counts as
|
|
evidence — the suggestion would manufacture its own proof.
|
|
|
|
Asked first and cheaply, so an arm on an install with no confirmed links
|
|
— every install, until someone judges one — never pays for a search.
|
|
"""
|
|
from scribe.services.access import readable_notes_clause
|
|
from scribe.services.rulebooks import _owned_rules_clause
|
|
|
|
async with async_session() as session:
|
|
rows = (await session.execute(
|
|
select(LessonRuleLink.lesson_id)
|
|
.join(Rule, Rule.id == LessonRuleLink.rule_id)
|
|
.join(Note, Note.id == LessonRuleLink.lesson_id)
|
|
.where(LessonRuleLink.state == CONFIRMED)
|
|
.where(Rule.deleted_at.is_(None))
|
|
.where(_owned_rules_clause(user_id))
|
|
.where(Note.deleted_at.is_(None))
|
|
.where(readable_notes_clause(user_id))
|
|
.distinct()
|
|
)).scalars().all()
|
|
return {int(i) for i in rows}
|
|
|
|
|
|
async def confirmed_rules_in_scope(
|
|
user_id: int, lesson_ids, project_id: int | None,
|
|
) -> dict[int, list]:
|
|
"""{lesson_id: [Rule]} — each lesson's CONFIRMED rules that may surface here.
|
|
|
|
"May surface here" is the rule arms' own scope (milestone 414): a global
|
|
rule, or a rule of THIS project. A lesson is project-independent and
|
|
surfaces everywhere, so without this a lesson linked to project A's rule
|
|
would carry that rule into project B's sessions — the leak milestone 414
|
|
closed for the direct arms, reopened through a side door.
|
|
"""
|
|
from scribe.services.rulebooks import _owned_rules_clause
|
|
|
|
ids = [int(i) for i in lesson_ids or []]
|
|
if not ids:
|
|
return {}
|
|
home = (
|
|
or_(Rule.topic_id.isnot(None), Rule.project_id == project_id)
|
|
if project_id else Rule.topic_id.isnot(None)
|
|
)
|
|
async with async_session() as session:
|
|
rows = (await session.execute(
|
|
select(LessonRuleLink.lesson_id, Rule)
|
|
.join(Rule, Rule.id == LessonRuleLink.rule_id)
|
|
.where(LessonRuleLink.lesson_id.in_(ids))
|
|
.where(LessonRuleLink.state == CONFIRMED)
|
|
.where(Rule.deleted_at.is_(None))
|
|
.where(_owned_rules_clause(user_id))
|
|
.where(home)
|
|
.order_by(Rule.id)
|
|
)).all()
|
|
out: dict[int, list] = {}
|
|
for lesson_id, rule in rows:
|
|
out.setdefault(int(lesson_id), []).append(rule)
|
|
return out
|
|
|
|
|
|
# ── Convergence: lessons with no rule that keep landing in one place (#4634) ─
|
|
#
|
|
# A lesson answered "no rule fits" is a situation nothing binds. One is a
|
|
# lesson. Several that resemble each other are what a missing rule looks like
|
|
# from the outside — the same moment met again and again, each time written
|
|
# down as advice. This is noticed at the WRITE, when the newest of them is
|
|
# answered, and never by a sweep or a timer (#4183): the reader is in the
|
|
# situation then, and a nudge arriving anywhere else is one nobody acts on.
|
|
#
|
|
# WHAT "RESEMBLE" MEANS (#5193). Not a fixed similarity. Lessons are written
|
|
# in one register — a sentence of advice and the moment it applies — so an
|
|
# embedder scores any two of them well above two unrelated texts, and a
|
|
# constant bar sits inside that band: every no-rule lesson then "resembles"
|
|
# most of the others, and the group grows with the pool rather than with any
|
|
# situation recurring. The bar is instead relative to each lesson's own
|
|
# background: how far a neighbour stands above the typical score that lesson
|
|
# gets against every lesson in its register, measured in that register's own
|
|
# spread (median and MAD, so the near neighbours being judged do not move the
|
|
# yardstick). An install with a different embedder, or lessons in a different
|
|
# style, gets a different band and the same bar.
|
|
#
|
|
# And resemblance must be MUTUAL, among every member, not just to the lesson
|
|
# being answered. A lesson broad enough to stand near many others is a hub,
|
|
# and hub-and-spoke is how three unrelated lessons used to make a "group"
|
|
# through one general one. A clique is the shape of one situation met again.
|
|
#
|
|
# DEFAULTS, stated as defaults (rules 32, 115). Three lessons — the new one
|
|
# and two it resembles — is the smallest group that is a pattern rather than
|
|
# a pair. One spread above the background is a modest bar for one direction
|
|
# alone; the strictness comes from requiring it of every pair in both
|
|
# directions, which unrelated lessons rarely all clear. Raise it if groups
|
|
# name lessons a reader would not call one situation; lower it if lessons a
|
|
# reader would group never meet.
|
|
CONVERGENCE_LESSONS = 3
|
|
CONVERGENCE_STANDOUT = 1.0
|
|
# Below this many other lessons a median and its spread describe too little
|
|
# to say what stands out, so nothing is named.
|
|
CONVERGENCE_MIN_BACKGROUND = 8
|
|
# How many of a lesson's register the background is read from. A register
|
|
# larger than this is measured on its nearest part, which reads the
|
|
# background HIGH — the bar errs strict, toward naming nothing, never toward
|
|
# a group that is only the pool's size.
|
|
_CONVERGENCE_BACKGROUND = 200
|
|
# Neighbours checked for mutual resemblance, nearest first. Each costs one
|
|
# more search at the write, and a group large enough to need more is named
|
|
# just as well by its nearest members.
|
|
_CONVERGENCE_CANDIDATES = 8
|
|
# The consistency constant that makes a MAD estimate a standard deviation's
|
|
# scale on normal data, so CONVERGENCE_STANDOUT reads as "spreads".
|
|
_MAD_SCALE = 1.4826
|
|
|
|
|
|
def convergence_group(members: list[dict]) -> dict | None:
|
|
"""Decide from a candidate group — the new lesson FIRST, then those it
|
|
resembles — whether it names a missing rule. Pure, so the bar is testable.
|
|
|
|
Each member is {id, title, sources, project_id}. The bar counts DISTINCT
|
|
LESSONS, never incidents: a single broad lesson drawn from many incidents
|
|
is still one judgment about one situation, and incidents cannot stand in
|
|
for lessons. And when every member says what taught it and all of them
|
|
point at the same single incident, that is one event written up several
|
|
times, not a situation recurring — the group stays quiet.
|
|
"""
|
|
seen: set[int] = set()
|
|
group = []
|
|
for m in members:
|
|
if m["id"] not in seen:
|
|
seen.add(m["id"])
|
|
group.append(m)
|
|
if len(group) < CONVERGENCE_LESSONS:
|
|
return None
|
|
incidents = sorted({s for m in group for s in (m.get("sources") or [])})
|
|
if all(m.get("sources") for m in group) and len(incidents) < 2:
|
|
return None
|
|
projects = sorted({m["project_id"] for m in group if m.get("project_id")})
|
|
named = ", ".join(f"#{m['id']} “{m['title']}”" for m in group)
|
|
return {
|
|
"lessons": [{"id": m["id"], "title": m["title"]} for m in group],
|
|
"incidents": incidents,
|
|
"projects": projects,
|
|
"hint": (
|
|
f"{len(group)} lessons answered \"no rule fits\" and keep landing "
|
|
f"in one situation: {named}. A situation met this often may want a "
|
|
"rule — the binding choice they each circle. Draft it with "
|
|
"create_rule (it goes to the operator, as every rule does), then "
|
|
"point each lesson at it with update_lesson(lesson_id, "
|
|
"rule_ids=[<new rule id>]). If no single choice is right every "
|
|
"time, the lessons are the right record and nothing more is needed."
|
|
),
|
|
}
|
|
|
|
|
|
async def _no_rule_ids(lesson_ids) -> set[int]:
|
|
ids = [int(i) for i in lesson_ids or []]
|
|
if not ids:
|
|
return set()
|
|
async with async_session() as session:
|
|
rows = (await session.execute(
|
|
select(LessonNoRule.lesson_id).where(LessonNoRule.lesson_id.in_(ids))
|
|
)).scalars().all()
|
|
return {int(i) for i in rows}
|
|
|
|
|
|
async def convergence_for(user_id: int, lesson_id: int) -> dict | None:
|
|
"""The convergence a newly answered no-rule lesson completes, or None.
|
|
|
|
Fail-open: this decorates a write that already succeeded, so a failed
|
|
search leaves the response as it was (snippet #4286's reasoning).
|
|
"""
|
|
try:
|
|
return await _convergence_for(user_id, lesson_id)
|
|
except Exception:
|
|
logger.warning("lesson convergence could not be checked", exc_info=True)
|
|
return None
|
|
|
|
|
|
def standout(scores: dict[int, float]) -> dict[int, float]:
|
|
"""How far each lesson stands above this one's background, in spreads.
|
|
|
|
`scores` is one lesson's similarity to every other lesson in its register,
|
|
itself left out. Pure, so the bar is testable. Empty when there is too
|
|
little background to measure, or none of it varies.
|
|
"""
|
|
if len(scores) < CONVERGENCE_MIN_BACKGROUND:
|
|
return {}
|
|
middle = statistics.median(scores.values())
|
|
spread = statistics.median(abs(v - middle) for v in scores.values()) * _MAD_SCALE
|
|
if spread <= 0:
|
|
return {}
|
|
return {i: (v - middle) / spread for i, v in scores.items()}
|
|
|
|
|
|
def converging(lesson_id: int, standouts: dict[int, dict[int, float]],
|
|
candidates: list[int]) -> list[int]:
|
|
"""The lesson, then each candidate that stands out to EVERY member so far
|
|
and every member to it. Candidates are taken nearest first, so the clique
|
|
grows around the closest situation rather than the first id. Pure."""
|
|
def near(a: int, b: int) -> bool:
|
|
return (standouts.get(a, {}).get(b, float("-inf")) >= CONVERGENCE_STANDOUT
|
|
and standouts.get(b, {}).get(a, float("-inf")) >= CONVERGENCE_STANDOUT)
|
|
|
|
group = [lesson_id]
|
|
for c in candidates:
|
|
if c not in group and all(near(c, m) for m in group):
|
|
group.append(c)
|
|
return group
|
|
|
|
|
|
async def _convergence_for(user_id: int, lesson_id: int) -> dict | None:
|
|
from scribe.services import lessons as lessons_svc
|
|
from scribe.services.embeddings import semantic_search_notes, trigger_title
|
|
|
|
lesson = await lessons_svc.get_lesson(user_id, lesson_id)
|
|
if lesson is None:
|
|
return None
|
|
|
|
async def background_of(note) -> list[tuple[float, Note]]:
|
|
"""`note`'s similarity to every other lesson it can reach — the
|
|
background it is measured against. No threshold: the bar is relative,
|
|
and needs the scores a fixed bar would have turned away. No
|
|
supersession penalty either: it would shift some scores and not
|
|
others, and the background is a measure of resemblance alone."""
|
|
data = note.data if isinstance(note.data, dict) else {}
|
|
query = trigger_title(data.get("what") or note.title, lessons_svc.lesson_trigger(note))
|
|
return await semantic_search_notes(
|
|
user_id, query, exclude_ids={int(note.id)}, limit=_CONVERGENCE_BACKGROUND,
|
|
threshold=-1.0, note_type=(lessons_svc.LESSON_NOTE_TYPE,),
|
|
include_global_kinds=True, scope="browse", demote_superseded=False,
|
|
)
|
|
|
|
lid = int(lesson.id)
|
|
found = await background_of(lesson)
|
|
notes = {int(n.id): n for _s, n in found}
|
|
standouts = {lid: standout({int(n.id): s for s, n in found})}
|
|
standing = [i for i, z in standouts[lid].items() if z >= CONVERGENCE_STANDOUT]
|
|
answered = await _no_rule_ids(standing)
|
|
candidates = sorted(answered, key=lambda i: (-standouts[lid][i], i))[:_CONVERGENCE_CANDIDATES]
|
|
# Too few stand out from here for any clique to reach the bar — skip the
|
|
# searches the mutual check would cost.
|
|
if len(candidates) < CONVERGENCE_LESSONS - 1:
|
|
return None
|
|
for c in candidates:
|
|
standouts[c] = standout({int(n.id): s for s, n in await background_of(notes[c])})
|
|
|
|
def member(note) -> dict:
|
|
return {
|
|
"id": int(note.id), "title": note.title,
|
|
"sources": lessons_svc.lesson_sources(note), "project_id": note.project_id,
|
|
}
|
|
|
|
group = converging(lid, standouts, candidates)
|
|
return convergence_group([member(lesson)] + [member(notes[i]) for i in group[1:]])
|