feat(lessons): a lesson can be written, and it keeps every incident that taught it (#3731)

Milestone 385 step 4 — the write path.

ITS OWN TOOL MODULE, not create_note(note_type="lesson"), on the snippet and
process precedent and for the reason that precedent exists: a kind whose value
depends on one field being filled needs a door that ASKS for that field by
name. create_note would take a lesson through a generic body parameter and the
trigger — the whole of why a lesson is findable — would be something the writer
had to know to include.

THE TRIGGER IS REQUIRED, refused rather than flagged. Step 1 left the choice
open. Refusing is right for the same reason create_rule makes enforcement the
deciding question: a lesson with no trigger is not a weaker lesson, it is a
note that will never surface, and nothing downstream can tell the difference —
it saves, reads correctly in every listing, and is silently absent from the one
moment it was written for. A flag is a warning nobody is present to read; the
write path is where the writer still is. The message says SYMPTOM, because
"required" alone produces a topic where a situation was wanted.

The docstring carries the distinction this milestone exists to fix, in a line a
reader can apply: the difference between a lesson and a rule is FORCE, not
importance. If ignoring it would be a mistake it is a rule and needs the
operator's yes; if ignoring it just means someone re-derives it the slow way it
is a lesson, and nobody is bound.

CARDINALITY: a LIST, in notes.data under `taught_by`. The founding example
generalised three incidents into one claim about failure classes no CI lane can
see — generalising across incidents is the shape a good lesson HAS, and
arose_from_id holds one, so a single id keeps the first and drops two while
reading as complete. It lives in `data` rather than a join table for the reason
decision #4157 put the trigger there: a table would settle, for every note kind
at once, whether provenance is multi-valued — a question nothing has measured.
`arose_from_id` is filled only when there is exactly ONE source, because every
surface that renders it renders it as THE origin, and one of three would make
those surfaces state something false.

THE DUPLICATE GATE, which step 4 asked to check: a lesson is judged at a bar
ABOVE the sibling band, not the general 0.90. #2518 measured deliberately
parallel variants at 0.92 on a document that is mostly prose about the thing,
which is exactly a lesson's shape now — so at 0.90 two genuinely different
lessons about one area ("CI cannot see this class of failure") would refuse each
other. Its own constant rather than reusing the snippet's: the two are separate
facts that coincide today, and this number is inherited from a structurally
analogous corpus rather than measured on lessons, of which there are none yet.

Follows canon #2846 including the third registration point it names and this
change would otherwise have missed: get_lesson is in server._READ_ONLY_TOOLS
and the two writers in _WRITE_TOOLS, which test_mcp_auth requires.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01821k5B3Ysecp9fNYs92Kuy

src/scribe/services/lessons.py

@@ -95,6 +95,10 @@ LESSON_NOTE_TYPE = "lesson"
 # a second word for it.
 TRIGGER_KEY = "when_to_apply"
 
+# What taught this lesson: the ids of the issues, tasks or notes it was drawn
+# from. A LIST, and that is the whole decision — see `normalize_sources`.
+SOURCES_KEY = "taught_by"
+
 # The body's trigger line, and the pattern that reads it back. The body is the
 # readable form and the thing that gets embedded; `data` is the queryable
 # mirror. Reads prefer the mirror and fall back to this, which is the discipline
@@ -102,6 +106,12 @@ TRIGGER_KEY = "when_to_apply"
 # existed is still readable.
 _BODY_TRIGGER_RE = re.compile(r"^\*\*When to apply:\*\*\s*(.+?)\s*$", re.M)
 
+# The readable mirror of `data[SOURCES_KEY]`, and the pattern that reads it
+# back — the shape snippets use for `**Merged from:** #ids`, for the same
+# reason: the body is what a human sees and what survives a row with no `data`.
+_BODY_SOURCES_RE = re.compile(r"^\*\*Learned from:\*\*\s*(.+?)\s*$", re.M)
+_ID_RE = re.compile(r"#(\d+)")
+
 
 def lesson_trigger(note) -> str:
     """When this lesson applies, or "" — the mirror first, then the body.
@@ -120,6 +130,80 @@ def lesson_trigger(note) -> str:
     return match.group(1).strip() if match else ""
 
 
+def normalize_sources(entries: list | None) -> list[int]:
+    """The ids that taught this lesson — ints, de-duplicated, in the order given.
+
+    THE CARDINALITY DECISION, and why it is a list.
+
+    `arose_from_id` already exists and holds ONE id, which is the obvious first
+    answer and the wrong one. The lesson that started this milestone generalised
+    THREE incidents — a badge collision, an un-backfilled column, a duplicated
+    const — into one claim about failure classes no CI lane can see. Generalising
+    across incidents is the shape a good lesson HAS, not an edge case. A single
+    id would keep the first and silently drop the rest, and a record that drops
+    two of its three sources is worse than one that names none, because it reads
+    as complete.
+
+    It lives in `notes.data` rather than a join table for exactly the reason
+    decision #4157 put the trigger there: a join table would settle, for every
+    note kind at once, whether provenance is multi-valued — a question nothing
+    has measured. `data` is JSONB with a GIN index (0070), so the list is
+    queryable today and a table can be migrated to later if the need is shown.
+
+    Order is history, not sorting: the incidents stay in the sequence the writer
+    named them, which is the order they were learned in.
+    """
+    out: list[int] = []
+    seen: set[int] = set()
+    for raw in entries or []:
+        ident = raw.get("id") if isinstance(raw, dict) else raw
+        try:
+            i = int(ident)
+        except (TypeError, ValueError):
+            continue
+        if i > 0 and i not in seen:
+            seen.add(i)
+            out.append(i)
+    return out
+
+
+def lesson_sources(note) -> list[int]:
+    """What taught this lesson — the mirror first, then the body, then
+    `arose_from_id`.
+
+    Three fallbacks rather than two, because the third is what keeps this
+    honest on a record written before the kind existed: a note carrying only
+    `arose_from_id` has exactly one source and this returns it, so a caller
+    never has to ask which field to read.
+    """
+    data = getattr(note, "data", None) or {}
+    if isinstance(data, dict):
+        from_mirror = normalize_sources(data.get(SOURCES_KEY))
+        if from_mirror:
+            return from_mirror
+    match = _BODY_SOURCES_RE.search(getattr(note, "body", None) or "")
+    if match:
+        found = normalize_sources(_ID_RE.findall(match.group(1)))
+        if found:
+            return found
+    single = getattr(note, "arose_from_id", None)
+    return normalize_sources([single]) if single else []
+
+
+def sole_source(sources: list[int] | None) -> int | None:
+    """`arose_from_id` for this lesson: the id when there is exactly ONE.
+
+    Left NULL for a lesson drawn from several, deliberately. Every existing
+    surface that renders provenance reads `arose_from_id` and renders it as
+    THE origin; handing it one of three would make those surfaces state
+    something false. Showing nothing there is accurate — there is no single
+    origin — and `data[SOURCES_KEY]` carries all of them for the surfaces that
+    know to ask.
+    """
+    ids = normalize_sources(sources)
+    return ids[0] if len(ids) == 1 else None
+
+
 def compose_title(what: str, when_to_apply: str = "") -> str:
     """`{what} — {when it applies}`, the half of the document that ranks.
 
@@ -138,7 +222,9 @@ def compose_title(what: str, when_to_apply: str = "") -> str:
     return trigger_title(what, when_to_apply)
 
 
-def compose_body(insight: str, when_to_apply: str = "") -> str:
+def compose_body(
+    insight: str, when_to_apply: str = "", learned_from: list[int] | None = None,
+) -> str:
     """The lesson body — the trigger line first, the insight after.
 
     The mirror of `compose_title` on the other half of the document, and the
@@ -172,11 +258,19 @@ def compose_body(insight: str, when_to_apply: str = "") -> str:
     insight = (insight or "").strip()
     if insight:
         lines.append(insight)
+    sources = normalize_sources(learned_from)
+    if sources:
+        # LAST, not beside the trigger. The first line has to be what this
+        # lesson is FOR; a provenance line above the insight would push the
+        # thing the reader came for below a list of ids, and would put
+        # numbers where the trigger's second appearance does its work.
+        lines.append("**Learned from:** " + ", ".join(f"#{i}" for i in sources))
     return "\n\n".join(lines)
 
 
 def lesson_document(
     what: str, when_to_apply: str = "", insight: str = "",
+    learned_from: list[int] | None = None,
 ) -> tuple[str, str]:
     """The (title, body) a lesson is STORED — and therefore embedded — as.
 
@@ -193,4 +287,156 @@ def lesson_document(
     instead — the stored record IS the sharp document — which is why nothing
     re-embeds and `CHUNKER_VERSION` does not move.
     """
-    return compose_title(what, when_to_apply), compose_body(insight, when_to_apply)
+    return (
+        compose_title(what, when_to_apply),
+        compose_body(insight, when_to_apply, learned_from),
+    )
+
+
+def compose_data(
+    what: str, when_to_apply: str = "", learned_from: list[int] | None = None,
+) -> dict:
+    """The indexed mirror of the same fields the body renders (0070).
+
+    Written together with the body by the one caller that composes both, so the
+    two can never describe different things — the discipline `compose_data`
+    follows for snippets, and the reason `lesson_trigger` can prefer `data`
+    without checking whether it agrees with the prose.
+
+    Empty values are omitted so the column stays sparse: a lesson with no
+    sources has no `taught_by` key rather than an empty list, which keeps a
+    `?` containment query honest.
+    """
+    data: dict = {"what": what.strip()} if what and what.strip() else {}
+    trigger = (when_to_apply or "").strip()
+    if trigger:
+        data[TRIGGER_KEY] = trigger
+    sources = normalize_sources(learned_from)
+    if sources:
+        data[SOURCES_KEY] = sources
+    return data
+
+
+async def create_lesson(
+    user_id: int,
+    *,
+    what: str,
+    when_to_apply: str = "",
+    insight: str = "",
+    learned_from: list[int] | None = None,
+    tags: list[str] | None = None,
+    project_id: int | None = None,
+):
+    """Create a lesson note. Returns the created Note.
+
+    The title, the body and the `data` mirror are composed HERE from named
+    parameters rather than asked of the caller. That is the whole evidence base
+    for this design and not a convenience: the snippet corpus carries a trigger
+    on 164 of 164 records with no guard anywhere, because a service builds the
+    title from a parameter — what is at 100% is a named structured field, not an
+    agent typing a convention correctly.
+
+    `project_id` is accepted and kept, even though a lesson is reachable from
+    every project (step 3). Where it was learned is a fact worth keeping; it
+    simply stops being the limit of where it can be found.
+    """
+    from scribe.services import notes as notes_svc
+
+    sources = normalize_sources(learned_from)
+    title, body = lesson_document(what, when_to_apply, insight, sources)
+    return await notes_svc.create_note(
+        user_id,
+        title=title,
+        body=body,
+        note_type=LESSON_NOTE_TYPE,
+        tags=tags,
+        project_id=project_id,
+        # NULL unless there is exactly one source — see `sole_source`.
+        arose_from_id=sole_source(sources),
+        data=compose_data(what, when_to_apply, sources),
+    )
+
+
+async def get_lesson(user_id: int, lesson_id: int):
+    """Fetch a lesson by id, or None if it isn't one / isn't readable.
+
+    Share-aware (rule 78): a fetch by id is an explicit act, so it resolves the
+    caller's full read scope rather than ownership alone — without this, a
+    lesson a search legitimately surfaced could not then be opened (#2093).
+    """
+    from scribe.services import notes as notes_svc
+
+    result = await notes_svc.get_note_for_user(user_id, lesson_id)
+    if result is None:
+        return None
+    note, _permission = result
+    if note.note_type != LESSON_NOTE_TYPE or note.deleted_at is not None:
+        return None
+    return note
+
+
+async def update_lesson(
+    user_id: int,
+    lesson_id: int,
+    *,
+    what: str | None = None,
+    when_to_apply: str | None = None,
+    insight: str | None = None,
+    learned_from: list[int] | None = None,
+    tags: list[str] | None = None,
+):
+    """Update a lesson, re-composing title, body and mirror from the merged
+    fields. Returns the updated Note, or None if it isn't a readable lesson.
+
+    READ-MODIFY-WRITE over the whole record rather than patching one half.
+    The three fields are not independent: the trigger appears in the title AND
+    at the head of the body, so editing it in place would need two edits that
+    a caller could do one of. Re-composing from the merged values means a
+    partial update cannot leave the halves disagreeing — which, because the
+    document is what ranks, would be a lesson that still reads correctly and
+    quietly stops being retrievable.
+    """
+    from scribe.services import notes as notes_svc
+
+    note = await get_lesson(user_id, lesson_id)
+    if note is None:
+        return None
+
+    current = note.data if isinstance(note.data, dict) else {}
+    merged_what = current.get("what") or "" if what is None else what
+    merged_trigger = lesson_trigger(note) if when_to_apply is None else when_to_apply
+    merged_sources = (
+        lesson_sources(note) if learned_from is None
+        else normalize_sources(learned_from)
+    )
+    if insight is None:
+        insight = _strip_composed_lines(note.body)
+
+    title, body = lesson_document(
+        merged_what, merged_trigger, insight, merged_sources,
+    )
+    fields: dict = {
+        "title": title,
+        "body": body,
+        "arose_from_id": sole_source(merged_sources),
+        "data": compose_data(merged_what, merged_trigger, merged_sources),
+    }
+    if tags is not None:
+        fields["tags"] = tags
+    return await notes_svc.update_note(user_id, lesson_id, **fields)
+
+
+def _strip_composed_lines(body: str | None) -> str:
+    """The insight alone — the body with the lines `compose_body` wrote removed.
+
+    An update that keeps the insight has to hand it back to `compose_body`,
+    which will re-add the trigger and provenance lines. Without this the two
+    composed lines accumulate a copy per edit, and since the trigger line is
+    half of what makes the document rank, the duplicates would look like the
+    shape working rather than a bug.
+    """
+    kept = [
+        line for line in (body or "").splitlines()
+        if not _BODY_TRIGGER_RE.match(line) and not _BODY_SOURCES_RE.match(line)
+    ]
+    return "\n".join(kept).strip()