FabledGrimoire/src/scribe/services/lessons.py

"""Lesson service — a transferable insight, retrievable by situation.

A *lesson* is a Note with ``note_type='lesson'``: a better way to think about a
problem, or a solution that transfers, recorded so a later session meets it at
the moment it applies — and **without binding the reader**.

WHY THE KIND EXISTS (milestone 385, from note #3727)

Rules were the only surface that is global AND situation-keyed, so an agent
holding a transferable insight had one door, and that door binds. The observed
symptom was sessions offering rule proposals for things that should not be
rules.

The gap is a document-shape fact, not a threshold:

  - a note is embedded as ``title\\nbody`` and is findable by **what it is
    about**;
  - a rule is embedded as ``{title} — {trigger}`` with ``When to apply:``
    repeated at the head of the body, so the trigger appears twice in a short
    document and dominates the vector — findable by **when it applies**.

No tuning reaches across that: the field the query would match on is simply not
in a note's document. So a lesson carries a trigger and is embedded like a rule,
while staying a note in every other respect.

WHERE THE TRIGGER LIVES (decision #4157, milestone 385 step 1)

In ``notes.data`` under ``when_to_apply``, written through a named parameter and
mirrored into the title and the head of the body — the shape snippets already
use for ``when_to_use``. Not a column on ``notes``.

That decision was measured rather than assumed. The whole snippet corpus —
164 of 164 — carries a ``when_to_use`` with **no guard anywhere**, which refutes
the premise that an unenforced field gets skipped. What it does NOT show is that
an agent types a title convention correctly: ``compose_title`` builds the title
from the parameter, so what is at 100% is a named structured field. A column
would have bought enforceability at the price of deciding, for every note kind
at once, a question nothing had measured.

The mirror is what makes the vector sharp, and it is why nothing re-embeds:
``chunk_document`` is untouched, so ``CHUNKER_VERSION`` does not move. The
trigger reaches the document by being in the text, exactly as a snippet's is.

WHAT A LESSON INHERITS, AND THE CELLS LEFT EMPTY ON PURPOSE (#3163)

A new kind inherits the note machinery wholesale, and #3163 asks which parts it
should NOT get — so that an empty cell is a decision rather than an oversight.

Inherited, all deliberately:

  - **versions** — a lesson is reworded as understanding improves, and what it
    used to say is worth as much as any note's history.
  - **supersession** — the event this most needs. A lesson replaced by a better
    lesson is precisely what ``note_supersessions`` models, and the demotion
    penalty already exists.
  - **trash**, **the share ACL**, **tags**, **project and System tagging**,
    **chunked embeddings**, **the near-duplicate gate**.

NOT inherited, and each for a stated reason:

  - **status / task_kind / milestone_id** — a lesson is not work. ``is_task`` is
    ``status is not None``, so a lesson that acquired a status would become a
    task and appear in open-work listings. This is the one cell where filling it
    in by accident silently changes what the record IS.
  - **recurrence** — task-only, and a lesson does not recur.
  - **verify_with / expires_when** — available, because they are generic note
    fields, but not part of a lesson's contract and not asked for on create. The
    milestone-312 distinction is why: those mark a record that asserts a FACT
    about someone else's software and can go false unwatched. A lesson is closer
    to a norm — "a better way to think about this" has no truth value that rots
    on its own. A lesson that does assert such a fact can still carry them.

WHY THERE IS NO MIGRATION

``note_type`` carries **no CHECK constraint** — only ``task_kind`` does
(``notes_task_kind_check``, migrations 0056 / 0065). Migration 0036 added
``note_type`` as plain ``Text`` with a server default and nothing has gated it
since. So rule 36 has nothing to expand here, and the failure it guards against
— a value the database refuses on an instance predating its migration — cannot
arise for this column.

The real vocabulary is ``services.knowledge._FACETS``, which is where a kind
becomes reachable on the browse surface and validated at the door. That is one
table feeding both dialects of the type filter, so adding a kind there is a
single edit — a property #3161 recommended and that landed before this.
"""
from __future__ import annotations

import re

LESSON_NOTE_TYPE = "lesson"

# The key in `notes.data`. Named for the field it mirrors on `rules`, because it
# answers the same question and a reader who knows one should not have to learn
# a second word for it.
TRIGGER_KEY = "when_to_apply"

# The body's trigger line, and the pattern that reads it back. The body is the
# readable form and the thing that gets embedded; `data` is the queryable
# mirror. Reads prefer the mirror and fall back to this, which is the discipline
# `snippet_fields` follows and the reason a row written before the mirror
# existed is still readable.
_BODY_TRIGGER_RE = re.compile(r"^\*\*When to apply:\*\*\s*(.+?)\s*$", re.M)


def lesson_trigger(note) -> str:
    """When this lesson applies, or "" — the mirror first, then the body.

    Prefers `data` for the same reason every snippet read does: it is indexed,
    and parsing a body to answer a question the database can answer is how a
    hot path ends up regexing markdown. The fallback is not dead code — it is
    what makes a lesson readable if the mirror is ever absent, and an absent
    mirror must degrade to the right answer rather than to silence.
    """
    data = getattr(note, "data", None) or {}
    from_mirror = (data.get(TRIGGER_KEY) or "").strip() if isinstance(data, dict) else ""
    if from_mirror:
        return from_mirror
    match = _BODY_TRIGGER_RE.search(getattr(note, "body", None) or "")
    return match.group(1).strip() if match else ""


def compose_title(what: str, when_to_apply: str = "") -> str:
    """`{what} — {when it applies}`, the half of the document that ranks.

    Built HERE rather than asked of the caller, and that distinction is the
    whole evidence base for this design: the snippet corpus is at 100% on its
    trigger because a service composes the title from a named parameter, not
    because agents type separators reliably. A caller made to spell the
    convention is the option milestone 385 step 1 rejected.

    The join is `embeddings.trigger_title` — shared with rules and snippets, so
    the three kinds that rank on a trigger cannot drift apart in how they say
    so.
    """
    from scribe.services.embeddings import trigger_title

    return trigger_title(what, when_to_apply)


def compose_body(insight: str, when_to_apply: str = "") -> str:
    """The lesson body — the trigger line first, the insight after.

    The mirror of `compose_title` on the other half of the document, and the
    reason the pair is what makes a lesson findable: `chunk_document` joins
    them as `{title}\\n{body}`, so a lesson composed here states WHEN IT
    APPLIES in the title and again in the first line of the body. That is the
    twice-in-a-short-document shape note #2485 measured as the only sharp one
    in the corpus, reached the way a snippet reaches it — by being in the text
    — rather than by a second document builder at embed time.

    `**When to apply:**` rather than plain text: the body is the READABLE
    form, `data` is the queryable mirror, and `_BODY_TRIGGER_RE` reads this
    line back when the mirror is missing. Its markdown must therefore match
    what that pattern expects, which is why neither is written by hand
    anywhere else.

    The insight goes in the body rather than being held out of the document.
    `rule_document` excludes a rule's `why` because long dated narrative made
    sixteen dev-logs land on the centroid of "development" — but that finding
    predates chunking (#280). A body over the budget is now split into several
    chunks, EACH prefixed with the title, so a lesson's story no longer
    averages itself into its trigger: it occupies its own vectors, and every
    one of them still carries the trigger in its prefix. Holding it out would
    cost the reader the only part that explains the insight and would buy a
    sharpness the chunker already provides.
    """
    lines = []
    trigger = (when_to_apply or "").strip()
    if trigger:
        lines.append(f"**When to apply:** {trigger}")
    insight = (insight or "").strip()
    if insight:
        lines.append(insight)
    return "\n\n".join(lines)


def lesson_document(
    what: str, when_to_apply: str = "", insight: str = "",
) -> tuple[str, str]:
    """The (title, body) a lesson is STORED — and therefore embedded — as.

    One call so the two halves cannot be composed apart. A lesson whose title
    carried the trigger and whose body did not would embed as an ordinary
    note wearing a label, and nothing would report it: the record would look
    right in every listing and simply never be retrieved at the moment it
    applies.

    Deliberately returns what is STORED, not a separate embed-time shape.
    Rules need `rule_document` because a rule keeps its trigger in a column
    and its title is a plain name, so the sharp document has to be synthesised
    for the ranker and exists nowhere else. A lesson follows the snippet
    instead — the stored record IS the sharp document — which is why nothing
    re-embeds and `CHUNKER_VERSION` does not move.
    """
    return compose_title(what, when_to_apply), compose_body(insight, when_to_apply)