rename(547): the server is grimoire — package, image, DB defaults, backups

Step 2 of the Scribe -> Grimoire rename.

- src/scribe -> src/grimoire, and every import, patch target, module path,
  hypercorn entry point and `python -m grimoire.restore` with it; pyproject,
  uv.lock and the frontend package are named grimoire.
- Image git.fabledsword.com/bvandeusen/fabledgrimoire; CI's Postgres service
  runs as grimoire / grimoire_test.
- Compose files default the DB user, database and (prod) volume to grimoire,
  each overridable from .env, so an install created as Scribe keeps its data
  by setting POSTGRES_USER, POSTGRES_DB and PGDATA_VOLUME.
- Backups are written as grimoire-<ts>.tar with format grimoire-backup-tar.
  Archives named scribe-* with the scribe-backup-tar manifest still restore
  and are still counted at the destination (folder and S3), with tests.
- Exports, the MCP auth realm, the ASGI scope key and the persistent logger
  carry the new name.

Kept on purpose: the secret-sealing key label stays "scribe.secrets.v1",
because every stored secret was derived under it.

Plugin identity (plugin name, hooks, env vars, .scribe marker, skills) is
step 3.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

src/grimoire/services/rule_usage.py

@@ -0,0 +1,418 @@
+"""Rule usage telemetry — did a surfaced rule ever get read?
+
+The sibling of `note_usage`, for rules.
+
+Event streams, deliberately independent:
+
+  - SURFACED: a surface put this rule in front of the agent — a ranked arm,
+    unbidden, or a bulk delivery (see AMBIENT VS RANKED below).
+  - PULLED: someone then opened it in full (`get_rule`, or the REST detail
+    route).
+
+WHY RULES NEED THEIR OWN STREAM. `retrieval_logs` records what the ranker
+scored, never whether the hint was any use, so even an honest clear-rate cannot
+settle whether a rule arm earns its lines. The ratio these two streams produce
+is the missing half, and without it any threshold change is a number picked
+off a histogram. (A clear-rate can also mislead outright: an arm that logs
+only the calls that FOUND something reports a decline count of zero at any
+threshold — a statistic that cannot vary, read as a finding about the corpus.)
+
+Design notes, mirroring `note_usage`:
+  - Writes are fire-and-forget through `background.spawn`, so telemetry never
+    adds latency to — or can break — the surface it observes. This module does
+    NOT carry its own copy of the strong-reference dance; `background` is the
+    one place that gets it right, and another copy is how one of them drifts.
+  - Failures degrade, but never SILENTLY. `report_telemetry_failure` logs at
+    WARNING and drops one AppLog row per process per site. A telemetry
+    subsystem whose failures go to `logger.debug` can run at zero for weeks,
+    indistinguishable from "nobody uses this".
+  - Reads (`usage_for_rules`) are awaited and aggregated in one round-trip for
+    a whole page, never per row.
+
+AMBIENT VS RANKED. The note twin splits ranked surfacings from ambient ones
+because `enter_project` and the skill sync put records in front of the agent
+without choosing them, and counting those as surfacings makes recency read as
+popularity. Rules have exactly that shape: the SessionStart preload, and every
+`rules_payload` surface hand over the whole applicable set at once, chosen by
+nobody.
+
+Those bulk surfaces emit too, as ambient. Without them the always-on set's
+token cost would be certain and its usefulness UNFALSIFIABLE — the one surface
+whose value is actually in question would be the one exempt from the
+scoreboard that judges every other. The split is a readout-level `case()`, no
+migration, because `event` and `source` are plain Text with no CHECK
+constraint. `source` stays granular so a reader can still tell the preload
+from `enter_project` from a ranked arm.
+
+WHY THIS NAMES THE RANKED SOURCES AND THE TWIN NAMES THE AMBIENT ONES. A
+deliberate divergence, on the failure mode rather than on symmetry. Both shapes
+fail silently when someone adds a surface and forgets the list, so the question
+is which list changes more often — and here it is the ambient one. Ranked
+sources are added when somebody builds a ranker, which is rare and deliberate;
+bulk ones appear whenever a surface hands rules over, which is most of them.
+Naming the rare, slow-moving half means a newly-added bulk surface defaults to
+`ambient`, which merely under-counts it, instead of defaulting to `ranked`,
+which would quietly pad the pull-through denominator with surfacings nobody
+chose and make the arm look imprecise. Keep the list that must be remembered
+as short and as slow-moving as possible.
+"""
+from __future__ import annotations
+
+import logging
+
+from sqlalchemy import case, func, select
+
+from grimoire.models import async_session
+from grimoire.models.base import iso
+from grimoire.models.rule_usage import (
+    APPLIED, DEPARTED, OUTCOMES, PULLED, SURFACED, RuleUsageEvent,
+)
+from grimoire.services.background import report_telemetry_failure, spawn
+from grimoire.services.retrieval_pipeline import MOMENT_RULE_SOURCE, RULE_RANKED_SOURCES
+
+logger = logging.getLogger(__name__)
+
+# The surfaces that CHOSE the rules they showed. Everything else is ambient —
+# see the module docstring for why the rare half is the half that gets named.
+#
+# Membership is the whole definition of the pull-through denominator: a ranked
+# surfacing is a claim ("this rule may apply to what you are doing") that a pull
+# can confirm or refute, while an ambient one is a delivery nobody decided on.
+# The ranked half is read off the retrieval pipeline's specs: every rule arm
+# — the reply backstop included, which records only
+# the rule that HELD the reply — the preference slot, which is a ranker's
+# choice twice over, and the rule reached through a confirmed lesson link,
+# whose own source lets its pull-through read apart from a direct match.
+# An arm added there is counted here by construction.
+RANKED_SOURCES = (
+    *RULE_RANKED_SOURCES,
+    # A rule mounted on a moment. Not a ranker's pick, but
+    # not bulk either: somebody decided this rule applies at this moment, and
+    # that is exactly the claim a pull can confirm or refute. Its
+    # pull-through is the evidence for whether a mount earns its line.
+    MOMENT_RULE_SOURCE,
+)
+
+
+def is_ambient(source: str) -> bool:
+    """Was this surfacing a bulk delivery rather than a ranked choice?
+
+    One definition, read by both the per-rule badge readout and the aggregate
+    in `retrieval_telemetry`, so the two cannot disagree about what
+    "surfaced" counts.
+
+    Sync and pure, per the service canon, but deliberately PUBLIC where
+    that canon says such helpers stay `_private`. The departure is the point:
+    a module-private copy in each caller is exactly the second definition this
+    exists to prevent.
+    """
+    return source not in RANKED_SOURCES
+
+
+async def _report_failure(site: str) -> None:
+    await report_telemetry_failure("rule_usage", site)
+
+
+async def _insert_events(rows: list[dict]) -> None:
+    """Persist usage rows. Best-effort: failures degrade, visibly."""
+    try:
+        async with async_session() as session:
+            session.add_all([RuleUsageEvent(**row) for row in rows])
+            await session.commit()
+    except Exception:
+        await _report_failure("write")
+
+
+def _schedule(rows: list[dict]) -> None:
+    if not rows:
+        return
+    spawn(_insert_events(rows), site="rule_usage_write")
+
+
+def record_rule_surfaced(
+    *, user_id: int | None, rule_ids: list[int] | set[int], source: str,
+    detail: dict[int, str] | None = None,
+) -> None:
+    """Fire-and-forget: record that these rules were shown to the agent.
+
+    Takes the whole delivery at once — one insert per surfacing event, not per
+    rule — because a hint is a single decision and its rows should land
+    together.
+
+    Record what was actually SHOWN, never what was considered. For the ranked
+    arm that means the post-filter hits: it drops what the session already
+    holds (`exclude_rule_ids`) before it speaks, and a rule considered and not
+    shown was not surfaced. Counting those would inflate the denominator with
+    claims the agent never saw, which reads as a precision problem the arm does
+    not have.
+
+    Bulk surfaces pass their whole delivered set, which is the same rule read
+    from the other end — everything in a preload IS shown. `source` is what
+    separates the two afterwards (see `RANKED_SOURCES`); this function does not
+    care which kind it is recording.
+
+    `detail` maps a rule id to what to record beside its row — the moment a
+    mounted rule arrived at, so a mount's pull-through can be
+    read per moment rather than only per arm.
+    """
+    try:
+        rows = [
+            {
+                "user_id": user_id,
+                "rule_id": int(rid),
+                "event": SURFACED,
+                "source": source,
+                **({"detail": detail[rid]} if detail and rid in detail else {}),
+            }
+            for rid in rule_ids
+        ]
+    except Exception:
+        logger.debug("rule usage payload build failed", exc_info=True)
+        return
+    _schedule(rows)
+
+
+def record_rule_pulled(*, user_id: int | None, rule_id: int, source: str) -> None:
+    """Fire-and-forget: record that a rule was opened in full.
+
+    A PULL is somebody choosing to open one record. `enter_project` is NOT a
+    pull — they are bulk resident loads that hand over
+    every applicable rule at once, and counting them would swamp the signal
+    with the very ambient delivery the ratio exists to distinguish from.
+    """
+    try:
+        rows = [
+            {
+                "user_id": user_id,
+                "rule_id": int(rule_id),
+                "event": PULLED,
+                "source": source,
+            }
+        ]
+    except Exception:
+        logger.debug("rule usage payload build failed", exc_info=True)
+        return
+    _schedule(rows)
+
+
+def record_rule_outcome(
+    *,
+    user_id: int | None,
+    rule_id: int,
+    outcome: str,
+    source: str,
+    detail: str = "",
+) -> None:
+    """Fire-and-forget: record what a rule ACTUALLY CHANGED.
+
+    The third stream. `surfaced` says the system offered a rule; `pulled`
+    says somebody opened it. Neither says whether it made any difference, so
+    without this a rule that fires constantly and is always obeyed and a rule
+    that fires constantly and is never obeyed produce identical telemetry.
+    The second is far the more urgent.
+
+    Two outcomes, because there are only two a judge can honestly report:
+
+      APPLIED  — the rule changed what was done, or confirmed it. No `detail`
+                 required: following a rule is the unremarkable case and
+                 charging prose for it is how an event stops being recorded.
+      DEPARTED — read, and deliberately not followed. `detail` is REQUIRED
+                 and is the entire value of the event. A departure without
+                 its reason reads back as a miss, which collapses the two
+                 states this exists to separate.
+
+    THERE IS NO THIRD CALL, and the absence is the design. Read-and-silently-
+    unchanged is real — it is the failure this stream exists to expose — but
+    it cannot be reported, because an agent that knew it was ignoring a rule
+    would not be ignoring it. It is derived: a rule pulled, with no outcome
+    behind it. See `outcome_state`.
+
+    Guarded rather than trusting: a bad outcome or a reasonless departure is
+    dropped and REPORTED, never written. Telemetry that lies is worse than
+    telemetry that is missing, and a `departed` row with an empty
+    reason is a lie the readout cannot detect.
+    """
+    if outcome not in OUTCOMES:
+        logger.warning("rule outcome rejected: unknown outcome %r", outcome)
+        spawn(_report_failure("outcome_unknown"), site="rule_usage_outcome")
+        return
+    if outcome == DEPARTED and not (detail or "").strip():
+        logger.warning("rule outcome rejected: departure with no reason")
+        spawn(_report_failure("outcome_no_reason"), site="rule_usage_outcome")
+        return
+    try:
+        rows = [
+            {
+                "user_id": user_id,
+                "rule_id": int(rule_id),
+                "event": outcome,
+                "source": source,
+                "detail": (detail or "").strip() or None,
+            }
+        ]
+    except Exception:
+        logger.debug("rule usage payload build failed", exc_info=True)
+        return
+    _schedule(rows)
+
+
+# What a rule's usage says happened to it, in one word. The four states are
+# ordered by how much the system actually knows; recorded outcomes are what
+# keep UNACTED from hiding inside FOLLOWED.
+UNREAD = "unread"        # surfaced, never opened
+UNACTED = "unacted"      # opened, and nothing recorded after — the blind spot
+FOLLOWED = "followed"    # opened and applied
+DEPARTED_FROM = "departed"   # opened and deliberately not followed, with a why
+
+
+def outcome_state(usage: dict) -> str:
+    """Which of the four states a rule's usage is in.
+
+    Pure, and reading only the aggregate `usage_for_rules` already returns —
+    so the readout, the badge and any later session summary all answer this
+    question the same way. Two callers computing "was this followed" from raw
+    counts would drift.
+
+    PRECEDENCE, and it is deliberate: a departure outranks an application.
+    A rule both applied and departed from in the same window is a rule
+    someone argued with, and the argument is the interesting half — reporting
+    it as plain compliance would hide the one row a reader most wants.
+
+    UNACTED is the derived state and the reason this function exists. It is
+    not "no data"; it is a rule that was surfaced, deliberately OPENED, and
+    then left no trace of having mattered. That is a much stronger signal
+    than never having been opened at all, and without recorded outcomes it
+    is indistinguishable from compliance.
+    """
+    if int(usage.get("departed_count") or 0):
+        return DEPARTED_FROM
+    if int(usage.get("applied_count") or 0):
+        return FOLLOWED
+    if int(usage.get("pull_count") or 0):
+        return UNACTED
+    return UNREAD
+
+
+def empty_rule_usage() -> dict:
+    """The zero readout — what a rule with no recorded events looks like.
+
+    Callers render this shape unconditionally, so a rule predating the table
+    reads as "never surfaced, never pulled" rather than as a missing key. That
+    distinction matters more here than for notes: every rule in an install
+    predates this table, so for a while "no events" is the normal state and it
+    must not look like a broken readout.
+
+    `surfaced_count` is RANKED surfacings only; `ambient_count` is the bulk
+    deliveries (see `RANKED_SOURCES`). The split is what keeps the badge's
+    "shown often, opened never → dead weight" reading honest: every rule in an
+    always-on set is delivered every session, so an unsplit counter would rank
+    the resident set as the most-surfaced rules in the install purely for being
+    resident.
+    """
+    return {
+        "surfaced_count": 0,
+        "ambient_count": 0,
+        "pull_count": 0,
+        # The outcome half. Zero here means "nothing recorded", which for a
+        # rule that was also never pulled is simply silence — and for one
+        # that WAS pulled is the blind spot (UNACTED).
+        # `outcome_state` is what tells those apart; no caller should be
+        # reading these counts raw to decide it.
+        #
+        # The REASON for a departure is on the row (`detail`), not here. One
+        # GROUP BY cannot carry the text of the latest departure without a
+        # DISTINCT ON alongside it, and a key that the aggregate could never
+        # fill would read as "no reason given" on every rule that has one —
+        # a permanently-null field that lies. The readout that needs the
+        # prose reads the rows.
+        "applied_count": 0,
+        "departed_count": 0,
+        "last_outcome_at": None,
+        "last_surfaced_at": None,
+        "last_pulled_at": None,
+    }
+
+
+async def usage_for_rules(rule_ids: list[int]) -> dict[int, dict]:
+    """Aggregate usage for a set of rules: {rule_id: {counts + timestamps}}.
+
+    One GROUP BY for the whole page rather than a query per row — this feeds a
+    list view, so the per-row shape would be N+1 by construction. Rules with no
+    events come back with `empty_rule_usage()`, so the caller never has to tell
+    "no events" from "not in the result".
+    """
+    ids = [int(r) for r in rule_ids]
+    out: dict[int, dict] = {rid: empty_rule_usage() for rid in ids}
+    if not ids:
+        return out
+
+    # Classified in SQL so the group stays small: per rule we get at most
+    # (surfaced-ranked, surfaced-ambient, pulled) rather than a row per distinct
+    # source. ONE labelled expression, bound to a variable and reused in the
+    # GROUP BY — a second `case()` instance there renders its own expanding-IN
+    # bind names under asyncpg, so the database sees two DIFFERENT expressions
+    # and rejects the query with a GroupingError. The note twin carries the
+    # same warning for the same reason: the rejection is swallowed and every
+    # counter reads zero while the writes are landing fine.
+    ambient = case(
+        (RuleUsageEvent.source.notin_(RANKED_SOURCES), True),
+        else_=False,
+    ).label("ambient")
+    try:
+        async with async_session() as session:
+            rows = (
+                await session.execute(
+                    select(
+                        RuleUsageEvent.rule_id,
+                        RuleUsageEvent.event,
+                        func.count().label("n"),
+                        func.max(RuleUsageEvent.created_at).label("last_at"),
+                        ambient,
+                    )
+                    .where(RuleUsageEvent.rule_id.in_(ids))
+                    .group_by(
+                        RuleUsageEvent.rule_id,
+                        RuleUsageEvent.event,
+                        ambient,
+                    )
+                )
+            ).all()
+    except Exception:
+        # A telemetry readout must not be able to break the list it decorates —
+        # but it must say it failed, or a broken readout is indistinguishable
+        # from a corpus nobody uses.
+        await _report_failure("readout")
+        return out
+
+    for rule_id, event, n, last_at, is_amb in rows:
+        slot = out.get(int(rule_id))
+        if slot is None:
+            continue
+        # Outcomes first, and never split by ambient. `ambient` asks whether
+        # a RANKER chose to show the rule; an outcome is reported by a judge
+        # after the fact and has no ranker behind it, so the flag is noise
+        # here. Branching on it would silently drop every outcome row into a
+        # bucket nothing reads.
+        if event in OUTCOMES:
+            key = "applied_count" if event == APPLIED else "departed_count"
+            slot[key] = slot[key] + int(n)
+            prev = slot["last_outcome_at"]
+            now = iso(last_at)
+            if now and (prev is None or now > prev):
+                slot["last_outcome_at"] = now
+        elif event == SURFACED and is_amb:
+            slot["ambient_count"] = int(n)
+        elif event == SURFACED:
+            slot["surfaced_count"] = int(n)
+            slot["last_surfaced_at"] = iso(last_at)
+        elif event == PULLED:
+            # Pulls are pulls regardless of what surfaced the rule — "did
+            # anyone ever open this?" does not depend on how it was found. Both
+            # halves accumulate, so this ADDS rather than assigns: a rule can
+            # now be pulled after a ranked hint and after a preload, and the
+            # split arrives as two rows.
+            slot["pull_count"] = slot["pull_count"] + int(n)
+            latest = iso(last_at)
+            if latest and (slot["last_pulled_at"] or "") < latest:
+                slot["last_pulled_at"] = latest
+    return out