"""Rule usage telemetry — did a surfaced rule ever get read? The sibling of `note_usage`, for the one retrieval surface in Scribe that could not be measured at all. Two event streams, deliberately independent: - SURFACED: the write-path standing-rule arm put this rule in front of the agent, unbidden, during a write. - PULLED: someone then opened it in full (`get_rule`, or the REST detail route). WHY THIS ARM AND NOT ANOTHER. Every other surface declines most of the time — `write_path` returns nothing on 78% of calls, `reuse_slot` on 79%, auto-inject on 39%. The rule arm APPEARED never to have returned nothing (#3311), and this docstring used to put that forward as the puzzle worth measuring: "either a perfectly tuned surface or a bar it cannot fail to clear". It was neither, and the correction belongs here rather than being quietly deleted. The arm wrote its `retrieval_logs` row only on calls that FOUND something (#3497), so `zero_result_calls` sat at 0 and `cleared_threshold` at `calls` because of the shape of the code — at any threshold whatsoever. A statistic that could not vary was read as a finding about the corpus. It is the #2663 failure mode one level up: there the broken readout was a zero, here it was a hundred percent, which is far better camouflage. The reason to measure this arm survives the correction, and is stronger for it. `retrieval_logs` records what the ranker scored, never whether the hint was any use, so even an honest clear-rate would not settle the question. The ratio these two streams produce is the missing half, and without it any threshold change is a number picked off a histogram. Design notes, mirroring `note_usage`: - Writes are fire-and-forget through `background.spawn`, so telemetry never adds latency to — or can break — the surface it observes. This module does NOT carry its own copy of the strong-reference dance; `background` is the one place that gets it right, and a fourth copy is how one of them drifts. - Failures degrade, but never SILENTLY. `report_telemetry_failure` logs at WARNING and drops one AppLog row per process per site. #2663 is the record of this exact subsystem class running at zero for weeks — indistinguishable from "nobody uses this" — because every failure went to `logger.debug`. - Reads (`usage_for_rules`) are awaited and aggregated in one round-trip for a whole page, never per row. AMBIENT VS RANKED. The note twin splits ranked surfacings from ambient ones because `enter_project` and the skill sync put records in front of the agent without choosing them, and counting those as surfacings makes recency read as popularity (#2477). Rules have exactly that shape: the SessionStart preload, and every `rules_payload` surface hand over the whole applicable set at once, chosen by nobody. Until 2026-09-03 those bulk surfaces emitted nothing, and this module said so — "an empty `AMBIENT_SOURCES` would be machinery pretending to a distinction the data does not yet contain". True as far as it went, but it had a consequence worth naming, because it is the reason the bucket exists now: the always-on set's token cost was certain and its usefulness was UNFALSIFIABLE, permanently and by construction. The one surface whose value was actually in question was the one surface exempt from the scoreboard that judges every other. They emit now. The split is the readout-level change the old note promised — a `case()`, no migration, because `event` and `source` are plain Text with no CHECK constraint. `source` stays granular so a reader can still tell the preload from `enter_project` from the ranked arm. WHY THIS NAMES THE RANKED SOURCES AND THE TWIN NAMES THE AMBIENT ONES. A deliberate divergence, on the failure mode rather than on symmetry. Both shapes fail silently when someone adds a surface and forgets the list, so the question is which list changes more often — and here it is emphatically the ambient one: there are TWO ranked rule sources (the write-path arm and the pre-tool arm) against the seven bulk ones the preload alone contributes. Ranked sources are added when somebody builds a ranker, which is rare and deliberate; bulk ones appear whenever a surface hands rules over, which is most of them. Naming the rare, slow-moving half means a newly-added bulk surface defaults to `ambient`, which merely under-counts it, instead of defaulting to `ranked`, which would quietly pad the pull-through denominator with surfacings nobody chose and make the arm look imprecise. Same argument #3191 and #3430 make against hand-kept lists: keep the list that must be remembered as short and as slow-moving as possible. """ from __future__ import annotations import logging from sqlalchemy import case, func, select from scribe.models import async_session from scribe.models.base import iso from scribe.models.rule_usage import ( APPLIED, DEPARTED, OUTCOMES, PULLED, SURFACED, RuleUsageEvent, ) from scribe.services.background import report_telemetry_failure, spawn logger = logging.getLogger(__name__) # The surfaces that CHOSE the rules they showed. Everything else is ambient — # see the module docstring for why the rare half is the half that gets named. # # Membership is the whole definition of the pull-through denominator: a ranked # surfacing is a claim ("this rule may apply to what you are doing") that a pull # can confirm or refute, while an ambient one is a delivery nobody decided on. # Add a source here only when a ranker picked it. RANKED_SOURCES = ( "write_path_rule", "pre_tool_rule", "prompt_rule", # A reserved slot is a ranker's choice twice over — it ran a query AND # decided a kind was worth guaranteeing a place. Left out, its line would # be counted as bulk delivery and drop out of the denominator, so the one # surface built because a record class kept losing would be the one whose # hits nobody could confirm. "preference_slot", # The completion-report lookup on update_task (milestone 409 step 4). It # runs its own query and shows only what cleared the bar — a ranker. "report_preference", ) def is_ambient(source: str) -> bool: """Was this surfacing a bulk delivery rather than a ranked choice? One definition, read by both the per-rule badge readout and the aggregate in `retrieval_telemetry` — the two used to be able to disagree about what "surfaced" counted, which is the class of drift #3246 found across the rules system. Sync and pure, per the service canon (#2860), but deliberately PUBLIC where that canon says such helpers stay `_private`. The departure is the point: a module-private copy in each caller is exactly the second definition this exists to prevent. """ return source not in RANKED_SOURCES async def _report_failure(site: str) -> None: await report_telemetry_failure("rule_usage", site) async def _insert_events(rows: list[dict]) -> None: """Persist usage rows. Best-effort: failures degrade, visibly.""" try: async with async_session() as session: session.add_all([RuleUsageEvent(**row) for row in rows]) await session.commit() except Exception: await _report_failure("write") def _schedule(rows: list[dict]) -> None: if not rows: return spawn(_insert_events(rows), site="rule_usage_write") def record_rule_surfaced( *, user_id: int | None, rule_ids: list[int] | set[int], source: str ) -> None: """Fire-and-forget: record that these rules were shown to the agent. Takes the whole delivery at once — one insert per surfacing event, not per rule — because a hint is a single decision and its rows should land together. Record what was actually SHOWN, never what was considered. For the ranked arm that means the post-filter hits: it drops what the session already holds (`exclude_rule_ids`) before it speaks, and a rule considered and not shown was not surfaced. Counting those would inflate the denominator with claims the agent never saw, which reads as a precision problem the arm does not have. Bulk surfaces pass their whole delivered set, which is the same rule read from the other end — everything in a preload IS shown. `source` is what separates the two afterwards (see `RANKED_SOURCES`); this function does not care which kind it is recording. """ try: rows = [ { "user_id": user_id, "rule_id": int(rid), "event": SURFACED, "source": source, } for rid in rule_ids ] except Exception: logger.debug("rule usage payload build failed", exc_info=True) return _schedule(rows) def record_rule_pulled(*, user_id: int | None, rule_id: int, source: str) -> None: """Fire-and-forget: record that a rule was opened in full. A PULL is somebody choosing to open one record. `enter_project` is NOT a pull — they are bulk resident loads that hand over every applicable rule at once, and counting them would swamp the signal with the very ambient delivery the ratio exists to distinguish from. """ try: rows = [ { "user_id": user_id, "rule_id": int(rule_id), "event": PULLED, "source": source, } ] except Exception: logger.debug("rule usage payload build failed", exc_info=True) return _schedule(rows) def record_rule_outcome( *, user_id: int | None, rule_id: int, outcome: str, source: str, detail: str = "", ) -> None: """Fire-and-forget: record what a rule ACTUALLY CHANGED (#4212). The third stream, and the one milestone 419 exists for. `surfaced` says the system offered a rule; `pulled` says somebody opened it. Neither says whether it made any difference, so a rule that fires constantly and is always obeyed and a rule that fires constantly and is never obeyed have, until now, produced identical telemetry. The second is far the more urgent and is precisely the one the readout could not name. Two outcomes, because there are only two a judge can honestly report: APPLIED — the rule changed what was done, or confirmed it. No `detail` required: following a rule is the unremarkable case and charging prose for it is how an event stops being recorded. DEPARTED — read, and deliberately not followed. `detail` is REQUIRED and is the entire value of the event. A departure without its reason reads back as a miss, which collapses the two states this exists to separate. THERE IS NO THIRD CALL, and the absence is the design. Read-and-silently- unchanged is real — it is the failure this milestone was opened on — but it cannot be reported, because an agent that knew it was ignoring a rule would not be ignoring it. It is derived: a rule pulled, with no outcome behind it. See `outcome_state`. Guarded rather than trusting: a bad outcome or a reasonless departure is dropped and REPORTED, never written. Telemetry that lies is worse than telemetry that is missing (#2663), and a `departed` row with an empty reason is a lie the readout cannot detect. """ if outcome not in OUTCOMES: logger.warning("rule outcome rejected: unknown outcome %r", outcome) spawn(_report_failure("outcome_unknown"), site="rule_usage_outcome") return if outcome == DEPARTED and not (detail or "").strip(): logger.warning("rule outcome rejected: departure with no reason") spawn(_report_failure("outcome_no_reason"), site="rule_usage_outcome") return try: rows = [ { "user_id": user_id, "rule_id": int(rule_id), "event": outcome, "source": source, "detail": (detail or "").strip() or None, } ] except Exception: logger.debug("rule usage payload build failed", exc_info=True) return _schedule(rows) # What a rule's usage says happened to it, in one word. The four states are # ordered by how much the system actually knows, and only the last two are # new — the point of the milestone is that UNACTED used to be invisible # inside APPLIED. UNREAD = "unread" # surfaced, never opened UNACTED = "unacted" # opened, and nothing recorded after — the blind spot FOLLOWED = "followed" # opened and applied DEPARTED_FROM = "departed" # opened and deliberately not followed, with a why def outcome_state(usage: dict) -> str: """The three states milestone 419 asked to be able to tell apart, plus the one that already existed. Pure, and reading only the aggregate `usage_for_rules` already returns — so the readout, the badge and any later session summary all answer this question the same way. Two callers computing "was this followed" from raw counts is the drift #3246 found across the rules system, arriving again. PRECEDENCE, and it is deliberate: a departure outranks an application. A rule both applied and departed from in the same window is a rule someone argued with, and the argument is the interesting half — reporting it as plain compliance would hide the one row a reader most wants. UNACTED is the derived state and the reason this function exists. It is not "no data"; it is a rule that was surfaced, deliberately OPENED, and then left no trace of having mattered. That is a much stronger signal than never having been opened at all, and it is the signal that was previously indistinguishable from compliance. """ if int(usage.get("departed_count") or 0): return DEPARTED_FROM if int(usage.get("applied_count") or 0): return FOLLOWED if int(usage.get("pull_count") or 0): return UNACTED return UNREAD def empty_rule_usage() -> dict: """The zero readout — what a rule with no recorded events looks like. Callers render this shape unconditionally, so a rule predating the table reads as "never surfaced, never pulled" rather than as a missing key. That distinction matters more here than for notes: every rule in an install predates this table, so for a while "no events" is the normal state and it must not look like a broken readout. `surfaced_count` is RANKED surfacings only; `ambient_count` is the bulk deliveries (see `RANKED_SOURCES`). The split is what keeps the badge's "shown often, opened never → dead weight" reading honest: every rule in an always-on set is delivered every session, so an unsplit counter would rank the resident set as the most-surfaced rules in the install purely for being resident. """ return { "surfaced_count": 0, "ambient_count": 0, "pull_count": 0, # The outcome half (#4212). Zero here means "nothing recorded", which # for a rule that was also never pulled is simply silence — and for # one that WAS pulled is the blind spot this milestone is named for. # `outcome_state` is what tells those apart; no caller should be # reading these counts raw to decide it. # # The REASON for a departure is on the row (`detail`), not here. One # GROUP BY cannot carry the text of the latest departure without a # DISTINCT ON alongside it, and a key that the aggregate could never # fill would read as "no reason given" on every rule that has one — # a permanently-null field that lies. The readout that needs the # prose reads the rows (#4213). "applied_count": 0, "departed_count": 0, "last_outcome_at": None, "last_surfaced_at": None, "last_pulled_at": None, } async def usage_for_rules(rule_ids: list[int]) -> dict[int, dict]: """Aggregate usage for a set of rules: {rule_id: {counts + timestamps}}. One GROUP BY for the whole page rather than a query per row — this feeds a list view, so the per-row shape would be N+1 by construction. Rules with no events come back with `empty_rule_usage()`, so the caller never has to tell "no events" from "not in the result". """ ids = [int(r) for r in rule_ids] out: dict[int, dict] = {rid: empty_rule_usage() for rid in ids} if not ids: return out # Classified in SQL so the group stays small: per rule we get at most # (surfaced-ranked, surfaced-ambient, pulled) rather than a row per distinct # source. ONE labelled expression, bound to a variable and reused in the # GROUP BY — a second `case()` instance there renders its own expanding-IN # bind names under asyncpg, so the database sees two DIFFERENT expressions # and rejects the query with a GroupingError. The note twin carries the # same warning for the same reason, and #2663 is what it cost: the # rejection was swallowed and every counter read zero in production while # the writes were landing fine. ambient = case( (RuleUsageEvent.source.notin_(RANKED_SOURCES), True), else_=False, ).label("ambient") try: async with async_session() as session: rows = ( await session.execute( select( RuleUsageEvent.rule_id, RuleUsageEvent.event, func.count().label("n"), func.max(RuleUsageEvent.created_at).label("last_at"), ambient, ) .where(RuleUsageEvent.rule_id.in_(ids)) .group_by( RuleUsageEvent.rule_id, RuleUsageEvent.event, ambient, ) ) ).all() except Exception: # A telemetry readout must not be able to break the list it decorates — # but it must say it failed, or a broken readout is indistinguishable # from a corpus nobody uses (#2663). await _report_failure("readout") return out for rule_id, event, n, last_at, is_amb in rows: slot = out.get(int(rule_id)) if slot is None: continue # Outcomes first, and never split by ambient. `ambient` asks whether # a RANKER chose to show the rule; an outcome is reported by a judge # after the fact and has no ranker behind it, so the flag is noise # here. Branching on it would silently drop every outcome row into a # bucket nothing reads. if event in OUTCOMES: key = "applied_count" if event == APPLIED else "departed_count" slot[key] = slot[key] + int(n) prev = slot["last_outcome_at"] now = iso(last_at) if now and (prev is None or now > prev): slot["last_outcome_at"] = now elif event == SURFACED and is_amb: slot["ambient_count"] = int(n) elif event == SURFACED: slot["surfaced_count"] = int(n) slot["last_surfaced_at"] = iso(last_at) elif event == PULLED: # Pulls are pulls regardless of what surfaced the rule — "did # anyone ever open this?" does not depend on how it was found. Both # halves accumulate, so this ADDS rather than assigns: a rule can # now be pulled after a ranked hint and after a preload, and the # split arrives as two rows. slot["pull_count"] = slot["pull_count"] + int(n) latest = iso(last_at) if latest and (slot["last_pulled_at"] or "") < latest: slot["last_pulled_at"] = latest return out