feat(telemetry): the preload emits, and the always-on set stops being unfalsifiable (#3473)

The ranked rule arm became measurable in M333. The preload did not — and
that is the surface whose value is actually in question. `list_always_on_rules`,
the SessionStart block and every `rules_payload` caller handed rules over
wholesale and emitted nothing, so the resident set's token cost was certain
and its usefulness could not be tested even in principle.

Bulk deliveries now record as AMBIENT, beside the ranked count and never
inside pull-through. Folding them in would mean growing the always-on set
depressed the arm's measured precision and trimming it flattered the arm,
neither for any reason to do with the arm.

`RANKED_SOURCES` inverts the note twin's `AMBIENT_SOURCES` deliberately: there
is one ranked rule source and this change adds seven bulk ones, so naming the
rare half makes a forgotten surface default to ambient — under-counting it —
rather than padding the denominator with surfacings nobody chose.

Two lookalike call sites are deliberately left silent, with a test to keep
them that way: the write-path etag arm and `rules_etag_for` read the rules to
build or compare a MARKER and show nobody anything.

No migration — `event` and `source` are plain Text with no CHECK (rule 36
does not apply). Snippet #2858 updated to the new `rules_payload` contract.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011cPyzNnegXHr5iRMzzy5KJ

src/scribe/services/planning.py

@@ -60,7 +60,7 @@ async def start_planning(user_id: int, project_id: int, title: str) -> dict:
 
     return {
         "milestone": milestone.to_dict(),
-        **rulebooks_svc.rules_payload(applicable),
+        **rulebooks_svc.rules_payload(applicable, user_id=user_id, source="start_planning"),
         "project_goal": getattr(project, "goal", "") or "",
         "open_task_count": open_count,
     }

src/scribe/services/plugin_context.py

@@ -1375,6 +1375,24 @@ async def build_session_context(
     # exclusion (milestone 297) takes a rulebook out of this block, and is
     # named below so the departure is visible rather than silent.
     rules = await rulebooks_svc.list_always_on_rules(user_id, project_id=project_id)
+    # AMBIENT source, and the one that matters most: this is the preload — the
+    # block every session opens with, chosen by nobody, paid for every turn.
+    #
+    # It emitted nothing until 2026-09-03, which made the resident set's cost
+    # certain and its usefulness unfalsifiable at the same time (#3473). Note
+    # #3089 is the argument this measurement finally lets someone test: that a
+    # rule arriving with thirty others, none of them relevant, is read as
+    # preamble rather than as a claim — so presence is not surfacing, and a
+    # tier-1 set can grow without anybody noticing it stopped working.
+    #
+    # Recorded even when the hook truncates the block below: the rules WERE
+    # delivered, and counting only the untruncated ones would quietly shrink
+    # the denominator exactly where the set is too big to read.
+    record_rule_surfaced(
+        user_id=user_id,
+        rule_ids=[r.id for r in rules],
+        source="session_start",
+    )
     excluded = (
         await rulebooks_svc.excluded_always_on_rulebooks(user_id, project_id)
         if project_id else []

src/scribe/services/retrieval_telemetry.py

@@ -30,6 +30,7 @@ from scribe.models.note_usage import PULLED, SURFACED, NoteUsageEvent
 from scribe.models.rule_usage import PULLED as RULE_PULLED
 from scribe.models.rule_usage import SURFACED as RULE_SURFACED
 from scribe.models.rule_usage import RuleUsageEvent
+from scribe.services.rule_usage import is_ambient
 from scribe.models.retrieval_log import RetrievalLog
 
 logger = logging.getLogger(__name__)
@@ -428,10 +429,15 @@ async def retrieval_summary(user_id: int | None, *, days: int = 30) -> dict:
                         .group_by(RuleUsageEvent.event, RuleUsageEvent.source)
                     )
                 ).all()
-                # No AMBIENT exclusion here, unlike the note twin: nothing
-                # surfaces a rule un-ranked yet. `list_always_on_rules` and
-                # `enter_project` deliver rules wholesale but emit no event, so
-                # there is no ambient class to subtract (milestone 333 step 1).
+                # The rows carry `source`, so the ranked/ambient split is done
+                # below rather than in SQL — the bulk surfaces started emitting
+                # on 2026-09-03 (#3473), so there IS an ambient class now.
+                #
+                # `distinct_rules_surfaced` deliberately counts BOTH classes. It
+                # answers "how many distinct rules did this install put in front
+                # of an agent at all", which is the denominator for dead weight
+                # — and a rule delivered by the preload a hundred times and
+                # never opened is the most important case that question has.
                 distinct_rules_surfaced = (
                     await session.execute(
                         select(func.count(func.distinct(RuleUsageEvent.rule_id)))
@@ -545,11 +551,18 @@ async def retrieval_summary(user_id: int | None, *, days: int = 30) -> dict:
     # been comparing across windows, without telling them it now measures
     # something else.
     #
-    # No `ambient` key, unlike its twin. Nothing surfaces a rule un-ranked yet;
-    # the absence is a fact about the data rather than an oversight, and it
-    # returns the moment a bulk loader starts emitting.
+    # `ambient` now carries the bulk deliveries — the SessionStart preload,
+    # `list_always_on_rules`, and every `rules_payload` surface (#3473). Before
+    # they emitted, this block had no ambient key and said the absence was a
+    # fact about the data. It was, and it was also the thing that made the
+    # always-on set impossible to judge: the largest rule surface in the
+    # product was the one surface its own scoreboard could not see.
+    #
+    # READ THE TWO SEPARATELY, ALWAYS. `surfaced` is a claim a ranker made and
+    # a pull can settle. `ambient` is a delivery nobody chose, so a high count
+    # says the set is large and resident, never that it is useful.
     rule_usage = {
-        "surfaced": 0,
+        "surfaced": 0, "ambient": 0,
         "pulled": 0, "pulled_by_agent": 0, "pulled_by_human": 0,
         "distinct_rules_surfaced": int(distinct_rules_surfaced or 0),
         "distinct_rules_pulled": int(distinct_rules_pulled or 0),
@@ -565,7 +578,14 @@ async def retrieval_summary(user_id: int | None, *, days: int = 30) -> dict:
         for event, source, n in rule_rows:
             n = int(n)
             if event == RULE_SURFACED:
-                rule_usage["surfaced"] += n
+                # One definition of ranked-vs-ambient, imported rather than
+                # restated — the per-rule badge readout reads the same
+                # predicate, and two spellings of "what counts as surfaced" is
+                # precisely the uneven wiring #3246 found across this system.
+                if is_ambient(source):
+                    rule_usage["ambient"] += n
+                else:
+                    rule_usage["surfaced"] += n
             elif event == RULE_PULLED:
                 rule_usage["pulled"] += n
                 # Same split, and it carries MORE weight here than for notes.
@@ -582,6 +602,15 @@ async def retrieval_summary(user_id: int | None, *, days: int = 30) -> dict:
     # empty numerator AND denominator that is a claim the data does not
     # support, and it is the reading that would make a brand-new install look
     # like a broken one.
+    #
+    # RANKED SURFACINGS ONLY in the denominator, and this is the load-bearing
+    # line of the whole change. Pull-through asks "was that hint any use", and
+    # only a surface that CHOSE what it showed can be judged by it. Folding the
+    # preload in would divide the same pulls by a number that grows with every
+    # session and every rule added to the resident set — so enlarging the
+    # always-on set would DEPRESS the arm's measured precision, and trimming it
+    # would flatter it, neither for any reason to do with the arm. The ambient
+    # count sits beside it, unaveraged, and is read as size rather than skill.
     rule_usage["pull_through"] = (
         round(rule_usage["pulled_by_agent"] / rule_usage["surfaced"], 4)
         if rule_usage["surfaced"] else None

src/scribe/services/rule_usage.py

@@ -30,23 +30,43 @@ Design notes, mirroring `note_usage`:
   - Reads (`usage_for_rules`) are awaited and aggregated in one round-trip for
     a whole page, never per row.
 
-NO AMBIENT BUCKET, YET — and that is a decision, not an omission. The note twin
-splits ranked surfacings from ambient ones because `enter_project` and the
-skill sync put records in front of the agent without choosing them, and
-counting those as surfacings makes recency read as popularity (#2477). Rules
-have the same shape of problem waiting: `list_always_on_rules` and
-`enter_project` load rules wholesale on every session. They do not emit here
-today, so there is nothing to bucket, and an empty `AMBIENT_SOURCES` would be
-machinery pretending to a distinction the data does not yet contain. When a
-bulk surface starts emitting, the split is a readout-level change — a tuple and
-a `case()`, exactly as in the twin — and needs no migration. Keep it that way:
-`source` stays granular so the choice remains available.
+AMBIENT VS RANKED. The note twin splits ranked surfacings from ambient ones
+because `enter_project` and the skill sync put records in front of the agent
+without choosing them, and counting those as surfacings makes recency read as
+popularity (#2477). Rules have exactly that shape: the SessionStart preload,
+`list_always_on_rules`, and every `rules_payload` surface hand over the whole
+applicable set at once, chosen by nobody.
+
+Until 2026-09-03 those bulk surfaces emitted nothing, and this module said so —
+"an empty `AMBIENT_SOURCES` would be machinery pretending to a distinction the
+data does not yet contain". True as far as it went, but it had a consequence
+worth naming, because it is the reason the bucket exists now: the always-on
+set's token cost was certain and its usefulness was UNFALSIFIABLE, permanently
+and by construction. The one surface whose value was actually in question was
+the one surface exempt from the scoreboard that judges every other.
+
+They emit now. The split is the readout-level change the old note promised — a
+`case()`, no migration, because `event` and `source` are plain Text with no
+CHECK constraint. `source` stays granular so a reader can still tell the
+preload from `enter_project` from the ranked arm.
+
+WHY THIS NAMES THE RANKED SOURCES AND THE TWIN NAMES THE AMBIENT ONES. A
+deliberate divergence, on the failure mode rather than on symmetry. Both shapes
+fail silently when someone adds a surface and forgets the list, so the question
+is which list changes more often — and here it is emphatically the ambient one:
+there is exactly ONE ranked rule source, and this change alone adds seven bulk
+ones. Naming the rare, stable half means a newly-added bulk surface defaults to
+`ambient`, which merely under-counts it, instead of defaulting to `ranked`,
+which would quietly pad the pull-through denominator with surfacings nobody
+chose and make the arm look imprecise. Same argument #3191 and #3430 make
+against hand-kept lists: keep the list that must be remembered as short and as
+slow-moving as possible.
 """
 from __future__ import annotations
 
 import logging
 
-from sqlalchemy import func, select
+from sqlalchemy import case, func, select
 
 from scribe.models import async_session
 from scribe.models.base import iso
@@ -55,6 +75,31 @@ from scribe.services.background import report_telemetry_failure, spawn
 
 logger = logging.getLogger(__name__)
 
+# The surfaces that CHOSE the rules they showed. Everything else is ambient —
+# see the module docstring for why the rare half is the half that gets named.
+#
+# Membership is the whole definition of the pull-through denominator: a ranked
+# surfacing is a claim ("this rule may apply to what you are doing") that a pull
+# can confirm or refute, while an ambient one is a delivery nobody decided on.
+# Add a source here only when a ranker picked it.
+RANKED_SOURCES = ("write_path_rule",)
+
+
+def is_ambient(source: str) -> bool:
+    """Was this surfacing a bulk delivery rather than a ranked choice?
+
+    One definition, read by both the per-rule badge readout and the aggregate
+    in `retrieval_telemetry` — the two used to be able to disagree about what
+    "surfaced" counted, which is the class of drift #3246 found across the
+    rules system.
+
+    Sync and pure, per the service canon (#2860), but deliberately PUBLIC where
+    that canon says such helpers stay `_private`. The departure is the point:
+    a module-private copy in each caller is exactly the second definition this
+    exists to prevent.
+    """
+    return source not in RANKED_SOURCES
+
 
 async def _report_failure(site: str) -> None:
     await report_telemetry_failure("rule_usage", site)
@@ -81,14 +126,21 @@ def record_rule_surfaced(
 ) -> None:
     """Fire-and-forget: record that these rules were shown to the agent.
 
-    Takes the whole hint at once — one insert per surfacing event, not per rule
-    — because a hint is a single decision and its rows should land together.
+    Takes the whole delivery at once — one insert per surfacing event, not per
+    rule — because a hint is a single decision and its rows should land
+    together.
 
-    Record the RANKED hits only. The arm filters candidates before it speaks
-    (`exclude_rule_ids` drops what the session already holds), and a rule that
-    was considered and not shown was not surfaced. Counting those would inflate
-    the denominator with claims the agent never saw, which reads as a precision
-    problem the arm does not have.
+    Record what was actually SHOWN, never what was considered. For the ranked
+    arm that means the post-filter hits: it drops what the session already
+    holds (`exclude_rule_ids`) before it speaks, and a rule considered and not
+    shown was not surfaced. Counting those would inflate the denominator with
+    claims the agent never saw, which reads as a precision problem the arm does
+    not have.
+
+    Bulk surfaces pass their whole delivered set, which is the same rule read
+    from the other end — everything in a preload IS shown. `source` is what
+    separates the two afterwards (see `RANKED_SOURCES`); this function does not
+    care which kind it is recording.
     """
     try:
         rows = [
@@ -137,9 +189,17 @@ def empty_rule_usage() -> dict:
     distinction matters more here than for notes: every rule in an install
     predates this table, so for a while "no events" is the normal state and it
     must not look like a broken readout.
+
+    `surfaced_count` is RANKED surfacings only; `ambient_count` is the bulk
+    deliveries (see `RANKED_SOURCES`). The split is what keeps the badge's
+    "shown often, opened never → dead weight" reading honest: every rule in an
+    always-on set is delivered every session, so an unsplit counter would rank
+    the resident set as the most-surfaced rules in the install purely for being
+    resident.
     """
     return {
         "surfaced_count": 0,
+        "ambient_count": 0,
         "pull_count": 0,
         "last_surfaced_at": None,
         "last_pulled_at": None,
@@ -159,6 +219,19 @@ async def usage_for_rules(rule_ids: list[int]) -> dict[int, dict]:
     if not ids:
         return out
 
+    # Classified in SQL so the group stays small: per rule we get at most
+    # (surfaced-ranked, surfaced-ambient, pulled) rather than a row per distinct
+    # source. ONE labelled expression, bound to a variable and reused in the
+    # GROUP BY — a second `case()` instance there renders its own expanding-IN
+    # bind names under asyncpg, so the database sees two DIFFERENT expressions
+    # and rejects the query with a GroupingError. The note twin carries the
+    # same warning for the same reason, and #2663 is what it cost: the
+    # rejection was swallowed and every counter read zero in production while
+    # the writes were landing fine.
+    ambient = case(
+        (RuleUsageEvent.source.notin_(RANKED_SOURCES), True),
+        else_=False,
+    ).label("ambient")
     try:
         async with async_session() as session:
             rows = (
@@ -168,9 +241,14 @@ async def usage_for_rules(rule_ids: list[int]) -> dict[int, dict]:
                         RuleUsageEvent.event,
                         func.count().label("n"),
                         func.max(RuleUsageEvent.created_at).label("last_at"),
+                        ambient,
                     )
                     .where(RuleUsageEvent.rule_id.in_(ids))
-                    .group_by(RuleUsageEvent.rule_id, RuleUsageEvent.event)
+                    .group_by(
+                        RuleUsageEvent.rule_id,
+                        RuleUsageEvent.event,
+                        ambient,
+                    )
                 )
             ).all()
     except Exception:
@@ -180,14 +258,23 @@ async def usage_for_rules(rule_ids: list[int]) -> dict[int, dict]:
         await _report_failure("readout")
         return out
 
-    for rule_id, event, n, last_at in rows:
+    for rule_id, event, n, last_at, is_amb in rows:
         slot = out.get(int(rule_id))
         if slot is None:
             continue
-        if event == SURFACED:
+        if event == SURFACED and is_amb:
+            slot["ambient_count"] = int(n)
+        elif event == SURFACED:
             slot["surfaced_count"] = int(n)
             slot["last_surfaced_at"] = iso(last_at)
         elif event == PULLED:
-            slot["pull_count"] = int(n)
-            slot["last_pulled_at"] = iso(last_at)
+            # Pulls are pulls regardless of what surfaced the rule — "did
+            # anyone ever open this?" does not depend on how it was found. Both
+            # halves accumulate, so this ADDS rather than assigns: a rule can
+            # now be pulled after a ranked hint and after a preload, and the
+            # split arrives as two rows.
+            slot["pull_count"] = slot["pull_count"] + int(n)
+            latest = iso(last_at)
+            if latest and (slot["last_pulled_at"] or "") < latest:
+                slot["last_pulled_at"] = latest
     return out

src/scribe/services/rulebooks.py

@@ -23,6 +23,7 @@ from scribe.services.verification import (
 )
 from scribe.services import rule_versions
 from scribe.models.rule_version import RuleVersion
+from scribe.services.rule_usage import record_rule_surfaced
 
 logger = logging.getLogger(__name__)
 
@@ -1395,7 +1396,7 @@ async def get_applicable_rules(
     }
 
 
-def rules_payload(applicable: dict) -> dict:
+def rules_payload(applicable: dict, *, user_id: int | None, source: str) -> dict:
     """The caller-facing shape of a get_applicable_rules() result.
 
     Every surface that hands rules to an agent (enter_project, get_project,
@@ -1406,7 +1407,34 @@ def rules_payload(applicable: dict) -> dict:
     `excluded_always_on` (milestone 297) names the always-on rulebooks this
     project decided NOT to inherit, so the departure is visible wherever the
     rules are.
+
+    IT ALSO RECORDS THE SURFACING, which is why it now takes a caller and a
+    source. Every one of those surfaces is a bulk delivery — the applicable set
+    handed over whole, chosen by nobody — so this is the one place that has to
+    emit for all of them. Doing it per-caller instead would be five sites to
+    remember, and #3430 gap 2 is what that costs: the process→skill sync went
+    un-emitted through an entire dedicated telemetry survey because nothing
+    forced its surface to be accounted for.
+
+    `source` stays the CALLER's name rather than a constant, so the readout can
+    still separate the session handshake from a mid-session milestone read;
+    `RANKED_SOURCES` in `rule_usage` is what folds them back together.
+
+    Emitting from here is safe in a way emitting from `get_applicable_rules`
+    would not be: this function is only ever called to BUILD A REPLY. The two
+    other callers of the rules machinery — the write-path etag arm
+    (`plugin_context`) and `rules_etag_for` — compute a marker and show nobody
+    anything, and counting those would put rules in the denominator that no
+    agent ever saw.
     """
+    record_rule_surfaced(
+        user_id=user_id,
+        rule_ids=(
+            [r["id"] for r in applicable.get("rules", [])]
+            + [r["id"] for r in applicable.get("project_rules", [])]
+        ),
+        source=source,
+    )
     return {
         "applicable_rules": applicable["rules"],
         "applicable_rules_truncated": applicable["truncated"],