feat(telemetry): the readout names rules that were opened and changed nothing (#4213)

Milestone 419 step 2. Step 1 made an outcome recordable; this makes it
readable. `retrieval_summary`'s rule block gains `applied`, `departed` and
`distinct_rules_acted`, and `_compute_warnings` gains two codes.

TWO CODES, NOT ONE WITH A ZERO IN IT. `read_and_unacted` reports rules that
were opened and left no outcome, against the ones that did. It only fires once
outcomes exist anywhere in the window, because a window with none cannot tell
"every rule was ignored" from "nothing calls `rule_outcome` yet" — and on
every install the day this ships, the truth is the second. Claiming the first
there would be #3311's failure exactly: a statistic that could not vary being
read as a fact about the corpus. The cold case gets its own code,
`outcomes_never_recorded`, whose prose says in as many words that it does NOT
mean the rules were ignored.

`applied` AND `departed` ARE NOT SUMMED. A departure carries the reason the
agent gave and is evidence about the RULE; an application is evidence about
the agent. Folded together they would say only "an outcome exists", which is
true of both and useful about neither. `distinct_rules_acted` counts either,
because for the unacted arithmetic the distinction does not matter.

An outcome is not a pull. The fold branches on OUTCOMES first and never routes
an outcome through the surfaced/ambient split: `source` on an outcome row
names the door the outcome came through, not a ranker, so the ambient
distinction has nothing to say about it. An integration test holds that line —
if an outcome leaked into the pull counters the silently-unchanged rule would
vanish into a compliant-looking total, which is the confusion #4212 was opened
to end.

Verified by lifting the shipped `_compute_warnings` out of source with `ast`
and exercising it against the six populations the new tests assert: cold
instrument, warm instrument, departures-only, full compliance, nothing opened,
and a failed read. The integration tests for the new counts run against real
Postgres in CI — count(distinct) with an IN over an unconstrained column is a
SQL shape a mock would agree with whatever it did, which is what #2663 was.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01821k5B3Ysecp9fNYs92Kuy

src/scribe/mcp/tools/search.py

@@ -336,6 +336,18 @@ It is an UPPER BOUND per surface: a pull records the door it came
     `pulled_by_human`, the distinct-rule counts, and `pull_through` on the same
     definition (agent pulls over RANKED surfacings).
 
+    `applied`, `departed` AND `distinct_rules_acted` ARE WHAT HAPPENED AFTER
+    THE RULE WAS OPENED (#4213). A pull says the rule was read; these say it
+    changed something. `applied` counts rules followed, `departed` rules
+    deliberately not followed — kept apart rather than summed, because a
+    departure carries the reason the agent gave and is evidence about the
+    RULE, while an application is evidence about the agent. There is
+    deliberately no count of rules read and quietly ignored: that state is what
+    is left over when a rule was pulled and neither outcome arrived, and
+    `read_and_unacted` below is where it is reported. Asking an agent to
+    declare it would be asking it to notice an omission it is defined by not
+    noticing.
+
     A SEPARATE BLOCK, not folded into `usage`, and reading it as one number
     with that is the mistake to avoid. The corpora differ by orders of
     magnitude — a few dozen eligible rules against thousands of notes — so a
@@ -418,6 +430,21 @@ It is an UPPER BOUND per surface: a pull records the door it came
     - `surfaced_never_pulled` — distinct records shown and never opened, per
       corpus. Read their titles before touching a threshold: a record nobody
       opens is usually one whose title does not say when it matters.
+    - `read_and_unacted` — distinct rules OPENED in the window that recorded
+      no outcome, against the ones that did. The failure milestone 419 was
+      opened on, and the worse sibling of `surfaced_never_pulled` above: a
+      rule nobody opens is cheap, while a rule read and silently unchanged is
+      indistinguishable from one that worked. It does not say which of the two
+      causes it is — a rule mis-triggering, arriving where it does not apply,
+      or a rule being ignored — and those want opposite fixes, so read the
+      rules before moving anything.
+    - `outcomes_never_recorded` — rules were opened and NOT ONE outcome exists
+      anywhere in the window. Deliberately a separate code, and not a
+      `read_and_unacted` with a zero in it: a window with no outcomes at all
+      cannot tell "every rule was ignored" from "nothing on this install calls
+      `rule_outcome` yet", and reporting the first would manufacture a finding
+      out of an unwired feature. Wire the outcome call before reading this as
+      a fact about the corpus.
     - `unregistered_source` — rows under a source missing from
       `retrieval_registry`. Its numbers are real; no verdict could be
       computed, because nothing says whether it was asked or fired unbidden.

src/scribe/services/retrieval_telemetry.py

@@ -28,6 +28,9 @@ from scribe.models import async_session
 from scribe.models.base import iso
 from scribe.models.note import Note
 from scribe.models.note_usage import PULLED, SURFACED, NoteUsageEvent
+from scribe.models.rule_usage import OUTCOMES as RULE_OUTCOMES
+from scribe.models.rule_usage import APPLIED as RULE_APPLIED
+from scribe.models.rule_usage import DEPARTED as RULE_DEPARTED
 from scribe.models.rule_usage import PULLED as RULE_PULLED
 from scribe.models.rule_usage import SURFACED as RULE_SURFACED
 from scribe.models.rule_usage import RuleUsageEvent
@@ -636,6 +639,52 @@ def _compute_warnings(sources: dict, usage: dict, rule_usage: dict,
                 surfaced=int(shown), pulled=int(pulled or 0), never_pulled=never,
             ))
 
+    # ── A rule that was read and changed nothing (#4213, milestone 419) ──
+    #
+    # The failure this milestone was opened on, and the one number that could
+    # not previously be computed. `surfaced_never_pulled` above catches a rule
+    # nobody opens; this catches the worse case — a rule the agent DID open,
+    # deliberately, and then left no trace of having acted on. Until #4212
+    # those were arithmetically identical to compliance.
+    opened = int(rule_usage.get("distinct_rules_pulled") or 0)
+    acted = int(rule_usage.get("distinct_rules_acted") or 0)
+    applied = int(rule_usage.get("applied") or 0)
+    departed = int(rule_usage.get("departed") or 0)
+
+    if opened and not (applied or departed):
+        # THE HONEST ANSWER WHILE THE INSTRUMENT IS COLD, and the reason this
+        # is a separate code rather than a zero fed into the check below.
+        # Outcomes only started being recorded in milestone 419; a window
+        # containing none cannot tell "every rule was ignored" from "nothing
+        # reports outcomes yet". Emitting the ignored-rules warning here would
+        # manufacture a finding out of an unwired feature — #3311's mistake
+        # exactly, where a statistic that could not vary was read as a fact
+        # about the corpus.
+        out.append(_warn(
+            "outcomes_never_recorded",
+            f"{opened} distinct rules were opened in this window and not one "
+            f"recorded an outcome. This does NOT mean they were ignored — it "
+            f"means nothing is calling `rule_outcome`, so the difference "
+            f"between a rule that worked and a rule that was read and "
+            f"forgotten is still unmeasured here.",
+            source=None, opened=opened, applied=0, departed=0,
+        ))
+    elif opened:
+        unacted = opened - acted
+        if unacted > 0:
+            out.append(_warn(
+                "read_and_unacted",
+                f"{unacted} of {opened} distinct rules were opened in this "
+                f"window and left no outcome, against {acted} that did. A rule "
+                f"read and silently unchanged looks exactly like one that "
+                f"worked; these are the ones where nobody can tell. Either the "
+                f"rule is mis-triggering — it arrives, gets read, and does not "
+                f"apply — or it is being ignored, and the two want opposite "
+                f"fixes.",
+                source=None, opened=opened, acted=acted, unacted=unacted,
+                applied=applied, departed=departed,
+            ))
+
     return out
 
 
@@ -1063,10 +1112,26 @@ async def retrieval_summary(
                         )
                     )
                 ).scalar_one()
+                # Rules that were OPENED AND THEN ACTED ON (#4212). Its own
+                # distinct count for the same reason the two above have one:
+                # "how many rules did anything come of" cannot be summed from
+                # the per-source group without double-counting a rule that was
+                # applied once and departed from once.
+                distinct_rules_acted = (
+                    await session.execute(
+                        select(func.count(func.distinct(RuleUsageEvent.rule_id)))
+                        .where(
+                            RuleUsageEvent.created_at >= since,
+                            RuleUsageEvent.user_id == user_id,
+                            RuleUsageEvent.event.in_(RULE_OUTCOMES),
+                        )
+                    )
+                ).scalar_one()
             except Exception:
                 logger.warning("rule usage read failed", exc_info=True)
                 rule_rows = None
                 distinct_rules_surfaced = distinct_rules_pulled = 0
+                distinct_rules_acted = 0
     except Exception:
         logger.warning("retrieval summary read failed", exc_info=True)
         out["read_failed"] = True
@@ -1175,6 +1240,13 @@ async def retrieval_summary(
         "pulled": 0, "pulled_by_agent": 0, "pulled_by_human": 0,
         "distinct_rules_surfaced": int(distinct_rules_surfaced or 0),
         "distinct_rules_pulled": int(distinct_rules_pulled or 0),
+        # The outcome half (#4212, milestone 419). `pulled` says a rule was
+        # opened; these say whether anything came of it. Until this existed, a
+        # rule obeyed every time and a rule ignored every time produced
+        # identical rows, and the second is the one worth finding.
+        "applied": 0,
+        "departed": 0,
+        "distinct_rules_acted": int(distinct_rules_acted or 0),
     }
     if rule_rows is None:
         # The FLAG is added, the shape is kept — matching `by_source_failed`
@@ -1205,6 +1277,15 @@ async def retrieval_summary(
                     rule_usage["pulled_by_agent"] += n
                 else:
                     rule_usage["pulled_by_human"] += n
+            elif event == RULE_APPLIED:
+                rule_usage["applied"] += n
+            elif event == RULE_DEPARTED:
+                # Kept apart from `applied` rather than summed into a single
+                # "acted" count. A departure is a rule someone ARGUED with,
+                # and an install where every outcome is a departure is telling
+                # you something quite different from one where none is —
+                # folding them together would hide exactly that.
+                rule_usage["departed"] += n
 
     # None, not 0.0, when nothing was surfaced — matching the note block. A
     # ratio of zero asserts "we showed rules and none were opened"; with an