feat(telemetry): retrieval_telemetry reports rule pull-through where it reported nothing (#3317)

Milestone 333 step 3, the read half. Steps 1 and 2 built the table and filled
it; until now nothing read it, and `usage` — sourced entirely from
note_usage_events — described notes only while `sources` happily listed a
write_path_rule row above it. A reader takes the aggregate as covering
everything named above it. It did not.

A SEPARATE `rule_usage` BLOCK, not folded into `usage`. Two reasons, and the
second is the one that bites: the corpora differ by orders of magnitude, so a
blended ratio would be the note ratio with noise on it and the rule arm would
stay invisible inside it; and `usage` is what existing callers already read and
compare across windows, so silently changing what it counts would move a number
nobody was told had changed meaning. There is a test asserting rule events stay
out of the note block.

No `ambient` key, unlike the twin. Nothing surfaces a rule un-ranked —
list_always_on_rules and enter_project hand rules over wholesale but emit no
event — so there is no ambient class to subtract. The absence is a fact about
the data, not an oversight, and it returns when a bulk loader starts emitting.

Guarded separately, like `by_source`. This table did not exist a commit ago,
and an instance running upgraded code against un-migrated schema would
otherwise take down two readouts that work perfectly in order to report a third
that cannot. On failure the FLAG is added and the SHAPE is kept — a caller must
not have to choose between crashing on a missing key and quietly rendering
zeros it has no right to.

`pull_through` is None rather than 0.0 on an empty window, matching the note
block. A ratio of zero asserts "rules were shown and none opened"; with an
empty numerator and denominator that is a claim the data does not support, and
it is the reading that would make a brand-new install look like a broken one.

Also fixed, from #3311: the rule arm never timed its search, so it was the one
source in the readout reporting a null p90_duration_ms — a gap that reads as
"this surface is somehow not measurable" rather than "nobody passed the
number".

Both docstrings updated in the same change. The tool's is the agent-facing
contract (rule 119) and it explicitly said rule surfacings were absent and had
"no usage counter at all". Leaving that would have had a reader conclude the
arm has zero pull-through rather than a separate one.

Tests are integration for the reason the block above them is: real GROUP BYs
and count(distinct) against a table a commit old, in a module whose one
production outage was a SQL shape the database rejected inside a broad except.
A mock would agree with whatever the code does, including nothing.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TcCs1CcQ1ormdnzSshKqvN

tests/test_services_retrieval_telemetry.py

@@ -359,3 +359,155 @@ async def test_the_agent_pull_filter_does_not_treat_its_underscore_as_a_wildcard
         async with async_session() as s:
             await s.execute(delete(NoteUsageEvent).where(NoteUsageEvent.user_id == UID))
             await s.commit()
+
+
+# ─── rule usage (milestone 333 step 3) ───────────────────────────────────────
+# Integration, for the same reason the block above is: these are real GROUP BYs
+# and count(distinct) against a table that did not exist a commit ago, in a
+# module whose one production outage (#2663) was a SQL shape the database
+# rejected inside a broad except. A mock would agree with whatever the code
+# does, including nothing.
+
+
+async def _rule_events(uid, rows):
+    """Write (event, source) pairs for one rule and hand back a cleanup."""
+    from sqlalchemy import delete
+
+    from scribe.models import async_session
+    from scribe.models.rule_usage import RuleUsageEvent
+
+    async with async_session() as s:
+        s.add_all([
+            RuleUsageEvent(user_id=uid, rule_id=rid, event=ev, source=src)
+            for rid, ev, src in rows
+        ])
+        await s.commit()
+
+    async def cleanup():
+        async with async_session() as s:
+            await s.execute(
+                delete(RuleUsageEvent).where(RuleUsageEvent.user_id == uid)
+            )
+            await s.commit()
+
+    return cleanup
+
+
+@pytest.mark.integration
+@pytest.mark.asyncio
+async def test_rule_usage_is_a_coherent_zero_on_a_fresh_install(_dispose_engine):
+    """Every rule in an existing install predates this table, so "no events" is
+    the normal state for a while. It must read as zero, not as a missing key
+    and not as a failure — the same "no rows" / "read broke" distinction the
+    rest of this readout keeps (#2663).
+
+    `pull_through` is None rather than 0.0, matching the note block: a ratio of
+    zero asserts "rules were shown and none opened", which with an empty
+    numerator AND denominator is a claim the data does not support.
+    """
+    from scribe.services.retrieval_telemetry import retrieval_summary
+
+    out = await retrieval_summary(990010, days=30)
+    assert out["read_failed"] is False
+    assert "rule_usage_failed" not in out["rule_usage"]
+    assert out["rule_usage"]["surfaced"] == 0
+    assert out["rule_usage"]["pulled"] == 0
+    assert out["rule_usage"]["distinct_rules_surfaced"] == 0
+    assert out["rule_usage"]["pull_through"] is None
+
+
+@pytest.mark.integration
+@pytest.mark.asyncio
+async def test_an_agent_reading_a_surfaced_rule_is_what_moves_the_ratio(_dispose_engine):
+    """The whole point of the milestone: the arm can now be told apart from a
+    bar it cannot fail to clear."""
+    from scribe.services.retrieval_telemetry import retrieval_summary
+
+    cleanup = await _rule_events(990011, [
+        (5001, "surfaced", "write_path_rule"),
+        (5002, "surfaced", "write_path_rule"),
+        (5001, "pulled", "mcp_get_rule"),
+    ])
+    try:
+        ru = (await retrieval_summary(990011, days=30))["rule_usage"]
+        assert ru["surfaced"] == 2
+        assert ru["pulled"] == 1
+        assert ru["pulled_by_agent"] == 1
+        assert ru["pulled_by_human"] == 0
+        assert ru["distinct_rules_surfaced"] == 2
+        assert ru["distinct_rules_pulled"] == 1
+        assert ru["pull_through"] == 0.5
+    finally:
+        await cleanup()
+
+
+@pytest.mark.integration
+@pytest.mark.asyncio
+async def test_a_person_browsing_the_rule_list_does_not_move_the_ratio(_dispose_engine):
+    """The mcp_/rest_ split, and it carries more weight here than for notes.
+
+    The arm's claim is "this rule may apply to what you are writing". Only an
+    agent opening it says that claim landed; a person clicking through the rule
+    list in the web UI says nothing about the hint. Both are still counted in
+    `pulled`, so "is this rule dead weight?" stays answerable.
+    """
+    from scribe.services.retrieval_telemetry import retrieval_summary
+
+    cleanup = await _rule_events(990012, [
+        (5003, "surfaced", "write_path_rule"),
+        (5003, "pulled", "rest_rule"),
+    ])
+    try:
+        ru = (await retrieval_summary(990012, days=30))["rule_usage"]
+        assert ru["pulled"] == 1
+        assert ru["pulled_by_human"] == 1
+        assert ru["pulled_by_agent"] == 0
+        # Surfaced once, opened by nobody who matters to this question.
+        assert ru["pull_through"] == 0.0
+    finally:
+        await cleanup()
+
+
+@pytest.mark.integration
+@pytest.mark.asyncio
+async def test_rule_events_stay_out_of_the_note_block(_dispose_engine):
+    """The separation, asserted rather than assumed.
+
+    `usage` is what existing callers already read and compare across windows.
+    If rule events leaked into it, that number would move for a reason nobody
+    was told about — and the rule arm would still be invisible, because a few
+    dozen rules against thousands of notes is noise on the note ratio.
+    """
+    from scribe.services.retrieval_telemetry import retrieval_summary
+
+    cleanup = await _rule_events(990013, [
+        (5004, "surfaced", "write_path_rule"),
+        (5004, "pulled", "mcp_get_rule"),
+    ])
+    try:
+        out = await retrieval_summary(990013, days=30)
+        assert out["rule_usage"]["surfaced"] == 1
+        # The note block saw none of it.
+        assert out["usage"]["surfaced"] == 0
+        assert out["usage"]["pulled"] == 0
+        assert out["usage"]["pull_through"] is None
+    finally:
+        await cleanup()
+
+
+@pytest.mark.integration
+@pytest.mark.asyncio
+async def test_rule_usage_sees_only_its_own_users_events(_dispose_engine):
+    """Same access rule as the rest of the readout — the owner filter IS the
+    rule for telemetry, which is not a shared record kind."""
+    from scribe.services.retrieval_telemetry import retrieval_summary
+
+    cleanup = await _rule_events(990014, [
+        (5005, "surfaced", "write_path_rule"),
+        (5005, "pulled", "mcp_get_rule"),
+    ])
+    try:
+        assert (await retrieval_summary(990015, days=30))["rule_usage"]["surfaced"] == 0
+        assert (await retrieval_summary(990014, days=30))["rule_usage"]["surfaced"] == 1
+    finally:
+        await cleanup()