feat(telemetry): the preload emits, and the always-on set stops being unfalsifiable (#3473)
CI & Build / Python lint (push) Successful in 3s
CI & Build / Plugin hooks (push) Successful in 11s
CI & Build / TypeScript typecheck (push) Successful in 23s
CI & Build / integration (push) Successful in 31s
CI & Build / Python tests (push) Successful in 1m6s
CI & Build / Build & push image (push) Successful in 27s

The ranked rule arm became measurable in M333. The preload did not — and
that is the surface whose value is actually in question. `list_always_on_rules`,
the SessionStart block and every `rules_payload` caller handed rules over
wholesale and emitted nothing, so the resident set's token cost was certain
and its usefulness could not be tested even in principle.

Bulk deliveries now record as AMBIENT, beside the ranked count and never
inside pull-through. Folding them in would mean growing the always-on set
depressed the arm's measured precision and trimming it flattered the arm,
neither for any reason to do with the arm.

`RANKED_SOURCES` inverts the note twin's `AMBIENT_SOURCES` deliberately: there
is one ranked rule source and this change adds seven bulk ones, so naming the
rare half makes a forgotten surface default to ambient — under-counting it —
rather than padding the denominator with surfacings nobody chose.

Two lookalike call sites are deliberately left silent, with a test to keep
them that way: the write-path etag arm and `rules_etag_for` read the rules to
build or compare a MARKER and show nobody anything.

No migration — `event` and `source` are plain Text with no CHECK (rule 36
does not apply). Snippet #2858 updated to the new `rules_payload` contract.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011cPyzNnegXHr5iRMzzy5KJ
This commit is contained in:
2026-09-02 23:06:40 -04:00
co-authored by Claude Opus 5
parent 6627cfc2f0
commit 8b9b3a1d9b
14 changed files with 505 additions and 48 deletions
@@ -511,3 +511,68 @@ async def test_rule_usage_sees_only_its_own_users_events(_dispose_engine):
assert (await retrieval_summary(990014, days=30))["rule_usage"]["surfaced"] == 1
finally:
await cleanup()
@pytest.mark.integration
@pytest.mark.asyncio
async def test_the_preload_lands_in_ambient_and_never_in_the_ratio(_dispose_engine):
"""The split that makes the always-on set judgeable (#3473).
Pull-through asks "was that hint any use", and only a surface that CHOSE
what it showed can be judged by it. If the preload counted toward the
denominator, growing the always-on set would DEPRESS the arm's measured
precision and trimming it would flatter it — neither for any reason to do
with the arm. So the resident deliveries are counted, reported, and kept
out of the ratio.
"""
from scribe.services.retrieval_telemetry import retrieval_summary
cleanup = await _rule_events(990012, [
# One rule the arm actually chose, and opened.
(5101, "surfaced", "write_path_rule"),
(5101, "pulled", "mcp_get_rule"),
# Four bulk deliveries across every shape of preload. Nobody chose any
# of them, and none may touch the denominator.
(5102, "surfaced", "session_start"),
(5103, "surfaced", "list_always_on_rules"),
(5104, "surfaced", "enter_project"),
(5105, "surfaced", "get_milestone"),
])
try:
ru = (await retrieval_summary(990012, days=30))["rule_usage"]
assert ru["surfaced"] == 1, "only the arm chose a rule"
assert ru["ambient"] == 4, "the four bulk deliveries are reported, not dropped"
# 1 agent pull over 1 RANKED surfacing. Were the ambient four folded in
# the ratio would read 0.2 — the arm looking four times worse for
# having a large resident set beside it.
assert ru["pull_through"] == 1.0
# Dead-weight detection needs both classes: a rule delivered by the
# preload and never opened is the case that reading matters most for.
assert ru["distinct_rules_surfaced"] == 5
assert ru["distinct_rules_pulled"] == 1
finally:
await cleanup()
@pytest.mark.integration
@pytest.mark.asyncio
async def test_ambient_alone_reports_no_ratio(_dispose_engine):
"""A brand-new install loads rules every session and may never trigger the
arm. That must read as "no ranked surfacings yet", not as a precision of
zero — the reading that would make a working install look broken."""
from scribe.services.retrieval_telemetry import retrieval_summary
cleanup = await _rule_events(990013, [
(5201, "surfaced", "session_start"),
(5202, "surfaced", "session_start"),
])
try:
ru = (await retrieval_summary(990013, days=30))["rule_usage"]
assert ru["ambient"] == 2
assert ru["surfaced"] == 0
assert ru["pull_through"] is None
finally:
await cleanup()