feat(telemetry): the preload emits, and the always-on set stops being unfalsifiable (#3473)
CI & Build / Python lint (push) Successful in 3s
CI & Build / Plugin hooks (push) Successful in 11s
CI & Build / TypeScript typecheck (push) Successful in 23s
CI & Build / integration (push) Successful in 31s
CI & Build / Python tests (push) Successful in 1m6s
CI & Build / Build & push image (push) Successful in 27s

The ranked rule arm became measurable in M333. The preload did not — and
that is the surface whose value is actually in question. `list_always_on_rules`,
the SessionStart block and every `rules_payload` caller handed rules over
wholesale and emitted nothing, so the resident set's token cost was certain
and its usefulness could not be tested even in principle.

Bulk deliveries now record as AMBIENT, beside the ranked count and never
inside pull-through. Folding them in would mean growing the always-on set
depressed the arm's measured precision and trimming it flattered the arm,
neither for any reason to do with the arm.

`RANKED_SOURCES` inverts the note twin's `AMBIENT_SOURCES` deliberately: there
is one ranked rule source and this change adds seven bulk ones, so naming the
rare half makes a forgotten surface default to ambient — under-counting it —
rather than padding the denominator with surfacings nobody chose.

Two lookalike call sites are deliberately left silent, with a test to keep
them that way: the write-path etag arm and `rules_etag_for` read the rules to
build or compare a MARKER and show nobody anything.

No migration — `event` and `source` are plain Text with no CHECK (rule 36
does not apply). Snippet #2858 updated to the new `rules_payload` contract.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011cPyzNnegXHr5iRMzzy5KJ
This commit is contained in:
2026-09-02 23:06:40 -04:00
co-authored by Claude Opus 5
parent 6627cfc2f0
commit 8b9b3a1d9b
14 changed files with 505 additions and 48 deletions
+20 -6
View File
@@ -197,17 +197,31 @@ It is an UPPER BOUND per surface: a pull records the door it came
query failed while the rest of the readout stood.
`rule_usage` — the same question for RULES, from `rule_usage_events`:
`surfaced`, `pulled` split into `pulled_by_agent` / `pulled_by_human`, the
distinct-rule counts, and `pull_through` on the same definition (agent
pulls over surfacings).
`surfaced` and `ambient`, `pulled` split into `pulled_by_agent` /
`pulled_by_human`, the distinct-rule counts, and `pull_through` on the same
definition (agent pulls over RANKED surfacings).
A SEPARATE BLOCK, not folded into `usage`, and reading it as one number
with that is the mistake to avoid. The corpora differ by orders of
magnitude — a few dozen eligible rules against thousands of notes — so a
blended ratio would be the note ratio with noise on it and would hide the
rule arm entirely. It also has no `ambient` key, because nothing surfaces a
rule un-ranked: `list_always_on_rules` and `enter_project` hand over rules
wholesale but emit no event, so there is no ambient class to separate.
rule arm entirely.
`surfaced` VS `ambient` IS THE READING THAT MATTERS HERE. `surfaced` counts
rules a ranker chose — today only the write-path arm — and those are claims
a pull can settle. `ambient` counts BULK DELIVERIES: the SessionStart
preload, `list_always_on_rules`, and the `rules_payload` surfaces
(`enter_project`, `get_project`, `get_milestone`, `start_planning`,
`get_task`), which hand over the whole applicable set at once with nobody
choosing anything. A large `ambient` says the resident set is big and
arrives often — never that it is useful, and never that it is read.
`pull_through` therefore divides by `surfaced` alone. Fold the preload in
and growing the always-on set would depress the arm's measured precision
while trimming it would flatter it, for reasons having nothing to do with
the arm. To judge the PRELOAD instead, compare `ambient` against pulls of
those same rules over time: a resident set surfaced thousands of times and
opened never is the dead-weight signal, one tier up.
Read it against `sources["write_path_rule"]`. That surface has never once
declined to fire, and until this block existed there was no way to tell a