feat(telemetry): the preload emits, and the always-on set stops being unfalsifiable (#3473)
CI & Build / Python lint (push) Successful in 3s
CI & Build / Plugin hooks (push) Successful in 11s
CI & Build / TypeScript typecheck (push) Successful in 23s
CI & Build / integration (push) Successful in 31s
CI & Build / Python tests (push) Successful in 1m6s
CI & Build / Build & push image (push) Successful in 27s
CI & Build / Python lint (push) Successful in 3s
CI & Build / Plugin hooks (push) Successful in 11s
CI & Build / TypeScript typecheck (push) Successful in 23s
CI & Build / integration (push) Successful in 31s
CI & Build / Python tests (push) Successful in 1m6s
CI & Build / Build & push image (push) Successful in 27s
The ranked rule arm became measurable in M333. The preload did not — and that is the surface whose value is actually in question. `list_always_on_rules`, the SessionStart block and every `rules_payload` caller handed rules over wholesale and emitted nothing, so the resident set's token cost was certain and its usefulness could not be tested even in principle. Bulk deliveries now record as AMBIENT, beside the ranked count and never inside pull-through. Folding them in would mean growing the always-on set depressed the arm's measured precision and trimming it flattered the arm, neither for any reason to do with the arm. `RANKED_SOURCES` inverts the note twin's `AMBIENT_SOURCES` deliberately: there is one ranked rule source and this change adds seven bulk ones, so naming the rare half makes a forgotten surface default to ambient — under-counting it — rather than padding the denominator with surfacings nobody chose. Two lookalike call sites are deliberately left silent, with a test to keep them that way: the write-path etag arm and `rules_etag_for` read the rules to build or compare a MARKER and show nobody anything. No migration — `event` and `source` are plain Text with no CHECK (rule 36 does not apply). Snippet #2858 updated to the new `rules_payload` contract. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011cPyzNnegXHr5iRMzzy5KJ
This commit is contained in:
@@ -197,17 +197,31 @@ It is an UPPER BOUND per surface: a pull records the door it came
|
||||
query failed while the rest of the readout stood.
|
||||
|
||||
`rule_usage` — the same question for RULES, from `rule_usage_events`:
|
||||
`surfaced`, `pulled` split into `pulled_by_agent` / `pulled_by_human`, the
|
||||
distinct-rule counts, and `pull_through` on the same definition (agent
|
||||
pulls over surfacings).
|
||||
`surfaced` and `ambient`, `pulled` split into `pulled_by_agent` /
|
||||
`pulled_by_human`, the distinct-rule counts, and `pull_through` on the same
|
||||
definition (agent pulls over RANKED surfacings).
|
||||
|
||||
A SEPARATE BLOCK, not folded into `usage`, and reading it as one number
|
||||
with that is the mistake to avoid. The corpora differ by orders of
|
||||
magnitude — a few dozen eligible rules against thousands of notes — so a
|
||||
blended ratio would be the note ratio with noise on it and would hide the
|
||||
rule arm entirely. It also has no `ambient` key, because nothing surfaces a
|
||||
rule un-ranked: `list_always_on_rules` and `enter_project` hand over rules
|
||||
wholesale but emit no event, so there is no ambient class to separate.
|
||||
rule arm entirely.
|
||||
|
||||
`surfaced` VS `ambient` IS THE READING THAT MATTERS HERE. `surfaced` counts
|
||||
rules a ranker chose — today only the write-path arm — and those are claims
|
||||
a pull can settle. `ambient` counts BULK DELIVERIES: the SessionStart
|
||||
preload, `list_always_on_rules`, and the `rules_payload` surfaces
|
||||
(`enter_project`, `get_project`, `get_milestone`, `start_planning`,
|
||||
`get_task`), which hand over the whole applicable set at once with nobody
|
||||
choosing anything. A large `ambient` says the resident set is big and
|
||||
arrives often — never that it is useful, and never that it is read.
|
||||
|
||||
`pull_through` therefore divides by `surfaced` alone. Fold the preload in
|
||||
and growing the always-on set would depress the arm's measured precision
|
||||
while trimming it would flatter it, for reasons having nothing to do with
|
||||
the arm. To judge the PRELOAD instead, compare `ambient` against pulls of
|
||||
those same rules over time: a resident set surfaced thousands of times and
|
||||
opened never is the dead-weight signal, one tier up.
|
||||
|
||||
Read it against `sources["write_path_rule"]`. That surface has never once
|
||||
declined to fire, and until this block existed there was no way to tell a
|
||||
|
||||
Reference in New Issue
Block a user