feat(telemetry): the preload emits, and the always-on set stops being unfalsifiable (#3473)
CI & Build / Python lint (push) Successful in 3s
CI & Build / Plugin hooks (push) Successful in 11s
CI & Build / TypeScript typecheck (push) Successful in 23s
CI & Build / integration (push) Successful in 31s
CI & Build / Python tests (push) Successful in 1m6s
CI & Build / Build & push image (push) Successful in 27s
CI & Build / Python lint (push) Successful in 3s
CI & Build / Plugin hooks (push) Successful in 11s
CI & Build / TypeScript typecheck (push) Successful in 23s
CI & Build / integration (push) Successful in 31s
CI & Build / Python tests (push) Successful in 1m6s
CI & Build / Build & push image (push) Successful in 27s
The ranked rule arm became measurable in M333. The preload did not — and that is the surface whose value is actually in question. `list_always_on_rules`, the SessionStart block and every `rules_payload` caller handed rules over wholesale and emitted nothing, so the resident set's token cost was certain and its usefulness could not be tested even in principle. Bulk deliveries now record as AMBIENT, beside the ranked count and never inside pull-through. Folding them in would mean growing the always-on set depressed the arm's measured precision and trimming it flattered the arm, neither for any reason to do with the arm. `RANKED_SOURCES` inverts the note twin's `AMBIENT_SOURCES` deliberately: there is one ranked rule source and this change adds seven bulk ones, so naming the rare half makes a forgotten surface default to ambient — under-counting it — rather than padding the denominator with surfacings nobody chose. Two lookalike call sites are deliberately left silent, with a test to keep them that way: the write-path etag arm and `rules_etag_for` read the rules to build or compare a MARKER and show nobody anything. No migration — `event` and `source` are plain Text with no CHECK (rule 36 does not apply). Snippet #2858 updated to the new `rules_payload` contract. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011cPyzNnegXHr5iRMzzy5KJ
This commit is contained in:
@@ -1375,6 +1375,24 @@ async def build_session_context(
|
||||
# exclusion (milestone 297) takes a rulebook out of this block, and is
|
||||
# named below so the departure is visible rather than silent.
|
||||
rules = await rulebooks_svc.list_always_on_rules(user_id, project_id=project_id)
|
||||
# AMBIENT source, and the one that matters most: this is the preload — the
|
||||
# block every session opens with, chosen by nobody, paid for every turn.
|
||||
#
|
||||
# It emitted nothing until 2026-09-03, which made the resident set's cost
|
||||
# certain and its usefulness unfalsifiable at the same time (#3473). Note
|
||||
# #3089 is the argument this measurement finally lets someone test: that a
|
||||
# rule arriving with thirty others, none of them relevant, is read as
|
||||
# preamble rather than as a claim — so presence is not surfacing, and a
|
||||
# tier-1 set can grow without anybody noticing it stopped working.
|
||||
#
|
||||
# Recorded even when the hook truncates the block below: the rules WERE
|
||||
# delivered, and counting only the untruncated ones would quietly shrink
|
||||
# the denominator exactly where the set is too big to read.
|
||||
record_rule_surfaced(
|
||||
user_id=user_id,
|
||||
rule_ids=[r.id for r in rules],
|
||||
source="session_start",
|
||||
)
|
||||
excluded = (
|
||||
await rulebooks_svc.excluded_always_on_rulebooks(user_id, project_id)
|
||||
if project_id else []
|
||||
|
||||
Reference in New Issue
Block a user