feat(telemetry): the preload emits, and the always-on set stops being unfalsifiable (#3473)
CI & Build / Python lint (push) Successful in 3s
CI & Build / Plugin hooks (push) Successful in 11s
CI & Build / TypeScript typecheck (push) Successful in 23s
CI & Build / integration (push) Successful in 31s
CI & Build / Python tests (push) Successful in 1m6s
CI & Build / Build & push image (push) Successful in 27s

The ranked rule arm became measurable in M333. The preload did not — and
that is the surface whose value is actually in question. `list_always_on_rules`,
the SessionStart block and every `rules_payload` caller handed rules over
wholesale and emitted nothing, so the resident set's token cost was certain
and its usefulness could not be tested even in principle.

Bulk deliveries now record as AMBIENT, beside the ranked count and never
inside pull-through. Folding them in would mean growing the always-on set
depressed the arm's measured precision and trimming it flattered the arm,
neither for any reason to do with the arm.

`RANKED_SOURCES` inverts the note twin's `AMBIENT_SOURCES` deliberately: there
is one ranked rule source and this change adds seven bulk ones, so naming the
rare half makes a forgotten surface default to ambient — under-counting it —
rather than padding the denominator with surfacings nobody chose.

Two lookalike call sites are deliberately left silent, with a test to keep
them that way: the write-path etag arm and `rules_etag_for` read the rules to
build or compare a MARKER and show nobody anything.

No migration — `event` and `source` are plain Text with no CHECK (rule 36
does not apply). Snippet #2858 updated to the new `rules_payload` contract.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011cPyzNnegXHr5iRMzzy5KJ
This commit is contained in:
2026-09-02 23:06:40 -04:00
co-authored by Claude Opus 5
parent 6627cfc2f0
commit 8b9b3a1d9b
14 changed files with 505 additions and 48 deletions
+29 -1
View File
@@ -23,6 +23,7 @@ from scribe.services.verification import (
)
from scribe.services import rule_versions
from scribe.models.rule_version import RuleVersion
from scribe.services.rule_usage import record_rule_surfaced
logger = logging.getLogger(__name__)
@@ -1395,7 +1396,7 @@ async def get_applicable_rules(
}
def rules_payload(applicable: dict) -> dict:
def rules_payload(applicable: dict, *, user_id: int | None, source: str) -> dict:
"""The caller-facing shape of a get_applicable_rules() result.
Every surface that hands rules to an agent (enter_project, get_project,
@@ -1406,7 +1407,34 @@ def rules_payload(applicable: dict) -> dict:
`excluded_always_on` (milestone 297) names the always-on rulebooks this
project decided NOT to inherit, so the departure is visible wherever the
rules are.
IT ALSO RECORDS THE SURFACING, which is why it now takes a caller and a
source. Every one of those surfaces is a bulk delivery — the applicable set
handed over whole, chosen by nobody — so this is the one place that has to
emit for all of them. Doing it per-caller instead would be five sites to
remember, and #3430 gap 2 is what that costs: the process→skill sync went
un-emitted through an entire dedicated telemetry survey because nothing
forced its surface to be accounted for.
`source` stays the CALLER's name rather than a constant, so the readout can
still separate the session handshake from a mid-session milestone read;
`RANKED_SOURCES` in `rule_usage` is what folds them back together.
Emitting from here is safe in a way emitting from `get_applicable_rules`
would not be: this function is only ever called to BUILD A REPLY. The two
other callers of the rules machinery — the write-path etag arm
(`plugin_context`) and `rules_etag_for` — compute a marker and show nobody
anything, and counting those would put rules in the denominator that no
agent ever saw.
"""
record_rule_surfaced(
user_id=user_id,
rule_ids=(
[r["id"] for r in applicable.get("rules", [])]
+ [r["id"] for r in applicable.get("project_rules", [])]
),
source=source,
)
return {
"applicable_rules": applicable["rules"],
"applicable_rules_truncated": applicable["truncated"],