feat(telemetry): the preload emits, and the always-on set stops being unfalsifiable (#3473)
CI & Build / Python lint (push) Successful in 3s
CI & Build / Plugin hooks (push) Successful in 11s
CI & Build / TypeScript typecheck (push) Successful in 23s
CI & Build / integration (push) Successful in 31s
CI & Build / Python tests (push) Successful in 1m6s
CI & Build / Build & push image (push) Successful in 27s
CI & Build / Python lint (push) Successful in 3s
CI & Build / Plugin hooks (push) Successful in 11s
CI & Build / TypeScript typecheck (push) Successful in 23s
CI & Build / integration (push) Successful in 31s
CI & Build / Python tests (push) Successful in 1m6s
CI & Build / Build & push image (push) Successful in 27s
The ranked rule arm became measurable in M333. The preload did not — and that is the surface whose value is actually in question. `list_always_on_rules`, the SessionStart block and every `rules_payload` caller handed rules over wholesale and emitted nothing, so the resident set's token cost was certain and its usefulness could not be tested even in principle. Bulk deliveries now record as AMBIENT, beside the ranked count and never inside pull-through. Folding them in would mean growing the always-on set depressed the arm's measured precision and trimming it flattered the arm, neither for any reason to do with the arm. `RANKED_SOURCES` inverts the note twin's `AMBIENT_SOURCES` deliberately: there is one ranked rule source and this change adds seven bulk ones, so naming the rare half makes a forgotten surface default to ambient — under-counting it — rather than padding the denominator with surfacings nobody chose. Two lookalike call sites are deliberately left silent, with a test to keep them that way: the write-path etag arm and `rules_etag_for` read the rules to build or compare a MARKER and show nobody anything. No migration — `event` and `source` are plain Text with no CHECK (rule 36 does not apply). Snippet #2858 updated to the new `rules_payload` contract. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011cPyzNnegXHr5iRMzzy5KJ
This commit is contained in:
@@ -23,6 +23,7 @@ from scribe.services.verification import (
|
||||
)
|
||||
from scribe.services import rule_versions
|
||||
from scribe.models.rule_version import RuleVersion
|
||||
from scribe.services.rule_usage import record_rule_surfaced
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
@@ -1395,7 +1396,7 @@ async def get_applicable_rules(
|
||||
}
|
||||
|
||||
|
||||
def rules_payload(applicable: dict) -> dict:
|
||||
def rules_payload(applicable: dict, *, user_id: int | None, source: str) -> dict:
|
||||
"""The caller-facing shape of a get_applicable_rules() result.
|
||||
|
||||
Every surface that hands rules to an agent (enter_project, get_project,
|
||||
@@ -1406,7 +1407,34 @@ def rules_payload(applicable: dict) -> dict:
|
||||
`excluded_always_on` (milestone 297) names the always-on rulebooks this
|
||||
project decided NOT to inherit, so the departure is visible wherever the
|
||||
rules are.
|
||||
|
||||
IT ALSO RECORDS THE SURFACING, which is why it now takes a caller and a
|
||||
source. Every one of those surfaces is a bulk delivery — the applicable set
|
||||
handed over whole, chosen by nobody — so this is the one place that has to
|
||||
emit for all of them. Doing it per-caller instead would be five sites to
|
||||
remember, and #3430 gap 2 is what that costs: the process→skill sync went
|
||||
un-emitted through an entire dedicated telemetry survey because nothing
|
||||
forced its surface to be accounted for.
|
||||
|
||||
`source` stays the CALLER's name rather than a constant, so the readout can
|
||||
still separate the session handshake from a mid-session milestone read;
|
||||
`RANKED_SOURCES` in `rule_usage` is what folds them back together.
|
||||
|
||||
Emitting from here is safe in a way emitting from `get_applicable_rules`
|
||||
would not be: this function is only ever called to BUILD A REPLY. The two
|
||||
other callers of the rules machinery — the write-path etag arm
|
||||
(`plugin_context`) and `rules_etag_for` — compute a marker and show nobody
|
||||
anything, and counting those would put rules in the denominator that no
|
||||
agent ever saw.
|
||||
"""
|
||||
record_rule_surfaced(
|
||||
user_id=user_id,
|
||||
rule_ids=(
|
||||
[r["id"] for r in applicable.get("rules", [])]
|
||||
+ [r["id"] for r in applicable.get("project_rules", [])]
|
||||
),
|
||||
source=source,
|
||||
)
|
||||
return {
|
||||
"applicable_rules": applicable["rules"],
|
||||
"applicable_rules_truncated": applicable["truncated"],
|
||||
|
||||
Reference in New Issue
Block a user