Compare commits

..
9 Commits
Author SHA1 Message Date
bvandeusen 14ff41faf5 Merge pull request 'feat(rules): rule creation becomes propose-then-approve (#3557)' (#141) from dev into main
CI & Build / Python lint (push) Successful in 3s
CI & Build / Plugin hooks (push) Successful in 8s
CI & Build / TypeScript typecheck (push) Successful in 22s
CI & Build / integration (push) Successful in 30s
CI & Build / Python tests (push) Successful in 1m2s
CI & Build / Build & push image (push) Successful in 19s
2026-09-04 22:01:47 -04:00
bvandeusen 67df41ae00 Merge pull request 'feat(rules): the instruction surfaces say to RETRIEVE a rule, not only to receive one (#3523)' (#140) from dev into main
CI & Build / Python lint (push) Successful in 3s
CI & Build / Plugin hooks (push) Successful in 10s
CI & Build / integration (push) Successful in 32s
CI & Build / Python tests (push) Successful in 1m5s
CI & Build / TypeScript typecheck (push) Successful in 3m18s
CI & Build / Build & push image (push) Successful in 16s
2026-09-03 22:07:10 -04:00
bvandeusen 0915c48bb0 Merge pull request 'feat(telemetry): tell a ranker decline from a repeat before the observation window opens (#3497)' (#139) from dev into main
CI & Build / Python lint (push) Successful in 3s
CI & Build / Plugin hooks (push) Successful in 9s
CI & Build / integration (push) Successful in 30s
CI & Build / Python tests (push) Successful in 1m6s
CI & Build / TypeScript typecheck (push) Successful in 5m20s
CI & Build / Build & push image (push) Successful in 17s
2026-09-03 21:23:01 -04:00
bvandeusen aea7b63b62 Merge pull request 'fix(telemetry): both rule arms logged only their hits, so the clear-rate could only read 100% (#3497)' (#138) from dev into main
CI & Build / Python lint (push) Successful in 4s
CI & Build / Plugin hooks (push) Successful in 9s
CI & Build / integration (push) Successful in 30s
CI & Build / TypeScript typecheck (push) Successful in 33s
CI & Build / Python tests (push) Successful in 1m5s
CI & Build / Build & push image (push) Successful in 17s
2026-09-03 07:19:08 -04:00
bvandeusen 5b02908dfd dev → main: rules become measurable at the preload, and retrievable at the tool call (#137)
CI & Build / Python lint (push) Successful in 4s
CI & Build / Plugin hooks (push) Successful in 10s
CI & Build / integration (push) Successful in 29s
CI & Build / TypeScript typecheck (push) Successful in 33s
CI & Build / Python tests (push) Successful in 1m6s
CI & Build / Build & push image (push) Successful in 17s
2026-09-02 23:41:09 -04:00
bvandeusen 34cd389371 dev → main: rule usage telemetry, the plugin's derived version (#136)
CI & Build / Python lint (push) Successful in 3s
CI & Build / Plugin hooks (push) Successful in 10s
CI & Build / TypeScript typecheck (push) Successful in 23s
CI & Build / integration (push) Successful in 30s
CI & Build / Python tests (push) Successful in 1m5s
CI & Build / Build & push image (push) Successful in 16s
2026-09-02 18:52:20 -04:00
bvandeusen b267037911 Two milestones: a note can carry its own check (317), and a rule keeps what it used to say (323) (#135)
CI & Build / Python lint (push) Successful in 3s
CI & Build / Plugin hooks (push) Successful in 11s
CI & Build / TypeScript typecheck (push) Successful in 38s
CI & Build / integration (push) Successful in 55s
CI & Build / Python tests (push) Successful in 1m37s
CI & Build / Build & push image (push) Successful in 29s
2026-08-31 00:01:15 -04:00
bvandeusen 93d660b710 Tool disambiguators, kind badges, and a badge layer that clears AA (#3123, #3124, #3132)
CI & Build / Python lint (push) Successful in 4s
CI & Build / Plugin hooks (push) Successful in 10s
CI & Build / integration (push) Successful in 31s
CI & Build / TypeScript typecheck (push) Successful in 33s
CI & Build / Python tests (push) Successful in 1m17s
CI & Build / Build & push image (push) Successful in 15s
2026-08-27 21:28:48 -04:00
bvandeusen 056c7c75da A task's kind is correctable — the Kind select stops lying (#3129)
CI & Build / Python lint (push) Successful in 3s
CI & Build / Plugin hooks (push) Successful in 7s
CI & Build / integration (push) Successful in 28s
CI & Build / TypeScript typecheck (push) Successful in 33s
CI & Build / Python tests (push) Successful in 1m6s
CI & Build / Build & push image (push) Successful in 15s
2026-08-27 18:05:16 -04:00
6 changed files with 22 additions and 419 deletions
+4 -31
View File
@@ -237,37 +237,10 @@ It is an UPPER BOUND per surface: a pull records the door it came
those same rules over time: a resident set surfaced thousands of times and
opened never is the dead-weight signal, one tier up.
Read it against `sources["write_path_rule"]`. That arm was once believed
never to decline — the reading that scoped #3311 — but it was the arm's
`retrieval_logs` row being written only on calls that FOUND something, so
the zeros were missing rather than absent (#3497). Measured since, it
declines the large majority of its calls like any other surface.
EVERY COUNTER BLOCK CARRIES ITS OWN COVERAGE — `complete_from` and
`covers_window`. `complete_from` is when the number became trustworthy:
for one source, its first recorded row; for a section that sums several,
the LATEST of theirs, because a total is complete only once every
contributor was being written. `covers_window: false` means the window
reaches back further than the recording does, so the count is a fraction
of the period it appears to describe.
READ IT BEFORE COMPARING TWO NUMBERS, and especially before comparing
across a deploy. A counter added last week, read over a 30-day window,
reports a real count against an imagined denominator — and the result is
a plausible fraction rather than an obvious zero, which is what makes it
dangerous. That reading cost milestone #379 five steps aimed at a defect
that did not exist.
`covers_window` is null, never false, when nothing was ever recorded:
"no measurement" is not "partial measurement", the same distinction
`suppression`'s null carries a few paragraphs up.
A SOURCE SHOWING `calls: 0` WAS RECORDING AND MADE NO CALLS. `sources`
lists every source the table has ever held, not only those active in the
window, so a surface that stopped firing stays visible rather than
disappearing — being absent is reserved for a source that has never
recorded at all. Its score fields are null, not zero: the calls are a
real observation, the distribution is not one.
Read it against `sources["write_path_rule"]`. That surface has never once
declined to fire, and until this block existed there was no way to tell a
well-tuned arm from a bar it cannot fail to clear (#3311). `pull_through`
is the number that tells them apart.
`rule_usage_failed: true` means that read failed while the rest of the
readout stood. The counts are still present so a caller can render, but
+4 -11
View File
@@ -774,17 +774,10 @@ async def semantic_search_rules(
is the surfacing question, and it has its own machinery
(get_applicable_rules) rather than a second, subtly different copy here.
`tier` narrows to one tier, and NONE is the ordinary case. The write-path
and pre-tool hints deliberately pass nothing: an always-on rule is already
in the session, but being in a list from turn zero is not the same as being
in front of the reader when the action it governs is taken, and filtering
on tier made a whole class of rules permanently ineligible for the one
mechanism that surfaces a rule AT the moment. Relevance is the threshold's
job; see the block above RULEHINT_LIMIT in services/plugin_context.py for
the argument and for what the resulting scores are being read against.
Pass a tier when a caller genuinely wants one class — a listing, an audit,
a UI that renders the tiers apart. Not to approximate relevance.
`tier` narrows to one tier. The write-path hint passes "conditional",
because an always-on rule is ALREADY in the session — surfacing it again as
a suggestion is pure noise, and noise on a hint that fires on every write
is how a hint gets ignored.
Collapses to best-chunk-per-rule like the note search, so a long rule split
across chunks competes once rather than crowding the results with itself.
+12 -60
View File
@@ -94,17 +94,13 @@ WRITEPATH_DEFAULT_THRESHOLD = 0.68
# THE STRUCTURAL ARGUMENT, which is the only kind admissible here (rule 115).
# Two facts hold on any install, including one with six rules and no telemetry:
#
# 1. The eligible corpus is SMALL — every rule an install owns, still only
# a few dozen documents against thousands of notes. A top-k over a small
# pool always returns something, so "the best match cleared the bar"
# drifts from "a good match exists" toward "N things were ranked". A bar
# calibrated for best-of-thousands is cleared by best-of-forty as
# arithmetic rather than relevance.
# This argument WEAKENED when the arms stopped filtering to one tier
# (see the note on that below): a larger pool makes clearing the bar
# mean more, not less. The threshold was deliberately left where it was
# anyway — moving two variables at once would make the resulting
# distribution unreadable, and this one errs toward silence on purpose.
# 1. The eligible corpus is TINY. The arm searches `tier="conditional"`
# rules only — a handful to a few dozen documents against thousands of
# notes. A top-k over forty candidates always returns something, so
# "the best match cleared the bar" stops meaning "a good match exists"
# and starts meaning "forty things were ranked". A bar calibrated for
# best-of-thousands is cleared by best-of-forty as arithmetic, not
# relevance.
# 2. Rules are short imperative technical English — a far more HOMOGENEOUS
# corpus than note prose. #2223 measured the floor for code against prose
# at 0.55-0.63 and set 0.68 above it. A more homogeneous corpus has a
@@ -132,14 +128,9 @@ RULEHINT_DEFAULT_THRESHOLD = 0.72
# ONE rule per write, not two — and this is deliberately NOT a knob.
#
# With a corpus this small, top-k does as much damage as the threshold: k=2
# over a few dozen candidates means the second line is almost always the
# second-best noise, arriving with the same confident framing as the first.
# Halving k halves that regardless of where the bar sits.
#
# It also BOUNDS the blast radius of widening the pool (below): with k=1 a
# wider corpus can change WHICH rule surfaces and how often one does, but it
# can never make a single hint longer. The loudness of one hint and the
# eligibility of a rule are separate controls, and only one of them moved.
# over forty candidates means the second line is almost always the second-best
# noise, arriving with the same confident framing as the first. Halving k
# halves that regardless of where the bar sits.
#
# It stays a constant because it is a decision about how LOUD one hint may be,
# not a per-install tuning question. The hint already carries prior art, shape
@@ -149,45 +140,6 @@ RULEHINT_DEFAULT_THRESHOLD = 0.72
# adds a way to misconfigure the surface (rule 25 cuts both ways).
RULEHINT_LIMIT = 1
# WHY THE ARMS NO LONGER FILTER TO ONE TIER (#3702).
#
# Both arms used to pass `tier="conditional"`, on the reasoning that an
# always-on rule is already in the session, so surfacing it again is pure
# noise. That reasoning conflates two different things:
#
# PRESENT IN CONTEXT — the rule was delivered at session start.
# SALIENT AT THE MOMENT — the rule is in front of the reader when the
# action it governs is about to be taken.
#
# A rule handed over in a list at turn zero is present while a session writes
# a config value three hundred turns later. It is not surfaced. So the filter
# did not merely skip a redundant hint — it made a whole class of rules
# permanently ineligible for the only mechanism that puts a rule in front of
# an agent AT the moment, and the more important a rule is, the more likely
# it was in that class.
#
# The deeper defect is that the filter was doing the THRESHOLD's job. Whether
# a rule belongs in this hint is a relevance question, and a similarity bar is
# the control for relevance. A categorical exclusion standing in for a
# relevance judgment cannot be tuned, cannot be measured, and cannot be wrong
# in a way anybody notices.
#
# THIS IS A MEASURED CHANGE, NOT A SETTLED ONE. The old comment's fear is
# real — a hint that fires on every write and says obvious things teaches the
# reader to skip the block, and the surface is then lost along with its true
# positives. That fear had simply never been checked. `retrieval_logs` already
# records top_score, result_count and the query for every call, so the
# evidence now arrives on its own:
#
# - rules clear the bar often and at high scores -> the fear was justified,
# the filter was a crude proxy for a bar set too low, and the WORK IS THE
# BAR. Any reinstated filter should then carry a measured reason.
# - rules clear rarely, in a thin band near the bar -> the filter was never
# the right instrument and relevance was always sufficient.
#
# Only the eligibility moved. The bar and k=1 were both left exactly where
# they were, so the resulting distribution has one cause.
# How much of a command reaches the embedding (#3476). A shell call is not a
# file: most are short, and the ones that are not are usually a heredoc or a
# pasted script whose bulk says nothing about which rule applies. The VERB AND
@@ -1253,7 +1205,7 @@ async def build_write_path_hint(
rule_t0 = time.perf_counter()
hits = await semantic_search_rules(
user_id, code or path, limit=RULEHINT_LIMIT,
threshold=cfg["rule_threshold"],
threshold=cfg["rule_threshold"], tier="conditional",
)
rule_ms = (time.perf_counter() - rule_t0) * 1000.0
fresh = [(score, rule) for score, rule in hits if rule.id not in already]
@@ -1385,7 +1337,7 @@ async def build_tool_rule_hint(
t0 = time.perf_counter()
hits = await semantic_search_rules(
user_id, query, limit=RULEHINT_LIMIT,
threshold=cfg["rule_threshold"],
threshold=cfg["rule_threshold"], tier="conditional",
)
duration_ms = (time.perf_counter() - t0) * 1000.0
+1 -99
View File
@@ -213,70 +213,10 @@ def _bucket(rows: list) -> dict:
}
# The aggregate row Postgres would have returned for a source with no rows in
# the window: nothing counted, nothing scored. Positional, matching the SELECT
# `_bucket` unpacks — calls, zero, cleared, p10, p50, p90, min, max, avg_n,
# dur, measured, supp_calls, supp_zero. The three counts are 0 because zero
# calls is a real observation; everything else is None because a distribution
# nobody sampled has no value, and rendering it as 0.0 would state one.
_NO_ROWS_IN_WINDOW = [0, 0, 0, None, None, None, None, None, None, None, 0, 0, 0]
def _round(v, places: int = 4):
return None if v is None else round(float(v), places)
async def _complete_from(session, model, user_id) -> dict[str, Any]:
"""When each source in `model` started being recorded, and the instant the
WHOLE table is complete from. Returns {source: earliest_row, "*": latest}.
THE GRAIN IS THE SOURCE, and that is the whole point. `retrieval_logs` has
rows going back months, so a table-level "earliest row" says months and
tells a reader their window is fully covered — while a source added last
week has a week of rows and a counter that silently means something else.
Per-source is the only grain at which partial coverage is visible.
THE AGGREGATE USES THE LATEST, NOT THE EARLIEST. A number that sums several
sources is complete only once EVERY contributor was recording, so "*" is a
max over the sources, not a min. Taking the min here would reproduce the
exact reading this exists to prevent: the oldest source vouching for the
youngest.
All-time, deliberately unfiltered by the window — a query bounded by
`since` can only ever report something at or after `since`, which answers
nothing.
"""
rows = (
await session.execute(
select(model.source, func.min(model.created_at))
.where(model.user_id == user_id)
.group_by(model.source)
)
).all()
out: dict[str, Any] = {src: ts for src, ts in rows if ts is not None}
stamps = list(out.values())
out["*"] = max(stamps) if stamps else None
return out
def _coverage(complete_from, since) -> dict:
"""The two keys every counter block carries, from one timestamp.
`covers_window` is None — never False — when nothing was ever recorded.
"No rows at all" is not "partial coverage", it is no measurement, and the
null convention #3497 established for `suppression` holds here for the
same reason: absent must not read as a verdict.
"""
return {
# iso() already returns None for an unset value (#2845) — the guard
# belongs on covers_window, which is a verdict, not a serialisation.
"complete_from": iso(complete_from),
"covers_window": (
None if complete_from is None else complete_from <= since
),
}
async def retrieval_summary(user_id: int | None, *, days: int = 30) -> dict:
"""What the retrieval telemetry says, per surface, over a window.
@@ -343,9 +283,6 @@ async def retrieval_summary(user_id: int | None, *, days: int = 30) -> dict:
by_source_rows = None
rule_rows = None
distinct_rules_surfaced = distinct_rules_pulled = 0
# None means the coverage read did not happen — distinct from a table with
# no rows, which is {"*": None}. Same reason `read_failed` exists.
note_complete = rule_complete = None
try:
async with async_session() as session:
@@ -374,34 +311,8 @@ async def retrieval_summary(user_id: int | None, *, days: int = 30) -> dict:
.group_by(RetrievalLog.source)
)
).all()
log_complete = await _complete_from(session, RetrievalLog, user_id)
for row in rows:
source = row[0]
bucket = _bucket(list(row[1:]))
# Per SOURCE, not per table: retrieval_logs goes back months
# while any individual arm may be days old, and the table's
# age would vouch for an arm that has barely started.
bucket.update(_coverage(log_complete.get(source), since))
out["sources"][source] = bucket
# A source with rows in the table but NONE in this window would
# otherwise be absent from the readout — and absent is exactly how
# a source that never existed renders, so a surface that WAS
# recording and went silent is unreadable (#3720). That is #2663
# one level up: the failure that looks like the correct answer.
#
# Zero here is a real measurement, not a manufactured one. The
# all-time query proves the source was recording, and it made no
# calls across a window it fully covers — which is why no
# `covers_window` special case is needed: a source whose first row
# fell after `since` would have that row IN the window and already
# hold a bucket, so anything reaching here began before it.
for src, first_row in log_complete.items():
if src == "*" or first_row is None or src in out["sources"]:
continue
quiet = _bucket(list(_NO_ROWS_IN_WINDOW))
quiet.update(_coverage(first_row, since))
out["sources"][src] = quiet
out["sources"][row[0]] = _bucket(list(row[1:]))
# The corpus side, at its own grain. `ambient` mirrors
# note_usage.usage_for_notes: an ambient surfacing was not a scored
@@ -428,7 +339,6 @@ async def retrieval_summary(user_id: int | None, *, days: int = 30) -> dict:
.group_by(NoteUsageEvent.event, NoteUsageEvent.source)
)
).all()
note_complete = await _complete_from(session, NoteUsageEvent, user_id)
# Distinct-note counts need their OWN queries, and this is not
# fussiness: count(distinct note_id) per (event, source) group
@@ -556,9 +466,6 @@ async def retrieval_summary(user_id: int | None, *, days: int = 30) -> dict:
.group_by(RuleUsageEvent.event, RuleUsageEvent.source)
)
).all()
rule_complete = await _complete_from(
session, RuleUsageEvent, user_id,
)
# The rows carry `source`, so the ranked/ambient split is done
# below rather than in SQL — the bulk surfaces started emitting
# on 2026-09-03 (#3473), so there IS an ambient class now.
@@ -668,10 +575,6 @@ async def retrieval_summary(user_id: int | None, *, days: int = 30) -> dict:
}
usage["by_source"] = by_source
# The SECTION's coverage, from the latest source to start recording — a
# figure that sums several sources is complete only once every one of them
# was being written. `_complete_from` computes that as "*".
usage.update(_coverage((note_complete or {}).get("*"), since))
out["usage"] = usage
# ── Rules, deliberately a SEPARATE block ────────────────────────────
@@ -749,7 +652,6 @@ async def retrieval_summary(user_id: int | None, *, days: int = 30) -> dict:
round(rule_usage["pulled_by_agent"] / rule_usage["surfaced"], 4)
if rule_usage["surfaced"] else None
)
rule_usage.update(_coverage((rule_complete or {}).get("*"), since))
out["rule_usage"] = rule_usage
return out
+1 -9
View File
@@ -160,15 +160,7 @@ async def test_the_arm_searches_on_its_OWN_bar_not_the_code_one():
kw = search.await_args.kwargs
assert kw["threshold"] == 0.81, "the arm is still using the code threshold"
assert kw["limit"] == pc.RULEHINT_LIMIT
# NO tier filter (#3702). The arms search every rule the caller owns,
# because "already in the session" is not the same as "in front of the
# reader at the moment it applies" — and relevance is the threshold's
# job, not a category's. If this assertion is failing because a tier
# argument came back, read the block above RULEHINT_LIMIT first: the
# filter may legitimately return, but only carrying a measured reason.
assert "tier" not in kw or kw["tier"] is None, (
"the arm is filtering the rule corpus by tier again"
)
assert kw["tier"] == "conditional"
@pytest.mark.asyncio
-209
View File
@@ -638,212 +638,3 @@ async def test_ambient_alone_reports_no_ratio(_dispose_engine):
assert ru["pull_through"] is None
finally:
await cleanup()
# ── Window coverage (#3712) ────────────────────────────────────────────
#
# A counter added last week, read over a 30-day window, reports a real count
# against an imagined denominator. The result is a plausible FRACTION rather
# than an obvious zero, which is what makes it dangerous — #379 spent five
# planned steps on a defect that turned out to be a window opening before the
# recording it was measuring existed.
def test_coverage_says_nothing_rather_than_false_when_nothing_was_recorded():
"""Null, never False. "No measurement" is not "partial measurement".
The same distinction `suppression`'s null carries (#3497): absent must not
read as a verdict. A False here would assert the window is under-covered,
which is a claim nobody is in a position to make.
"""
from datetime import datetime, timezone
from scribe.services.retrieval_telemetry import _coverage
since = datetime(2026, 9, 1, tzinfo=timezone.utc)
assert _coverage(None, since) == {
"complete_from": None, "covers_window": None,
}
def test_coverage_reads_a_start_before_the_window_as_covered():
from datetime import datetime, timezone
from scribe.services.retrieval_telemetry import _coverage
since = datetime(2026, 9, 1, tzinfo=timezone.utc)
older = datetime(2026, 8, 1, tzinfo=timezone.utc)
newer = datetime(2026, 9, 5, tzinfo=timezone.utc)
assert _coverage(older, since)["covers_window"] is True
assert _coverage(newer, since)["covers_window"] is False, (
"a counter that started inside the window covers only part of it"
)
assert _coverage(newer, since)["complete_from"] == newer.isoformat()
@pytest.mark.integration
@pytest.mark.asyncio
async def test_coverage_is_per_source_because_the_table_is_older_than_its_arms(
_dispose_engine,
):
"""THE grain question, and the reason a per-table answer is useless.
`retrieval_logs` accumulates for months. A table-level "earliest row"
therefore says months for every source it holds — including one added
days ago whose counter means something quite different. The old source
would vouch for the young one, which is exactly the reading this exists
to prevent.
"""
from datetime import datetime, timedelta, timezone
from sqlalchemy import delete
from scribe.models import async_session
from scribe.models.retrieval_log import RetrievalLog
from scribe.services.retrieval_telemetry import retrieval_summary
UID = 990077
now = datetime.now(timezone.utc)
async with async_session() as s:
# An old surface, recording since well before any window we ask for,
# AND still recording inside it. Both rows are needed: `complete_from`
# comes from the all-time query, but a source only gets a bucket at all
# if it has rows in the window, so the 90-day row alone would leave
# nothing to assert on.
s.add(RetrievalLog(
user_id=UID, source="auto_inject", result_count=1,
created_at=now - timedelta(days=90),
))
s.add(RetrievalLog(
user_id=UID, source="auto_inject", result_count=1,
created_at=now - timedelta(days=1),
))
# A young arm, first written INSIDE the window below.
s.add(RetrievalLog(
user_id=UID, source="pre_tool_rule", result_count=1,
created_at=now - timedelta(days=2),
))
await s.commit()
try:
out = await retrieval_summary(UID, days=30)
assert out["sources"]["auto_inject"]["covers_window"] is True
assert out["sources"]["pre_tool_rule"]["covers_window"] is False, (
"the young arm was reported as covering a 30-day window — the "
"table's age has been allowed to vouch for one of its sources"
)
finally:
async with async_session() as s:
await s.execute(delete(RetrievalLog).where(RetrievalLog.user_id == UID))
await s.commit()
@pytest.mark.integration
@pytest.mark.asyncio
async def test_a_surface_that_went_silent_is_not_the_same_as_one_that_never_ran(
_dispose_engine,
):
"""#3720 — absent is how "never existed" renders, so it cannot also be how
"stopped recording" renders.
A surface losing its recorder is one of the failures this milestone exists
to make visible, and dropping it from the readout is the most complete way
to hide it. Zero here is a real measurement: the table proves the source
was recording, and it made no calls across a window it fully covers.
"""
from datetime import datetime, timedelta, timezone
from sqlalchemy import delete
from scribe.models import async_session
from scribe.models.retrieval_log import RetrievalLog
from scribe.services.retrieval_telemetry import retrieval_summary
UID = 990078
now = datetime.now(timezone.utc)
async with async_session() as s:
# Recorded once, well before the window, and never since.
s.add(RetrievalLog(
user_id=UID, source="auto_inject", result_count=3, top_score=0.81,
created_at=now - timedelta(days=60),
))
await s.commit()
try:
out = await retrieval_summary(UID, days=7)
assert "auto_inject" in out["sources"], (
"a source with rows in the table but none in the window was "
"dropped from the readout — a surface that stopped recording now "
"reads exactly like one that never existed"
)
quiet = out["sources"]["auto_inject"]
assert quiet["calls"] == 0
# The window IS covered; what was observed across it is nothing.
assert quiet["covers_window"] is True
# ...but nothing was sampled, so no distribution may be claimed. A
# zeroed score would assert a measurement, which is #3311's mistake.
assert quiet["top_score"] == {
"p10": None, "p50": None, "p90": None, "min": None, "max": None,
}
assert quiet["suppression"] is None
assert quiet["avg_result_count"] is None
finally:
async with async_session() as s:
await s.execute(delete(RetrievalLog).where(RetrievalLog.user_id == UID))
await s.commit()
@pytest.mark.integration
@pytest.mark.asyncio
async def test_a_section_is_complete_only_from_its_latest_contributor(
_dispose_engine,
):
"""A sum is complete once EVERY contributor was being written — so the
section takes the LATEST first-row, not the earliest.
Taking the earliest would be worse than reporting nothing: it would pick
the oldest source in the table and use it to certify a total that a
newer source is still only partly contributing to. That is the original
error in miniature.
"""
from datetime import datetime, timedelta, timezone
from sqlalchemy import delete
from scribe.models import async_session
from scribe.models.rule_usage import RuleUsageEvent
from scribe.services.retrieval_telemetry import retrieval_summary
UID = 990078
now = datetime.now(timezone.utc)
old = now - timedelta(days=90)
young = now - timedelta(days=2)
async with async_session() as s:
s.add_all([
RuleUsageEvent(
user_id=UID, rule_id=1, event="surfaced",
source="list_always_on_rules", created_at=old,
),
RuleUsageEvent(
user_id=UID, rule_id=2, event="surfaced",
source="pre_tool_rule", created_at=young,
),
])
await s.commit()
try:
out = await retrieval_summary(UID, days=30)
ru = out["rule_usage"]
assert ru["complete_from"] == young.isoformat(), (
"the section reported completeness from its OLDEST source; a "
"total is only as complete as its newest contributor"
)
assert ru["covers_window"] is False
finally:
async with async_session() as s:
await s.execute(delete(RuleUsageEvent).where(RuleUsageEvent.user_id == UID))
await s.commit()