Compare commits
9
Commits
dev
..
14ff41faf5
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
14ff41faf5 | ||
|
|
67df41ae00 | ||
|
|
0915c48bb0 | ||
|
|
aea7b63b62 | ||
|
|
5b02908dfd | ||
|
|
34cd389371 | ||
|
|
b267037911 | ||
|
|
93d660b710 | ||
|
|
056c7c75da |
@@ -237,37 +237,10 @@ It is an UPPER BOUND per surface: a pull records the door it came
|
||||
those same rules over time: a resident set surfaced thousands of times and
|
||||
opened never is the dead-weight signal, one tier up.
|
||||
|
||||
Read it against `sources["write_path_rule"]`. That arm was once believed
|
||||
never to decline — the reading that scoped #3311 — but it was the arm's
|
||||
`retrieval_logs` row being written only on calls that FOUND something, so
|
||||
the zeros were missing rather than absent (#3497). Measured since, it
|
||||
declines the large majority of its calls like any other surface.
|
||||
|
||||
EVERY COUNTER BLOCK CARRIES ITS OWN COVERAGE — `complete_from` and
|
||||
`covers_window`. `complete_from` is when the number became trustworthy:
|
||||
for one source, its first recorded row; for a section that sums several,
|
||||
the LATEST of theirs, because a total is complete only once every
|
||||
contributor was being written. `covers_window: false` means the window
|
||||
reaches back further than the recording does, so the count is a fraction
|
||||
of the period it appears to describe.
|
||||
|
||||
READ IT BEFORE COMPARING TWO NUMBERS, and especially before comparing
|
||||
across a deploy. A counter added last week, read over a 30-day window,
|
||||
reports a real count against an imagined denominator — and the result is
|
||||
a plausible fraction rather than an obvious zero, which is what makes it
|
||||
dangerous. That reading cost milestone #379 five steps aimed at a defect
|
||||
that did not exist.
|
||||
|
||||
`covers_window` is null, never false, when nothing was ever recorded:
|
||||
"no measurement" is not "partial measurement", the same distinction
|
||||
`suppression`'s null carries a few paragraphs up.
|
||||
|
||||
A SOURCE SHOWING `calls: 0` WAS RECORDING AND MADE NO CALLS. `sources`
|
||||
lists every source the table has ever held, not only those active in the
|
||||
window, so a surface that stopped firing stays visible rather than
|
||||
disappearing — being absent is reserved for a source that has never
|
||||
recorded at all. Its score fields are null, not zero: the calls are a
|
||||
real observation, the distribution is not one.
|
||||
Read it against `sources["write_path_rule"]`. That surface has never once
|
||||
declined to fire, and until this block existed there was no way to tell a
|
||||
well-tuned arm from a bar it cannot fail to clear (#3311). `pull_through`
|
||||
is the number that tells them apart.
|
||||
|
||||
`rule_usage_failed: true` means that read failed while the rest of the
|
||||
readout stood. The counts are still present so a caller can render, but
|
||||
|
||||
@@ -774,17 +774,10 @@ async def semantic_search_rules(
|
||||
is the surfacing question, and it has its own machinery
|
||||
(get_applicable_rules) rather than a second, subtly different copy here.
|
||||
|
||||
`tier` narrows to one tier, and NONE is the ordinary case. The write-path
|
||||
and pre-tool hints deliberately pass nothing: an always-on rule is already
|
||||
in the session, but being in a list from turn zero is not the same as being
|
||||
in front of the reader when the action it governs is taken, and filtering
|
||||
on tier made a whole class of rules permanently ineligible for the one
|
||||
mechanism that surfaces a rule AT the moment. Relevance is the threshold's
|
||||
job; see the block above RULEHINT_LIMIT in services/plugin_context.py for
|
||||
the argument and for what the resulting scores are being read against.
|
||||
|
||||
Pass a tier when a caller genuinely wants one class — a listing, an audit,
|
||||
a UI that renders the tiers apart. Not to approximate relevance.
|
||||
`tier` narrows to one tier. The write-path hint passes "conditional",
|
||||
because an always-on rule is ALREADY in the session — surfacing it again as
|
||||
a suggestion is pure noise, and noise on a hint that fires on every write
|
||||
is how a hint gets ignored.
|
||||
|
||||
Collapses to best-chunk-per-rule like the note search, so a long rule split
|
||||
across chunks competes once rather than crowding the results with itself.
|
||||
|
||||
@@ -94,17 +94,13 @@ WRITEPATH_DEFAULT_THRESHOLD = 0.68
|
||||
# THE STRUCTURAL ARGUMENT, which is the only kind admissible here (rule 115).
|
||||
# Two facts hold on any install, including one with six rules and no telemetry:
|
||||
#
|
||||
# 1. The eligible corpus is SMALL — every rule an install owns, still only
|
||||
# a few dozen documents against thousands of notes. A top-k over a small
|
||||
# pool always returns something, so "the best match cleared the bar"
|
||||
# drifts from "a good match exists" toward "N things were ranked". A bar
|
||||
# calibrated for best-of-thousands is cleared by best-of-forty as
|
||||
# arithmetic rather than relevance.
|
||||
# This argument WEAKENED when the arms stopped filtering to one tier
|
||||
# (see the note on that below): a larger pool makes clearing the bar
|
||||
# mean more, not less. The threshold was deliberately left where it was
|
||||
# anyway — moving two variables at once would make the resulting
|
||||
# distribution unreadable, and this one errs toward silence on purpose.
|
||||
# 1. The eligible corpus is TINY. The arm searches `tier="conditional"`
|
||||
# rules only — a handful to a few dozen documents against thousands of
|
||||
# notes. A top-k over forty candidates always returns something, so
|
||||
# "the best match cleared the bar" stops meaning "a good match exists"
|
||||
# and starts meaning "forty things were ranked". A bar calibrated for
|
||||
# best-of-thousands is cleared by best-of-forty as arithmetic, not
|
||||
# relevance.
|
||||
# 2. Rules are short imperative technical English — a far more HOMOGENEOUS
|
||||
# corpus than note prose. #2223 measured the floor for code against prose
|
||||
# at 0.55-0.63 and set 0.68 above it. A more homogeneous corpus has a
|
||||
@@ -132,14 +128,9 @@ RULEHINT_DEFAULT_THRESHOLD = 0.72
|
||||
# ONE rule per write, not two — and this is deliberately NOT a knob.
|
||||
#
|
||||
# With a corpus this small, top-k does as much damage as the threshold: k=2
|
||||
# over a few dozen candidates means the second line is almost always the
|
||||
# second-best noise, arriving with the same confident framing as the first.
|
||||
# Halving k halves that regardless of where the bar sits.
|
||||
#
|
||||
# It also BOUNDS the blast radius of widening the pool (below): with k=1 a
|
||||
# wider corpus can change WHICH rule surfaces and how often one does, but it
|
||||
# can never make a single hint longer. The loudness of one hint and the
|
||||
# eligibility of a rule are separate controls, and only one of them moved.
|
||||
# over forty candidates means the second line is almost always the second-best
|
||||
# noise, arriving with the same confident framing as the first. Halving k
|
||||
# halves that regardless of where the bar sits.
|
||||
#
|
||||
# It stays a constant because it is a decision about how LOUD one hint may be,
|
||||
# not a per-install tuning question. The hint already carries prior art, shape
|
||||
@@ -149,45 +140,6 @@ RULEHINT_DEFAULT_THRESHOLD = 0.72
|
||||
# adds a way to misconfigure the surface (rule 25 cuts both ways).
|
||||
RULEHINT_LIMIT = 1
|
||||
|
||||
# WHY THE ARMS NO LONGER FILTER TO ONE TIER (#3702).
|
||||
#
|
||||
# Both arms used to pass `tier="conditional"`, on the reasoning that an
|
||||
# always-on rule is already in the session, so surfacing it again is pure
|
||||
# noise. That reasoning conflates two different things:
|
||||
#
|
||||
# PRESENT IN CONTEXT — the rule was delivered at session start.
|
||||
# SALIENT AT THE MOMENT — the rule is in front of the reader when the
|
||||
# action it governs is about to be taken.
|
||||
#
|
||||
# A rule handed over in a list at turn zero is present while a session writes
|
||||
# a config value three hundred turns later. It is not surfaced. So the filter
|
||||
# did not merely skip a redundant hint — it made a whole class of rules
|
||||
# permanently ineligible for the only mechanism that puts a rule in front of
|
||||
# an agent AT the moment, and the more important a rule is, the more likely
|
||||
# it was in that class.
|
||||
#
|
||||
# The deeper defect is that the filter was doing the THRESHOLD's job. Whether
|
||||
# a rule belongs in this hint is a relevance question, and a similarity bar is
|
||||
# the control for relevance. A categorical exclusion standing in for a
|
||||
# relevance judgment cannot be tuned, cannot be measured, and cannot be wrong
|
||||
# in a way anybody notices.
|
||||
#
|
||||
# THIS IS A MEASURED CHANGE, NOT A SETTLED ONE. The old comment's fear is
|
||||
# real — a hint that fires on every write and says obvious things teaches the
|
||||
# reader to skip the block, and the surface is then lost along with its true
|
||||
# positives. That fear had simply never been checked. `retrieval_logs` already
|
||||
# records top_score, result_count and the query for every call, so the
|
||||
# evidence now arrives on its own:
|
||||
#
|
||||
# - rules clear the bar often and at high scores -> the fear was justified,
|
||||
# the filter was a crude proxy for a bar set too low, and the WORK IS THE
|
||||
# BAR. Any reinstated filter should then carry a measured reason.
|
||||
# - rules clear rarely, in a thin band near the bar -> the filter was never
|
||||
# the right instrument and relevance was always sufficient.
|
||||
#
|
||||
# Only the eligibility moved. The bar and k=1 were both left exactly where
|
||||
# they were, so the resulting distribution has one cause.
|
||||
|
||||
# How much of a command reaches the embedding (#3476). A shell call is not a
|
||||
# file: most are short, and the ones that are not are usually a heredoc or a
|
||||
# pasted script whose bulk says nothing about which rule applies. The VERB AND
|
||||
@@ -1253,7 +1205,7 @@ async def build_write_path_hint(
|
||||
rule_t0 = time.perf_counter()
|
||||
hits = await semantic_search_rules(
|
||||
user_id, code or path, limit=RULEHINT_LIMIT,
|
||||
threshold=cfg["rule_threshold"],
|
||||
threshold=cfg["rule_threshold"], tier="conditional",
|
||||
)
|
||||
rule_ms = (time.perf_counter() - rule_t0) * 1000.0
|
||||
fresh = [(score, rule) for score, rule in hits if rule.id not in already]
|
||||
@@ -1385,7 +1337,7 @@ async def build_tool_rule_hint(
|
||||
t0 = time.perf_counter()
|
||||
hits = await semantic_search_rules(
|
||||
user_id, query, limit=RULEHINT_LIMIT,
|
||||
threshold=cfg["rule_threshold"],
|
||||
threshold=cfg["rule_threshold"], tier="conditional",
|
||||
)
|
||||
duration_ms = (time.perf_counter() - t0) * 1000.0
|
||||
|
||||
|
||||
@@ -213,70 +213,10 @@ def _bucket(rows: list) -> dict:
|
||||
}
|
||||
|
||||
|
||||
# The aggregate row Postgres would have returned for a source with no rows in
|
||||
# the window: nothing counted, nothing scored. Positional, matching the SELECT
|
||||
# `_bucket` unpacks — calls, zero, cleared, p10, p50, p90, min, max, avg_n,
|
||||
# dur, measured, supp_calls, supp_zero. The three counts are 0 because zero
|
||||
# calls is a real observation; everything else is None because a distribution
|
||||
# nobody sampled has no value, and rendering it as 0.0 would state one.
|
||||
_NO_ROWS_IN_WINDOW = [0, 0, 0, None, None, None, None, None, None, None, 0, 0, 0]
|
||||
|
||||
|
||||
def _round(v, places: int = 4):
|
||||
return None if v is None else round(float(v), places)
|
||||
|
||||
|
||||
async def _complete_from(session, model, user_id) -> dict[str, Any]:
|
||||
"""When each source in `model` started being recorded, and the instant the
|
||||
WHOLE table is complete from. Returns {source: earliest_row, "*": latest}.
|
||||
|
||||
THE GRAIN IS THE SOURCE, and that is the whole point. `retrieval_logs` has
|
||||
rows going back months, so a table-level "earliest row" says months and
|
||||
tells a reader their window is fully covered — while a source added last
|
||||
week has a week of rows and a counter that silently means something else.
|
||||
Per-source is the only grain at which partial coverage is visible.
|
||||
|
||||
THE AGGREGATE USES THE LATEST, NOT THE EARLIEST. A number that sums several
|
||||
sources is complete only once EVERY contributor was recording, so "*" is a
|
||||
max over the sources, not a min. Taking the min here would reproduce the
|
||||
exact reading this exists to prevent: the oldest source vouching for the
|
||||
youngest.
|
||||
|
||||
All-time, deliberately unfiltered by the window — a query bounded by
|
||||
`since` can only ever report something at or after `since`, which answers
|
||||
nothing.
|
||||
"""
|
||||
rows = (
|
||||
await session.execute(
|
||||
select(model.source, func.min(model.created_at))
|
||||
.where(model.user_id == user_id)
|
||||
.group_by(model.source)
|
||||
)
|
||||
).all()
|
||||
out: dict[str, Any] = {src: ts for src, ts in rows if ts is not None}
|
||||
stamps = list(out.values())
|
||||
out["*"] = max(stamps) if stamps else None
|
||||
return out
|
||||
|
||||
|
||||
def _coverage(complete_from, since) -> dict:
|
||||
"""The two keys every counter block carries, from one timestamp.
|
||||
|
||||
`covers_window` is None — never False — when nothing was ever recorded.
|
||||
"No rows at all" is not "partial coverage", it is no measurement, and the
|
||||
null convention #3497 established for `suppression` holds here for the
|
||||
same reason: absent must not read as a verdict.
|
||||
"""
|
||||
return {
|
||||
# iso() already returns None for an unset value (#2845) — the guard
|
||||
# belongs on covers_window, which is a verdict, not a serialisation.
|
||||
"complete_from": iso(complete_from),
|
||||
"covers_window": (
|
||||
None if complete_from is None else complete_from <= since
|
||||
),
|
||||
}
|
||||
|
||||
|
||||
async def retrieval_summary(user_id: int | None, *, days: int = 30) -> dict:
|
||||
"""What the retrieval telemetry says, per surface, over a window.
|
||||
|
||||
@@ -343,9 +283,6 @@ async def retrieval_summary(user_id: int | None, *, days: int = 30) -> dict:
|
||||
by_source_rows = None
|
||||
rule_rows = None
|
||||
distinct_rules_surfaced = distinct_rules_pulled = 0
|
||||
# None means the coverage read did not happen — distinct from a table with
|
||||
# no rows, which is {"*": None}. Same reason `read_failed` exists.
|
||||
note_complete = rule_complete = None
|
||||
|
||||
try:
|
||||
async with async_session() as session:
|
||||
@@ -374,34 +311,8 @@ async def retrieval_summary(user_id: int | None, *, days: int = 30) -> dict:
|
||||
.group_by(RetrievalLog.source)
|
||||
)
|
||||
).all()
|
||||
log_complete = await _complete_from(session, RetrievalLog, user_id)
|
||||
for row in rows:
|
||||
source = row[0]
|
||||
bucket = _bucket(list(row[1:]))
|
||||
# Per SOURCE, not per table: retrieval_logs goes back months
|
||||
# while any individual arm may be days old, and the table's
|
||||
# age would vouch for an arm that has barely started.
|
||||
bucket.update(_coverage(log_complete.get(source), since))
|
||||
out["sources"][source] = bucket
|
||||
|
||||
# A source with rows in the table but NONE in this window would
|
||||
# otherwise be absent from the readout — and absent is exactly how
|
||||
# a source that never existed renders, so a surface that WAS
|
||||
# recording and went silent is unreadable (#3720). That is #2663
|
||||
# one level up: the failure that looks like the correct answer.
|
||||
#
|
||||
# Zero here is a real measurement, not a manufactured one. The
|
||||
# all-time query proves the source was recording, and it made no
|
||||
# calls across a window it fully covers — which is why no
|
||||
# `covers_window` special case is needed: a source whose first row
|
||||
# fell after `since` would have that row IN the window and already
|
||||
# hold a bucket, so anything reaching here began before it.
|
||||
for src, first_row in log_complete.items():
|
||||
if src == "*" or first_row is None or src in out["sources"]:
|
||||
continue
|
||||
quiet = _bucket(list(_NO_ROWS_IN_WINDOW))
|
||||
quiet.update(_coverage(first_row, since))
|
||||
out["sources"][src] = quiet
|
||||
out["sources"][row[0]] = _bucket(list(row[1:]))
|
||||
|
||||
# The corpus side, at its own grain. `ambient` mirrors
|
||||
# note_usage.usage_for_notes: an ambient surfacing was not a scored
|
||||
@@ -428,7 +339,6 @@ async def retrieval_summary(user_id: int | None, *, days: int = 30) -> dict:
|
||||
.group_by(NoteUsageEvent.event, NoteUsageEvent.source)
|
||||
)
|
||||
).all()
|
||||
note_complete = await _complete_from(session, NoteUsageEvent, user_id)
|
||||
|
||||
# Distinct-note counts need their OWN queries, and this is not
|
||||
# fussiness: count(distinct note_id) per (event, source) group
|
||||
@@ -556,9 +466,6 @@ async def retrieval_summary(user_id: int | None, *, days: int = 30) -> dict:
|
||||
.group_by(RuleUsageEvent.event, RuleUsageEvent.source)
|
||||
)
|
||||
).all()
|
||||
rule_complete = await _complete_from(
|
||||
session, RuleUsageEvent, user_id,
|
||||
)
|
||||
# The rows carry `source`, so the ranked/ambient split is done
|
||||
# below rather than in SQL — the bulk surfaces started emitting
|
||||
# on 2026-09-03 (#3473), so there IS an ambient class now.
|
||||
@@ -668,10 +575,6 @@ async def retrieval_summary(user_id: int | None, *, days: int = 30) -> dict:
|
||||
}
|
||||
usage["by_source"] = by_source
|
||||
|
||||
# The SECTION's coverage, from the latest source to start recording — a
|
||||
# figure that sums several sources is complete only once every one of them
|
||||
# was being written. `_complete_from` computes that as "*".
|
||||
usage.update(_coverage((note_complete or {}).get("*"), since))
|
||||
out["usage"] = usage
|
||||
|
||||
# ── Rules, deliberately a SEPARATE block ────────────────────────────
|
||||
@@ -749,7 +652,6 @@ async def retrieval_summary(user_id: int | None, *, days: int = 30) -> dict:
|
||||
round(rule_usage["pulled_by_agent"] / rule_usage["surfaced"], 4)
|
||||
if rule_usage["surfaced"] else None
|
||||
)
|
||||
rule_usage.update(_coverage((rule_complete or {}).get("*"), since))
|
||||
out["rule_usage"] = rule_usage
|
||||
|
||||
return out
|
||||
|
||||
@@ -160,15 +160,7 @@ async def test_the_arm_searches_on_its_OWN_bar_not_the_code_one():
|
||||
kw = search.await_args.kwargs
|
||||
assert kw["threshold"] == 0.81, "the arm is still using the code threshold"
|
||||
assert kw["limit"] == pc.RULEHINT_LIMIT
|
||||
# NO tier filter (#3702). The arms search every rule the caller owns,
|
||||
# because "already in the session" is not the same as "in front of the
|
||||
# reader at the moment it applies" — and relevance is the threshold's
|
||||
# job, not a category's. If this assertion is failing because a tier
|
||||
# argument came back, read the block above RULEHINT_LIMIT first: the
|
||||
# filter may legitimately return, but only carrying a measured reason.
|
||||
assert "tier" not in kw or kw["tier"] is None, (
|
||||
"the arm is filtering the rule corpus by tier again"
|
||||
)
|
||||
assert kw["tier"] == "conditional"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
|
||||
@@ -638,212 +638,3 @@ async def test_ambient_alone_reports_no_ratio(_dispose_engine):
|
||||
assert ru["pull_through"] is None
|
||||
finally:
|
||||
await cleanup()
|
||||
|
||||
|
||||
# ── Window coverage (#3712) ────────────────────────────────────────────
|
||||
#
|
||||
# A counter added last week, read over a 30-day window, reports a real count
|
||||
# against an imagined denominator. The result is a plausible FRACTION rather
|
||||
# than an obvious zero, which is what makes it dangerous — #379 spent five
|
||||
# planned steps on a defect that turned out to be a window opening before the
|
||||
# recording it was measuring existed.
|
||||
|
||||
|
||||
def test_coverage_says_nothing_rather_than_false_when_nothing_was_recorded():
|
||||
"""Null, never False. "No measurement" is not "partial measurement".
|
||||
|
||||
The same distinction `suppression`'s null carries (#3497): absent must not
|
||||
read as a verdict. A False here would assert the window is under-covered,
|
||||
which is a claim nobody is in a position to make.
|
||||
"""
|
||||
from datetime import datetime, timezone
|
||||
|
||||
from scribe.services.retrieval_telemetry import _coverage
|
||||
|
||||
since = datetime(2026, 9, 1, tzinfo=timezone.utc)
|
||||
assert _coverage(None, since) == {
|
||||
"complete_from": None, "covers_window": None,
|
||||
}
|
||||
|
||||
|
||||
def test_coverage_reads_a_start_before_the_window_as_covered():
|
||||
from datetime import datetime, timezone
|
||||
|
||||
from scribe.services.retrieval_telemetry import _coverage
|
||||
|
||||
since = datetime(2026, 9, 1, tzinfo=timezone.utc)
|
||||
older = datetime(2026, 8, 1, tzinfo=timezone.utc)
|
||||
newer = datetime(2026, 9, 5, tzinfo=timezone.utc)
|
||||
|
||||
assert _coverage(older, since)["covers_window"] is True
|
||||
assert _coverage(newer, since)["covers_window"] is False, (
|
||||
"a counter that started inside the window covers only part of it"
|
||||
)
|
||||
assert _coverage(newer, since)["complete_from"] == newer.isoformat()
|
||||
|
||||
|
||||
@pytest.mark.integration
|
||||
@pytest.mark.asyncio
|
||||
async def test_coverage_is_per_source_because_the_table_is_older_than_its_arms(
|
||||
_dispose_engine,
|
||||
):
|
||||
"""THE grain question, and the reason a per-table answer is useless.
|
||||
|
||||
`retrieval_logs` accumulates for months. A table-level "earliest row"
|
||||
therefore says months for every source it holds — including one added
|
||||
days ago whose counter means something quite different. The old source
|
||||
would vouch for the young one, which is exactly the reading this exists
|
||||
to prevent.
|
||||
"""
|
||||
from datetime import datetime, timedelta, timezone
|
||||
|
||||
from sqlalchemy import delete
|
||||
|
||||
from scribe.models import async_session
|
||||
from scribe.models.retrieval_log import RetrievalLog
|
||||
from scribe.services.retrieval_telemetry import retrieval_summary
|
||||
|
||||
UID = 990077
|
||||
now = datetime.now(timezone.utc)
|
||||
async with async_session() as s:
|
||||
# An old surface, recording since well before any window we ask for,
|
||||
# AND still recording inside it. Both rows are needed: `complete_from`
|
||||
# comes from the all-time query, but a source only gets a bucket at all
|
||||
# if it has rows in the window, so the 90-day row alone would leave
|
||||
# nothing to assert on.
|
||||
s.add(RetrievalLog(
|
||||
user_id=UID, source="auto_inject", result_count=1,
|
||||
created_at=now - timedelta(days=90),
|
||||
))
|
||||
s.add(RetrievalLog(
|
||||
user_id=UID, source="auto_inject", result_count=1,
|
||||
created_at=now - timedelta(days=1),
|
||||
))
|
||||
# A young arm, first written INSIDE the window below.
|
||||
s.add(RetrievalLog(
|
||||
user_id=UID, source="pre_tool_rule", result_count=1,
|
||||
created_at=now - timedelta(days=2),
|
||||
))
|
||||
await s.commit()
|
||||
|
||||
try:
|
||||
out = await retrieval_summary(UID, days=30)
|
||||
|
||||
assert out["sources"]["auto_inject"]["covers_window"] is True
|
||||
assert out["sources"]["pre_tool_rule"]["covers_window"] is False, (
|
||||
"the young arm was reported as covering a 30-day window — the "
|
||||
"table's age has been allowed to vouch for one of its sources"
|
||||
)
|
||||
finally:
|
||||
async with async_session() as s:
|
||||
await s.execute(delete(RetrievalLog).where(RetrievalLog.user_id == UID))
|
||||
await s.commit()
|
||||
|
||||
|
||||
@pytest.mark.integration
|
||||
@pytest.mark.asyncio
|
||||
async def test_a_surface_that_went_silent_is_not_the_same_as_one_that_never_ran(
|
||||
_dispose_engine,
|
||||
):
|
||||
"""#3720 — absent is how "never existed" renders, so it cannot also be how
|
||||
"stopped recording" renders.
|
||||
|
||||
A surface losing its recorder is one of the failures this milestone exists
|
||||
to make visible, and dropping it from the readout is the most complete way
|
||||
to hide it. Zero here is a real measurement: the table proves the source
|
||||
was recording, and it made no calls across a window it fully covers.
|
||||
"""
|
||||
from datetime import datetime, timedelta, timezone
|
||||
|
||||
from sqlalchemy import delete
|
||||
|
||||
from scribe.models import async_session
|
||||
from scribe.models.retrieval_log import RetrievalLog
|
||||
from scribe.services.retrieval_telemetry import retrieval_summary
|
||||
|
||||
UID = 990078
|
||||
now = datetime.now(timezone.utc)
|
||||
async with async_session() as s:
|
||||
# Recorded once, well before the window, and never since.
|
||||
s.add(RetrievalLog(
|
||||
user_id=UID, source="auto_inject", result_count=3, top_score=0.81,
|
||||
created_at=now - timedelta(days=60),
|
||||
))
|
||||
await s.commit()
|
||||
|
||||
try:
|
||||
out = await retrieval_summary(UID, days=7)
|
||||
|
||||
assert "auto_inject" in out["sources"], (
|
||||
"a source with rows in the table but none in the window was "
|
||||
"dropped from the readout — a surface that stopped recording now "
|
||||
"reads exactly like one that never existed"
|
||||
)
|
||||
quiet = out["sources"]["auto_inject"]
|
||||
assert quiet["calls"] == 0
|
||||
# The window IS covered; what was observed across it is nothing.
|
||||
assert quiet["covers_window"] is True
|
||||
# ...but nothing was sampled, so no distribution may be claimed. A
|
||||
# zeroed score would assert a measurement, which is #3311's mistake.
|
||||
assert quiet["top_score"] == {
|
||||
"p10": None, "p50": None, "p90": None, "min": None, "max": None,
|
||||
}
|
||||
assert quiet["suppression"] is None
|
||||
assert quiet["avg_result_count"] is None
|
||||
finally:
|
||||
async with async_session() as s:
|
||||
await s.execute(delete(RetrievalLog).where(RetrievalLog.user_id == UID))
|
||||
await s.commit()
|
||||
|
||||
|
||||
@pytest.mark.integration
|
||||
@pytest.mark.asyncio
|
||||
async def test_a_section_is_complete_only_from_its_latest_contributor(
|
||||
_dispose_engine,
|
||||
):
|
||||
"""A sum is complete once EVERY contributor was being written — so the
|
||||
section takes the LATEST first-row, not the earliest.
|
||||
|
||||
Taking the earliest would be worse than reporting nothing: it would pick
|
||||
the oldest source in the table and use it to certify a total that a
|
||||
newer source is still only partly contributing to. That is the original
|
||||
error in miniature.
|
||||
"""
|
||||
from datetime import datetime, timedelta, timezone
|
||||
|
||||
from sqlalchemy import delete
|
||||
|
||||
from scribe.models import async_session
|
||||
from scribe.models.rule_usage import RuleUsageEvent
|
||||
from scribe.services.retrieval_telemetry import retrieval_summary
|
||||
|
||||
UID = 990078
|
||||
now = datetime.now(timezone.utc)
|
||||
old = now - timedelta(days=90)
|
||||
young = now - timedelta(days=2)
|
||||
async with async_session() as s:
|
||||
s.add_all([
|
||||
RuleUsageEvent(
|
||||
user_id=UID, rule_id=1, event="surfaced",
|
||||
source="list_always_on_rules", created_at=old,
|
||||
),
|
||||
RuleUsageEvent(
|
||||
user_id=UID, rule_id=2, event="surfaced",
|
||||
source="pre_tool_rule", created_at=young,
|
||||
),
|
||||
])
|
||||
await s.commit()
|
||||
|
||||
try:
|
||||
out = await retrieval_summary(UID, days=30)
|
||||
ru = out["rule_usage"]
|
||||
|
||||
assert ru["complete_from"] == young.isoformat(), (
|
||||
"the section reported completeness from its OLDEST source; a "
|
||||
"total is only as complete as its newest contributor"
|
||||
)
|
||||
assert ru["covers_window"] is False
|
||||
finally:
|
||||
async with async_session() as s:
|
||||
await s.execute(delete(RuleUsageEvent).where(RuleUsageEvent.user_id == UID))
|
||||
await s.commit()
|
||||
|
||||
Reference in New Issue
Block a user