bvandeusenandClaude Opus 5
e7c1af32a0
CI & Build / Python lint (push) Successful in 3s
CI & Build / Plugin hooks (push) Successful in 8s
CI & Build / TypeScript typecheck (push) Successful in 23s
CI & Build / integration (push) Successful in 33s
CI & Build / Python tests (push) Successful in 1m6s
CI & Build / Build & push image (push) Successful in 28s
`cleared_threshold` was documented as the number to read first. It was a tautology. The search applies the threshold before returning, so every returned result cleared it by construction and a call with no results has no top_score to compare — the condition was true exactly when `result_count > 0`. It was `calls - zero_result_calls` under a name that promised a second opinion, and `zero + cleared == calls` held on all nineteen source/window readings ever taken, today's live seven included. The reading procedure built on it asked the reader to compare a number with itself, and a threshold change was unobservable through it: raise the bar and both numbers move together, so the field could never show a bar set too high. REPLACED, NOT JUST REMOVED. The question the table exists to answer is whether the bar is in the right place, and that is only answerable from the calls that returned NOTHING: how close did the best rejected candidate come? A 0.72 bar turning away a stream of 0.71s is set too high by a hair; the same bar turning away 0.30s is working. Both render as a zero-result call today and nothing separates them, because the losing score is discarded inside the search. So both searches now rank WITHOUT the bar and apply it in Python. The qualifying set is provably identical — rows arrive ordered by distance, so every above-bar row sorts ahead of every below-bar one, and an over-fetch that returned N above-bar rows returns the same N plus some losers. What changes is that the losers are visible instead of dropped in the query. `report` carries the score out without changing what a search RETURNS: eight of eleven call sites want hits and nothing else. New column (migration 0096), nullable and unbackfilled. A row written before this genuinely does not know, and a 0.0 would read as "the corpus held nothing remotely relevant" — a claim invented out of a caller's silence, which is the substitution this whole milestone corrects. The new aggregate is a percentile_cont WITHIN GROUP over a CASE, one step from the shape that produced #2663, where a rejected query was swallowed by the broad except and every counter read zero. It carries an integration guard for that reason: only real Postgres can say it parses, and the symptom of failure is silence. Also adds a guard that no int field in a bucket equals `calls - zero_result_calls`. That identity is what `cleared_threshold` satisfied for its whole life, and it survived because it had its own name and nobody added the two numbers beside it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011cPyzNnegXHr5iRMzzy5KJ
94 lines
4.8 KiB
Python
94 lines
4.8 KiB
Python
from datetime import datetime, timezone
|
|
|
|
from sqlalchemy import Boolean, DateTime, Float, Index, Integer, Text
|
|
from sqlalchemy.dialects.postgresql import JSONB
|
|
from sqlalchemy.orm import Mapped, mapped_column
|
|
|
|
from scribe.models import Base
|
|
from scribe.models.base import iso
|
|
|
|
|
|
class RetrievalLog(Base):
|
|
"""One row per semantic-retrieval call, for KB-injection tuning.
|
|
|
|
Captures what a query asked for, what came back, and the score
|
|
distribution of the results — the empirical basis for tuning the
|
|
similarity threshold and top-k per surface. `result_ids` holds the ranked
|
|
hits (id + score + rank) so a later pass can correlate "what we surfaced"
|
|
against "what the agent then fetched/referenced".
|
|
|
|
Deliberately FK-free on user_id (mirrors AppLog): telemetry should outlive
|
|
the row it describes, and a deleted user shouldn't cascade away history.
|
|
"""
|
|
|
|
__tablename__ = "retrieval_logs"
|
|
|
|
id: Mapped[int] = mapped_column(primary_key=True)
|
|
# Declared here rather than via CreatedAtMixin on purpose: the composite
|
|
# index below orders on `created_at.desc()`, which needs the column object
|
|
# in this class body — a mixin's column is not in scope there.
|
|
created_at: Mapped[datetime] = mapped_column(
|
|
DateTime(timezone=True), default=lambda: datetime.now(timezone.utc)
|
|
)
|
|
user_id: Mapped[int | None] = mapped_column(Integer, nullable=True)
|
|
# Retrieval surface: 'mcp_search' | 'rest_search' | 'auto_inject' | ...
|
|
source: Mapped[str] = mapped_column(Text, nullable=False)
|
|
query: Mapped[str | None] = mapped_column(Text, nullable=True)
|
|
# Effective parameters actually used for this call.
|
|
threshold: Mapped[float | None] = mapped_column(Float, nullable=True)
|
|
limit_n: Mapped[int | None] = mapped_column(Integer, nullable=True)
|
|
project_id: Mapped[int | None] = mapped_column(Integer, nullable=True)
|
|
# The content-type filter as passed to semantic_search_notes: True=tasks,
|
|
# False=notes, NULL=any.
|
|
is_task: Mapped[bool | None] = mapped_column(Boolean, nullable=True)
|
|
result_count: Mapped[int] = mapped_column(Integer, nullable=False, default=0)
|
|
# How many scored hits this call DROPPED because the session had already
|
|
# been shown them. NULLABLE, and the null is load-bearing: it means "this
|
|
# surface does not report suppression", which must not read as "nothing was
|
|
# suppressed". `result_count == 0` alone conflates two different events —
|
|
# the ranker found nothing above threshold, and the ranker found something
|
|
# the reader already had — and only the first says a threshold is too high.
|
|
# Reading a zero as a ranker decline is how #3311 mis-scoped a milestone;
|
|
# an unmeasured value that renders as 0 is the same mistake with a nicer
|
|
# face, so surfaces that filter INSIDE the search leave this null.
|
|
suppressed_count: Mapped[int | None] = mapped_column(Integer, nullable=True)
|
|
top_score: Mapped[float | None] = mapped_column(Float, nullable=True)
|
|
min_score: Mapped[float | None] = mapped_column(Float, nullable=True)
|
|
# The best score the ranker COULD have offered, before the threshold — as
|
|
# against `top_score`, which is the best it DID offer. They are equal on
|
|
# any call that returned something, and only this one exists on a call
|
|
# that returned nothing, which is the only place a bar can be judged from
|
|
# (#3670). Null means the caller did not measure it, never "nothing was
|
|
# close": a 0.0 there would read as a corpus with no relevant records at
|
|
# all, which is an artifact standing in for a measurement.
|
|
best_available_score: Mapped[float | None] = mapped_column(Float, nullable=True)
|
|
# [{"id": int, "score": float, "rank": int}, ...], highest-first.
|
|
result_ids: Mapped[list] = mapped_column(JSONB, nullable=False, default=list)
|
|
duration_ms: Mapped[float | None] = mapped_column(Float, nullable=True)
|
|
|
|
__table_args__ = (
|
|
Index("ix_retrieval_logs_created_at", "created_at"),
|
|
Index("ix_retrieval_logs_user_id", "user_id"),
|
|
Index("ix_retrieval_logs_source", "source"),
|
|
Index("ix_retrieval_logs_source_created_at", "source", created_at.desc()),
|
|
)
|
|
|
|
def to_dict(self) -> dict:
|
|
return {
|
|
"id": self.id,
|
|
"created_at": iso(self.created_at),
|
|
"user_id": self.user_id,
|
|
"source": self.source,
|
|
"query": self.query,
|
|
"threshold": self.threshold,
|
|
"limit_n": self.limit_n,
|
|
"project_id": self.project_id,
|
|
"is_task": self.is_task,
|
|
"result_count": self.result_count,
|
|
"suppressed_count": self.suppressed_count,
|
|
"top_score": self.top_score,
|
|
"min_score": self.min_score,
|
|
"result_ids": self.result_ids,
|
|
"duration_ms": self.duration_ms,
|
|
}
|