bvandeusenandClaude Opus 5
8be555d6dd
CI & Build / Python lint (push) Successful in 3s
CI & Build / Plugin hooks (push) Successful in 8s
CI & Build / integration (push) Successful in 31s
CI & Build / TypeScript typecheck (push) Successful in 37s
CI & Build / Python tests (push) Successful in 1m4s
CI & Build / Build & push image (push) Successful in 28s
Making the rule arms log every call exposed a second ambiguity in the same
row. `result_count == 0` is two unrelated events wearing one number:
- the ranker found nothing above the bar — the only evidence a threshold is
set too high; and
- the ranker found only what this session had already been shown — which
says nothing whatever about the bar.
A long session excludes its way into the second, so the arm reads worse the
longer it runs correctly. Rows written now carry the ambiguity permanently,
which is why this lands before any watch period rather than after.
`retrieval_logs.suppressed_count` (0095, nullable) holds what the caller
dropped as already-shown. Both rule arms report it; they filter in Python and
always know. The note arms pass exclusions INTO semantic_search_notes and
never see what was dropped, so they store NULL.
THE NULL IS LOAD-BEARING. It means "not measured here", and the readout
renders it as `suppression: null` rather than a zeroed dict. Defaulting to 0
would let an unmeasured surface read as a perfectly clean one — the same
substitution of an artifact for a measurement that #3311 made. No backfill,
for the same reason: existing rows genuinely do not know.
`retrieval_telemetry`'s `sources` gains `suppression` with `measured_calls`,
`calls_with_suppression` and `zero_because_already_shown`; subtract the last
from `zero_result_calls` for the true ranker declines. The MCP tool docstring
says to read the two together and warns against reading the null as a zero.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011cPyzNnegXHr5iRMzzy5KJ
86 lines
4.2 KiB
Python
86 lines
4.2 KiB
Python
from datetime import datetime, timezone
|
|
|
|
from sqlalchemy import Boolean, DateTime, Float, Index, Integer, Text
|
|
from sqlalchemy.dialects.postgresql import JSONB
|
|
from sqlalchemy.orm import Mapped, mapped_column
|
|
|
|
from scribe.models import Base
|
|
from scribe.models.base import iso
|
|
|
|
|
|
class RetrievalLog(Base):
|
|
"""One row per semantic-retrieval call, for KB-injection tuning.
|
|
|
|
Captures what a query asked for, what came back, and the score
|
|
distribution of the results — the empirical basis for tuning the
|
|
similarity threshold and top-k per surface. `result_ids` holds the ranked
|
|
hits (id + score + rank) so a later pass can correlate "what we surfaced"
|
|
against "what the agent then fetched/referenced".
|
|
|
|
Deliberately FK-free on user_id (mirrors AppLog): telemetry should outlive
|
|
the row it describes, and a deleted user shouldn't cascade away history.
|
|
"""
|
|
|
|
__tablename__ = "retrieval_logs"
|
|
|
|
id: Mapped[int] = mapped_column(primary_key=True)
|
|
# Declared here rather than via CreatedAtMixin on purpose: the composite
|
|
# index below orders on `created_at.desc()`, which needs the column object
|
|
# in this class body — a mixin's column is not in scope there.
|
|
created_at: Mapped[datetime] = mapped_column(
|
|
DateTime(timezone=True), default=lambda: datetime.now(timezone.utc)
|
|
)
|
|
user_id: Mapped[int | None] = mapped_column(Integer, nullable=True)
|
|
# Retrieval surface: 'mcp_search' | 'rest_search' | 'auto_inject' | ...
|
|
source: Mapped[str] = mapped_column(Text, nullable=False)
|
|
query: Mapped[str | None] = mapped_column(Text, nullable=True)
|
|
# Effective parameters actually used for this call.
|
|
threshold: Mapped[float | None] = mapped_column(Float, nullable=True)
|
|
limit_n: Mapped[int | None] = mapped_column(Integer, nullable=True)
|
|
project_id: Mapped[int | None] = mapped_column(Integer, nullable=True)
|
|
# The content-type filter as passed to semantic_search_notes: True=tasks,
|
|
# False=notes, NULL=any.
|
|
is_task: Mapped[bool | None] = mapped_column(Boolean, nullable=True)
|
|
result_count: Mapped[int] = mapped_column(Integer, nullable=False, default=0)
|
|
# How many scored hits this call DROPPED because the session had already
|
|
# been shown them. NULLABLE, and the null is load-bearing: it means "this
|
|
# surface does not report suppression", which must not read as "nothing was
|
|
# suppressed". `result_count == 0` alone conflates two different events —
|
|
# the ranker found nothing above threshold, and the ranker found something
|
|
# the reader already had — and only the first says a threshold is too high.
|
|
# Reading a zero as a ranker decline is how #3311 mis-scoped a milestone;
|
|
# an unmeasured value that renders as 0 is the same mistake with a nicer
|
|
# face, so surfaces that filter INSIDE the search leave this null.
|
|
suppressed_count: Mapped[int | None] = mapped_column(Integer, nullable=True)
|
|
top_score: Mapped[float | None] = mapped_column(Float, nullable=True)
|
|
min_score: Mapped[float | None] = mapped_column(Float, nullable=True)
|
|
# [{"id": int, "score": float, "rank": int}, ...], highest-first.
|
|
result_ids: Mapped[list] = mapped_column(JSONB, nullable=False, default=list)
|
|
duration_ms: Mapped[float | None] = mapped_column(Float, nullable=True)
|
|
|
|
__table_args__ = (
|
|
Index("ix_retrieval_logs_created_at", "created_at"),
|
|
Index("ix_retrieval_logs_user_id", "user_id"),
|
|
Index("ix_retrieval_logs_source", "source"),
|
|
Index("ix_retrieval_logs_source_created_at", "source", created_at.desc()),
|
|
)
|
|
|
|
def to_dict(self) -> dict:
|
|
return {
|
|
"id": self.id,
|
|
"created_at": iso(self.created_at),
|
|
"user_id": self.user_id,
|
|
"source": self.source,
|
|
"query": self.query,
|
|
"threshold": self.threshold,
|
|
"limit_n": self.limit_n,
|
|
"project_id": self.project_id,
|
|
"is_task": self.is_task,
|
|
"result_count": self.result_count,
|
|
"suppressed_count": self.suppressed_count,
|
|
"top_score": self.top_score,
|
|
"min_score": self.min_score,
|
|
"result_ids": self.result_ids,
|
|
"duration_ms": self.duration_ms,
|
|
}
|