Telemetry coverage, and the rule arms stop filtering to one tier #143

Merged
bvandeusen merged 4 commits from dev into main 2026-09-08 11:11:40 -04:00
Showing only changes of commit 950c93c5d4 - Show all commits
+29 -29
View File
@@ -217,35 +217,6 @@ def _round(v, places: int = 4):
return None if v is None else round(float(v), places)
async def retrieval_summary(user_id: int | None, *, days: int = 30) -> dict:
"""What the retrieval telemetry says, per surface, over a window.
Three aggregates side by side, each read from the table built for it — NOT
a join. `usage` is notes, `rule_usage` is rules, and they stay apart
because a few dozen eligible rules blended into thousands of notes is the
note ratio with noise on it (milestone 333). `NoteUsageEvent`'s own docstring is explicit that the two are
complements ("RetrievalLog tunes the threshold, this tunes the corpus") and
that RetrievalLog's JSONB `result_ids` "can't be indexed at" the per-note
grain. So the score distribution comes from `retrieval_logs` on its indexed
columns, and surfaced-vs-pulled comes from `note_usage_events` at the grain
it was built for. Reading each from its own table is both cheaper and more
honest than correlating them through JSONB.
`usage["by_source"]` is the one join, and it stays INSIDE
`note_usage_events` — surfaced rows against pulled rows on note_id. That
answers "of the notes this surface chose, how many were opened", which the
top-level ratio averages away. It does not cross into `retrieval_logs`, so
the sentence above still holds.
Scoped to one user's own telemetry. There is no sharing model for a
retrieval log — it records what THIS user's agent asked for, including the
query text — so an owner filter is the whole access rule here rather than a
shortcut around `services/access.py` (P#78 governs shared record kinds).
Never raises: a telemetry readout that can break its caller is worse than
no readout. It does distinguish "no rows" from "the read failed", because
#2663 is exactly the bug where those two looked identical for weeks.
"""
async def _complete_from(session, model, user_id) -> dict[str, Any]:
"""When each source in `model` started being recorded, and the instant the
WHOLE table is complete from. Returns {source: earliest_row, "*": latest}.
@@ -297,6 +268,35 @@ def _coverage(complete_from, since) -> dict:
}
async def retrieval_summary(user_id: int | None, *, days: int = 30) -> dict:
"""What the retrieval telemetry says, per surface, over a window.
Three aggregates side by side, each read from the table built for it — NOT
a join. `usage` is notes, `rule_usage` is rules, and they stay apart
because a few dozen eligible rules blended into thousands of notes is the
note ratio with noise on it (milestone 333). `NoteUsageEvent`'s own docstring is explicit that the two are
complements ("RetrievalLog tunes the threshold, this tunes the corpus") and
that RetrievalLog's JSONB `result_ids` "can't be indexed at" the per-note
grain. So the score distribution comes from `retrieval_logs` on its indexed
columns, and surfaced-vs-pulled comes from `note_usage_events` at the grain
it was built for. Reading each from its own table is both cheaper and more
honest than correlating them through JSONB.
`usage["by_source"]` is the one join, and it stays INSIDE
`note_usage_events` — surfaced rows against pulled rows on note_id. That
answers "of the notes this surface chose, how many were opened", which the
top-level ratio averages away. It does not cross into `retrieval_logs`, so
the sentence above still holds.
Scoped to one user's own telemetry. There is no sharing model for a
retrieval log — it records what THIS user's agent asked for, including the
query text — so an owner filter is the whole access rule here rather than a
shortcut around `services/access.py` (P#78 governs shared record kinds).
Never raises: a telemetry readout that can break its caller is worse than
no readout. It does distinguish "no rows" from "the read failed", because
#2663 is exactly the bug where those two looked identical for weeks.
"""
since = datetime.now(timezone.utc) - timedelta(days=max(1, int(days)))
out: dict = {
"window_days": int(days),