fix(showcase): over-sample + random-order to break near-dup clustering — #699
TABLESAMPLE SYSTEM_ROWS reads CONTIGUOUS rows from each sampled page, so
sequentially-imported near-duplicates (multi-image posts, variant sets) came
back adjacent and clustered in the showcase ("three near-identical in a row").
Sample limit*5 rows (spanning more pages) then ORDER BY random() before taking
limit — breaks the physical adjacency for much better spread, still cheap
(random() over a few hundred rows, not the whole table).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -20,11 +20,20 @@ class ShowcaseService:
|
||||
async def random_sample(self, limit: int = 60) -> list[dict]:
|
||||
if limit < 1 or limit > 200:
|
||||
raise ValueError("limit must be between 1 and 200")
|
||||
# Over-sample then random-order (#699): SYSTEM_ROWS reads CONTIGUOUS rows
|
||||
# from each sampled page, so sequentially-imported near-duplicates
|
||||
# (multi-image posts, variant sets) come back adjacent and cluster in the
|
||||
# showcase ("three near-identical in a row"). Sampling a multiple of
|
||||
# `limit` spans more pages, and ORDER BY random() before taking `limit`
|
||||
# breaks the physical adjacency — far better spread, still cheap
|
||||
# (random() over a few hundred rows, not the whole table).
|
||||
oversample = min(limit * 5, 1000)
|
||||
stmt = select(ImageRecord).from_statement(
|
||||
text(
|
||||
"SELECT * FROM image_record "
|
||||
"TABLESAMPLE SYSTEM_ROWS(:n)"
|
||||
).bindparams(n=limit)
|
||||
"SELECT * FROM ("
|
||||
" SELECT * FROM image_record TABLESAMPLE SYSTEM_ROWS(:o)"
|
||||
") sub ORDER BY random() LIMIT :n"
|
||||
).bindparams(o=oversample, n=limit)
|
||||
)
|
||||
rows = (await self.session.execute(stmt)).scalars().all()
|
||||
return [
|
||||
|
||||
Reference in New Issue
Block a user