fix: variant artwork was dropped as a near-duplicate even at threshold 0 (4223)
CI / lint (push) Failing after 3s
CI / extension-version (push) Successful in 3s
Build images / sign-extension (push) Successful in 4s
Build images / build-agent (push) Successful in 7s
CI / frontend-build (push) Successful in 30s
CI / backend-lint-and-test (push) Successful in 1m8s
Build images / build-web (push) Successful in 1m28s
Build images / smoke-web (push) Skipped
CI / integration (push) Successful in 2m51s
Build images / build-ml (push) Successful in 2m59s
Build images / promote (push) Skipped
CI / lint (push) Failing after 3s
CI / extension-version (push) Successful in 3s
Build images / sign-extension (push) Successful in 4s
Build images / build-agent (push) Successful in 7s
CI / frontend-build (push) Successful in 30s
CI / backend-lint-and-test (push) Successful in 1m8s
Build images / build-web (push) Successful in 1m28s
Build images / smoke-web (push) Skipped
CI / integration (push) Successful in 2m51s
Build images / build-ml (push) Successful in 2m59s
Build images / promote (push) Skipped
The operator reported a 15-image variant pack landing as 3 records, then reported variants STILL being dropped with phash_threshold at 0 — the floor of the dial. No setting could have fixed it: at hash_size=8 a pHash is 64 bits of coarse light/dark layout, so two variants sharing a composition produce the SAME bits. Distance 0 meant "identical hash", not "identical image", and the dial was simultaneously too coarse to keep variants and too tight to catch a re-encoded rescale. The hash no longer decides a merge on its own. find_similar now runs three gates, cheapest first: the threshold proposes candidates, aspect ratio (ASPECT_TOL, matching the tier-1 video path) rejects crops and re-canvases, and a pixel-level confirm on the two files accepts. Every gate fails closed — unknown dimensions, an unreadable candidate, a hash of the wrong width all mean "not a duplicate", because too strict keeps a redundant copy the operator can see while too loose deletes artwork only a source re-walk returns. - utils/phash.py: HASH_SIZE 8 -> 16 (256-bit, what ImageRepo always used); aspect_matches, fingerprint/fingerprint_path/fingerprints_match (PIL-only, mean drift + changed-pixel fraction), find_similar gains `confirm`. - importer: _pixel_confirmer supplies gate 3 on both dedup sites, lazily and cached, so a non-matching import costs no extra I/O. - 0098: widens image_record.phash to 64 chars and NULLs every value — a stored 64-bit hash cannot be compared to a 256-bit one, and backfill_phash is NULL-only, keyset-paginated and now on the daily beat, so the library re-hashes itself. Dedup degrades to sha256 until it finishes. - phash_threshold counts bits and the denominator went 64 -> 256, so the setting is reset to the new default of 24 (there is no honest carry-over) and the slider is rescaled to 0-64. - gallery_service dup_threshold 8 -> 32: the same fraction of the hash, so the Explore rail keeps the variance the operator tuned in on 2026-07-01. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR
This commit is contained in:
@@ -146,12 +146,17 @@ async def test_similar_collapses_near_duplicate_phashes(db):
|
||||
dupes = []
|
||||
for n in range(2, 7): # 5 near-identical reposts
|
||||
r = await _img(db, n, _vec(1, 0.01 * n))
|
||||
r.phash = "ffffffffffffffff" # identical perceptual hash
|
||||
r.phash = "f" * 64 # identical perceptual hash
|
||||
dupes.append(r)
|
||||
# 64 hex chars = the 256-bit hash utils.phash emits (#4223, migration
|
||||
# 0098). The widths must match the real ones: _diversify_similar's
|
||||
# dup_threshold moved 8 -> 32 with the hash, so 64-bit fixtures would
|
||||
# now read as near-duplicates of each other and collapse the very rows
|
||||
# this asserts come through.
|
||||
distinct_a = await _img(db, 7, _vec(1, 1))
|
||||
distinct_a.phash = "0000000000000000"
|
||||
distinct_a.phash = "0" * 64 # 256 bits away from the dupes
|
||||
distinct_b = await _img(db, 8, _vec(0, 1))
|
||||
distinct_b.phash = "0f0f0f0f0f0f0f0f"
|
||||
distinct_b.phash = "0f" * 32 # 128 bits away
|
||||
await db.flush()
|
||||
|
||||
res = await GalleryService(db).similar(src.id, limit=10)
|
||||
|
||||
Reference in New Issue
Block a user