CI / lint (push) Failing after 3s
CI / extension-version (push) Successful in 3s
Build images / sign-extension (push) Successful in 4s
Build images / build-agent (push) Successful in 7s
CI / frontend-build (push) Successful in 30s
CI / backend-lint-and-test (push) Successful in 1m8s
Build images / build-web (push) Successful in 1m28s
Build images / smoke-web (push) Skipped
CI / integration (push) Successful in 2m51s
Build images / build-ml (push) Successful in 2m59s
Build images / promote (push) Skipped
The operator reported a 15-image variant pack landing as 3 records, then reported variants STILL being dropped with phash_threshold at 0 — the floor of the dial. No setting could have fixed it: at hash_size=8 a pHash is 64 bits of coarse light/dark layout, so two variants sharing a composition produce the SAME bits. Distance 0 meant "identical hash", not "identical image", and the dial was simultaneously too coarse to keep variants and too tight to catch a re-encoded rescale. The hash no longer decides a merge on its own. find_similar now runs three gates, cheapest first: the threshold proposes candidates, aspect ratio (ASPECT_TOL, matching the tier-1 video path) rejects crops and re-canvases, and a pixel-level confirm on the two files accepts. Every gate fails closed — unknown dimensions, an unreadable candidate, a hash of the wrong width all mean "not a duplicate", because too strict keeps a redundant copy the operator can see while too loose deletes artwork only a source re-walk returns. - utils/phash.py: HASH_SIZE 8 -> 16 (256-bit, what ImageRepo always used); aspect_matches, fingerprint/fingerprint_path/fingerprints_match (PIL-only, mean drift + changed-pixel fraction), find_similar gains `confirm`. - importer: _pixel_confirmer supplies gate 3 on both dedup sites, lazily and cached, so a non-matching import costs no extra I/O. - 0098: widens image_record.phash to 64 chars and NULLs every value — a stored 64-bit hash cannot be compared to a 256-bit one, and backfill_phash is NULL-only, keyset-paginated and now on the daily beat, so the library re-hashes itself. Dedup degrades to sha256 until it finishes. - phash_threshold counts bits and the denominator went 64 -> 256, so the setting is reset to the new default of 24 (there is no honest carry-over) and the slider is rescaled to 0-64. - gallery_service dup_threshold 8 -> 32: the same fraction of the hash, so the Explore rail keeps the variance the operator tuned in on 2026-07-01. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR
141 lines
5.1 KiB
Python
141 lines
5.1 KiB
Python
import io
|
|
|
|
import imagehash
|
|
from PIL import Image
|
|
|
|
from backend.app.utils.phash import (
|
|
aspect_matches,
|
|
compute_phash,
|
|
find_similar,
|
|
fingerprint,
|
|
fingerprints_match,
|
|
)
|
|
|
|
|
|
def _img(color, size=(64, 64)):
|
|
return Image.new("RGB", size, color)
|
|
|
|
|
|
def _split(orient, size=64):
|
|
"""A structured image (half black / half white) so it has real DCT
|
|
content — solid colors all phash-collapse to the same value."""
|
|
im = Image.new("L", (size, size), 0)
|
|
px = im.load()
|
|
for y in range(size):
|
|
for x in range(size):
|
|
if (x if orient == "v" else y) >= size // 2:
|
|
px[x, y] = 255
|
|
return im.convert("RGB")
|
|
|
|
|
|
def test_compute_phash_stable_and_hex():
|
|
h1 = compute_phash(_img((10, 120, 200)))
|
|
h2 = compute_phash(_img((10, 120, 200)))
|
|
assert isinstance(h1, str) and len(h1) == 64 # hash_size=16 -> 256-bit -> 64 hex
|
|
assert h1 == h2
|
|
|
|
|
|
def test_compute_phash_none_for_non_image():
|
|
assert compute_phash(io.BytesIO(b"not an image")) is None
|
|
|
|
|
|
def test_find_similar_none_when_far():
|
|
# Vertical vs horizontal split = structurally orthogonal → far apart.
|
|
new = compute_phash(_split("v"))
|
|
cand = (compute_phash(_split("h")), 100, 100, 7)
|
|
rel, mid = find_similar(new, 50, 50, [cand], threshold=2)
|
|
assert rel == "none" and mid is None
|
|
|
|
|
|
def test_find_similar_larger_exists_skips_new():
|
|
h = compute_phash(_img((123, 50, 7)))
|
|
rel, mid = find_similar(h, 100, 100, [(h, 400, 400, 9)], threshold=2)
|
|
assert rel == "larger_exists" and mid == 9
|
|
|
|
|
|
def test_find_similar_smaller_exists_supersede():
|
|
h = compute_phash(_img((123, 50, 7)))
|
|
rel, mid = find_similar(h, 800, 800, [(h, 100, 100, 4)], threshold=2)
|
|
assert rel == "smaller_exists" and mid == 4
|
|
|
|
|
|
def test_find_similar_threshold_boundary_inclusive_and_first_match():
|
|
h = compute_phash(_img((10, 10, 10)))
|
|
far = compute_phash(_img((240, 240, 240)))
|
|
d = imagehash.hex_to_hash(h) - imagehash.hex_to_hash(far)
|
|
# at exactly the distance, it counts as similar (<=)
|
|
rel, _ = find_similar(h, 10, 10, [(far, 999, 999, 1)], threshold=d)
|
|
assert rel == "larger_exists"
|
|
rel2, _ = find_similar(h, 10, 10, [(far, 999, 999, 1)], threshold=d - 1)
|
|
assert rel2 == "none"
|
|
|
|
|
|
# --- Gate 2: aspect ratio (#4223) -------------------------------------------
|
|
|
|
def test_aspect_matches_tolerates_rounding_but_not_a_crop():
|
|
assert aspect_matches(1000, 1000, 250, 250)
|
|
assert aspect_matches(1999, 1000, 1000, 500) # off-by-one on a rescale
|
|
assert not aspect_matches(1000, 1000, 1000, 500) # 1:1 vs 2:1
|
|
# Unmeasurable is not a proven duplicate — the gate fails closed.
|
|
assert not aspect_matches(1000, 1000, None, None)
|
|
assert not aspect_matches(0, 0, 100, 100)
|
|
|
|
|
|
def test_find_similar_skips_a_hash_twin_with_a_different_aspect():
|
|
"""A crop or re-canvas keeps the composition, so it can land inside the
|
|
threshold. Dimensions are what say it is not a rescale."""
|
|
h = compute_phash(_img((123, 50, 7)))
|
|
rel, mid = find_similar(h, 100, 100, [(h, 400, 200, 9)], threshold=64)
|
|
assert rel == "none" and mid is None
|
|
|
|
|
|
# --- Gate 3: the pixel confirm (#4223) --------------------------------------
|
|
|
|
def test_fingerprints_match_across_a_rescale():
|
|
assert fingerprints_match(
|
|
fingerprint(_split("v", 64)), fingerprint(_split("v", 512))
|
|
)
|
|
|
|
|
|
def test_fingerprints_reject_a_local_change():
|
|
"""The case the hash cannot see: same composition, one region redrawn."""
|
|
base = _split("v", 128)
|
|
variant = base.copy()
|
|
for y in range(20, 60):
|
|
for x in range(80, 120): # a patch on the white half
|
|
variant.putpixel((x, y), (0, 0, 0))
|
|
assert not fingerprints_match(fingerprint(base), fingerprint(variant))
|
|
|
|
|
|
def test_fingerprints_match_is_false_when_either_side_is_missing():
|
|
assert not fingerprints_match(fingerprint(_split("v")), None)
|
|
assert not fingerprints_match(None, None)
|
|
|
|
|
|
def test_find_similar_confirm_can_veto_a_hash_and_aspect_match():
|
|
h = compute_phash(_img((123, 50, 7)))
|
|
cand = [(h, 400, 400, 9)]
|
|
rel, mid = find_similar(h, 100, 100, cand, threshold=64, confirm=lambda _: True)
|
|
assert rel == "larger_exists" and mid == 9
|
|
rel2, mid2 = find_similar(h, 100, 100, cand, threshold=64, confirm=lambda _: False)
|
|
assert rel2 == "none" and mid2 is None
|
|
|
|
|
|
def test_find_similar_keeps_looking_after_a_veto():
|
|
"""A vetoed candidate must not end the search, or one false pre-filter hit
|
|
would hide the real duplicate sitting behind it."""
|
|
h = compute_phash(_img((123, 50, 7)))
|
|
candidates = [(h, 400, 400, 1), (h, 400, 400, 2)]
|
|
rel, mid = find_similar(
|
|
h, 100, 100, candidates, threshold=64, confirm=lambda cid: cid == 2
|
|
)
|
|
assert rel == "larger_exists" and mid == 2
|
|
|
|
|
|
def test_find_similar_skips_a_hash_of_the_wrong_width():
|
|
"""Mid-re-hash (migration 0098): a leftover 64-bit hash cannot be compared
|
|
to a 256-bit one. Skipping it degrades to no dedup, never to a merge."""
|
|
h = compute_phash(_img((123, 50, 7)))
|
|
rel, mid = find_similar(h, 100, 100, [("ffffffffffffffff", 400, 400, 3)], threshold=64)
|
|
assert rel == "none" and mid is None
|