fix: variant artwork was dropped as a near-duplicate even at threshold 0 (4223)
CI / lint (push) Failing after 3s
CI / extension-version (push) Successful in 3s
Build images / sign-extension (push) Successful in 4s
Build images / build-agent (push) Successful in 7s
CI / frontend-build (push) Successful in 30s
CI / backend-lint-and-test (push) Successful in 1m8s
Build images / build-web (push) Successful in 1m28s
Build images / smoke-web (push) Skipped
CI / integration (push) Successful in 2m51s
Build images / build-ml (push) Successful in 2m59s
Build images / promote (push) Skipped

The operator reported a 15-image variant pack landing as 3 records, then
reported variants STILL being dropped with phash_threshold at 0 — the floor
of the dial. No setting could have fixed it: at hash_size=8 a pHash is 64
bits of coarse light/dark layout, so two variants sharing a composition
produce the SAME bits. Distance 0 meant "identical hash", not "identical
image", and the dial was simultaneously too coarse to keep variants and too
tight to catch a re-encoded rescale.

The hash no longer decides a merge on its own. find_similar now runs three
gates, cheapest first: the threshold proposes candidates, aspect ratio
(ASPECT_TOL, matching the tier-1 video path) rejects crops and re-canvases,
and a pixel-level confirm on the two files accepts. Every gate fails closed —
unknown dimensions, an unreadable candidate, a hash of the wrong width all
mean "not a duplicate", because too strict keeps a redundant copy the
operator can see while too loose deletes artwork only a source re-walk
returns.

- utils/phash.py: HASH_SIZE 8 -> 16 (256-bit, what ImageRepo always used);
  aspect_matches, fingerprint/fingerprint_path/fingerprints_match (PIL-only,
  mean drift + changed-pixel fraction), find_similar gains `confirm`.
- importer: _pixel_confirmer supplies gate 3 on both dedup sites, lazily and
  cached, so a non-matching import costs no extra I/O.
- 0098: widens image_record.phash to 64 chars and NULLs every value — a
  stored 64-bit hash cannot be compared to a 256-bit one, and backfill_phash
  is NULL-only, keyset-paginated and now on the daily beat, so the library
  re-hashes itself. Dedup degrades to sha256 until it finishes.
- phash_threshold counts bits and the denominator went 64 -> 256, so the
  setting is reset to the new default of 24 (there is no honest carry-over)
  and the slider is rescaled to 0-64.
- gallery_service dup_threshold 8 -> 32: the same fraction of the hash, so
  the Explore rail keeps the variance the operator tuned in on 2026-07-01.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR
This commit is contained in:
2026-09-21 00:16:14 -04:00
co-authored by Claude Opus 5
parent 3fe9d0a612
commit a05cb66635
13 changed files with 501 additions and 45 deletions
+35 -1
View File
@@ -39,7 +39,7 @@ from ..utils.paths import (
hash_suffixed_name,
safe_ext,
)
from ..utils.phash import compute_phash, find_similar
from ..utils.phash import compute_phash, find_similar, fingerprint_path, fingerprints_match
from ..utils.sidecar import find_sidecar, parse_sidecar
from ..utils.slug import slugify
from .archive_extractor import extract_archive, is_archive
@@ -234,6 +234,38 @@ class Importer:
(phash, width or 0, height or 0, image_id)
)
def _pixel_confirmer(self, source: Path):
"""Build `find_similar`'s gate-3 callback for an incoming file.
pHash proposes; this accepts. A candidate is a duplicate only if its
file really is the same picture as `source` at a different size —
which is the only merge the operator asked for (#4223). Everything
else (a missing file, an unreadable one, a deleted row) returns
False: the destructive outcomes here are dropping a download and
overwriting a kept file, so an unanswerable question must not read
as "yes".
Both sides' fingerprints are computed lazily and cached, so an
import that matches nothing costs no I/O at all and an archive
member that keeps hitting the same candidate pays for it once.
"""
new_fp: list = []
cand_fps: dict[int, object] = {}
def confirm(candidate_id: int) -> bool:
if not new_fp:
new_fp.append(fingerprint_path(source))
if new_fp[0] is None:
return False
if candidate_id not in cand_fps:
rec = self.session.get(ImageRecord, candidate_id)
cand_fps[candidate_id] = (
fingerprint_path(Path(rec.path)) if rec and rec.path else None
)
return fingerprints_match(new_fp[0], cand_fps[candidate_id])
return confirm
def _get_or_create(self, stmt, factory):
"""Race-safe find-or-create. Run `stmt` (scalar_one_or_none); if a
row exists, return it. Otherwise open a savepoint and INSERT
@@ -862,6 +894,7 @@ class Importer:
rel, match_id = find_similar(
phash, width or 0, height or 0,
candidates, self.settings.phash_threshold,
confirm=self._pixel_confirmer(source),
)
if rel == "larger_exists":
# Enrich-on-duplicate (parity with attach_in_place).
@@ -1241,6 +1274,7 @@ class Importer:
rel, match_id = find_similar(
phash, width or 0, height or 0,
candidates, self.settings.phash_threshold,
confirm=self._pixel_confirmer(path),
)
if rel == "larger_exists":
# Enrich-on-duplicate: link the near-dup's post to the