fix: variant artwork was dropped as a near-duplicate even at threshold 0 (4223)

The operator reported a 15-image variant pack landing as 3 records, then
reported variants STILL being dropped with phash_threshold at 0 — the floor
of the dial. No setting could have fixed it: at hash_size=8 a pHash is 64
bits of coarse light/dark layout, so two variants sharing a composition
produce the SAME bits. Distance 0 meant "identical hash", not "identical
image", and the dial was simultaneously too coarse to keep variants and too
tight to catch a re-encoded rescale.

The hash no longer decides a merge on its own. find_similar now runs three
gates, cheapest first: the threshold proposes candidates, aspect ratio
(ASPECT_TOL, matching the tier-1 video path) rejects crops and re-canvases,
and a pixel-level confirm on the two files accepts. Every gate fails closed —
unknown dimensions, an unreadable candidate, a hash of the wrong width all
mean "not a duplicate", because too strict keeps a redundant copy the
operator can see while too loose deletes artwork only a source re-walk
returns.

- utils/phash.py: HASH_SIZE 8 -> 16 (256-bit, what ImageRepo always used);
  aspect_matches, fingerprint/fingerprint_path/fingerprints_match (PIL-only,
  mean drift + changed-pixel fraction), find_similar gains `confirm`.
- importer: _pixel_confirmer supplies gate 3 on both dedup sites, lazily and
  cached, so a non-matching import costs no extra I/O.
- 0098: widens image_record.phash to 64 chars and NULLs every value — a
  stored 64-bit hash cannot be compared to a 256-bit one, and backfill_phash
  is NULL-only, keyset-paginated and now on the daily beat, so the library
  re-hashes itself. Dedup degrades to sha256 until it finishes.
- phash_threshold counts bits and the denominator went 64 -> 256, so the
  setting is reset to the new default of 24 (there is no honest carry-over)
  and the slider is rescaled to 0-64.
- gallery_service dup_threshold 8 -> 32: the same fraction of the hash, so
  the Explore rail keeps the variance the operator tuned in on 2026-07-01.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR

backend/app/celery_app.py

@@ -120,6 +120,14 @@ def make_celery() -> Celery:
                 "schedule": 86400.0,  # daily — sweep .part/.partial left by a
                 # download/import killed mid-write (graceful-shutdown fallout)
             },
+            "backfill-phash-daily": {
+                "task": "backend.app.tasks.maintenance.backfill_phash",
+                "schedule": 86400.0,  # daily — NULL-only, so a no-op once the
+                # library is hashed. This is what makes migration 0098's
+                # re-hash happen on its own: 0098 NULLs every phash, and
+                # without a scheduled refill the library would sit
+                # dedup-disabled until someone ran a deep scan (#4223).
+            },
             "train-heads-nightly": {
                 "task": "backend.app.tasks.ml.scheduled_train_heads",
                 "schedule": 86400.0,  # passive cadence; manual retrain stays available

backend/app/models/image_record.py

@@ -64,7 +64,11 @@ class ImageRecord(Base):
     # that 0001 also built was an exact duplicate of it — dropped in 0089
     # (#3301). Lookups by sha256 use the constraint's index.
     sha256: Mapped[str] = mapped_column(String(64), nullable=False)
-    phash: Mapped[str | None] = mapped_column(String(32), nullable=True, index=True)
+    # 64 hex chars = the 256-bit hash utils.phash emits at hash_size=16. Was
+    # String(32) (64-bit) until migration 0098; the narrow column was the
+    # reason for the undersized hash, and the undersized hash was collapsing
+    # variant artwork into one record (#4223).
+    phash: Mapped[str | None] = mapped_column(String(64), nullable=True, index=True)
     size_bytes: Mapped[int] = mapped_column(BigInteger, nullable=False)
     mime: Mapped[str] = mapped_column(String(64), nullable=False)
     width: Mapped[int | None] = mapped_column(Integer, nullable=True)

backend/app/models/import_settings.py

@@ -42,7 +42,12 @@ class ImportSettings(Base):
     single_color_threshold: Mapped[float] = mapped_column(Float, nullable=False, default=0.95, server_default="0.95")
     single_color_tolerance: Mapped[int] = mapped_column(Integer, nullable=False, default=30, server_default="30")
 
-    phash_threshold: Mapped[int] = mapped_column(Integer, nullable=False, default=10, server_default="10")
+    # Hamming distance over a 256-bit pHash (utils.phash, hash_size=16). The
+    # unit CHANGED in migration 0098 — it used to be bits out of 64 — so the
+    # old default of 10 is not this scale's 10, and 0098 resets every row.
+    # This is now the cheap PRE-FILTER: the aspect + pixel gates decide, which
+    # is what lets it be generous enough to catch a re-encoded rescale.
+    phash_threshold: Mapped[int] = mapped_column(Integer, nullable=False, default=24, server_default="24")
 
     # FC-3c downloader knobs
     download_rate_limit_seconds: Mapped[float] = mapped_column(

backend/app/services/gallery_service.py

@@ -322,7 +322,7 @@ def _gallery_images(rows, artists: dict[int, dict]) -> list[GalleryImage]:
     ]
 
 
-def _diversify_similar(src, rows, limit, *, dup_threshold=8, lam=0.40):
+def _diversify_similar(src, rows, limit, *, dup_threshold=32, lam=0.40):
     """Trim a nearest-cosine candidate pool down to `limit` diverse picks.
 
     1. pHash collapse: drop any candidate whose perceptual hash is within
@@ -338,6 +338,11 @@ def _diversify_similar(src, rows, limit, *, dup_threshold=8, lam=0.40):
     2026-07-01 — dropped 0.55→0.40, dup 6→8, paired with a wider pool in
     `similar()`).
 
+    `dup_threshold` counts Hamming bits, so it moved 8→32 when the pHash went
+    from 64 to 256 bits (#4223, migration 0098) — the same fraction of the
+    hash, i.e. the tuning the operator chose, unchanged. This collapse is
+    DISPLAY-only: it hides a near-dup from one rail, it never drops a record.
+
     Falls back to nearest-order (`rows[:limit]`) on any failure or a small pool.
     """
     if len(rows) <= 1:

backend/app/services/importer.py

@@ -39,7 +39,7 @@ from ..utils.paths import (
     hash_suffixed_name,
     safe_ext,
 )
-from ..utils.phash import compute_phash, find_similar
+from ..utils.phash import compute_phash, find_similar, fingerprint_path, fingerprints_match
 from ..utils.sidecar import find_sidecar, parse_sidecar
 from ..utils.slug import slugify
 from .archive_extractor import extract_archive, is_archive
@@ -234,6 +234,38 @@ class Importer:
                 (phash, width or 0, height or 0, image_id)
             )
 
+    def _pixel_confirmer(self, source: Path):
+        """Build `find_similar`'s gate-3 callback for an incoming file.
+
+        pHash proposes; this accepts. A candidate is a duplicate only if its
+        file really is the same picture as `source` at a different size —
+        which is the only merge the operator asked for (#4223). Everything
+        else (a missing file, an unreadable one, a deleted row) returns
+        False: the destructive outcomes here are dropping a download and
+        overwriting a kept file, so an unanswerable question must not read
+        as "yes".
+
+        Both sides' fingerprints are computed lazily and cached, so an
+        import that matches nothing costs no I/O at all and an archive
+        member that keeps hitting the same candidate pays for it once.
+        """
+        new_fp: list = []
+        cand_fps: dict[int, object] = {}
+
+        def confirm(candidate_id: int) -> bool:
+            if not new_fp:
+                new_fp.append(fingerprint_path(source))
+            if new_fp[0] is None:
+                return False
+            if candidate_id not in cand_fps:
+                rec = self.session.get(ImageRecord, candidate_id)
+                cand_fps[candidate_id] = (
+                    fingerprint_path(Path(rec.path)) if rec and rec.path else None
+                )
+            return fingerprints_match(new_fp[0], cand_fps[candidate_id])
+
+        return confirm
+
     def _get_or_create(self, stmt, factory):
         """Race-safe find-or-create. Run `stmt` (scalar_one_or_none); if a
         row exists, return it. Otherwise open a savepoint and INSERT
@@ -862,6 +894,7 @@ class Importer:
                 rel, match_id = find_similar(
                     phash, width or 0, height or 0,
                     candidates, self.settings.phash_threshold,
+                    confirm=self._pixel_confirmer(source),
                 )
                 if rel == "larger_exists":
                     # Enrich-on-duplicate (parity with attach_in_place).
@@ -1241,6 +1274,7 @@ class Importer:
                 rel, match_id = find_similar(
                     phash, width or 0, height or 0,
                     candidates, self.settings.phash_threshold,
+                    confirm=self._pixel_confirmer(path),
                 )
                 if rel == "larger_exists":
                     # Enrich-on-duplicate: link the near-dup's post to the

backend/app/tasks/maintenance.py

@@ -502,10 +502,16 @@ def prune_task_runs() -> dict:
     soft_time_limit=1800, time_limit=2100,
 )
 def backfill_phash() -> int:
-    """Recompute phash for stored images that have none (imported before
-    FC-2d-i+ii). Keyset-paginated by id (restart-safe), NULL-only fill,
-    idempotent. Videos legitimately keep phash NULL. A missing/unreadable
-    file is logged and left NULL — never fails the task."""
+    """Recompute phash for stored images that have none. Keyset-paginated by
+    id (restart-safe), NULL-only fill, idempotent. Videos legitimately keep
+    phash NULL. A missing/unreadable file is logged and left NULL — never
+    fails the task.
+
+    Two sources of NULLs: images imported before FC-2d-i+ii, and migration
+    0098, which cleared every phash so the library could be re-hashed at
+    hash_size=16 (#4223). The daily beat entry exists for the second — until
+    a row is refilled it takes no part in dedup, which is why this runs on a
+    schedule rather than waiting for a deep scan."""
     SessionLocal = _sync_session_factory()
     updated = 0
     last_id = 0

backend/app/utils/phash.py

@@ -1,57 +1,206 @@
 """Perceptual-hash dedup helpers (ported from ImageRepo).
 
-hash_size=8 -> 64-bit hash -> 16-hex-char string, which fits the existing
-ImageRecord.phash String(32) column (no image_record migration). IR uses
-hash_size=16; the deliberate FC deviation keeps the schema unchanged. The
-Hamming threshold is the operator-exposed dial (ImportSettings).
+hash_size=16 -> 256-bit hash -> 64-hex-char string, which is why
+`ImageRecord.phash` is String(64) (widened in migration 0098).
+
+## Why 16, after running at 8 from FC-2d until 2026-09-21
+
+`hash_size=8` keeps only the top-left 8x8 block of the DCT — 64 bits
+describing an image's coarse light/dark layout and nothing else. Variant
+artwork that shares a composition (same pose and framing, a different
+outfit / expression / overlay) collided OUTRIGHT at that size, so the
+operator's near-duplicate dial could not separate variants from rescales
+even at its floor: `phash_threshold=0` means "the same 64 bits", not "the
+same image", and packs of 15 variants were landing as 3 records
+(issue #4223). IR always used 16; FC's deviation to 8 was made to fit the
+old String(32) column without a migration — a schema convenience that cost
+the operator artwork.
+
+## The hash no longer decides a merge on its own
+
+Dropping a file or superseding one is destructive, so `find_similar` now
+runs three gates, cheapest first:
+
+  1. Hamming distance within the operator's `phash_threshold` — the cheap
+     indexed pre-filter that PROPOSES candidates.
+  2. Aspect ratio within ASPECT_TOL. A crop or a re-canvas is not a
+     rescale, and this is the same identity test the tier-1 video near-dup
+     path already uses (`_VIDEO_DUP_ASPECT_TOL`).
+  3. A pixel-level confirm on the candidate's actual file, supplied by the
+     caller (the importer opens the files; this module stays I/O-free apart
+     from `fingerprint_path`).
+
+Because the confirm is what ACCEPTS a match, the threshold can stay
+generous enough to tolerate a re-encode without putting variants at risk.
+
+Every gate fails CLOSED: unknown dimensions, an unreadable candidate, a
+hash that won't parse — all mean "not a duplicate". The two failure
+directions are not symmetrical. Too strict keeps a redundant lower-res copy,
+which the operator can see and delete; too loose deletes artwork that only
+a re-walk of the source can bring back.
 """
 
 import imagehash
+from PIL import Image, ImageChops, ImageStat
 
-HASH_SIZE = 8
+HASH_SIZE = 16
+
+# Gate 2. Matching `importer._VIDEO_DUP_ASPECT_TOL` — a rescale preserves
+# aspect ratio to within rounding, so this only has to absorb off-by-one
+# pixel dimensions.
+ASPECT_TOL = 0.02
+
+# Gate 3. Both images are reduced to one FINGERPRINT_SIZE-square grayscale
+# thumbnail and compared directly, which is the question actually being
+# asked: "are these the same picture at different resolutions?"
+#
+# Two criteria, because they catch different things. MEAN absolute
+# difference catches a global change (a recolour, a filter, a different
+# shading pass) that leaves the composition intact. The CHANGED-PIXEL
+# fraction catches a LOCAL one — an added overlay, a different expression,
+# an alternate outfit on part of the figure — which a mean over 4096 pixels
+# would otherwise dilute into noise.
+#
+# Tuned to fail closed (see the module docstring). A true rescale lands near
+# zero on both; these ceilings sit well above that and well below a variant.
+FINGERPRINT_SIZE = 64
+FINGERPRINT_MAX_MEAN_DIFF = 6.0
+FINGERPRINT_CHANGED_LEVEL = 64
+FINGERPRINT_MAX_CHANGED_FRACTION = 0.02
+
+
+def _seek_first_frame(pil_image) -> None:
+    """Animated images (multi-frame WebP/GIF/APNG) are hashed and
+    fingerprinted on frame 0 — the conventional choice for animated
+    content, and the one that keeps PIL from iterating every frame."""
+    if getattr(pil_image, "is_animated", False):
+        try:
+            pil_image.seek(0)
+        except Exception:
+            pass
 
 
 def compute_phash(pil_image) -> str | None:
     """Perceptual hash of an opened PIL image, as a hex string. None on any
     failure (videos/unreadable/non-image).
 
-    For animated images (multi-frame WebP/GIF/APNG), explicitly seek to
-    frame 0 first. Without this, some PIL operations downstream of
-    imagehash.phash (convert("L"), resize) can iterate all frames and
-    blow past Celery's hard time limit on large animations
-    (operator-flagged 2026-05-26 against animated WebPs). The pHash of
-    frame 0 is the conventional choice for animated content.
+    Frame 0 for animated images: without the seek, PIL operations
+    downstream of imagehash.phash (convert("L"), resize) can iterate all
+    frames and blow past Celery's hard time limit on large animations
+    (operator-flagged 2026-05-26 against animated WebPs).
     """
     try:
-        if getattr(pil_image, "is_animated", False):
-            try:
-                pil_image.seek(0)
-            except Exception:
-                pass
+        _seek_first_frame(pil_image)
         return str(imagehash.phash(pil_image, hash_size=HASH_SIZE))
     except Exception:
         return None
 
 
+def fingerprint(pil_image):
+    """A small grayscale thumbnail of an opened PIL image, for gate 3.
+
+    Returns a detached PIL image (so the caller may close the original) or
+    None on any failure. PIL-only on purpose: numpy is an imagehash
+    transitive dependency, not a declared one for this path.
+    """
+    try:
+        _seek_first_frame(pil_image)
+        return pil_image.convert("L").resize(
+            (FINGERPRINT_SIZE, FINGERPRINT_SIZE), Image.LANCZOS
+        )
+    except Exception:
+        return None
+
+
+def fingerprint_path(path) -> "Image.Image | None":
+    """`fingerprint` for a file on disk. None if it cannot be read — which
+    the gate treats as "not a duplicate"."""
+    try:
+        with Image.open(path) as im:
+            return fingerprint(im)
+    except Exception:
+        return None
+
+
+def fingerprints_match(
+    a, b,
+    *,
+    max_mean_diff: float = FINGERPRINT_MAX_MEAN_DIFF,
+    changed_level: int = FINGERPRINT_CHANGED_LEVEL,
+    max_changed_fraction: float = FINGERPRINT_MAX_CHANGED_FRACTION,
+) -> bool:
+    """True when two fingerprints are the same picture: no large global
+    drift AND no meaningful local region that differs. False on any
+    failure."""
+    if a is None or b is None:
+        return False
+    try:
+        diff = ImageChops.difference(a, b)
+        if ImageStat.Stat(diff).mean[0] > max_mean_diff:
+            return False
+        hist = diff.histogram()
+        total = sum(hist)
+        if not total:
+            return False
+        changed = sum(hist[changed_level:])
+        return (changed / total) <= max_changed_fraction
+    except Exception:
+        return False
+
+
+def aspect_matches(
+    width: int | None, height: int | None,
+    cand_width: int | None, cand_height: int | None,
+    *, tol: float = ASPECT_TOL,
+) -> bool:
+    """Gate 2. False when either side's dimensions are unknown — an
+    unmeasurable candidate is not a proven duplicate."""
+    if not width or not height or not cand_width or not cand_height:
+        return False
+    a, b = width / height, cand_width / cand_height
+    if a <= 0 or b <= 0:
+        return False
+    return abs(a - b) / max(a, b) <= tol
+
+
 def find_similar(
     phash_hex: str,
     width: int,
     height: int,
     candidates: list[tuple[str, int, int, int]],
     threshold: int,
+    *,
+    confirm=None,
 ) -> tuple[str, int | None]:
     """candidates: (phash_hex, width, height, image_id). Returns one of
     ("none", None) / ("larger_exists", id) / ("smaller_exists", id).
-    First qualifying candidate wins (IR loop order)."""
+    First candidate to pass EVERY gate wins (IR loop order).
+
+    `confirm` is gate 3: an optional callable(image_id) -> bool, called only
+    for a candidate that already passed the hash and aspect gates, and
+    expected to compare the two files' pixels. A rejected candidate does not
+    end the search — the loop moves on, so a false pre-filter hit cannot
+    mask a real duplicate further down the list. Omitting it leaves the
+    hash+aspect behaviour, which is what the unit tests exercise.
+    """
     new_h = imagehash.hex_to_hash(phash_hex)
     for cand_hex, cw, ch, cid in candidates:
         try:
             dist = new_h - imagehash.hex_to_hash(cand_hex)
         except Exception:
+            # Includes the mismatched-length case while a library re-hash
+            # (migration 0098) is still in flight: an old 64-bit hash cannot
+            # be compared to a new 256-bit one, and skipping it degrades to
+            # "no dedup yet" rather than to a wrong merge.
             continue
-        if dist <= threshold:
-            if cw >= width and ch >= height:
-                return ("larger_exists", cid)
-            if width > cw or height > ch:
-                return ("smaller_exists", cid)
+        if dist > threshold:
+            continue
+        if not aspect_matches(width, height, cw, ch):
+            continue
+        if confirm is not None and not confirm(cid):
+            continue
+        if cw >= width and ch >= height:
+            return ("larger_exists", cid)
+        if width > cw or height > ch:
+            return ("smaller_exists", cid)
     return ("none", None)