fix: variant artwork was dropped as a near-duplicate even at threshold 0 (4223)
CI / lint (push) Failing after 3s
CI / extension-version (push) Successful in 3s
Build images / sign-extension (push) Successful in 4s
Build images / build-agent (push) Successful in 7s
CI / frontend-build (push) Successful in 30s
CI / backend-lint-and-test (push) Successful in 1m8s
Build images / build-web (push) Successful in 1m28s
Build images / smoke-web (push) Skipped
CI / integration (push) Successful in 2m51s
Build images / build-ml (push) Successful in 2m59s
Build images / promote (push) Skipped

The operator reported a 15-image variant pack landing as 3 records, then
reported variants STILL being dropped with phash_threshold at 0 — the floor
of the dial. No setting could have fixed it: at hash_size=8 a pHash is 64
bits of coarse light/dark layout, so two variants sharing a composition
produce the SAME bits. Distance 0 meant "identical hash", not "identical
image", and the dial was simultaneously too coarse to keep variants and too
tight to catch a re-encoded rescale.

The hash no longer decides a merge on its own. find_similar now runs three
gates, cheapest first: the threshold proposes candidates, aspect ratio
(ASPECT_TOL, matching the tier-1 video path) rejects crops and re-canvases,
and a pixel-level confirm on the two files accepts. Every gate fails closed —
unknown dimensions, an unreadable candidate, a hash of the wrong width all
mean "not a duplicate", because too strict keeps a redundant copy the
operator can see while too loose deletes artwork only a source re-walk
returns.

- utils/phash.py: HASH_SIZE 8 -> 16 (256-bit, what ImageRepo always used);
  aspect_matches, fingerprint/fingerprint_path/fingerprints_match (PIL-only,
  mean drift + changed-pixel fraction), find_similar gains `confirm`.
- importer: _pixel_confirmer supplies gate 3 on both dedup sites, lazily and
  cached, so a non-matching import costs no extra I/O.
- 0098: widens image_record.phash to 64 chars and NULLs every value — a
  stored 64-bit hash cannot be compared to a 256-bit one, and backfill_phash
  is NULL-only, keyset-paginated and now on the daily beat, so the library
  re-hashes itself. Dedup degrades to sha256 until it finishes.
- phash_threshold counts bits and the denominator went 64 -> 256, so the
  setting is reset to the new default of 24 (there is no honest carry-over)
  and the slider is rescaled to 0-64.
- gallery_service dup_threshold 8 -> 32: the same fraction of the hash, so
  the Explore rail keeps the variance the operator tuned in on 2026-07-01.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR
This commit is contained in:
2026-09-21 00:16:14 -04:00
co-authored by Claude Opus 5
parent 3fe9d0a612
commit a05cb66635
13 changed files with 501 additions and 45 deletions
@@ -10,16 +10,20 @@
<div class="fc-phash">
<div class="fc-phash__title">Near-duplicate sensitivity</div>
<div class="fc-help mb-1">
How aggressively imports merge look-alike images (perceptual-hash
distance). <strong>Lower it if edits/variants of the same image are
being dropped as duplicates;</strong> raise it to collapse more
look-alikes. Applies to new imports.
How wide a net imports cast when looking for the same image at a
different resolution (perceptual-hash distance, out of 256 bits).
A match is only merged if the two files also share an aspect ratio
<em>and</em> their pixels agree, so variant artwork a different
outfit, expression or overlay on the same pose is kept even at a
generous setting. <strong>Raise it if higher-resolution re-uploads
are landing as separate copies;</strong> lower it to merge less.
Applies to new imports.
</div>
<v-row no-gutters class="align-center">
<v-col cols="12" sm="9">
<v-slider
v-model="local.phash_threshold"
:min="0" :max="16" :step="1"
:min="0" :max="64" :step="1"
:ticks="PHASH_TICKS" show-ticks="always" tick-size="4"
thumb-label color="accent" hide-details
class="fc-phash__slider"
@@ -139,8 +143,10 @@ const store = useImportStore()
// instead of relying on the old tab's mount hook.
onMounted(() => { if (!store.settings) store.loadSettings() })
// Labelled stops so the less-initiated get the gist without knowing what a
// Hamming distance is. 0 = byte-for-byte only; 10 = the shipped default.
const PHASH_TICKS = { 0: 'Exact', 4: 'Strict', 10: 'Default', 16: 'Loose' }
// Hamming distance is. 0 = an identical hash only; 24 = the shipped default.
// Bits out of 256 (utils/phash hash_size=16). The scale changed with
// migration 0098 — these are NOT the old 0-16 stops renamed (#4223).
const PHASH_TICKS = { 0: 'Exact', 12: 'Strict', 24: 'Default', 48: 'Loose' }
// Downloader + schedule-defaults fields moved to
// /subscriptions?tab=settings (operator decision 2026-05-27). This form
// now only owns image-import filters.
@@ -148,7 +154,7 @@ const local = reactive({
min_width: 0, min_height: 0,
skip_transparent: false, transparency_threshold: 0.9,
skip_single_color: false, single_color_threshold: 0.95,
phash_threshold: 10,
phash_threshold: 24,
wip_title_tagging_enabled: true,
wip_soft_title_tagging_enabled: false,
})