bvandeusenandClaude Opus 5
a05cb66635
CI / lint (push) Failing after 3s
CI / extension-version (push) Successful in 3s
Build images / sign-extension (push) Successful in 4s
Build images / build-agent (push) Successful in 7s
CI / frontend-build (push) Successful in 30s
CI / backend-lint-and-test (push) Successful in 1m8s
Build images / build-web (push) Successful in 1m28s
Build images / smoke-web (push) Skipped
CI / integration (push) Successful in 2m51s
Build images / build-ml (push) Successful in 2m59s
Build images / promote (push) Skipped
The operator reported a 15-image variant pack landing as 3 records, then reported variants STILL being dropped with phash_threshold at 0 — the floor of the dial. No setting could have fixed it: at hash_size=8 a pHash is 64 bits of coarse light/dark layout, so two variants sharing a composition produce the SAME bits. Distance 0 meant "identical hash", not "identical image", and the dial was simultaneously too coarse to keep variants and too tight to catch a re-encoded rescale. The hash no longer decides a merge on its own. find_similar now runs three gates, cheapest first: the threshold proposes candidates, aspect ratio (ASPECT_TOL, matching the tier-1 video path) rejects crops and re-canvases, and a pixel-level confirm on the two files accepts. Every gate fails closed — unknown dimensions, an unreadable candidate, a hash of the wrong width all mean "not a duplicate", because too strict keeps a redundant copy the operator can see while too loose deletes artwork only a source re-walk returns. - utils/phash.py: HASH_SIZE 8 -> 16 (256-bit, what ImageRepo always used); aspect_matches, fingerprint/fingerprint_path/fingerprints_match (PIL-only, mean drift + changed-pixel fraction), find_similar gains `confirm`. - importer: _pixel_confirmer supplies gate 3 on both dedup sites, lazily and cached, so a non-matching import costs no extra I/O. - 0098: widens image_record.phash to 64 chars and NULLs every value — a stored 64-bit hash cannot be compared to a 256-bit one, and backfill_phash is NULL-only, keyset-paginated and now on the daily beat, so the library re-hashes itself. Dedup degrades to sha256 until it finishes. - phash_threshold counts bits and the denominator went 64 -> 256, so the setting is reset to the new default of 24 (there is no honest carry-over) and the slider is rescaled to 0-64. - gallery_service dup_threshold 8 -> 32: the same fraction of the hash, so the Explore rail keeps the variance the operator tuned in on 2026-07-01. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR
91 lines
3.5 KiB
Python
91 lines
3.5 KiB
Python
"""Widen image_record.phash to 256-bit and re-hash the library (issue #4223).
|
|
|
|
The operator reported a 15-image variant pack landing as 3 records, and then
|
|
that variants were STILL being dropped with `phash_threshold` at 0. Zero was
|
|
already the floor of the dial, so no setting could have fixed it: at
|
|
`hash_size=8` a pHash is 64 bits of coarse light/dark layout, and variant
|
|
artwork sharing a composition produces the SAME 64 bits. Distance 0 meant
|
|
"identical hash", never "identical image".
|
|
|
|
`utils/phash.py` moves to `hash_size=16` (256 bits, what ImageRepo always
|
|
used) and adds an aspect-ratio gate plus a pixel-level confirm, so a merge is
|
|
accepted on the files rather than on the hash.
|
|
|
|
## Why this NULLs every phash
|
|
|
|
Widening the column does not correct the values already in it. Every stored
|
|
hash is a 64-bit hash of an image the app will now hash at 256 bits, and the
|
|
two cannot be compared — `find_similar` skips a mismatched-length candidate
|
|
rather than guessing, so leaving them would silently mean "no dedup, forever,
|
|
for everything imported before today". NULL is the state `backfill_phash`
|
|
already knows how to repair: it is NULL-only, keyset-paginated and
|
|
restart-safe, and the beat schedule runs it daily.
|
|
|
|
Until that backfill finishes, image dedup degrades to sha256 only —
|
|
duplicates may be kept. That is the safe direction, and the only one
|
|
available: the alternative is comparing hashes of different widths, which
|
|
would drop artwork. NOTHING here deletes or supersedes a file.
|
|
|
|
## Why the threshold is reset rather than carried over
|
|
|
|
`phash_threshold` counts bits, and the denominator went from 64 to 256. The
|
|
stored number would keep its value while meaning something four times
|
|
tighter. There is no honest carry-over, so every row goes to the new default
|
|
of 24 — including the operator's 0, which was a workaround for the bug this
|
|
revision fixes.
|
|
|
|
Revision ID: 0098
|
|
Revises: 0097
|
|
Create Date: 2026-09-21
|
|
|
|
"""
|
|
from typing import Sequence, Union
|
|
|
|
import sqlalchemy as sa
|
|
from alembic import op
|
|
|
|
revision: str = "0098"
|
|
down_revision: Union[str, None] = "0097"
|
|
branch_labels: Union[str, Sequence[str], None] = None
|
|
depends_on: Union[str, Sequence[str], None] = None
|
|
|
|
|
|
def upgrade() -> None:
|
|
# varchar(32) -> varchar(64): widening a length limit is a catalog-only
|
|
# change in Postgres, so this does not rewrite the table or its index.
|
|
op.alter_column(
|
|
"image_record", "phash",
|
|
existing_type=sa.String(32),
|
|
type_=sa.String(64),
|
|
existing_nullable=True,
|
|
)
|
|
op.execute("UPDATE image_record SET phash = NULL WHERE phash IS NOT NULL")
|
|
op.alter_column(
|
|
"import_settings", "phash_threshold",
|
|
existing_type=sa.Integer(),
|
|
server_default="24",
|
|
existing_nullable=False,
|
|
)
|
|
op.execute("UPDATE import_settings SET phash_threshold = 24")
|
|
|
|
|
|
def downgrade() -> None:
|
|
# The 64-bit hashes this replaced are gone, and a 64-char value does not
|
|
# fit back into varchar(32) — so the column is cleared again on the way
|
|
# down and left for backfill_phash to refill at whatever HASH_SIZE the
|
|
# code is running. Rule #22: no legacy to preserve.
|
|
op.execute("UPDATE image_record SET phash = NULL WHERE phash IS NOT NULL")
|
|
op.alter_column(
|
|
"image_record", "phash",
|
|
existing_type=sa.String(64),
|
|
type_=sa.String(32),
|
|
existing_nullable=True,
|
|
)
|
|
op.alter_column(
|
|
"import_settings", "phash_threshold",
|
|
existing_type=sa.Integer(),
|
|
server_default="10",
|
|
existing_nullable=False,
|
|
)
|
|
op.execute("UPDATE import_settings SET phash_threshold = 10")
|