feat(import): Tier-1 video near-dup by duration+aspect (#871)
Videos deduped on sha256 only (pHash is images-only), so a different encode/remux of the same clip imported as a distinct record — the "same video from multiple sources" clutter surfaced by #859. Tier-1 metadata fingerprint: identity = container duration (±1.0s) + matching aspect ratio, scoped to the same artist; quality axis = pixel dimensions (mirrors image pHash: larger_exists→skip+link, smaller_exists→supersede). Codec/bitrate are deliberately NOT part of identity (the point is matching across re-encodes). Tight tolerances because a wrong video merge is destructive. - image_record.duration_seconds (Float, nullable; migration 0052). NULL for images. - safe_probe.probe_video also reads format=duration (one extra ffprobe field on the call that already runs); ProbeResult.duration. - _find_similar_video(duration,w,h,artist) shared by both import pipelines. - _import_media (filesystem/archive path): captures duration, video near-dup branch, persists duration. - attach_in_place (download path — handles #859's videos, previously didn't probe video at all): best-effort probe for dims+duration (LENIENT — never newly rejects a downloaded video on probe failure), video near-dup branch, persists duration. - _supersede carries duration onto the kept row. Reuses SkipReason.duplicate_phash so the existing download/external dup-cleanup (path-safe unlink, #859) applies unchanged. Tests: skip-smaller, supersede-larger (+ duration adopted), and distinct-durations-not-merged (false-merge guard). Follow-up (Phase 2, #871): a backfill to re-probe NULL-duration existing videos so the current library participates in dedup; retroactive merge of existing dups is a separate destructive maintenance action. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,32 @@
|
||||
"""image_record: duration_seconds (Tier-1 video near-dup key)
|
||||
|
||||
#871. Videos previously deduped on sha256 only (pHash is images-only), so a
|
||||
different encode/remux of the same video imported as a distinct record. Persist
|
||||
the container duration so the importer can treat same-artist videos with matching
|
||||
duration (+ aspect ratio) as the same content and dedup/supersede like images.
|
||||
NULL for images and for video rows imported before this column existed (a
|
||||
backfill re-probes those so they participate in dedup).
|
||||
|
||||
Revision ID: 0052
|
||||
Revises: 0051
|
||||
Create Date: 2026-06-16
|
||||
"""
|
||||
from typing import Sequence, Union
|
||||
|
||||
import sqlalchemy as sa
|
||||
from alembic import op
|
||||
|
||||
revision: str = "0052"
|
||||
down_revision: Union[str, None] = "0051"
|
||||
branch_labels: Union[str, Sequence[str], None] = None
|
||||
depends_on: Union[str, Sequence[str], None] = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
op.add_column(
|
||||
"image_record", sa.Column("duration_seconds", sa.Float(), nullable=True)
|
||||
)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
op.drop_column("image_record", "duration_seconds")
|
||||
Reference in New Issue
Block a user