feat(import): Tier-1 video near-dup by duration+aspect (#871)
CI / frontend-build (push) Successful in 21s
CI / backend-lint-and-test (push) Successful in 28s
CI / lint (push) Successful in 3s
CI / integration (push) Successful in 3m13s

Videos deduped on sha256 only (pHash is images-only), so a different encode/remux
of the same clip imported as a distinct record — the "same video from multiple
sources" clutter surfaced by #859.

Tier-1 metadata fingerprint: identity = container duration (±1.0s) + matching
aspect ratio, scoped to the same artist; quality axis = pixel dimensions (mirrors
image pHash: larger_exists→skip+link, smaller_exists→supersede). Codec/bitrate
are deliberately NOT part of identity (the point is matching across re-encodes).
Tight tolerances because a wrong video merge is destructive.

- image_record.duration_seconds (Float, nullable; migration 0052). NULL for images.
- safe_probe.probe_video also reads format=duration (one extra ffprobe field on the
  call that already runs); ProbeResult.duration.
- _find_similar_video(duration,w,h,artist) shared by both import pipelines.
- _import_media (filesystem/archive path): captures duration, video near-dup
  branch, persists duration.
- attach_in_place (download path — handles #859's videos, previously didn't probe
  video at all): best-effort probe for dims+duration (LENIENT — never newly rejects
  a downloaded video on probe failure), video near-dup branch, persists duration.
- _supersede carries duration onto the kept row.

Reuses SkipReason.duplicate_phash so the existing download/external dup-cleanup
(path-safe unlink, #859) applies unchanged. Tests: skip-smaller, supersede-larger
(+ duration adopted), and distinct-durations-not-merged (false-merge guard).

Follow-up (Phase 2, #871): a backfill to re-probe NULL-duration existing videos so
the current library participates in dedup; retroactive merge of existing dups is a
separate destructive maintenance action.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
2026-06-15 22:17:36 -04:00
parent b48ba60830
commit f154603811
5 changed files with 267 additions and 6 deletions
+5
View File
@@ -13,6 +13,7 @@ from sqlalchemy import (
BigInteger,
DateTime,
Enum,
Float,
ForeignKey,
Integer,
String,
@@ -39,6 +40,10 @@ class ImageRecord(Base):
mime: Mapped[str] = mapped_column(String(64), nullable=False)
width: Mapped[int | None] = mapped_column(Integer, nullable=True)
height: Mapped[int | None] = mapped_column(Integer, nullable=True)
# Video container duration (seconds); NULL for images. The Tier-1 video
# near-dup key (#871): two videos of the same artist with matching duration
# (+ aspect) are the same content across re-encodes — dedup like image pHash.
duration_seconds: Mapped[float | None] = mapped_column(Float, nullable=True)
# Integrity verification status. FC-2e populates this; FC-2a leaves rows at 'unknown'.
# Values: 'unknown' (default), 'ok', 'corrupt', 'failed_verification'.