feat(import): Tier-1 video near-dup by duration+aspect (#871)
CI / frontend-build (push) Successful in 21s
CI / backend-lint-and-test (push) Successful in 28s
CI / lint (push) Successful in 3s
CI / integration (push) Successful in 3m13s

Videos deduped on sha256 only (pHash is images-only), so a different encode/remux
of the same clip imported as a distinct record — the "same video from multiple
sources" clutter surfaced by #859.

Tier-1 metadata fingerprint: identity = container duration (±1.0s) + matching
aspect ratio, scoped to the same artist; quality axis = pixel dimensions (mirrors
image pHash: larger_exists→skip+link, smaller_exists→supersede). Codec/bitrate
are deliberately NOT part of identity (the point is matching across re-encodes).
Tight tolerances because a wrong video merge is destructive.

- image_record.duration_seconds (Float, nullable; migration 0052). NULL for images.
- safe_probe.probe_video also reads format=duration (one extra ffprobe field on the
  call that already runs); ProbeResult.duration.
- _find_similar_video(duration,w,h,artist) shared by both import pipelines.
- _import_media (filesystem/archive path): captures duration, video near-dup
  branch, persists duration.
- attach_in_place (download path — handles #859's videos, previously didn't probe
  video at all): best-effort probe for dims+duration (LENIENT — never newly rejects
  a downloaded video on probe failure), video near-dup branch, persists duration.
- _supersede carries duration onto the kept row.

Reuses SkipReason.duplicate_phash so the existing download/external dup-cleanup
(path-safe unlink, #859) applies unchanged. Tests: skip-smaller, supersede-larger
(+ duration adopted), and distinct-durations-not-merged (false-merge guard).

Follow-up (Phase 2, #871): a backfill to re-probe NULL-duration existing videos so
the current library participates in dedup; retroactive merge of existing dups is a
separate destructive maintenance action.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
2026-06-15 22:17:36 -04:00
parent b48ba60830
commit f154603811
5 changed files with 267 additions and 6 deletions
+16 -2
View File
@@ -60,6 +60,9 @@ class ProbeResult:
reason: str | None = None
width: int | None = None
height: int | None = None
# Container duration in seconds (videos only) — the Tier-1 video near-dup
# key (#871). None when not a video / ffprobe didn't report it.
duration: float | None = None
def probe_video(path: Path, *, timeout: float = VIDEO_PROBE_TIMEOUT_SECONDS) -> ProbeResult:
@@ -69,7 +72,10 @@ def probe_video(path: Path, *, timeout: float = VIDEO_PROBE_TIMEOUT_SECONDS) ->
[
"ffprobe", "-v", "error",
"-select_streams", "v:0",
"-show_entries", "stream=width,height",
# format=duration is the reliable container duration (stream
# duration is often N/A for remuxed/streamed mp4) — the #871
# video near-dup key.
"-show_entries", "stream=width,height:format=duration",
"-of", "json", str(path),
],
capture_output=True, text=True, timeout=timeout,
@@ -87,13 +93,21 @@ def probe_video(path: Path, *, timeout: float = VIDEO_PROBE_TIMEOUT_SECONDS) ->
reason=f"ffprobe rejected the file: {out.stderr.strip()[:200]}",
)
try:
streams = (json.loads(out.stdout) or {}).get("streams") or []
parsed = json.loads(out.stdout) or {}
except json.JSONDecodeError as exc:
return ProbeResult(ok=False, crashed=False, reason=f"ffprobe output parse failed: {exc}")
streams = parsed.get("streams") or []
if not streams:
return ProbeResult(ok=False, crashed=False, reason="no decodable video stream")
duration = None
raw_dur = (parsed.get("format") or {}).get("duration")
try:
duration = float(raw_dur) if raw_dur is not None else None
except (TypeError, ValueError):
duration = None
return ProbeResult(
ok=True, width=streams[0].get("width"), height=streams[0].get("height"),
duration=duration,
)