feat(import): Tier-1 video near-dup by duration+aspect (#871)
Videos deduped on sha256 only (pHash is images-only), so a different encode/remux of the same clip imported as a distinct record — the "same video from multiple sources" clutter surfaced by #859. Tier-1 metadata fingerprint: identity = container duration (±1.0s) + matching aspect ratio, scoped to the same artist; quality axis = pixel dimensions (mirrors image pHash: larger_exists→skip+link, smaller_exists→supersede). Codec/bitrate are deliberately NOT part of identity (the point is matching across re-encodes). Tight tolerances because a wrong video merge is destructive. - image_record.duration_seconds (Float, nullable; migration 0052). NULL for images. - safe_probe.probe_video also reads format=duration (one extra ffprobe field on the call that already runs); ProbeResult.duration. - _find_similar_video(duration,w,h,artist) shared by both import pipelines. - _import_media (filesystem/archive path): captures duration, video near-dup branch, persists duration. - attach_in_place (download path — handles #859's videos, previously didn't probe video at all): best-effort probe for dims+duration (LENIENT — never newly rejects a downloaded video on probe failure), video near-dup branch, persists duration. - _supersede carries duration onto the kept row. Reuses SkipReason.duplicate_phash so the existing download/external dup-cleanup (path-safe unlink, #859) applies unchanged. Tests: skip-smaller, supersede-larger (+ duration adopted), and distinct-durations-not-merged (false-merge guard). Follow-up (Phase 2, #871): a backfill to re-probe NULL-duration existing videos so the current library participates in dedup; retroactive merge of existing dups is a separate destructive maintenance action. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
@@ -60,6 +60,9 @@ class ProbeResult:
|
||||
reason: str | None = None
|
||||
width: int | None = None
|
||||
height: int | None = None
|
||||
# Container duration in seconds (videos only) — the Tier-1 video near-dup
|
||||
# key (#871). None when not a video / ffprobe didn't report it.
|
||||
duration: float | None = None
|
||||
|
||||
|
||||
def probe_video(path: Path, *, timeout: float = VIDEO_PROBE_TIMEOUT_SECONDS) -> ProbeResult:
|
||||
@@ -69,7 +72,10 @@ def probe_video(path: Path, *, timeout: float = VIDEO_PROBE_TIMEOUT_SECONDS) ->
|
||||
[
|
||||
"ffprobe", "-v", "error",
|
||||
"-select_streams", "v:0",
|
||||
"-show_entries", "stream=width,height",
|
||||
# format=duration is the reliable container duration (stream
|
||||
# duration is often N/A for remuxed/streamed mp4) — the #871
|
||||
# video near-dup key.
|
||||
"-show_entries", "stream=width,height:format=duration",
|
||||
"-of", "json", str(path),
|
||||
],
|
||||
capture_output=True, text=True, timeout=timeout,
|
||||
@@ -87,13 +93,21 @@ def probe_video(path: Path, *, timeout: float = VIDEO_PROBE_TIMEOUT_SECONDS) ->
|
||||
reason=f"ffprobe rejected the file: {out.stderr.strip()[:200]}",
|
||||
)
|
||||
try:
|
||||
streams = (json.loads(out.stdout) or {}).get("streams") or []
|
||||
parsed = json.loads(out.stdout) or {}
|
||||
except json.JSONDecodeError as exc:
|
||||
return ProbeResult(ok=False, crashed=False, reason=f"ffprobe output parse failed: {exc}")
|
||||
streams = parsed.get("streams") or []
|
||||
if not streams:
|
||||
return ProbeResult(ok=False, crashed=False, reason="no decodable video stream")
|
||||
duration = None
|
||||
raw_dur = (parsed.get("format") or {}).get("duration")
|
||||
try:
|
||||
duration = float(raw_dur) if raw_dur is not None else None
|
||||
except (TypeError, ValueError):
|
||||
duration = None
|
||||
return ProbeResult(
|
||||
ok=True, width=streams[0].get("width"), height=streams[0].get("height"),
|
||||
duration=duration,
|
||||
)
|
||||
|
||||
|
||||
|
||||
Reference in New Issue
Block a user