Tier-1 video dedup: import-time + retroactive cleanup (#871) #110

Merged

bvandeusen merged 2 commits from dev into main 2026-06-16 08:55:39 -04:00

Conversation 0 Commits 2 Files Changed 12

+768 -6

bvandeusen commented 2026-06-16 08:55:32 -04:00

Owner

Copy Link

Copy Source

Merges #871 (CI-green, tip 41652db) to main / :latest.

• Import-time dedup — same-artist videos matching on duration (±1s) + aspect ratio are treated as the same content across re-encodes; the higher-res copy is kept (mirrors image pHash supersede). Wired into both import pipelines, including the download path that handles the #859 multi-source case. Adds image_record.duration_seconds (migration 0052) + format=duration to the existing ffprobe call.
• Retroactive cleanup — Settings → Maintenance → Deduplicate videos: Preview (re-probes old videos, reports groups/redundant/reclaimable bytes, non-destructive) then Apply (re-points each post to the keeper, then deletes the redundant copies + files). For cleaning the existing #859 duplicate clutter.

One new migration (0052). No new runtime deps. After merge: force a real re-pull of :latest, then the new "Deduplicate videos" action (and the still-pending #718 "re-extract archives") are available under Settings → Maintenance.

🤖 Generated with Claude Code

Merges #871 (CI-green, tip `41652db`) to main / `:latest`. - **Import-time dedup** — same-artist videos matching on duration (±1s) + aspect ratio are treated as the same content across re-encodes; the higher-res copy is kept (mirrors image pHash supersede). Wired into both import pipelines, including the download path that handles the #859 multi-source case. Adds `image_record.duration_seconds` (migration **0052**) + `format=duration` to the existing ffprobe call. - **Retroactive cleanup** — `Settings → Maintenance → Deduplicate videos`: Preview (re-probes old videos, reports groups/redundant/reclaimable bytes, non-destructive) then Apply (re-points each post to the keeper, then deletes the redundant copies + files). For cleaning the existing #859 duplicate clutter. One new migration (0052). No new runtime deps. **After merge:** force a real re-pull of `:latest`, then the new "Deduplicate videos" action (and the still-pending #718 "re-extract archives") are available under Settings → Maintenance. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

bvandeusen added 2 commits 2026-06-16 08:55:32 -04:00

feat(import): Tier-1 video near-dup by duration+aspect (#871) ... f154603811

Videos deduped on sha256 only (pHash is images-only), so a different encode/remux
of the same clip imported as a distinct record — the "same video from multiple
sources" clutter surfaced by #859.

Tier-1 metadata fingerprint: identity = container duration (±1.0s) + matching
aspect ratio, scoped to the same artist; quality axis = pixel dimensions (mirrors
image pHash: larger_exists→skip+link, smaller_exists→supersede). Codec/bitrate
are deliberately NOT part of identity (the point is matching across re-encodes).
Tight tolerances because a wrong video merge is destructive.

- image_record.duration_seconds (Float, nullable; migration 0052). NULL for images.
- safe_probe.probe_video also reads format=duration (one extra ffprobe field on the
  call that already runs); ProbeResult.duration.
- _find_similar_video(duration,w,h,artist) shared by both import pipelines.
- _import_media (filesystem/archive path): captures duration, video near-dup
  branch, persists duration.
- attach_in_place (download path — handles #859's videos, previously didn't probe
  video at all): best-effort probe for dims+duration (LENIENT — never newly rejects
  a downloaded video on probe failure), video near-dup branch, persists duration.
- _supersede carries duration onto the kept row.

Reuses SkipReason.duplicate_phash so the existing download/external dup-cleanup
(path-safe unlink, #859) applies unchanged. Tests: skip-smaller, supersede-larger
(+ duration adopted), and distinct-durations-not-merged (false-merge guard).

Follow-up (Phase 2, #871): a backfill to re-probe NULL-duration existing videos so
the current library participates in dedup; retroactive merge of existing dups is a
separate destructive maintenance action.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

feat(maintenance): retroactive video-dedup action — preview + apply (#871) ... 41652db20f

Phase 2 of #871: clean up the duplicate videos already in the library (the #859
"same video from multiple sources" clutter). Import-time dedup (Phase 1) only
prevents NEW dups; this is the operator-triggered cleanup of existing ones.

cleanup_service.dedup_videos(dry_run):
- backfill_video_durations: re-probe NULL-duration videos (pre-#871 rows) so the
  existing library participates; idempotent (only NULL rows), writes a negative
  sentinel for un-probeable files so they're neither re-probed forever nor matched.
- find_video_dup_groups: cluster same-artist videos by duration (±tol) + aspect,
  anchored per cluster to bound the span (no chain drift); keeper = highest pixel
  area then bytes. Reuses the importer's _VIDEO_DUP_* tolerances.
- apply: re-point each loser's post links to the keeper (so no post loses the
  video) THEN delete the redundant records + files via delete_images (cascade).
  dry_run shares the same discovery predicate and returns the projection only
  (rule 93). Tags on a loser are NOT merged (noted; videos rarely hand-curated).

- dedup_videos_task (maintenance queue; summary → task_run.metadata).
- POST /maintenance/dedup-videos {dry_run} + GET /maintenance/task-result/<id> so
  the card shows the dry-run projection before the destructive apply.
- VideoDedupCard: Preview → shows groups/redundant/reclaimable, then Apply behind
  a confirm dialog. Mounted in the Maintenance panel.

Tests: dedup collapses + re-links the loser's post to the keeper + removes the
file; dry-run deletes nothing; distinct durations aren't grouped; task registered.
(Migration 0052 for duration_seconds already shipped with Phase 1.)

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

bvandeusen merged commit c209e3b37e into main 2026-06-16 08:55:39 -04:00

Sign in to join this conversation.

Reviewers

No Reviewers

Labels

No items

No Label

Milestone

No items

No Milestone

Projects

Clear projects

No project

Assignees

Clear assignees

bvandeusen

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: bvandeusen/FabledCurator#110