Merges #871 (CI-green, tip 41652db) to main / :latest.
Import-time dedup — same-artist videos matching on duration (±1s) + aspect ratio are treated as the same content across re-encodes; the higher-res copy is kept (mirrors image pHash supersede). Wired into both import pipelines, including the download path that handles the #859 multi-source case. Adds image_record.duration_seconds (migration 0052) + format=duration to the existing ffprobe call.
Retroactive cleanup — Settings → Maintenance → Deduplicate videos: Preview (re-probes old videos, reports groups/redundant/reclaimable bytes, non-destructive) then Apply (re-points each post to the keeper, then deletes the redundant copies + files). For cleaning the existing #859 duplicate clutter.
One new migration (0052). No new runtime deps. After merge: force a real re-pull of :latest, then the new "Deduplicate videos" action (and the still-pending #718 "re-extract archives") are available under Settings → Maintenance.
Merges #871 (CI-green, tip `41652db`) to main / `:latest`.
- **Import-time dedup** — same-artist videos matching on duration (±1s) + aspect ratio are treated as the same content across re-encodes; the higher-res copy is kept (mirrors image pHash supersede). Wired into both import pipelines, including the download path that handles the #859 multi-source case. Adds `image_record.duration_seconds` (migration **0052**) + `format=duration` to the existing ffprobe call.
- **Retroactive cleanup** — `Settings → Maintenance → Deduplicate videos`: Preview (re-probes old videos, reports groups/redundant/reclaimable bytes, non-destructive) then Apply (re-points each post to the keeper, then deletes the redundant copies + files). For cleaning the existing #859 duplicate clutter.
One new migration (0052). No new runtime deps. **After merge:** force a real re-pull of `:latest`, then the new "Deduplicate videos" action (and the still-pending #718 "re-extract archives") are available under Settings → Maintenance.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Videos deduped on sha256 only (pHash is images-only), so a different encode/remux
of the same clip imported as a distinct record — the "same video from multiple
sources" clutter surfaced by #859.
Tier-1 metadata fingerprint: identity = container duration (±1.0s) + matching
aspect ratio, scoped to the same artist; quality axis = pixel dimensions (mirrors
image pHash: larger_exists→skip+link, smaller_exists→supersede). Codec/bitrate
are deliberately NOT part of identity (the point is matching across re-encodes).
Tight tolerances because a wrong video merge is destructive.
- image_record.duration_seconds (Float, nullable; migration 0052). NULL for images.
- safe_probe.probe_video also reads format=duration (one extra ffprobe field on the
call that already runs); ProbeResult.duration.
- _find_similar_video(duration,w,h,artist) shared by both import pipelines.
- _import_media (filesystem/archive path): captures duration, video near-dup
branch, persists duration.
- attach_in_place (download path — handles #859's videos, previously didn't probe
video at all): best-effort probe for dims+duration (LENIENT — never newly rejects
a downloaded video on probe failure), video near-dup branch, persists duration.
- _supersede carries duration onto the kept row.
Reuses SkipReason.duplicate_phash so the existing download/external dup-cleanup
(path-safe unlink, #859) applies unchanged. Tests: skip-smaller, supersede-larger
(+ duration adopted), and distinct-durations-not-merged (false-merge guard).
Follow-up (Phase 2, #871): a backfill to re-probe NULL-duration existing videos so
the current library participates in dedup; retroactive merge of existing dups is a
separate destructive maintenance action.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Phase 2 of #871: clean up the duplicate videos already in the library (the #859
"same video from multiple sources" clutter). Import-time dedup (Phase 1) only
prevents NEW dups; this is the operator-triggered cleanup of existing ones.
cleanup_service.dedup_videos(dry_run):
- backfill_video_durations: re-probe NULL-duration videos (pre-#871 rows) so the
existing library participates; idempotent (only NULL rows), writes a negative
sentinel for un-probeable files so they're neither re-probed forever nor matched.
- find_video_dup_groups: cluster same-artist videos by duration (±tol) + aspect,
anchored per cluster to bound the span (no chain drift); keeper = highest pixel
area then bytes. Reuses the importer's _VIDEO_DUP_* tolerances.
- apply: re-point each loser's post links to the keeper (so no post loses the
video) THEN delete the redundant records + files via delete_images (cascade).
dry_run shares the same discovery predicate and returns the projection only
(rule 93). Tags on a loser are NOT merged (noted; videos rarely hand-curated).
- dedup_videos_task (maintenance queue; summary → task_run.metadata).
- POST /maintenance/dedup-videos {dry_run} + GET /maintenance/task-result/<id> so
the card shows the dry-run projection before the destructive apply.
- VideoDedupCard: Preview → shows groups/redundant/reclaimable, then Apply behind
a confirm dialog. Mounted in the Maintenance panel.
Tests: dedup collapses + re-links the loser's post to the keeper + removes the
file; dry-run deletes nothing; distinct durations aren't grouped; task registered.
(Migration 0052 for duration_seconds already shipped with Phase 1.)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Merges #871 (CI-green, tip
41652db) to main /:latest.image_record.duration_seconds(migration 0052) +format=durationto the existing ffprobe call.Settings → Maintenance → Deduplicate videos: Preview (re-probes old videos, reports groups/redundant/reclaimable bytes, non-destructive) then Apply (re-points each post to the keeper, then deletes the redundant copies + files). For cleaning the existing #859 duplicate clutter.One new migration (0052). No new runtime deps. After merge: force a real re-pull of
:latest, then the new "Deduplicate videos" action (and the still-pending #718 "re-extract archives") are available under Settings → Maintenance.🤖 Generated with Claude Code