Videos deduped on sha256 only (pHash is images-only), so a different encode/remux
of the same clip imported as a distinct record — the "same video from multiple
sources" clutter surfaced by #859.
Tier-1 metadata fingerprint: identity = container duration (±1.0s) + matching
aspect ratio, scoped to the same artist; quality axis = pixel dimensions (mirrors
image pHash: larger_exists→skip+link, smaller_exists→supersede). Codec/bitrate
are deliberately NOT part of identity (the point is matching across re-encodes).
Tight tolerances because a wrong video merge is destructive.
- image_record.duration_seconds (Float, nullable; migration 0052). NULL for images.
- safe_probe.probe_video also reads format=duration (one extra ffprobe field on the
call that already runs); ProbeResult.duration.
- _find_similar_video(duration,w,h,artist) shared by both import pipelines.
- _import_media (filesystem/archive path): captures duration, video near-dup
branch, persists duration.
- attach_in_place (download path — handles #859's videos, previously didn't probe
video at all): best-effort probe for dims+duration (LENIENT — never newly rejects
a downloaded video on probe failure), video near-dup branch, persists duration.
- _supersede carries duration onto the kept row.
Reuses SkipReason.duplicate_phash so the existing download/external dup-cleanup
(path-safe unlink, #859) applies unchanged. Tests: skip-smaller, supersede-larger
(+ duration adopted), and distinct-durations-not-merged (false-merge guard).
Follow-up (Phase 2, #871): a backfill to re-probe NULL-duration existing videos so
the current library participates in dedup; retroactive merge of existing dups is a
separate destructive maintenance action.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Milestone #67 step 2. On the native core ingester the Post becomes the single
authoritative record for body/links/metadata, captured once per post by the
post-record; the per-media import only links image provenance + localization.
Before: every per-media sidecar carried the full post body, so a post with N
images wrote the body N+1 times (post-record + N media) — redundant on disk and
a divergence risk (#753). gallery-dl is unchanged (its sidecar is still the only
body source).
- patreon_downloader: the per-media sidecar is now minimal — {category, id,
source_url} only, no body. `_write_sidecar_data(minimal=True)` skips the body
resolution + detail-fetch (the post-record, written first in the walk, already
did it). Body no longer duplicated next to each image.
- importer: new per-instance `post_first` flag (Importer is per-task). When set,
`_apply_sidecar` still writes source_filehash + provenance + primary_post_id
but SKIPS `_apply_post_fields` (the post-record owns body/links/raw_metadata,
so applying a body-less sidecar would clobber raw_metadata + re-sync links off
empty data). Default False keeps gallery-dl writing post fields.
- download_service: `_phase3_persist` sets importer.post_first =
uses_native_ingester(platform) — the future-proof seam, so a platform migrating
onto the native core flips to post-first automatically (step 3). Media imports
before post-records but both unify on external_post_id, so the post ends with
its body either way.
Tests: per-media sidecar is minimal + never hits the detail fetcher; attach
post_first=True links provenance/localization but writes no post body/title;
post_first=False (gallery-dl) still applies them.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The native Patreon backfill flooded the feed with bare 'Post <id>' shells
(1589 for Anduo). Root cause: PostAttachment.sha256 was GLOBALLY unique, so a
non-art file reused across posts only ever linked to the first one, and
_capture_attachment created the Post before that dedup check — leaving later
posts with no image and no attachment. Duplicate IMAGES had the mirror gap:
attach_in_place returned duplicate_hash/duplicate_phash before _apply_sidecar,
so the second post got no provenance row, and the feed only rendered via
primary_post_id (one post per image).
Operator requirement: a duplicate item must show on EVERY post it appears in.
Unify the fix as link-not-suppress:
- importer: on duplicate_hash / duplicate_phash(larger_exists), append an
image_provenance row for the new post (keep primary on the first). Both the
download path (attach_in_place) and the filesystem path (_import_media).
- post_feed_service: render thumbnails by image_provenance UNION primary_post_id,
so a cross-posted image shows on every post (and legacy primary-only images
still show).
- PostAttachment: per-post uniqueness — drop UNIQUE(sha256), add partial
UNIQUE(post_id, sha256) + partial UNIQUE(sha256) WHERE post_id IS NULL
(migration 0043); _capture_attachment dedups per-(post,sha) over the shared
sha-addressed blob, so no post is left bare.
- cleanup: new prune-bare-posts maintenance action (cleanup_service
_bare_post_conditions shared by preview/count/delete per preview/apply parity;
admin endpoint; PostMaintenanceCard). Deletes posts with zero image links
(primary or provenance) AND zero attachments. Run after the feed fix so a
hidden provenance link spares the post instead of deleting it.
Tests: dup image shows on both posts; dup attachment shows on both posts; feed
renders provenance-linked duplicates; prune-bare delete-path == preview.
Operator redeploys (migration 0043) then runs the prune to clear the shells.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
attach_in_place mirrored only the media flow, so gallery-dl-downloaded
zips/PDFs/audio bounced back as `skipped+invalid_image`, which
download_service counted as an ingest error and flipped runs to
status="error" despite N successful image attaches. Lustria patreon
event #38998 (21 images + 1 OST zip) went red for exactly this reason.
Now attach_in_place dispatches the same way as import_one: archives →
_import_archive (extracts media members, captures archive as
PostAttachment), non-media → _capture_attachment. Download_service
accepts the new `attached` result and treats non-duplicate skips as
soft skips, not ingest errors.
Also: ShowcaseView always loadInitial() on mount, not just when the
store is empty — Pinia persists across navigations and operator wants
a fresh shuffle every time the showcase loads.
bvandeusen