feat(maintenance): reconcile duplicate posts (gallery-dl→native unify) #124
Assignees
bvandeusen
Clear assignees
bvandeusen
No Assignees
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: bvandeusen/FabledCurator#124
Reference in New Issue
Block a user
Delete Branch "dev"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
What
Unifies duplicate post records created when an artist was first downloaded by gallery-dl (posts keyed by the per-attachment id) and later walked by the native ingester (posts keyed by the real post id). They never dedup (
uq_post_source_external_idis onexternal_post_id) → duplicate rows (cheunart: 943→1109). Images are not touched (content-addressed/deduped already; operator-confirmed). Canonical key = the post id (the format the current native downloader produces), so future walks dedup and this can't recur.How
Post.raw_metadata["post_id"](both eras store the sidecar there). Confirmed on disk: legacyexternal_post_id= the sidecar's attachmentid, ≠post_id.reconcile_duplicate_posts(cleanup_service): group by(source_id, canonical post_id); keep the post-id-keyed row; conflict-safe re-point ofImageRecord.primary_post_id/ImageProvenance/PostAttachment/ExternalLink(drop the loser's row where the keeper already has the equivalent, per each table's uniqueness); backfill the keeper's empty date/title/body/raw_metadata from a loser; setexternal_post_id=post id + derivepost_url; delete losers.find_duplicate_post_groups(rule 93). API/api/admin/posts/reconcile-duplicates. UI: a second section onPostMaintenanceCard(preview groups+sample → confirm merge).Tests (all PASSED in CI)
merge + metadata backfill + image move; no-op when unique; provenance-collision dedup. Lint/unit/frontend green; reconcile integration tests confirmed green in run 1258.
Design: milestone #73. Forensics: note #917. Out of scope (flagged):
cheunartvsCheunartcase-variant artist dirs.🤖 Generated with Claude Code
An artist first downloaded by gallery-dl gets Post rows keyed by the per- attachment id; a later native walk keys the SAME real post by the post id. They never dedup (uq_post_source_external_id is on external_post_id) → duplicate post rows (cheunart: 943→1109). The real post id is recoverable in-DB from raw_metadata['post_id'] (both eras store the sidecar there). reconcile_duplicate_posts (cleanup_service): group posts by (source_id, canonical post_id = raw_metadata.post_id else external_post_id); for each group >1, keep the row already keyed by the post id (the format the CURRENT native downloader produces, so future walks dedup and this can't recur), re-point ImageRecord.primary_post_id / ImageProvenance / PostAttachment / ExternalLink onto it conflict-safe (drop the loser's row where the keeper already has the equivalent, per each table's uniqueness), backfill the keeper's empty date/title/body/raw_meta from a loser, set external_post_id=post_id + derive post_url, delete losers. IMAGES ARE NOT TOUCHED (content-addressed/deduped already; operator-confirmed). Preview/apply share find_duplicate_post_groups (rule 93). API /api/admin/posts/reconcile-duplicates (dry_run→{groups,posts_to_merge,sample}; apply→{groups,merged,sample}; optional source_id). UI: a second section on PostMaintenanceCard (preview groups+sample → confirm merge). Tests: merge + metadata backfill + image move, no-op when unique, provenance-collision dedup. Design: milestone #73. Forensics: note #917. Out of scope (flagged): cheunart vs Cheunart case-variant artist dirs/rows. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>