feat(maintenance): reconcile duplicate posts (gallery-dl→native unify)
An artist first downloaded by gallery-dl gets Post rows keyed by the per-
attachment id; a later native walk keys the SAME real post by the post id. They
never dedup (uq_post_source_external_id is on external_post_id) → duplicate post
rows (cheunart: 943→1109). The real post id is recoverable in-DB from
raw_metadata['post_id'] (both eras store the sidecar there).
reconcile_duplicate_posts (cleanup_service): group posts by (source_id, canonical
post_id = raw_metadata.post_id else external_post_id); for each group >1, keep the
row already keyed by the post id (the format the CURRENT native downloader
produces, so future walks dedup and this can't recur), re-point
ImageRecord.primary_post_id / ImageProvenance / PostAttachment / ExternalLink onto
it conflict-safe (drop the loser's row where the keeper already has the equivalent,
per each table's uniqueness), backfill the keeper's empty date/title/body/raw_meta
from a loser, set external_post_id=post_id + derive post_url, delete losers.
IMAGES ARE NOT TOUCHED (content-addressed/deduped already; operator-confirmed).
Preview/apply share find_duplicate_post_groups (rule 93). API
/api/admin/posts/reconcile-duplicates (dry_run→{groups,posts_to_merge,sample};
apply→{groups,merged,sample}; optional source_id). UI: a second section on
PostMaintenanceCard (preview groups+sample → confirm merge). Tests: merge +
metadata backfill + image move, no-op when unique, provenance-collision dedup.
Design: milestone #73. Forensics: note #917. Out of scope (flagged): cheunart vs
Cheunart case-variant artist dirs/rows.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
@@ -226,6 +226,33 @@ async def posts_prune_bare():
|
||||
return jsonify(result)
|
||||
|
||||
|
||||
@admin_bp.route("/posts/reconcile-duplicates", methods=["POST"])
|
||||
async def posts_reconcile_duplicates():
|
||||
"""Tier-A: unify duplicate post rows for the same real post — the gallery-dl
|
||||
(attachment-id) + native (post-id) duplicates — onto ONE post-id-keyed keeper,
|
||||
moving image/provenance/attachment/link rows over. Images are untouched.
|
||||
dry_run=true returns {groups, posts_to_merge, sample}; dry_run=false applies
|
||||
and returns {groups, merged, sample}. Optional source_id scopes to one source.
|
||||
Same find_duplicate_post_groups predicate drives preview + apply (rule 93)."""
|
||||
from ..services.cleanup_service import reconcile_duplicate_posts
|
||||
|
||||
body = await request.get_json(silent=True) or {}
|
||||
dry_run = bool(body.get("dry_run", False))
|
||||
raw_source = body.get("source_id")
|
||||
try:
|
||||
source_id = int(raw_source) if raw_source is not None else None
|
||||
except (TypeError, ValueError):
|
||||
return _bad("invalid_source_id", detail="source_id must be an integer")
|
||||
|
||||
async with get_session() as session:
|
||||
result = await session.run_sync(
|
||||
lambda sync_sess: reconcile_duplicate_posts(
|
||||
sync_sess, source_id=source_id, dry_run=dry_run,
|
||||
)
|
||||
)
|
||||
return jsonify(result)
|
||||
|
||||
|
||||
@admin_bp.route("/tags/purge-legacy", methods=["POST"])
|
||||
async def tags_purge_legacy():
|
||||
"""Tier-A: delete legacy IR-migration tags — archive/post/artist
|
||||
|
||||
Reference in New Issue
Block a user