fix(posts): link duplicate items to every post + prune bare shells
The native Patreon backfill flooded the feed with bare 'Post <id>' shells (1589 for Anduo). Root cause: PostAttachment.sha256 was GLOBALLY unique, so a non-art file reused across posts only ever linked to the first one, and _capture_attachment created the Post before that dedup check — leaving later posts with no image and no attachment. Duplicate IMAGES had the mirror gap: attach_in_place returned duplicate_hash/duplicate_phash before _apply_sidecar, so the second post got no provenance row, and the feed only rendered via primary_post_id (one post per image). Operator requirement: a duplicate item must show on EVERY post it appears in. Unify the fix as link-not-suppress: - importer: on duplicate_hash / duplicate_phash(larger_exists), append an image_provenance row for the new post (keep primary on the first). Both the download path (attach_in_place) and the filesystem path (_import_media). - post_feed_service: render thumbnails by image_provenance UNION primary_post_id, so a cross-posted image shows on every post (and legacy primary-only images still show). - PostAttachment: per-post uniqueness — drop UNIQUE(sha256), add partial UNIQUE(post_id, sha256) + partial UNIQUE(sha256) WHERE post_id IS NULL (migration 0043); _capture_attachment dedups per-(post,sha) over the shared sha-addressed blob, so no post is left bare. - cleanup: new prune-bare-posts maintenance action (cleanup_service _bare_post_conditions shared by preview/count/delete per preview/apply parity; admin endpoint; PostMaintenanceCard). Deletes posts with zero image links (primary or provenance) AND zero attachments. Run after the feed fix so a hidden provenance link spares the post instead of deleting it. Tests: dup image shows on both posts; dup attachment shows on both posts; feed renders provenance-linked duplicates; prune-bare delete-path == preview. Operator redeploys (migration 0043) then runs the prune to clear the shells. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -6,7 +6,8 @@ COALESCE(Post.post_date, Post.downloaded_at) so posts without a
|
||||
publish date sort by when we captured them.
|
||||
|
||||
Pure read-surface; no writes. The service composes the post dict
|
||||
(including thumbnails from ImageRecord.primary_post_id and non-media
|
||||
(thumbnails from every image linked to the post — its own primary images
|
||||
plus cross-posted duplicates via image_provenance — and non-media
|
||||
attachments from PostAttachment) so the API layer can jsonify directly.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
@@ -17,7 +18,14 @@ from datetime import datetime
|
||||
from sqlalchemy import and_, func, or_, select
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from ..models import Artist, ImageRecord, Post, PostAttachment, Source
|
||||
from ..models import (
|
||||
Artist,
|
||||
ImageProvenance,
|
||||
ImageRecord,
|
||||
Post,
|
||||
PostAttachment,
|
||||
Source,
|
||||
)
|
||||
from ..utils.text import html_to_plain, truncate_at_word
|
||||
from .gallery_service import thumbnail_url
|
||||
|
||||
@@ -205,28 +213,50 @@ class PostFeedService:
|
||||
"""
|
||||
if not post_ids:
|
||||
return {}
|
||||
# A post shows EVERY image linked to it — both its own primary images and
|
||||
# cross-posted duplicates linked via image_provenance (a near-dup of an
|
||||
# existing image gets a provenance row for the new post, not dropped; see
|
||||
# image_provenance docstring + the importer enrich-on-duplicate path). The
|
||||
# UNION dedups (image, post) pairs and also keeps any legacy image that has
|
||||
# a primary_post_id but no provenance row. Partition the window on the
|
||||
# link's post_id, not ImageRecord.primary_post_id, so a duplicate counts
|
||||
# under each post it belongs to.
|
||||
links = (
|
||||
select(
|
||||
ImageProvenance.image_record_id.label("image_id"),
|
||||
ImageProvenance.post_id.label("post_id"),
|
||||
)
|
||||
.where(ImageProvenance.post_id.in_(post_ids))
|
||||
.union(
|
||||
select(
|
||||
ImageRecord.id.label("image_id"),
|
||||
ImageRecord.primary_post_id.label("post_id"),
|
||||
).where(ImageRecord.primary_post_id.in_(post_ids))
|
||||
)
|
||||
.subquery()
|
||||
)
|
||||
# Rank images within each post; cap at `limit` rows per post when
|
||||
# limit is set, return all when limit is None.
|
||||
ranked = (
|
||||
select(
|
||||
ImageRecord.id,
|
||||
ImageRecord.primary_post_id,
|
||||
links.c.post_id,
|
||||
ImageRecord.sha256,
|
||||
ImageRecord.mime,
|
||||
ImageRecord.thumbnail_path,
|
||||
func.row_number().over(
|
||||
partition_by=ImageRecord.primary_post_id,
|
||||
partition_by=links.c.post_id,
|
||||
order_by=ImageRecord.id.asc(),
|
||||
).label("rn"),
|
||||
func.count(ImageRecord.id).over(
|
||||
partition_by=ImageRecord.primary_post_id,
|
||||
partition_by=links.c.post_id,
|
||||
).label("total"),
|
||||
)
|
||||
.where(ImageRecord.primary_post_id.in_(post_ids))
|
||||
.join(ImageRecord, ImageRecord.id == links.c.image_id)
|
||||
.subquery()
|
||||
)
|
||||
stmt = select(
|
||||
ranked.c.id, ranked.c.primary_post_id,
|
||||
ranked.c.id, ranked.c.post_id,
|
||||
ranked.c.sha256, ranked.c.mime, ranked.c.thumbnail_path, ranked.c.total,
|
||||
)
|
||||
if limit is not None:
|
||||
|
||||
Reference in New Issue
Block a user