fix(posts): link duplicate items to every post + prune bare shells
CI / lint (push) Successful in 2s
CI / frontend-build (push) Successful in 26s
CI / backend-lint-and-test (push) Successful in 30s
CI / integration (push) Successful in 3m20s

The native Patreon backfill flooded the feed with bare 'Post <id>' shells
(1589 for Anduo). Root cause: PostAttachment.sha256 was GLOBALLY unique, so a
non-art file reused across posts only ever linked to the first one, and
_capture_attachment created the Post before that dedup check — leaving later
posts with no image and no attachment. Duplicate IMAGES had the mirror gap:
attach_in_place returned duplicate_hash/duplicate_phash before _apply_sidecar,
so the second post got no provenance row, and the feed only rendered via
primary_post_id (one post per image).

Operator requirement: a duplicate item must show on EVERY post it appears in.
Unify the fix as link-not-suppress:

- importer: on duplicate_hash / duplicate_phash(larger_exists), append an
  image_provenance row for the new post (keep primary on the first). Both the
  download path (attach_in_place) and the filesystem path (_import_media).
- post_feed_service: render thumbnails by image_provenance UNION primary_post_id,
  so a cross-posted image shows on every post (and legacy primary-only images
  still show).
- PostAttachment: per-post uniqueness — drop UNIQUE(sha256), add partial
  UNIQUE(post_id, sha256) + partial UNIQUE(sha256) WHERE post_id IS NULL
  (migration 0043); _capture_attachment dedups per-(post,sha) over the shared
  sha-addressed blob, so no post is left bare.
- cleanup: new prune-bare-posts maintenance action (cleanup_service
  _bare_post_conditions shared by preview/count/delete per preview/apply parity;
  admin endpoint; PostMaintenanceCard). Deletes posts with zero image links
  (primary or provenance) AND zero attachments. Run after the feed fix so a
  hidden provenance link spares the post instead of deleting it.

Tests: dup image shows on both posts; dup attachment shows on both posts; feed
renders provenance-linked duplicates; prune-bare delete-path == preview.

Operator redeploys (migration 0043) then runs the prune to clear the shells.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-06-08 19:28:33 -04:00
parent df76bc0f58
commit a8f624a0f1
12 changed files with 581 additions and 19 deletions
+37 -7
View File
@@ -6,7 +6,8 @@ COALESCE(Post.post_date, Post.downloaded_at) so posts without a
publish date sort by when we captured them.
Pure read-surface; no writes. The service composes the post dict
(including thumbnails from ImageRecord.primary_post_id and non-media
(thumbnails from every image linked to the post — its own primary images
plus cross-posted duplicates via image_provenance — and non-media
attachments from PostAttachment) so the API layer can jsonify directly.
"""
from __future__ import annotations
@@ -17,7 +18,14 @@ from datetime import datetime
from sqlalchemy import and_, func, or_, select
from sqlalchemy.ext.asyncio import AsyncSession
from ..models import Artist, ImageRecord, Post, PostAttachment, Source
from ..models import (
Artist,
ImageProvenance,
ImageRecord,
Post,
PostAttachment,
Source,
)
from ..utils.text import html_to_plain, truncate_at_word
from .gallery_service import thumbnail_url
@@ -205,28 +213,50 @@ class PostFeedService:
"""
if not post_ids:
return {}
# A post shows EVERY image linked to it — both its own primary images and
# cross-posted duplicates linked via image_provenance (a near-dup of an
# existing image gets a provenance row for the new post, not dropped; see
# image_provenance docstring + the importer enrich-on-duplicate path). The
# UNION dedups (image, post) pairs and also keeps any legacy image that has
# a primary_post_id but no provenance row. Partition the window on the
# link's post_id, not ImageRecord.primary_post_id, so a duplicate counts
# under each post it belongs to.
links = (
select(
ImageProvenance.image_record_id.label("image_id"),
ImageProvenance.post_id.label("post_id"),
)
.where(ImageProvenance.post_id.in_(post_ids))
.union(
select(
ImageRecord.id.label("image_id"),
ImageRecord.primary_post_id.label("post_id"),
).where(ImageRecord.primary_post_id.in_(post_ids))
)
.subquery()
)
# Rank images within each post; cap at `limit` rows per post when
# limit is set, return all when limit is None.
ranked = (
select(
ImageRecord.id,
ImageRecord.primary_post_id,
links.c.post_id,
ImageRecord.sha256,
ImageRecord.mime,
ImageRecord.thumbnail_path,
func.row_number().over(
partition_by=ImageRecord.primary_post_id,
partition_by=links.c.post_id,
order_by=ImageRecord.id.asc(),
).label("rn"),
func.count(ImageRecord.id).over(
partition_by=ImageRecord.primary_post_id,
partition_by=links.c.post_id,
).label("total"),
)
.where(ImageRecord.primary_post_id.in_(post_ids))
.join(ImageRecord, ImageRecord.id == links.c.image_id)
.subquery()
)
stmt = select(
ranked.c.id, ranked.c.primary_post_id,
ranked.c.id, ranked.c.post_id,
ranked.c.sha256, ranked.c.mime, ranked.c.thumbnail_path, ranked.c.total,
)
if limit is not None: