fix(posts): link duplicate items to every post + prune bare shells
The native Patreon backfill flooded the feed with bare 'Post <id>' shells (1589 for Anduo). Root cause: PostAttachment.sha256 was GLOBALLY unique, so a non-art file reused across posts only ever linked to the first one, and _capture_attachment created the Post before that dedup check — leaving later posts with no image and no attachment. Duplicate IMAGES had the mirror gap: attach_in_place returned duplicate_hash/duplicate_phash before _apply_sidecar, so the second post got no provenance row, and the feed only rendered via primary_post_id (one post per image). Operator requirement: a duplicate item must show on EVERY post it appears in. Unify the fix as link-not-suppress: - importer: on duplicate_hash / duplicate_phash(larger_exists), append an image_provenance row for the new post (keep primary on the first). Both the download path (attach_in_place) and the filesystem path (_import_media). - post_feed_service: render thumbnails by image_provenance UNION primary_post_id, so a cross-posted image shows on every post (and legacy primary-only images still show). - PostAttachment: per-post uniqueness — drop UNIQUE(sha256), add partial UNIQUE(post_id, sha256) + partial UNIQUE(sha256) WHERE post_id IS NULL (migration 0043); _capture_attachment dedups per-(post,sha) over the shared sha-addressed blob, so no post is left bare. - cleanup: new prune-bare-posts maintenance action (cleanup_service _bare_post_conditions shared by preview/count/delete per preview/apply parity; admin endpoint; PostMaintenanceCard). Deletes posts with zero image links (primary or provenance) AND zero attachments. Run after the feed fix so a hidden provenance link spares the post instead of deleting it. Tests: dup image shows on both posts; dup attachment shows on both posts; feed renders provenance-linked duplicates; prune-bare delete-path == preview. Operator redeploys (migration 0043) then runs the prune to clear the shells. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -20,7 +20,15 @@ from typing import Any
|
||||
from sqlalchemy import func, or_, select, update
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from ..models import Artist, ImageRecord, LibraryAuditRun, Tag
|
||||
from ..models import (
|
||||
Artist,
|
||||
ImageProvenance,
|
||||
ImageRecord,
|
||||
LibraryAuditRun,
|
||||
Post,
|
||||
PostAttachment,
|
||||
Tag,
|
||||
)
|
||||
from ..models.series_chapter import SeriesChapter
|
||||
from ..models.series_page import SeriesPage
|
||||
from ..models.tag import image_tag
|
||||
@@ -407,6 +415,88 @@ def prune_unused_tags(session: Session, *, dry_run: bool = False) -> dict:
|
||||
return {"deleted": result.rowcount or 0, "sample_names": sample}
|
||||
|
||||
|
||||
def _bare_post_conditions() -> list:
|
||||
"""The WHERE conditions that define a 'bare' post — the SINGLE source of truth
|
||||
shared by find_bare_posts (preview sample), the dry-run count, AND the live
|
||||
delete, so the preview can NEVER diverge from what the delete removes
|
||||
([[feedback_preview_apply_parity]]).
|
||||
|
||||
A post is "bare" iff NOTHING is attached to it across every way content links
|
||||
to a post:
|
||||
- image_record.primary_post_id (a post's own canonical images)
|
||||
- image_provenance.post_id (cross-posted/duplicate images linked here)
|
||||
- post_attachment.post_id (preserved non-art files)
|
||||
|
||||
These are exactly the shells the empty-post flood produced: the native Patreon
|
||||
ingester synthesized a Post per walked post, but when its only content was a
|
||||
duplicate image/attachment that linked to an EARLIER post, the new post was
|
||||
left with none of the three (operator-flagged 2026-06-08). Every FK to post.id
|
||||
is SET NULL or CASCADE, so deleting a bare post is non-destructive by
|
||||
construction — there is nothing pointing at it to orphan. Must run AFTER the
|
||||
provenance-render fix so a post that DOES have a hidden provenance link is
|
||||
spared, not deleted.
|
||||
"""
|
||||
has_primary = select(ImageRecord.id).where(
|
||||
ImageRecord.primary_post_id == Post.id
|
||||
)
|
||||
has_provenance = select(ImageProvenance.id).where(
|
||||
ImageProvenance.post_id == Post.id
|
||||
)
|
||||
has_attachment = select(PostAttachment.id).where(
|
||||
PostAttachment.post_id == Post.id
|
||||
)
|
||||
return [
|
||||
~has_primary.exists(),
|
||||
~has_provenance.exists(),
|
||||
~has_attachment.exists(),
|
||||
]
|
||||
|
||||
|
||||
def find_bare_posts(
|
||||
session: Session, *, limit: int | None = None,
|
||||
) -> list[Post]:
|
||||
"""Posts with zero linked images (primary OR provenance) AND zero
|
||||
attachments — safe to sweep. Sorted by id. Shares its predicate with the
|
||||
live prune via _bare_post_conditions()."""
|
||||
stmt = select(Post).where(*_bare_post_conditions()).order_by(Post.id)
|
||||
if limit is not None:
|
||||
stmt = stmt.limit(limit)
|
||||
return list(session.execute(stmt).scalars().all())
|
||||
|
||||
|
||||
def _bare_post_label(post: Post) -> str:
|
||||
"""Human label for the preview sample — mirrors the feed's fallback title."""
|
||||
if post.post_title:
|
||||
return post.post_title
|
||||
return f"Post {post.external_post_id or post.id}"
|
||||
|
||||
|
||||
def prune_bare_posts(session: Session, *, dry_run: bool = False) -> dict:
|
||||
"""Find posts with no images and no attachments and (unless dry_run) delete
|
||||
them.
|
||||
|
||||
Returns:
|
||||
dry_run=True: {"count": N, "sample_names": [first 50]}
|
||||
dry_run=False: {"deleted": N, "sample_names": [first 50]}
|
||||
|
||||
The live delete runs a single DELETE with the SAME predicate
|
||||
(_bare_post_conditions) the preview uses, so the row count scales without
|
||||
binding every id as a parameter ([[reference_psycopg_65535_param_ceiling]])
|
||||
AND the delete can never remove a post the preview deemed kept.
|
||||
"""
|
||||
conditions = _bare_post_conditions()
|
||||
sample_rows = find_bare_posts(session, limit=50)
|
||||
sample = [_bare_post_label(p) for p in sample_rows]
|
||||
if dry_run:
|
||||
count = session.execute(
|
||||
select(func.count()).select_from(Post).where(*conditions)
|
||||
).scalar_one()
|
||||
return {"count": count, "sample_names": sample}
|
||||
result = session.execute(Post.__table__.delete().where(*conditions))
|
||||
session.commit()
|
||||
return {"deleted": result.rowcount or 0, "sample_names": sample}
|
||||
|
||||
|
||||
# Legacy tags FC no longer uses, in two shapes:
|
||||
# (1) kinds the tag input never produces — archive/post/artist.
|
||||
# provenance (post grouping) + archive membership are their own
|
||||
|
||||
Reference in New Issue
Block a user