fix(posts): link duplicate items to every post + prune bare shells
The native Patreon backfill flooded the feed with bare 'Post <id>' shells (1589 for Anduo). Root cause: PostAttachment.sha256 was GLOBALLY unique, so a non-art file reused across posts only ever linked to the first one, and _capture_attachment created the Post before that dedup check — leaving later posts with no image and no attachment. Duplicate IMAGES had the mirror gap: attach_in_place returned duplicate_hash/duplicate_phash before _apply_sidecar, so the second post got no provenance row, and the feed only rendered via primary_post_id (one post per image). Operator requirement: a duplicate item must show on EVERY post it appears in. Unify the fix as link-not-suppress: - importer: on duplicate_hash / duplicate_phash(larger_exists), append an image_provenance row for the new post (keep primary on the first). Both the download path (attach_in_place) and the filesystem path (_import_media). - post_feed_service: render thumbnails by image_provenance UNION primary_post_id, so a cross-posted image shows on every post (and legacy primary-only images still show). - PostAttachment: per-post uniqueness — drop UNIQUE(sha256), add partial UNIQUE(post_id, sha256) + partial UNIQUE(sha256) WHERE post_id IS NULL (migration 0043); _capture_attachment dedups per-(post,sha) over the shared sha-addressed blob, so no post is left bare. - cleanup: new prune-bare-posts maintenance action (cleanup_service _bare_post_conditions shared by preview/count/delete per preview/apply parity; admin endpoint; PostMaintenanceCard). Deletes posts with zero image links (primary or provenance) AND zero attachments. Run after the feed fix so a hidden provenance link spares the post instead of deleting it. Tests: dup image shows on both posts; dup attachment shows on both posts; feed renders provenance-linked duplicates; prune-bare delete-path == preview. Operator redeploys (migration 0043) then runs the prune to clear the shells. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -361,22 +361,39 @@ class Importer:
|
||||
artist = self._resolve_artist(source)
|
||||
post = self._post_for_sidecar(source, artist)
|
||||
sha = _sha256_of(source)
|
||||
select_existing = select(PostAttachment).where(PostAttachment.sha256 == sha)
|
||||
post_id = post.id if post else None
|
||||
# Dedup is PER-POST, not global: a non-art file the creator attaches to
|
||||
# many posts (a standard pdf/zip/link-card) must get a row on EVERY post
|
||||
# so none is left a bare shell. The on-disk blob stays sha-deduped
|
||||
# (attachments.store is sha-addressed + idempotent); only the rows are
|
||||
# per-post. PostAttachment's unique is partial (post_id, sha256) for
|
||||
# real posts and (sha256) for the NULL-post filesystem case. Before
|
||||
# 2026-06-08 the global UNIQUE(sha256) left every post after the first
|
||||
# with no attachment row → 1589 empty Anduo shells.
|
||||
if post_id is not None:
|
||||
select_existing = select(PostAttachment).where(
|
||||
PostAttachment.post_id == post_id,
|
||||
PostAttachment.sha256 == sha,
|
||||
)
|
||||
else:
|
||||
select_existing = select(PostAttachment).where(
|
||||
PostAttachment.post_id.is_(None),
|
||||
PostAttachment.sha256 == sha,
|
||||
)
|
||||
existing = self.session.execute(select_existing).scalar_one_or_none()
|
||||
if existing is not None:
|
||||
self.session.commit()
|
||||
return ImportResult(status="attached")
|
||||
# Savepoint + IntegrityError recovery — PostAttachment.sha256 is
|
||||
# UNIQUE, so two workers can both pass the SELECT and only the
|
||||
# second INSERT fails. Without savepoint, the outer transaction
|
||||
# poisons and the calling task crashes. attachments.store is
|
||||
# sha-addressed so both workers race to write the same target
|
||||
# path; shutil.copy2 + rename is idempotent. Audit 2026-06-02.
|
||||
# Savepoint + IntegrityError recovery — the partial UNIQUE means two
|
||||
# workers can both pass the SELECT and only the second INSERT fails.
|
||||
# Without savepoint, the outer transaction poisons and the calling task
|
||||
# crashes. attachments.store is sha-addressed so both workers race to
|
||||
# write the same target path; shutil.copy2 + rename is idempotent.
|
||||
sp = self.session.begin_nested()
|
||||
try:
|
||||
stored = self.attachments.store(source, sha)
|
||||
self.session.add(PostAttachment(
|
||||
post_id=post.id if post else None,
|
||||
post_id=post_id,
|
||||
artist_id=artist.id if artist else None,
|
||||
sha256=sha,
|
||||
path=stored,
|
||||
@@ -389,7 +406,7 @@ class Importer:
|
||||
sp.commit()
|
||||
except IntegrityError:
|
||||
sp.rollback()
|
||||
# Lost the race — the other worker's row is canonical.
|
||||
# Lost the race — the other worker's row for this (post, sha) wins.
|
||||
self.session.execute(select_existing).scalar_one()
|
||||
self.session.commit()
|
||||
return ImportResult(status="attached")
|
||||
@@ -555,6 +572,15 @@ class Importer:
|
||||
existing = self.session.execute(existing_stmt).scalar_one_or_none()
|
||||
if existing:
|
||||
if not self.deep:
|
||||
# Enrich-on-duplicate (parity with attach_in_place): a re-scanned
|
||||
# file that is a byte-dup of an existing image still links its
|
||||
# post via _apply_sidecar, so a cross-posted image shows on every
|
||||
# post. Artist is derived from the sidecar here (not yet resolved).
|
||||
self._apply_sidecar(
|
||||
existing, attribution_path, None,
|
||||
explicit_source=explicit_source,
|
||||
)
|
||||
self.session.commit()
|
||||
return ImportResult(
|
||||
status="skipped", skip_reason=SkipReason.duplicate_hash,
|
||||
image_id=existing.id, error="sha256 already present",
|
||||
@@ -581,6 +607,14 @@ class Importer:
|
||||
candidates, self.settings.phash_threshold,
|
||||
)
|
||||
if rel == "larger_exists":
|
||||
# Enrich-on-duplicate (parity with attach_in_place).
|
||||
larger = self.session.get(ImageRecord, match_id)
|
||||
if larger is not None:
|
||||
self._apply_sidecar(
|
||||
larger, attribution_path, None,
|
||||
explicit_source=explicit_source,
|
||||
)
|
||||
self.session.commit()
|
||||
return ImportResult(
|
||||
status="skipped",
|
||||
skip_reason=SkipReason.duplicate_phash,
|
||||
@@ -764,6 +798,15 @@ class Importer:
|
||||
select(ImageRecord).where(ImageRecord.sha256 == sha)
|
||||
).scalar_one_or_none()
|
||||
if existing:
|
||||
# Enrich-on-duplicate (image_provenance docstring / spec §3): the
|
||||
# same bytes appearing in another post must show on THAT post too,
|
||||
# so append a provenance row for the new post rather than dropping
|
||||
# the linkage. primary_post_id stays on the first post — _apply_sidecar
|
||||
# only sets it when NULL. Without this the new post is left with no
|
||||
# image AND (if its other content also deduped) no attachment, i.e. a
|
||||
# bare shell — operator-flagged 2026-06-08 (1589 empty Anduo posts).
|
||||
self._apply_sidecar(existing, path, artist, explicit_source=source)
|
||||
self.session.commit()
|
||||
return ImportResult(
|
||||
status="skipped", skip_reason=SkipReason.duplicate_hash,
|
||||
image_id=existing.id, error="sha256 already present",
|
||||
@@ -784,6 +827,15 @@ class Importer:
|
||||
candidates, self.settings.phash_threshold,
|
||||
)
|
||||
if rel == "larger_exists":
|
||||
# Enrich-on-duplicate: link the near-dup's post to the
|
||||
# existing larger image so it shows on both (see duplicate_hash
|
||||
# above). smaller_exists already links via _supersede.
|
||||
larger = self.session.get(ImageRecord, match_id)
|
||||
if larger is not None:
|
||||
self._apply_sidecar(
|
||||
larger, path, artist, explicit_source=source,
|
||||
)
|
||||
self.session.commit()
|
||||
return ImportResult(
|
||||
status="skipped",
|
||||
skip_reason=SkipReason.duplicate_phash,
|
||||
|
||||
Reference in New Issue
Block a user