feat(provenance): capture which archive an extracted image came from (#87)
CI / lint (push) Successful in 2s
CI / frontend-build (push) Successful in 19s
CI / backend-lint-and-test (push) Successful in 27s
CI / integration (push) Failing after 3m20s

Images pulled out of a .zip/.rar previously kept no record of WHICH archive
they came from — the member->archive link was computed during extraction and
discarded, leaving only image->post. So the provenance modal could only scope
attachments to the whole post, showing every archive a 'High Resolution Files'
bundle carried instead of the one a given file lives in.

- ImageProvenance.from_attachment_id: nullable FK -> post_attachment.id
  (SET NULL), migration 0055.
- importer: _import_archive stamps from_attachment_id on every member's
  provenance row for the post (new + superseded + deduped members), resolving
  the archive's own PostAttachment by (post, sha). Post-pass UPDATE, NULL-only
  and idempotent, so it doesn't touch the dedup/supersede branches and the
  backfill is safe to re-run. Nested members link to the outer stored archive.
- provenance_service.for_image: when the originating post's provenance row
  records from_attachment_id, return ONLY that archive; else fall back to the
  primary-post scoping from 068def2.
- ProvenancePanel: heading pluralizes ('Attachment' for a single file).
- Backfill: re-running reextract_archive_attachments (ArchiveReextractCard)
  routes through _import_archive and stamps existing rows — no new code.

Tests: capture stamps on fresh import, nested-archive attribution, per-post
archive on dedup; for_image filters to the containing archive; reextract
backfill stamps the link.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-06-21 22:22:03 -04:00
parent 068def2f24
commit 5269cd0709
8 changed files with 273 additions and 19 deletions
+10
View File
@@ -41,6 +41,16 @@ class ImageProvenance(Base):
source_id: Mapped[int | None] = mapped_column(
ForeignKey("source.id", ondelete="SET NULL"), nullable=True, index=True
)
# The archive PostAttachment this image was extracted FROM, when it came
# out of a .zip/.rar rather than as a loose file (milestone #87). Lets the
# provenance UI show the exact archive a file lives inside instead of every
# attachment on the post. NULL for loose downloads and pre-backfill rows.
# SET NULL so deleting the archive attachment never destroys the (image,
# post) edge — it just forgets which archive it came from.
from_attachment_id: Mapped[int | None] = mapped_column(
ForeignKey("post_attachment.id", ondelete="SET NULL"),
nullable=True, index=True,
)
captured_metadata: Mapped[dict | None] = mapped_column(JSON, nullable=True)
captured_at: Mapped[datetime] = mapped_column(
DateTime(timezone=True), nullable=False, server_default=func.now()
+56 -1
View File
@@ -17,7 +17,7 @@ from enum import StrEnum
from pathlib import Path
from PIL import Image
from sqlalchemy import select
from sqlalchemy import select, update
from sqlalchemy.exc import IntegrityError
from sqlalchemy.orm import Session
@@ -506,6 +506,12 @@ class Importer:
artist_use = artist if artist is not None else self._resolve_artist(source)
post = self._post_for_sidecar(source, artist_use)
member_ids: list[int] = []
# Every member image touched (new + superseded + deduped), so the
# from_attachment_id stamp below covers files that already existed in the
# library and were merely re-linked to this post — those matter most
# (the HR copy a bundle re-ships). Separate from member_ids, which is
# the NEWLY-imported subset feeding the ImportResult contract.
member_record_ids: set[int] = set()
# Per-outcome tally so the "no images" reason names the ACTUAL cause
# (#718): nested-archive packs, all-deduped (benign), unsupported formats,
# or failed/corrupt members — instead of one catch-all string.
@@ -516,11 +522,19 @@ class Importer:
self._collect_archive_members(
source, attribution=source, source_row=source_row,
depth=0, member_ids=member_ids, counts=counts,
member_record_ids=member_record_ids,
)
# Preserve the archive itself (links to the same Post/Artist).
self._capture_attachment(
source, post=post, artist=artist_use, resolved=True
)
# Stamp each member's provenance row for THIS post with the archive it
# came out of (milestone #87). Done as a post-pass rather than threaded
# through _import_media/_apply_sidecar so the many dedup/supersede
# branches stay untouched. NULL-only so a re-extract never re-stamps and
# the backfill (reextract task → this same path) is idempotent. Nested
# members link to this OUTER archive — the only one stored as a blob.
self._stamp_member_archive(post.id, source, member_record_ids)
if member_ids:
return ImportResult(
status="imported", image_id=member_ids[0],
@@ -555,6 +569,7 @@ class Importer:
self, archive_path: Path, *, attribution: Path,
source_row: Source | None, depth: int,
member_ids: list[int], counts: dict,
member_record_ids: set[int],
) -> None:
"""Extract `archive_path` and import its image/video members, RECURSING
into nested archives (#718). Members attribute to `attribution` — the
@@ -590,6 +605,7 @@ class Importer:
member_path, attribution=attribution,
source_row=source_row, depth=depth + 1,
member_ids=member_ids, counts=counts,
member_record_ids=member_record_ids,
)
continue
counts["media"] += 1
@@ -601,10 +617,16 @@ class Importer:
)
if res.status in ("imported", "superseded") and res.image_id:
member_ids.append(res.image_id)
member_record_ids.add(res.image_id)
elif res.status == "skipped" and res.skip_reason in (
SkipReason.duplicate_hash, SkipReason.duplicate_phash
):
counts["deduped"] += 1
# A deduped member still links provenance to this post
# (enrich-on-duplicate); record it so its archive origin
# gets stamped too.
if res.image_id:
member_record_ids.add(res.image_id)
else:
counts["failed"] += 1
except Exception as exc: # noqa: BLE001 — defensive per level; keep going
@@ -613,6 +635,39 @@ class Importer:
archive_path.name, depth, exc,
)
def _stamp_member_archive(
self, post_id: int | None, archive_source: Path, member_record_ids: set[int],
) -> None:
"""Record which archive each extracted member came from (milestone #87).
Resolves the archive's own PostAttachment (by post + sha — it was just
captured) and stamps from_attachment_id on every member's provenance row
FOR THIS POST. NULL-only, so re-extracting the same archive (the backfill
path) never overwrites and stays idempotent. No-op when the archive isn't
post-attached (filesystem import with no post) or yielded no members.
"""
if post_id is None or not member_record_ids:
return
sha = _sha256_of(archive_source)
att_id = self.session.execute(
select(PostAttachment.id).where(
PostAttachment.post_id == post_id,
PostAttachment.sha256 == sha,
)
).scalar_one_or_none()
if att_id is None:
return
self.session.execute(
update(ImageProvenance)
.where(
ImageProvenance.image_record_id.in_(member_record_ids),
ImageProvenance.post_id == post_id,
ImageProvenance.from_attachment_id.is_(None),
)
.values(from_attachment_id=att_id)
)
self.session.commit()
@staticmethod
def _video_aspect_matches(w, h, cw, ch) -> bool:
"""True when two (w,h) pairs share an aspect ratio within tolerance.
+30 -15
View File
@@ -66,6 +66,10 @@ class ProvenanceService:
).scalars().all()
return [_attachment_dict(a) for a in rows]
async def _attachment_by_id(self, attachment_id: int) -> list[dict]:
att = await self.session.get(PostAttachment, attachment_id)
return [_attachment_dict(att)] if att is not None else []
async def for_image(self, image_id: int) -> dict | None:
rec = await self.session.get(ImageRecord, image_id)
if rec is None:
@@ -85,22 +89,33 @@ class ProvenanceService:
)
rows = (await self.session.execute(stmt)).all()
post_ids = [ip.post_id for ip, _p, _s, _a in rows]
# Scope attachments to the image's ORIGINATING post only, not every
# post it's pHash-linked to. An image dupe gets a provenance row per
# post it reappears in (enrich-on-duplicate), and one of those is
# often a "High Resolution Files" mega-bundle carrying dozens of
# unrelated archives — aggregating across all linked posts ballooned
# the panel with files that have nothing to do with this image.
# primary_post_id is the post this file was actually captured from;
# fall back to all linked posts only when it's unset (older rows /
# filesystem imports). NB: FC stores archives as opaque blobs and
# never records which archive an extracted image came from, so we
# cannot scope tighter than the post — see milestone for the
# image->archive capture work.
attach_post_ids = (
[rec.primary_post_id] if rec.primary_post_id is not None else post_ids
# Prefer the EXACT archive this file came out of (milestone #87): if the
# originating post's provenance row records from_attachment_id, the image
# was extracted from that one .zip/.rar, so show only it — not the dozens
# of unrelated archives a "High Resolution Files" bundle post carries.
from_att_id = next(
(
ip.from_attachment_id
for ip, _p, _s, _a in rows
if ip.post_id == rec.primary_post_id
and ip.from_attachment_id is not None
),
None,
)
attachments = await self._attachments_for_posts(attach_post_ids)
if from_att_id is not None:
attachments = await self._attachment_by_id(from_att_id)
else:
# No recorded containing archive (loose download, or pre-backfill):
# scope to the originating post only, not every pHash-linked post.
# primary_post_id is the post this file was actually captured from;
# fall back to all linked posts when it's unset (older rows /
# filesystem imports).
attach_post_ids = (
[rec.primary_post_id]
if rec.primary_post_id is not None
else post_ids
)
attachments = await self._attachments_for_posts(attach_post_ids)
return {
"image_id": image_id,
"provenance": [