refactor(ingest): post-first — post-record is the sole body writer on the native path (#856)
CI / integration (push) Successful in 3m20s
CI / lint (push) Successful in 3s
CI / frontend-build (push) Successful in 16s
CI / backend-lint-and-test (push) Failing after 25s

Milestone #67 step 2. On the native core ingester the Post becomes the single
authoritative record for body/links/metadata, captured once per post by the
post-record; the per-media import only links image provenance + localization.

Before: every per-media sidecar carried the full post body, so a post with N
images wrote the body N+1 times (post-record + N media) — redundant on disk and
a divergence risk (#753). gallery-dl is unchanged (its sidecar is still the only
body source).

- patreon_downloader: the per-media sidecar is now minimal — {category, id,
  source_url} only, no body. `_write_sidecar_data(minimal=True)` skips the body
  resolution + detail-fetch (the post-record, written first in the walk, already
  did it). Body no longer duplicated next to each image.
- importer: new per-instance `post_first` flag (Importer is per-task). When set,
  `_apply_sidecar` still writes source_filehash + provenance + primary_post_id
  but SKIPS `_apply_post_fields` (the post-record owns body/links/raw_metadata,
  so applying a body-less sidecar would clobber raw_metadata + re-sync links off
  empty data). Default False keeps gallery-dl writing post fields.
- download_service: `_phase3_persist` sets importer.post_first =
  uses_native_ingester(platform) — the future-proof seam, so a platform migrating
  onto the native core flips to post-first automatically (step 3). Media imports
  before post-records but both unify on external_post_id, so the post ends with
  its body either way.

Tests: per-media sidecar is minimal + never hits the detail fetcher; attach
post_first=True links provenance/localization but writes no post body/title;
post_first=False (gallery-dl) still applies them.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
2026-06-15 10:10:21 -04:00
parent 00607a309b
commit dcbc3ae335
5 changed files with 139 additions and 41 deletions
+16 -1
View File
@@ -149,6 +149,14 @@ class Importer:
self.thumbnailer = thumbnailer
self.settings = settings
self.deep = deep
# Post-first ingest (#856, milestone #67): on the NATIVE core ingester the
# post-record (`upsert_post_record`) is the SOLE writer of a post's
# body/links/metadata; the per-media import only links image provenance +
# localization, NOT the post body. download_service sets this True for
# native platforms before the phase-3 media loop. Default False keeps the
# gallery-dl path writing post fields via `_apply_sidecar` (unchanged).
# Safe as a per-instance flag: Importer is constructed per Celery task.
self.post_first = False
self.attachments = AttachmentStore(images_root)
# phash near-dup candidate cache. Archive imports call _import_media
# per-member; without this cache the per-member SELECT *FROM
@@ -1110,7 +1118,14 @@ class Importer:
external_post_id=epid,
artist_id=artist.id,
)
self._apply_post_fields(post, sd)
# Post-first (#856): on the native path the post-record owns the body/
# links/metadata, so the per-media import must NOT write post fields — its
# minimal sidecar carries no body, and applying it would clobber
# raw_metadata + re-run link-sync off empty data. The image still links
# provenance + localization below. gallery-dl (post_first False) is
# unchanged: its sidecar IS the only body source, so it still applies.
if not self.post_first:
self._apply_post_fields(post, sd)
# Race-safe (image_record_id, post_id) upsert — mirrors the
# _find_or_create_source/post savepoint pattern. The plain