refactor(ingest): post-first — post-record is the sole body writer on the native path (#856)
Milestone #67 step 2. On the native core ingester the Post becomes the single authoritative record for body/links/metadata, captured once per post by the post-record; the per-media import only links image provenance + localization. Before: every per-media sidecar carried the full post body, so a post with N images wrote the body N+1 times (post-record + N media) — redundant on disk and a divergence risk (#753). gallery-dl is unchanged (its sidecar is still the only body source). - patreon_downloader: the per-media sidecar is now minimal — {category, id, source_url} only, no body. `_write_sidecar_data(minimal=True)` skips the body resolution + detail-fetch (the post-record, written first in the walk, already did it). Body no longer duplicated next to each image. - importer: new per-instance `post_first` flag (Importer is per-task). When set, `_apply_sidecar` still writes source_filehash + provenance + primary_post_id but SKIPS `_apply_post_fields` (the post-record owns body/links/raw_metadata, so applying a body-less sidecar would clobber raw_metadata + re-sync links off empty data). Default False keeps gallery-dl writing post fields. - download_service: `_phase3_persist` sets importer.post_first = uses_native_ingester(platform) — the future-proof seam, so a platform migrating onto the native core flips to post-first automatically (step 3). Media imports before post-records but both unify on external_post_id, so the post ends with its body either way. Tests: per-media sidecar is minimal + never hits the detail fetcher; attach post_first=True links provenance/localization but writes no post body/title; post_first=False (gallery-dl) still applies them. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
@@ -149,6 +149,14 @@ class Importer:
|
||||
self.thumbnailer = thumbnailer
|
||||
self.settings = settings
|
||||
self.deep = deep
|
||||
# Post-first ingest (#856, milestone #67): on the NATIVE core ingester the
|
||||
# post-record (`upsert_post_record`) is the SOLE writer of a post's
|
||||
# body/links/metadata; the per-media import only links image provenance +
|
||||
# localization, NOT the post body. download_service sets this True for
|
||||
# native platforms before the phase-3 media loop. Default False keeps the
|
||||
# gallery-dl path writing post fields via `_apply_sidecar` (unchanged).
|
||||
# Safe as a per-instance flag: Importer is constructed per Celery task.
|
||||
self.post_first = False
|
||||
self.attachments = AttachmentStore(images_root)
|
||||
# phash near-dup candidate cache. Archive imports call _import_media
|
||||
# per-member; without this cache the per-member SELECT *FROM
|
||||
@@ -1110,7 +1118,14 @@ class Importer:
|
||||
external_post_id=epid,
|
||||
artist_id=artist.id,
|
||||
)
|
||||
self._apply_post_fields(post, sd)
|
||||
# Post-first (#856): on the native path the post-record owns the body/
|
||||
# links/metadata, so the per-media import must NOT write post fields — its
|
||||
# minimal sidecar carries no body, and applying it would clobber
|
||||
# raw_metadata + re-run link-sync off empty data. The image still links
|
||||
# provenance + localization below. gallery-dl (post_first False) is
|
||||
# unchanged: its sidecar IS the only body source, so it still applies.
|
||||
if not self.post_first:
|
||||
self._apply_post_fields(post, sd)
|
||||
|
||||
# Race-safe (image_record_id, post_id) upsert — mirrors the
|
||||
# _find_or_create_source/post savepoint pattern. The plain
|
||||
|
||||
Reference in New Issue
Block a user