refactor(ingest): post-first — post-record is the sole body writer on the native path (#856)
CI / lint (push) Successful in 3s
CI / frontend-build (push) Successful in 16s
CI / backend-lint-and-test (push) Failing after 25s
CI / integration (push) Successful in 3m20s

Milestone #67 step 2. On the native core ingester the Post becomes the single
authoritative record for body/links/metadata, captured once per post by the
post-record; the per-media import only links image provenance + localization.

Before: every per-media sidecar carried the full post body, so a post with N
images wrote the body N+1 times (post-record + N media) — redundant on disk and
a divergence risk (#753). gallery-dl is unchanged (its sidecar is still the only
body source).

- patreon_downloader: the per-media sidecar is now minimal — {category, id,
  source_url} only, no body. `_write_sidecar_data(minimal=True)` skips the body
  resolution + detail-fetch (the post-record, written first in the walk, already
  did it). Body no longer duplicated next to each image.
- importer: new per-instance `post_first` flag (Importer is per-task). When set,
  `_apply_sidecar` still writes source_filehash + provenance + primary_post_id
  but SKIPS `_apply_post_fields` (the post-record owns body/links/raw_metadata,
  so applying a body-less sidecar would clobber raw_metadata + re-sync links off
  empty data). Default False keeps gallery-dl writing post fields.
- download_service: `_phase3_persist` sets importer.post_first =
  uses_native_ingester(platform) — the future-proof seam, so a platform migrating
  onto the native core flips to post-first automatically (step 3). Media imports
  before post-records but both unify on external_post_id, so the post ends with
  its body either way.

Tests: per-media sidecar is minimal + never hits the detail fetcher; attach
post_first=True links provenance/localization but writes no post body/title;
post_first=False (gallery-dl) still applies them.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
2026-06-15 10:10:21 -04:00
parent 00607a309b
commit dcbc3ae335
5 changed files with 139 additions and 41 deletions
+25 -16
View File
@@ -529,29 +529,38 @@ class PatreonDownloader:
def _write_sidecar(
self, post: dict, media_path: Path, *, source_url: str | None = None
) -> Path:
"""Write the importer-consumed sidecar next to `media_path`.
"""Write the per-media sidecar next to `media_path` — post-first (#856).
Patreon uses base-default sidecar keys (parse_sidecar maps
category->platform, id->external_post_id, title->post_title,
content->description, published_at->post_date, url->post_url). Patreon
registers no derive_post_url, so `url` is trusted as the permalink — we
pass the post's attributes.url.
`source_url` is THIS file's CDN URL (#830 Phase 2). The importer persists
its filehash on the ImageRecord so the post body's inline `<img src>` can
be remapped to the local copy at render time.
On the native ingester the POST-RECORD (`write_post_record` → `_post.json`)
is the sole writer of the post body/links/metadata, captured once per post
BEFORE its media in the walk. So the per-media sidecar carries ONLY
image-specific identity: `category` (platform) + `id` (external_post_id, to
link provenance to the right Post) + this file's `source_url` (its CDN URL,
#830 Phase 2 — the importer persists its filehash so the body's inline
`<img src>` remaps to the local copy at render time). No body: writing it
next to every image duplicated the post body N+1× and risked divergence
(milestone #67). The importer skips post fields for these (post_first).
"""
return self._write_sidecar_data(
post, media_path.with_suffix(".json"), source_url=source_url
post, media_path.with_suffix(".json"), source_url=source_url,
minimal=True,
)
def _write_sidecar_data(
self, post: dict, sidecar_path: Path, *, source_url: str | None = None
self, post: dict, sidecar_path: Path, *, source_url: str | None = None,
minimal: bool = False,
) -> Path:
"""Serialize the post's metadata to `sidecar_path`. Shared by the
per-media sidecar (next to each file) and the post-only sidecar
(`write_post_record`, for media-less posts). `source_url` is set only for
the per-media sidecar — a media-less post has no source file."""
"""Serialize the post's metadata to `sidecar_path`. The post-only record
(`write_post_record`) writes the FULL post (body/title/date/url); the
per-media sidecar (`_write_sidecar`, minimal=True) writes only image
identity (category/id/source_url) — post-first (#856). `source_url` is set
only for the per-media sidecar — a media-less post has no source file."""
if minimal:
data = {"category": "patreon", "id": str(post.get("id") or "")}
if source_url:
data["source_url"] = source_url
sidecar_path.write_text(json.dumps(data, indent=2))
return sidecar_path
attrs = post.get("attributes") or {}
title = attrs.get("title")
# Resolve the body HTML from the feed attrs: legacy flat `content`, else