refactor(ingest): post-first — post-record is the sole body writer on the native path (#856)
Milestone #67 step 2. On the native core ingester the Post becomes the single authoritative record for body/links/metadata, captured once per post by the post-record; the per-media import only links image provenance + localization. Before: every per-media sidecar carried the full post body, so a post with N images wrote the body N+1 times (post-record + N media) — redundant on disk and a divergence risk (#753). gallery-dl is unchanged (its sidecar is still the only body source). - patreon_downloader: the per-media sidecar is now minimal — {category, id, source_url} only, no body. `_write_sidecar_data(minimal=True)` skips the body resolution + detail-fetch (the post-record, written first in the walk, already did it). Body no longer duplicated next to each image. - importer: new per-instance `post_first` flag (Importer is per-task). When set, `_apply_sidecar` still writes source_filehash + provenance + primary_post_id but SKIPS `_apply_post_fields` (the post-record owns body/links/raw_metadata, so applying a body-less sidecar would clobber raw_metadata + re-sync links off empty data). Default False keeps gallery-dl writing post fields. - download_service: `_phase3_persist` sets importer.post_first = uses_native_ingester(platform) — the future-proof seam, so a platform migrating onto the native core flips to post-first automatically (step 3). Media imports before post-records but both unify on external_post_id, so the post ends with its body either way. Tests: per-media sidecar is minimal + never hits the detail fetcher; attach post_first=True links provenance/localization but writes no post body/title; post_first=False (gallery-dl) still applies them. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
@@ -529,29 +529,38 @@ class PatreonDownloader:
|
||||
def _write_sidecar(
|
||||
self, post: dict, media_path: Path, *, source_url: str | None = None
|
||||
) -> Path:
|
||||
"""Write the importer-consumed sidecar next to `media_path`.
|
||||
"""Write the per-media sidecar next to `media_path` — post-first (#856).
|
||||
|
||||
Patreon uses base-default sidecar keys (parse_sidecar maps
|
||||
category->platform, id->external_post_id, title->post_title,
|
||||
content->description, published_at->post_date, url->post_url). Patreon
|
||||
registers no derive_post_url, so `url` is trusted as the permalink — we
|
||||
pass the post's attributes.url.
|
||||
|
||||
`source_url` is THIS file's CDN URL (#830 Phase 2). The importer persists
|
||||
its filehash on the ImageRecord so the post body's inline `<img src>` can
|
||||
be remapped to the local copy at render time.
|
||||
On the native ingester the POST-RECORD (`write_post_record` → `_post.json`)
|
||||
is the sole writer of the post body/links/metadata, captured once per post
|
||||
BEFORE its media in the walk. So the per-media sidecar carries ONLY
|
||||
image-specific identity: `category` (platform) + `id` (external_post_id, to
|
||||
link provenance to the right Post) + this file's `source_url` (its CDN URL,
|
||||
#830 Phase 2 — the importer persists its filehash so the body's inline
|
||||
`<img src>` remaps to the local copy at render time). No body: writing it
|
||||
next to every image duplicated the post body N+1× and risked divergence
|
||||
(milestone #67). The importer skips post fields for these (post_first).
|
||||
"""
|
||||
return self._write_sidecar_data(
|
||||
post, media_path.with_suffix(".json"), source_url=source_url
|
||||
post, media_path.with_suffix(".json"), source_url=source_url,
|
||||
minimal=True,
|
||||
)
|
||||
|
||||
def _write_sidecar_data(
|
||||
self, post: dict, sidecar_path: Path, *, source_url: str | None = None
|
||||
self, post: dict, sidecar_path: Path, *, source_url: str | None = None,
|
||||
minimal: bool = False,
|
||||
) -> Path:
|
||||
"""Serialize the post's metadata to `sidecar_path`. Shared by the
|
||||
per-media sidecar (next to each file) and the post-only sidecar
|
||||
(`write_post_record`, for media-less posts). `source_url` is set only for
|
||||
the per-media sidecar — a media-less post has no source file."""
|
||||
"""Serialize the post's metadata to `sidecar_path`. The post-only record
|
||||
(`write_post_record`) writes the FULL post (body/title/date/url); the
|
||||
per-media sidecar (`_write_sidecar`, minimal=True) writes only image
|
||||
identity (category/id/source_url) — post-first (#856). `source_url` is set
|
||||
only for the per-media sidecar — a media-less post has no source file."""
|
||||
if minimal:
|
||||
data = {"category": "patreon", "id": str(post.get("id") or "")}
|
||||
if source_url:
|
||||
data["source_url"] = source_url
|
||||
sidecar_path.write_text(json.dumps(data, indent=2))
|
||||
return sidecar_path
|
||||
attrs = post.get("attributes") or {}
|
||||
title = attrs.get("title")
|
||||
# Resolve the body HTML from the feed attrs: legacy flat `content`, else
|
||||
|
||||
Reference in New Issue
Block a user