refactor(ingest): post-first — post-record is the sole body writer on the native path (#856)
Milestone #67 step 2. On the native core ingester the Post becomes the single authoritative record for body/links/metadata, captured once per post by the post-record; the per-media import only links image provenance + localization. Before: every per-media sidecar carried the full post body, so a post with N images wrote the body N+1 times (post-record + N media) — redundant on disk and a divergence risk (#753). gallery-dl is unchanged (its sidecar is still the only body source). - patreon_downloader: the per-media sidecar is now minimal — {category, id, source_url} only, no body. `_write_sidecar_data(minimal=True)` skips the body resolution + detail-fetch (the post-record, written first in the walk, already did it). Body no longer duplicated next to each image. - importer: new per-instance `post_first` flag (Importer is per-task). When set, `_apply_sidecar` still writes source_filehash + provenance + primary_post_id but SKIPS `_apply_post_fields` (the post-record owns body/links/raw_metadata, so applying a body-less sidecar would clobber raw_metadata + re-sync links off empty data). Default False keeps gallery-dl writing post fields. - download_service: `_phase3_persist` sets importer.post_first = uses_native_ingester(platform) — the future-proof seam, so a platform migrating onto the native core flips to post-first automatically (step 3). Media imports before post-records but both unify on external_post_id, so the post ends with its body either way. Tests: per-media sidecar is minimal + never hits the detail fetcher; attach post_first=True links provenance/localization but writes no post body/title; post_first=False (gallery-dl) still applies them. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
@@ -594,40 +594,34 @@ def test_range_ignored_by_server_refetches_clean(tmp_path, monkeypatch):
|
||||
# -- content enrichment (adaptive detail-fetch of an empty feed body) -------
|
||||
|
||||
|
||||
def test_enriches_empty_content_via_fetcher(tmp_path):
|
||||
"""An empty feed body is backfilled from the detail-fetch seam — once per
|
||||
post (memoized on the shared dict) — and lands in the importer sidecar."""
|
||||
def test_media_sidecar_is_minimal_post_first(tmp_path):
|
||||
"""Post-first (#856): the per-media sidecar carries ONLY image identity
|
||||
(category/id/source_url), NOT the post body — the post-record (`_post.json`)
|
||||
owns the body and is written first in the walk. So download_post never hits
|
||||
the detail fetcher (that's write_post_record's job), and the body is no longer
|
||||
duplicated next to every image."""
|
||||
calls: list[str] = []
|
||||
|
||||
def _fetcher(post_id: str) -> str:
|
||||
calls.append(post_id)
|
||||
return "<p>full body</p>"
|
||||
|
||||
dl = PatreonDownloader(
|
||||
images_root=tmp_path, cookies_path=None, validate=False,
|
||||
session=_FakeSession(), content_fetcher=_fetcher,
|
||||
session=_FakeSession(),
|
||||
content_fetcher=lambda pid: calls.append(pid) or "<p>full body</p>",
|
||||
)
|
||||
post = _post()
|
||||
post["attributes"]["content"] = "" # feed returned an empty body
|
||||
post["attributes"]["content"] = "" # even an empty feed body isn't fetched here
|
||||
dl.download_post(post, [_img("a.png"), _img("b.png")], "artist-x")
|
||||
|
||||
post_dir = tmp_path / "artist-x" / "patreon" / pd_mod._post_dir_name(post)
|
||||
sc = find_sidecar(post_dir / "01_a.png")
|
||||
assert sc is not None
|
||||
assert json.loads(sc.read_text())["content"] == "<p>full body</p>"
|
||||
# Fetched ONCE for the post, not once per media item.
|
||||
assert calls == ["1001"]
|
||||
|
||||
|
||||
def test_no_enrichment_when_feed_body_present(tmp_path):
|
||||
"""A non-empty feed body is trusted as-is; the detail endpoint isn't hit."""
|
||||
calls: list[str] = []
|
||||
dl = PatreonDownloader(
|
||||
images_root=tmp_path, cookies_path=None, validate=False,
|
||||
session=_FakeSession(),
|
||||
content_fetcher=lambda pid: calls.append(pid) or "x",
|
||||
)
|
||||
dl.download_post(_post(), [_img("a.png")], "artist-x") # _post body is non-empty
|
||||
data = json.loads(sc.read_text())
|
||||
assert data == {
|
||||
"category": "patreon",
|
||||
"id": "1001",
|
||||
"source_url": "https://cdn.patreon.com/a.png",
|
||||
}
|
||||
# No post body/title next to the image; the detail endpoint is never hit here.
|
||||
assert "content" not in data
|
||||
assert "title" not in data
|
||||
assert calls == []
|
||||
|
||||
|
||||
|
||||
Reference in New Issue
Block a user