refactor(ingest): post-first — post-record is the sole body writer on the native path (#856)
CI / lint (push) Successful in 3s
CI / frontend-build (push) Successful in 16s
CI / backend-lint-and-test (push) Failing after 25s
CI / integration (push) Successful in 3m20s

Milestone #67 step 2. On the native core ingester the Post becomes the single
authoritative record for body/links/metadata, captured once per post by the
post-record; the per-media import only links image provenance + localization.

Before: every per-media sidecar carried the full post body, so a post with N
images wrote the body N+1 times (post-record + N media) — redundant on disk and
a divergence risk (#753). gallery-dl is unchanged (its sidecar is still the only
body source).

- patreon_downloader: the per-media sidecar is now minimal — {category, id,
  source_url} only, no body. `_write_sidecar_data(minimal=True)` skips the body
  resolution + detail-fetch (the post-record, written first in the walk, already
  did it). Body no longer duplicated next to each image.
- importer: new per-instance `post_first` flag (Importer is per-task). When set,
  `_apply_sidecar` still writes source_filehash + provenance + primary_post_id
  but SKIPS `_apply_post_fields` (the post-record owns body/links/raw_metadata,
  so applying a body-less sidecar would clobber raw_metadata + re-sync links off
  empty data). Default False keeps gallery-dl writing post fields.
- download_service: `_phase3_persist` sets importer.post_first =
  uses_native_ingester(platform) — the future-proof seam, so a platform migrating
  onto the native core flips to post-first automatically (step 3). Media imports
  before post-records but both unify on external_post_id, so the post ends with
  its body either way.

Tests: per-media sidecar is minimal + never hits the detail fetcher; attach
post_first=True links provenance/localization but writes no post body/title;
post_first=False (gallery-dl) still applies them.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
2026-06-15 10:10:21 -04:00
parent 00607a309b
commit dcbc3ae335
5 changed files with 139 additions and 41 deletions
+18 -24
View File
@@ -594,40 +594,34 @@ def test_range_ignored_by_server_refetches_clean(tmp_path, monkeypatch):
# -- content enrichment (adaptive detail-fetch of an empty feed body) -------
def test_enriches_empty_content_via_fetcher(tmp_path):
"""An empty feed body is backfilled from the detail-fetch seam — once per
post (memoized on the shared dict) — and lands in the importer sidecar."""
def test_media_sidecar_is_minimal_post_first(tmp_path):
"""Post-first (#856): the per-media sidecar carries ONLY image identity
(category/id/source_url), NOT the post body — the post-record (`_post.json`)
owns the body and is written first in the walk. So download_post never hits
the detail fetcher (that's write_post_record's job), and the body is no longer
duplicated next to every image."""
calls: list[str] = []
def _fetcher(post_id: str) -> str:
calls.append(post_id)
return "<p>full body</p>"
dl = PatreonDownloader(
images_root=tmp_path, cookies_path=None, validate=False,
session=_FakeSession(), content_fetcher=_fetcher,
session=_FakeSession(),
content_fetcher=lambda pid: calls.append(pid) or "<p>full body</p>",
)
post = _post()
post["attributes"]["content"] = "" # feed returned an empty body
post["attributes"]["content"] = "" # even an empty feed body isn't fetched here
dl.download_post(post, [_img("a.png"), _img("b.png")], "artist-x")
post_dir = tmp_path / "artist-x" / "patreon" / pd_mod._post_dir_name(post)
sc = find_sidecar(post_dir / "01_a.png")
assert sc is not None
assert json.loads(sc.read_text())["content"] == "<p>full body</p>"
# Fetched ONCE for the post, not once per media item.
assert calls == ["1001"]
def test_no_enrichment_when_feed_body_present(tmp_path):
"""A non-empty feed body is trusted as-is; the detail endpoint isn't hit."""
calls: list[str] = []
dl = PatreonDownloader(
images_root=tmp_path, cookies_path=None, validate=False,
session=_FakeSession(),
content_fetcher=lambda pid: calls.append(pid) or "x",
)
dl.download_post(_post(), [_img("a.png")], "artist-x") # _post body is non-empty
data = json.loads(sc.read_text())
assert data == {
"category": "patreon",
"id": "1001",
"source_url": "https://cdn.patreon.com/a.png",
}
# No post body/title next to the image; the detail endpoint is never hit here.
assert "content" not in data
assert "title" not in data
assert calls == []