feat(patreon): capture media-less/text-only posts (post-only records)
CI / lint (push) Successful in 3s
CI / frontend-build (push) Successful in 21s
CI / backend-lint-and-test (push) Successful in 34s
CI / integration (push) Failing after 3m26s

Today the ingest core does `if not media: continue`, so a post with no
downloadable media (a pure-text post — which often holds the ONLY copy of an
external mega/gdrive/pixeldrain link) never upserts a Post. Now the native
ingester emits a post-only sidecar (`_post.json`) for every media-less post,
gated through the seen-ledger via a synthetic `post:<id>` key so the body is
detail-fetched + recorded ONCE (not re-walked every tick); recovery bypasses
the gate. Phase 3 imports these via Importer.upsert_post_record, keyed on
external_post_id so it UPDATES the same Post a media import would create —
never doubles, never clobbers a populated body with an empty one.

- gallery_dl.py: DownloadResult.post_record_paths (default []; gallery-dl path
  unaffected — all constructions are keyword).
- ingest_core.py: media-less branch (optional client/downloader seams via
  getattr; stub clients in tests skip it as before).
- patreon_client.py: post_record_key(post). patreon_downloader.py:
  write_post_record + _write_sidecar_data refactor (shared serializer).
- importer.py: upsert_post_record. download_service.py: phase-3 import loop.
- tests: client/downloader/ingester (gate + recovery)/importer (no-double).

Slice 0b of milestone #64. Refs FC #830.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-06-14 12:44:46 -04:00
parent 2c67c27044
commit 796e92540a
10 changed files with 294 additions and 3 deletions
+34
View File
@@ -577,3 +577,37 @@ def test_no_enrichment_when_feed_body_present(tmp_path):
)
dl.download_post(_post(), [_img("a.png")], "artist-x") # _post body is non-empty
assert calls == []
# -- write_post_record (media-less / text-only post capture) ----------------
def test_write_post_record_writes_enriched_post_only_sidecar(tmp_path):
calls: list[str] = []
def _fetcher(post_id: str) -> str:
calls.append(post_id)
return "<p>text post body</p>"
dl = PatreonDownloader(
images_root=tmp_path, cookies_path=None, validate=False,
session=_FakeSession(), content_fetcher=_fetcher,
)
post = _post()
post["attributes"]["content"] = "" # media-less post, empty feed body
sc = dl.write_post_record(post, "artist-x")
assert sc is not None
assert sc.name == "_post.json" # can't collide with a media `NN_*.json`
data = json.loads(sc.read_text())
assert data["id"] == "1001"
assert data["content"] == "<p>text post body</p>"
assert calls == ["1001"]
def test_write_post_record_none_without_post_id(tmp_path):
dl = PatreonDownloader(
images_root=tmp_path, cookies_path=None, validate=False,
session=_FakeSession(),
)
assert dl.write_post_record({"id": "", "attributes": {}}, "artist-x") is None