feat(patreon): capture media-less/text-only posts (post-only records)
CI / lint (push) Successful in 3s
CI / frontend-build (push) Successful in 21s
CI / backend-lint-and-test (push) Successful in 34s
CI / integration (push) Failing after 3m26s

Today the ingest core does `if not media: continue`, so a post with no
downloadable media (a pure-text post — which often holds the ONLY copy of an
external mega/gdrive/pixeldrain link) never upserts a Post. Now the native
ingester emits a post-only sidecar (`_post.json`) for every media-less post,
gated through the seen-ledger via a synthetic `post:<id>` key so the body is
detail-fetched + recorded ONCE (not re-walked every tick); recovery bypasses
the gate. Phase 3 imports these via Importer.upsert_post_record, keyed on
external_post_id so it UPDATES the same Post a media import would create —
never doubles, never clobbers a populated body with an empty one.

- gallery_dl.py: DownloadResult.post_record_paths (default []; gallery-dl path
  unaffected — all constructions are keyword).
- ingest_core.py: media-less branch (optional client/downloader seams via
  getattr; stub clients in tests skip it as before).
- patreon_client.py: post_record_key(post). patreon_downloader.py:
  write_post_record + _write_sidecar_data refactor (shared serializer).
- importer.py: upsert_post_record. download_service.py: phase-3 import loop.
- tests: client/downloader/ingester (gate + recovery)/importer (no-double).

Slice 0b of milestone #64. Refs FC #830.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-06-14 12:44:46 -04:00
parent 2c67c27044
commit 796e92540a
10 changed files with 294 additions and 3 deletions
+24
View File
@@ -118,10 +118,16 @@ class Ingester:
# tick has no resumable backfill state.
checkpoint = mode in ("backfill", "recovery")
ledger_key = self._ledger_key
# Optional seams (Patreon native ingester): capture pure-text posts that
# have NO downloadable media so the artist archive is complete. Absent on
# stub clients/downloaders (unit tests) → media-less posts skipped as before.
post_record_key = getattr(self.client, "post_record_key", None)
write_post_record = getattr(self.downloader, "write_post_record", None)
start = time.monotonic()
last_live = start # plan #709: last live-progress write timestamp
log_lines: list[str] = []
written: list[str] = []
post_records: list[str] = []
quarantined_paths: list[str] = []
downloaded = 0
errors = 0
@@ -158,6 +164,7 @@ class Ingester:
files_quarantined=quarantined,
quarantined_paths=list(quarantined_paths),
written_paths=written,
post_record_paths=list(post_records),
stdout="\n".join(log_lines),
stderr="",
return_code=return_code,
@@ -222,6 +229,23 @@ class Ingester:
chunk_new_posts += 1
media = self.client.extract_media(post, included)
if not media:
# No downloadable media — still capture the post itself
# (title/body/external links) so text-only posts aren't lost.
# Gate via the seen-ledger (synthetic post key) so a text post
# is detail-fetched + recorded ONCE, not re-walked every tick.
if post_record_key and write_post_record:
rk = post_record_key(post)
if rk is not None:
pkey, ppid = rk
already = (
set() if bypass_seen
else self._seen_keys(source_id, [pkey])
)
if pkey not in already:
rec_path = write_post_record(post, artist_slug)
if rec_path is not None:
post_records.append(str(rec_path))
self._mark_seen(source_id, [(pkey, ppid)])
continue
keys = [ledger_key(m) for m in media]