feat(patreon): capture media-less/text-only posts (post-only records)
Today the ingest core does `if not media: continue`, so a post with no downloadable media (a pure-text post — which often holds the ONLY copy of an external mega/gdrive/pixeldrain link) never upserts a Post. Now the native ingester emits a post-only sidecar (`_post.json`) for every media-less post, gated through the seen-ledger via a synthetic `post:<id>` key so the body is detail-fetched + recorded ONCE (not re-walked every tick); recovery bypasses the gate. Phase 3 imports these via Importer.upsert_post_record, keyed on external_post_id so it UPDATES the same Post a media import would create — never doubles, never clobbers a populated body with an empty one. - gallery_dl.py: DownloadResult.post_record_paths (default []; gallery-dl path unaffected — all constructions are keyword). - ingest_core.py: media-less branch (optional client/downloader seams via getattr; stub clients in tests skip it as before). - patreon_client.py: post_record_key(post). patreon_downloader.py: write_post_record + _write_sidecar_data refactor (shared serializer). - importer.py: upsert_post_record. download_service.py: phase-3 import loop. - tests: client/downloader/ingester (gate + recovery)/importer (no-double). Slice 0b of milestone #64. Refs FC #830. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -118,10 +118,16 @@ class Ingester:
|
||||
# tick has no resumable backfill state.
|
||||
checkpoint = mode in ("backfill", "recovery")
|
||||
ledger_key = self._ledger_key
|
||||
# Optional seams (Patreon native ingester): capture pure-text posts that
|
||||
# have NO downloadable media so the artist archive is complete. Absent on
|
||||
# stub clients/downloaders (unit tests) → media-less posts skipped as before.
|
||||
post_record_key = getattr(self.client, "post_record_key", None)
|
||||
write_post_record = getattr(self.downloader, "write_post_record", None)
|
||||
start = time.monotonic()
|
||||
last_live = start # plan #709: last live-progress write timestamp
|
||||
log_lines: list[str] = []
|
||||
written: list[str] = []
|
||||
post_records: list[str] = []
|
||||
quarantined_paths: list[str] = []
|
||||
downloaded = 0
|
||||
errors = 0
|
||||
@@ -158,6 +164,7 @@ class Ingester:
|
||||
files_quarantined=quarantined,
|
||||
quarantined_paths=list(quarantined_paths),
|
||||
written_paths=written,
|
||||
post_record_paths=list(post_records),
|
||||
stdout="\n".join(log_lines),
|
||||
stderr="",
|
||||
return_code=return_code,
|
||||
@@ -222,6 +229,23 @@ class Ingester:
|
||||
chunk_new_posts += 1
|
||||
media = self.client.extract_media(post, included)
|
||||
if not media:
|
||||
# No downloadable media — still capture the post itself
|
||||
# (title/body/external links) so text-only posts aren't lost.
|
||||
# Gate via the seen-ledger (synthetic post key) so a text post
|
||||
# is detail-fetched + recorded ONCE, not re-walked every tick.
|
||||
if post_record_key and write_post_record:
|
||||
rk = post_record_key(post)
|
||||
if rk is not None:
|
||||
pkey, ppid = rk
|
||||
already = (
|
||||
set() if bypass_seen
|
||||
else self._seen_keys(source_id, [pkey])
|
||||
)
|
||||
if pkey not in already:
|
||||
rec_path = write_post_record(post, artist_slug)
|
||||
if rec_path is not None:
|
||||
post_records.append(str(rec_path))
|
||||
self._mark_seen(source_id, [(pkey, ppid)])
|
||||
continue
|
||||
|
||||
keys = [ledger_key(m) for m in media]
|
||||
|
||||
Reference in New Issue
Block a user