feat(patreon): capture media-less/text-only posts (post-only records)
Today the ingest core does `if not media: continue`, so a post with no downloadable media (a pure-text post — which often holds the ONLY copy of an external mega/gdrive/pixeldrain link) never upserts a Post. Now the native ingester emits a post-only sidecar (`_post.json`) for every media-less post, gated through the seen-ledger via a synthetic `post:<id>` key so the body is detail-fetched + recorded ONCE (not re-walked every tick); recovery bypasses the gate. Phase 3 imports these via Importer.upsert_post_record, keyed on external_post_id so it UPDATES the same Post a media import would create — never doubles, never clobbers a populated body with an empty one. - gallery_dl.py: DownloadResult.post_record_paths (default []; gallery-dl path unaffected — all constructions are keyword). - ingest_core.py: media-less branch (optional client/downloader seams via getattr; stub clients in tests skip it as before). - patreon_client.py: post_record_key(post). patreon_downloader.py: write_post_record + _write_sidecar_data refactor (shared serializer). - importer.py: upsert_post_record. download_service.py: phase-3 import loop. - tests: client/downloader/ingester (gate + recovery)/importer (no-double). Slice 0b of milestone #64. Refs FC #830. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -380,6 +380,22 @@ class DownloadService:
|
||||
else:
|
||||
import_summary["errors"] += 1
|
||||
|
||||
# Post-only records: media-less posts (pure-text) the native ingester
|
||||
# captured so the artist archive is complete. Upsert each (keyed on
|
||||
# external_post_id → updates the same Post a media import would create,
|
||||
# never doubles). No file to clean up; the sidecar stays on disk.
|
||||
for rec_str in dl_result.post_record_paths or []:
|
||||
rec_path = Path(rec_str)
|
||||
if not rec_path.exists(): # noqa: ASYNC240
|
||||
continue
|
||||
|
||||
def _upsert(p=rec_path):
|
||||
return self.importer.upsert_post_record(
|
||||
p, artist=artist, source=source_row,
|
||||
)
|
||||
|
||||
await loop.run_in_executor(None, _upsert)
|
||||
|
||||
ev = (await self.async_session.execute(
|
||||
select(DownloadEvent).where(DownloadEvent.id == event_id)
|
||||
)).scalar_one()
|
||||
|
||||
Reference in New Issue
Block a user