Review of the #1–#9 ingester roadmap found two real-but-small gaps; this closes
both.
#5 (live posts progress) shipped at per-chunk granularity — _apply_backfill_
lifecycle accumulated DownloadResult.posts_processed AFTER each chunk, so the
badge didn't move during a chunk (up to ~14.5 min) and over-counted the
re-walked resume page. The plan called for within-chunk live updates. Move
ownership of _backfill_posts into the ingester: ingest_core writes a monotonic
absolute (posts_base + net-new) via _checkpoint_posts at each page boundary and
once at the end, EXCLUDING the resumed page so it no longer inflates across
chunks. download_service seeds posts_base from prior chunks and stops touching
the key (the lifecycle now carries the ingester's committed value forward).
#8 (per-media transient/permanent retry) covered only the plain-GET path
(_fetch_to_file); the Mux/video path returned None on any yt-dlp failure with no
retry. Give _run_ytdlp the same split: TimeoutExpired/OSError are transient
(back off + retry up to _MAX_MEDIA_RETRIES), a non-zero exit (CalledProcessError)
is permanent (yt-dlp already did its own network retries) → fail fast to the
per-item/dead-letter path.
Tests: live-posts absolute + resume-page exclusion + tick-doesn't-persist
(test_patreon_ingester); lifecycle-leaves-posts-to-ingester rewrite
(test_download_service); video transient-retry + permanent-fail-fast
(test_patreon_downloader).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Factor the native-ingest orchestration out of PatreonIngester into a reusable
ingest_core.Ingester base, parametrized by client/downloader/ledger-models/
constraints/key/platform/error_base. PatreonIngester becomes a thin adapter:
it resolves the Patreon client/downloader, wires the seen/dead-letter models +
UNIQUE-constraint names + _ledger_key into super().__init__, and overrides
_failure_result with the Patreon exception taxonomy. Behavior-preserving — no
table rename, no migration; the public surface (PatreonIngester, _ledger_key,
DEAD_LETTER_THRESHOLD, verify_patreon_credential) is unchanged.
This is the strategic seam: SubscribeStar/etc. now migrate by writing a
~40-line adapter, not by re-implementing the tick/backfill/recovery walk,
tiered skip, checkpoint, and dead-letter logic.
run() moved to ingest_core, so the budget test's monotonic patch repoints to
ingest_core.time.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
bvandeusen