refactor(ingest): extract platform-agnostic Ingester core — roadmap #9 (plan #706)
CI / lint (push) Successful in 3s
CI / frontend-build (push) Successful in 24s
CI / backend-lint-and-test (push) Successful in 25s
CI / integration (push) Successful in 3m0s

Factor the native-ingest orchestration out of PatreonIngester into a reusable
ingest_core.Ingester base, parametrized by client/downloader/ledger-models/
constraints/key/platform/error_base. PatreonIngester becomes a thin adapter:
it resolves the Patreon client/downloader, wires the seen/dead-letter models +
UNIQUE-constraint names + _ledger_key into super().__init__, and overrides
_failure_result with the Patreon exception taxonomy. Behavior-preserving — no
table rename, no migration; the public surface (PatreonIngester, _ledger_key,
DEAD_LETTER_THRESHOLD, verify_patreon_credential) is unchanged.

This is the strategic seam: SubscribeStar/etc. now migrate by writing a
~40-line adapter, not by re-implementing the tick/backfill/recovery walk,
tiered skip, checkpoint, and dead-letter logic.

run() moved to ingest_core, so the budget test's monotonic patch repoints to
ingest_core.time.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-06-06 00:24:23 -04:00
parent d592e0ca02
commit 9a2cd569c3
3 changed files with 505 additions and 424 deletions
+4 -2
View File
@@ -300,7 +300,9 @@ async def test_backfill_budget_cut_returns_partial_with_progress(
source_id, sync_engine, tmp_path, monkeypatch,
):
# Deterministic clock: start=0, post1 check=10 (ok), post2 check=200 (>budget).
import backend.app.services.patreon_ingester as mod
# run() lives in ingest_core now, so patch the clock there (the same global
# time module object, but we reference it through the module that uses it).
import backend.app.services.ingest_core as core
ticks = iter([0.0, 10.0, 200.0, 250.0])
last = [0.0]
@@ -311,7 +313,7 @@ async def test_backfill_budget_cut_returns_partial_with_progress(
pass
return last[0]
monkeypatch.setattr(mod.time, "monotonic", fake_monotonic)
monkeypatch.setattr(core.time, "monotonic", fake_monotonic)
pages = [
("CUR1", [("p1", [_media("p1", 1)])]),