fix(ingester): close #5 within-chunk live posts + #8 video transient retry
CI / lint (push) Successful in 2s
CI / frontend-build (push) Successful in 18s
CI / backend-lint-and-test (push) Successful in 25s
CI / integration (push) Successful in 2m59s

Review of the #1–#9 ingester roadmap found two real-but-small gaps; this closes
both.

#5 (live posts progress) shipped at per-chunk granularity — _apply_backfill_
lifecycle accumulated DownloadResult.posts_processed AFTER each chunk, so the
badge didn't move during a chunk (up to ~14.5 min) and over-counted the
re-walked resume page. The plan called for within-chunk live updates. Move
ownership of _backfill_posts into the ingester: ingest_core writes a monotonic
absolute (posts_base + net-new) via _checkpoint_posts at each page boundary and
once at the end, EXCLUDING the resumed page so it no longer inflates across
chunks. download_service seeds posts_base from prior chunks and stops touching
the key (the lifecycle now carries the ingester's committed value forward).

#8 (per-media transient/permanent retry) covered only the plain-GET path
(_fetch_to_file); the Mux/video path returned None on any yt-dlp failure with no
retry. Give _run_ytdlp the same split: TimeoutExpired/OSError are transient
(back off + retry up to _MAX_MEDIA_RETRIES), a non-zero exit (CalledProcessError)
is permanent (yt-dlp already did its own network retries) → fail fast to the
per-item/dead-letter path.

Tests: live-posts absolute + resume-page exclusion + tick-doesn't-persist
(test_patreon_ingester); lifecycle-leaves-posts-to-ingester rewrite
(test_download_service); video transient-retry + permanent-fail-fast
(test_patreon_downloader).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-06-06 09:56:26 -04:00
parent 9a2cd569c3
commit 697a86d31c
6 changed files with 194 additions and 25 deletions
+9 -7
View File
@@ -238,6 +238,9 @@ class DownloadService:
mode=mode,
resume_cursor=source_config.resume_cursor,
time_budget_seconds=source_config.timeout,
# plan #704 #5: seed the live posts badge from prior chunks so the
# ingester writes a monotonic absolute mid-walk (it owns the key).
posts_base=int(overrides.get("_backfill_posts", 0)),
),
)
return dl_result, resolved_campaign_id
@@ -515,13 +518,12 @@ class DownloadService:
new_overrides = dict(src.config_overrides or {})
chunks = int(new_overrides.get("_backfill_chunks", 0)) + 1
new_overrides["_backfill_chunks"] = chunks
# plan #704 (#5): accumulate posts-processed across chunks so the badge
# shows live walk progress, not just the chunk count. Cleared on
# (re)start. (Slight over-count: each chunk re-walks the resumed page —
# fine for a progress indicator.)
new_overrides["_backfill_posts"] = int(
new_overrides.get("_backfill_posts", 0)
) + int(getattr(dl_result, "posts_processed", 0) or 0)
# plan #704 (#5): _backfill_posts (the live progress badge) is OWNED by the
# ingester now — it writes a monotonic absolute mid-walk at each page
# boundary (ingest_core._checkpoint_posts), so the badge climbs DURING a
# chunk instead of jumping once per chunk here, and the re-walked resume
# page is no longer double-counted. new_overrides (read fresh above)
# carries the ingester's committed value forward untouched.
completed = (
dl_result.success