Review of the #1–#9 ingester roadmap found two real-but-small gaps; this closes both. #5 (live posts progress) shipped at per-chunk granularity — _apply_backfill_ lifecycle accumulated DownloadResult.posts_processed AFTER each chunk, so the badge didn't move during a chunk (up to ~14.5 min) and over-counted the re-walked resume page. The plan called for within-chunk live updates. Move ownership of _backfill_posts into the ingester: ingest_core writes a monotonic absolute (posts_base + net-new) via _checkpoint_posts at each page boundary and once at the end, EXCLUDING the resumed page so it no longer inflates across chunks. download_service seeds posts_base from prior chunks and stops touching the key (the lifecycle now carries the ingester's committed value forward). #8 (per-media transient/permanent retry) covered only the plain-GET path (_fetch_to_file); the Mux/video path returned None on any yt-dlp failure with no retry. Give _run_ytdlp the same split: TimeoutExpired/OSError are transient (back off + retry up to _MAX_MEDIA_RETRIES), a non-zero exit (CalledProcessError) is permanent (yt-dlp already did its own network retries) → fail fast to the per-item/dead-letter path. Tests: live-posts absolute + resume-page exclusion + tick-doesn't-persist (test_patreon_ingester); lifecycle-leaves-posts-to-ingester rewrite (test_download_service); video transient-retry + permanent-fail-fast (test_patreon_downloader). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -238,6 +238,9 @@ class DownloadService:
|
||||
mode=mode,
|
||||
resume_cursor=source_config.resume_cursor,
|
||||
time_budget_seconds=source_config.timeout,
|
||||
# plan #704 #5: seed the live posts badge from prior chunks so the
|
||||
# ingester writes a monotonic absolute mid-walk (it owns the key).
|
||||
posts_base=int(overrides.get("_backfill_posts", 0)),
|
||||
),
|
||||
)
|
||||
return dl_result, resolved_campaign_id
|
||||
@@ -515,13 +518,12 @@ class DownloadService:
|
||||
new_overrides = dict(src.config_overrides or {})
|
||||
chunks = int(new_overrides.get("_backfill_chunks", 0)) + 1
|
||||
new_overrides["_backfill_chunks"] = chunks
|
||||
# plan #704 (#5): accumulate posts-processed across chunks so the badge
|
||||
# shows live walk progress, not just the chunk count. Cleared on
|
||||
# (re)start. (Slight over-count: each chunk re-walks the resumed page —
|
||||
# fine for a progress indicator.)
|
||||
new_overrides["_backfill_posts"] = int(
|
||||
new_overrides.get("_backfill_posts", 0)
|
||||
) + int(getattr(dl_result, "posts_processed", 0) or 0)
|
||||
# plan #704 (#5): _backfill_posts (the live progress badge) is OWNED by the
|
||||
# ingester now — it writes a monotonic absolute mid-walk at each page
|
||||
# boundary (ingest_core._checkpoint_posts), so the badge climbs DURING a
|
||||
# chunk instead of jumping once per chunk here, and the re-walked resume
|
||||
# page is no longer double-counted. new_overrides (read fresh above)
|
||||
# carries the ingester's committed value forward untouched.
|
||||
|
||||
completed = (
|
||||
dl_result.success
|
||||
|
||||
Reference in New Issue
Block a user