Review of the #1–#9 ingester roadmap found two real-but-small gaps; this closes both. #5 (live posts progress) shipped at per-chunk granularity — _apply_backfill_ lifecycle accumulated DownloadResult.posts_processed AFTER each chunk, so the badge didn't move during a chunk (up to ~14.5 min) and over-counted the re-walked resume page. The plan called for within-chunk live updates. Move ownership of _backfill_posts into the ingester: ingest_core writes a monotonic absolute (posts_base + net-new) via _checkpoint_posts at each page boundary and once at the end, EXCLUDING the resumed page so it no longer inflates across chunks. download_service seeds posts_base from prior chunks and stops touching the key (the lifecycle now carries the ingester's committed value forward). #8 (per-media transient/permanent retry) covered only the plain-GET path (_fetch_to_file); the Mux/video path returned None on any yt-dlp failure with no retry. Give _run_ytdlp the same split: TimeoutExpired/OSError are transient (back off + retry up to _MAX_MEDIA_RETRIES), a non-zero exit (CalledProcessError) is permanent (yt-dlp already did its own network retries) → fail fast to the per-item/dead-letter path. Tests: live-posts absolute + resume-page exclusion + tick-doesn't-persist (test_patreon_ingester); lifecycle-leaves-posts-to-ingester rewrite (test_download_service); video transient-retry + permanent-fail-fast (test_patreon_downloader). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -649,17 +649,18 @@ async def test_backfill_stuck_guard_stalls_after_two_no_advance(
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_backfill_accumulates_posts_processed(
|
||||
async def test_backfill_lifecycle_leaves_posts_to_ingester(
|
||||
db, db_sync, tmp_path, seed_artist_and_source,
|
||||
):
|
||||
"""plan #704 (#5): the lifecycle accumulates DownloadResult.posts_processed
|
||||
into _backfill_posts across chunks for the live progress badge."""
|
||||
"""plan #704 (#5), revised: _backfill_posts is OWNED by the ingester now (it
|
||||
writes a monotonic absolute live mid-walk), so the post-chunk lifecycle must
|
||||
NOT touch it — it carries the ingester's committed value forward unchanged."""
|
||||
from unittest.mock import MagicMock
|
||||
|
||||
from backend.app.services.download_service import DownloadService
|
||||
|
||||
_artist, source = seed_artist_and_source
|
||||
source.config_overrides = {"_backfill_state": "running", "_backfill_posts": 5}
|
||||
source.config_overrides = {"_backfill_state": "running", "_backfill_posts": 12}
|
||||
source.backfill_runs_remaining = 5
|
||||
await db.commit()
|
||||
|
||||
@@ -669,7 +670,7 @@ async def test_backfill_accumulates_posts_processed(
|
||||
)
|
||||
ctx = {
|
||||
"source_id": source.id, "platform": "patreon",
|
||||
"config_overrides": {"_backfill_state": "running", "_backfill_posts": 5},
|
||||
"config_overrides": {"_backfill_state": "running", "_backfill_posts": 12},
|
||||
"backfill_runs_remaining": 5,
|
||||
}
|
||||
await svc._apply_backfill_lifecycle(
|
||||
@@ -679,7 +680,7 @@ async def test_backfill_accumulates_posts_processed(
|
||||
co = (await db.execute(
|
||||
select(Source.config_overrides).where(Source.id == source.id)
|
||||
)).scalar_one()
|
||||
assert co.get("_backfill_posts") == 15 # 5 (prior) + 10 (this chunk)
|
||||
assert co.get("_backfill_posts") == 12 # untouched — the ingester owns it
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
|
||||
Reference in New Issue
Block a user