fix(ingester): close #5 within-chunk live posts + #8 video transient retry
CI / lint (push) Successful in 2s
CI / frontend-build (push) Successful in 18s
CI / backend-lint-and-test (push) Successful in 25s
CI / integration (push) Successful in 2m59s

Review of the #1–#9 ingester roadmap found two real-but-small gaps; this closes
both.

#5 (live posts progress) shipped at per-chunk granularity — _apply_backfill_
lifecycle accumulated DownloadResult.posts_processed AFTER each chunk, so the
badge didn't move during a chunk (up to ~14.5 min) and over-counted the
re-walked resume page. The plan called for within-chunk live updates. Move
ownership of _backfill_posts into the ingester: ingest_core writes a monotonic
absolute (posts_base + net-new) via _checkpoint_posts at each page boundary and
once at the end, EXCLUDING the resumed page so it no longer inflates across
chunks. download_service seeds posts_base from prior chunks and stops touching
the key (the lifecycle now carries the ingester's committed value forward).

#8 (per-media transient/permanent retry) covered only the plain-GET path
(_fetch_to_file); the Mux/video path returned None on any yt-dlp failure with no
retry. Give _run_ytdlp the same split: TimeoutExpired/OSError are transient
(back off + retry up to _MAX_MEDIA_RETRIES), a non-zero exit (CalledProcessError)
is permanent (yt-dlp already did its own network retries) → fail fast to the
per-item/dead-letter path.

Tests: live-posts absolute + resume-page exclusion + tick-doesn't-persist
(test_patreon_ingester); lifecycle-leaves-posts-to-ingester rewrite
(test_download_service); video transient-retry + permanent-fail-fast
(test_patreon_downloader).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-06-06 09:56:26 -04:00
parent 9a2cd569c3
commit 697a86d31c
6 changed files with 194 additions and 25 deletions
+7 -6
View File
@@ -649,17 +649,18 @@ async def test_backfill_stuck_guard_stalls_after_two_no_advance(
@pytest.mark.asyncio
async def test_backfill_accumulates_posts_processed(
async def test_backfill_lifecycle_leaves_posts_to_ingester(
db, db_sync, tmp_path, seed_artist_and_source,
):
"""plan #704 (#5): the lifecycle accumulates DownloadResult.posts_processed
into _backfill_posts across chunks for the live progress badge."""
"""plan #704 (#5), revised: _backfill_posts is OWNED by the ingester now (it
writes a monotonic absolute live mid-walk), so the post-chunk lifecycle must
NOT touch it — it carries the ingester's committed value forward unchanged."""
from unittest.mock import MagicMock
from backend.app.services.download_service import DownloadService
_artist, source = seed_artist_and_source
source.config_overrides = {"_backfill_state": "running", "_backfill_posts": 5}
source.config_overrides = {"_backfill_state": "running", "_backfill_posts": 12}
source.backfill_runs_remaining = 5
await db.commit()
@@ -669,7 +670,7 @@ async def test_backfill_accumulates_posts_processed(
)
ctx = {
"source_id": source.id, "platform": "patreon",
"config_overrides": {"_backfill_state": "running", "_backfill_posts": 5},
"config_overrides": {"_backfill_state": "running", "_backfill_posts": 12},
"backfill_runs_remaining": 5,
}
await svc._apply_backfill_lifecycle(
@@ -679,7 +680,7 @@ async def test_backfill_accumulates_posts_processed(
co = (await db.execute(
select(Source.config_overrides).where(Source.id == source.id)
)).scalar_one()
assert co.get("_backfill_posts") == 15 # 5 (prior) + 10 (this chunk)
assert co.get("_backfill_posts") == 12 # untouched — the ingester owns it
@pytest.mark.asyncio