fix(ingester): close #5 within-chunk live posts + #8 video transient retry
CI / lint (push) Successful in 2s
CI / frontend-build (push) Successful in 18s
CI / backend-lint-and-test (push) Successful in 25s
CI / integration (push) Successful in 2m59s

Review of the #1–#9 ingester roadmap found two real-but-small gaps; this closes
both.

#5 (live posts progress) shipped at per-chunk granularity — _apply_backfill_
lifecycle accumulated DownloadResult.posts_processed AFTER each chunk, so the
badge didn't move during a chunk (up to ~14.5 min) and over-counted the
re-walked resume page. The plan called for within-chunk live updates. Move
ownership of _backfill_posts into the ingester: ingest_core writes a monotonic
absolute (posts_base + net-new) via _checkpoint_posts at each page boundary and
once at the end, EXCLUDING the resumed page so it no longer inflates across
chunks. download_service seeds posts_base from prior chunks and stops touching
the key (the lifecycle now carries the ingester's committed value forward).

#8 (per-media transient/permanent retry) covered only the plain-GET path
(_fetch_to_file); the Mux/video path returned None on any yt-dlp failure with no
retry. Give _run_ytdlp the same split: TimeoutExpired/OSError are transient
(back off + retry up to _MAX_MEDIA_RETRIES), a non-zero exit (CalledProcessError)
is permanent (yt-dlp already did its own network retries) → fail fast to the
per-item/dead-letter path.

Tests: live-posts absolute + resume-page exclusion + tick-doesn't-persist
(test_patreon_ingester); lifecycle-leaves-posts-to-ingester rewrite
(test_download_service); video transient-retry + permanent-fail-fast
(test_patreon_downloader).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-06-06 09:56:26 -04:00
parent 9a2cd569c3
commit 697a86d31c
6 changed files with 194 additions and 25 deletions
+41 -11
View File
@@ -353,6 +353,14 @@ class PatreonDownloader:
The output template uses `dest` without its extension; yt-dlp appends
the chosen container extension. We pass Referer/Origin (Mux JWT policy)
and the cookies file.
Mirrors the GET path's transient/permanent split (plan #705 #8): a hung
fetch (TimeoutExpired) or a spawn failure (OSError) is TRANSIENT — back
off and retry. A non-zero yt-dlp exit (CalledProcessError) is treated as
PERMANENT for this pass — yt-dlp already does its OWN internal network
retries, so a non-zero exit is effectively a real failure (private/gone/
geo-blocked), like a 4xx on the GET path: fail fast to the per-item error
→ dead-letter path.
"""
dest = Path(dest)
out_template = str(dest.with_suffix("")) + ".%(ext)s"
@@ -362,17 +370,39 @@ class PatreonDownloader:
if self.cookies_path and os.path.isfile(self.cookies_path):
cmd += ["--cookies", self.cookies_path]
cmd.append(url)
try:
subprocess.run(
cmd,
check=True,
capture_output=True,
text=True,
timeout=_TIMEOUT_SECONDS,
)
except (OSError, subprocess.SubprocessError) as exc:
log.warning("yt-dlp failed for %s: %s", url, exc)
return None
attempt = 0
while True:
try:
subprocess.run(
cmd,
check=True,
capture_output=True,
text=True,
timeout=_TIMEOUT_SECONDS,
)
break
except subprocess.CalledProcessError as exc:
# Permanent for this pass — fail fast (no retry).
log.warning(
"yt-dlp failed (exit %s) for %s: %s",
exc.returncode, url, (exc.stderr or "").strip() or exc,
)
return None
except (OSError, subprocess.TimeoutExpired) as exc:
# Transient — back off and retry, like a transport blip on a GET.
if attempt >= _MAX_MEDIA_RETRIES:
log.warning(
"yt-dlp transient failure exhausted for %s: %s", url, exc
)
return None
attempt += 1
delay = min(2.0 * (2 ** (attempt - 1)), _BACKOFF_CAP_SECONDS)
log.warning(
"yt-dlp transient failure (%s) — backing off %.1fs "
"(retry %d/%d): %s",
url, delay, attempt, _MAX_MEDIA_RETRIES, exc,
)
time.sleep(delay)
return self._existing_video_output(dest)
def _existing_video_output(self, dest: Path) -> Path | None: