Review of the #1–#9 ingester roadmap found two real-but-small gaps; this closes both. #5 (live posts progress) shipped at per-chunk granularity — _apply_backfill_ lifecycle accumulated DownloadResult.posts_processed AFTER each chunk, so the badge didn't move during a chunk (up to ~14.5 min) and over-counted the re-walked resume page. The plan called for within-chunk live updates. Move ownership of _backfill_posts into the ingester: ingest_core writes a monotonic absolute (posts_base + net-new) via _checkpoint_posts at each page boundary and once at the end, EXCLUDING the resumed page so it no longer inflates across chunks. download_service seeds posts_base from prior chunks and stops touching the key (the lifecycle now carries the ingester's committed value forward). #8 (per-media transient/permanent retry) covered only the plain-GET path (_fetch_to_file); the Mux/video path returned None on any yt-dlp failure with no retry. Give _run_ytdlp the same split: TimeoutExpired/OSError are transient (back off + retry up to _MAX_MEDIA_RETRIES), a non-zero exit (CalledProcessError) is permanent (yt-dlp already did its own network retries) → fail fast to the per-item/dead-letter path. Tests: live-posts absolute + resume-page exclusion + tick-doesn't-persist (test_patreon_ingester); lifecycle-leaves-posts-to-ingester rewrite (test_download_service); video transient-retry + permanent-fail-fast (test_patreon_downloader). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -353,6 +353,14 @@ class PatreonDownloader:
|
||||
The output template uses `dest` without its extension; yt-dlp appends
|
||||
the chosen container extension. We pass Referer/Origin (Mux JWT policy)
|
||||
and the cookies file.
|
||||
|
||||
Mirrors the GET path's transient/permanent split (plan #705 #8): a hung
|
||||
fetch (TimeoutExpired) or a spawn failure (OSError) is TRANSIENT — back
|
||||
off and retry. A non-zero yt-dlp exit (CalledProcessError) is treated as
|
||||
PERMANENT for this pass — yt-dlp already does its OWN internal network
|
||||
retries, so a non-zero exit is effectively a real failure (private/gone/
|
||||
geo-blocked), like a 4xx on the GET path: fail fast to the per-item error
|
||||
→ dead-letter path.
|
||||
"""
|
||||
dest = Path(dest)
|
||||
out_template = str(dest.with_suffix("")) + ".%(ext)s"
|
||||
@@ -362,17 +370,39 @@ class PatreonDownloader:
|
||||
if self.cookies_path and os.path.isfile(self.cookies_path):
|
||||
cmd += ["--cookies", self.cookies_path]
|
||||
cmd.append(url)
|
||||
try:
|
||||
subprocess.run(
|
||||
cmd,
|
||||
check=True,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=_TIMEOUT_SECONDS,
|
||||
)
|
||||
except (OSError, subprocess.SubprocessError) as exc:
|
||||
log.warning("yt-dlp failed for %s: %s", url, exc)
|
||||
return None
|
||||
attempt = 0
|
||||
while True:
|
||||
try:
|
||||
subprocess.run(
|
||||
cmd,
|
||||
check=True,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=_TIMEOUT_SECONDS,
|
||||
)
|
||||
break
|
||||
except subprocess.CalledProcessError as exc:
|
||||
# Permanent for this pass — fail fast (no retry).
|
||||
log.warning(
|
||||
"yt-dlp failed (exit %s) for %s: %s",
|
||||
exc.returncode, url, (exc.stderr or "").strip() or exc,
|
||||
)
|
||||
return None
|
||||
except (OSError, subprocess.TimeoutExpired) as exc:
|
||||
# Transient — back off and retry, like a transport blip on a GET.
|
||||
if attempt >= _MAX_MEDIA_RETRIES:
|
||||
log.warning(
|
||||
"yt-dlp transient failure exhausted for %s: %s", url, exc
|
||||
)
|
||||
return None
|
||||
attempt += 1
|
||||
delay = min(2.0 * (2 ** (attempt - 1)), _BACKOFF_CAP_SECONDS)
|
||||
log.warning(
|
||||
"yt-dlp transient failure (%s) — backing off %.1fs "
|
||||
"(retry %d/%d): %s",
|
||||
url, delay, attempt, _MAX_MEDIA_RETRIES, exc,
|
||||
)
|
||||
time.sleep(delay)
|
||||
return self._existing_video_output(dest)
|
||||
|
||||
def _existing_video_output(self, dest: Path) -> Path | None:
|
||||
|
||||
Reference in New Issue
Block a user