feat(patreon): resume partial media downloads via HTTP Range — B5 (plan #708)
Owning the media fetch means a mid-download transport cut no longer refetches from zero. _fetch_to_file now resumes: on a transient retry, if bytes already landed in the .part, it requests Range: bytes=<have>- and appends on a 206; falls back to a clean truncate-and-restart if the server ignores Range (200) or the range is past EOF (416). The .part staging means a non-range server never corrupts the output — worst case is the old behavior (refetch from zero). Tests: mid-stream cut resumes from the offset (asserts the Range header); a Range-ignoring server refetches clean (no double-write). Test session fakes updated to accept the new headers kwarg. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -301,18 +301,30 @@ class PatreonDownloader:
|
||||
|
||||
def _fetch_to_file(self, url: str, dest: Path) -> None:
|
||||
"""Stream a non-video URL to `dest` via the (stubbable) session, retrying
|
||||
TRANSIENT failures within the same pass (plan #705 #8).
|
||||
TRANSIENT failures within the same pass (plan #705 #8) and RESUMING from
|
||||
the bytes already on disk via a Range request when a retry follows a
|
||||
mid-download cut (plan #708 B5).
|
||||
|
||||
Retried (backoff): transport blips (connection reset / timeout /
|
||||
truncated stream — incl. mid-download), HTTP 429 (honoring Retry-After),
|
||||
and 5xx. Failed fast (no retry → HTTPError → per-item error → dead-letter
|
||||
path): 4xx other than 429 (404 gone, 403 forbidden) — re-fetching a
|
||||
permanent failure is pointless.
|
||||
|
||||
Resume: on a retry, if bytes already landed in `dest`, ask for the rest
|
||||
with `Range: bytes=<have>-`. A 206 means the server honored it → append; a
|
||||
200 means it ignored it (served the whole file) → start clean. The caller
|
||||
(_fetch_get) stages into a `.part`, so a non-range server never corrupts
|
||||
the output — the worst case is re-downloading from zero, as before.
|
||||
"""
|
||||
attempt = 0
|
||||
while True:
|
||||
have = dest.stat().st_size if dest.exists() else 0
|
||||
headers = {"Range": f"bytes={have}-"} if have > 0 else None
|
||||
try:
|
||||
resp = self.session.get(url, stream=True, timeout=_TIMEOUT_SECONDS)
|
||||
resp = self.session.get(
|
||||
url, stream=True, timeout=_TIMEOUT_SECONDS, headers=headers,
|
||||
)
|
||||
if (resp.status_code == 429 or resp.status_code >= 500) \
|
||||
and attempt < _MAX_MEDIA_RETRIES:
|
||||
attempt += 1
|
||||
@@ -324,10 +336,17 @@ class PatreonDownloader:
|
||||
)
|
||||
time.sleep(delay)
|
||||
continue
|
||||
# A Range that starts at/past EOF (we already have the whole file)
|
||||
# comes back 416 — the bytes we kept ARE the file.
|
||||
if have > 0 and resp.status_code == 416:
|
||||
return
|
||||
# 2xx → ok; 4xx-non-429 (or an exhausted 429/5xx) → HTTPError
|
||||
# (permanent for this pass) → not caught below → per-item error.
|
||||
resp.raise_for_status()
|
||||
with open(dest, "wb") as fh:
|
||||
# 206 → server honored the Range; append after the kept bytes.
|
||||
# Anything else (200) → it served the whole file → start clean.
|
||||
mode = "ab" if (have > 0 and resp.status_code == 206) else "wb"
|
||||
with open(dest, mode) as fh:
|
||||
for chunk in resp.iter_content(chunk_size=_CHUNK):
|
||||
if chunk:
|
||||
fh.write(chunk)
|
||||
|
||||
Reference in New Issue
Block a user