fix(download): preserve partial output + classify timeouts richer
Operator-flagged 2026-05-30: "the fail state of timeouts doesn't show anything other than that the task timedout and was cleaned up. I can't tell why it ran over or if it was stuck failed or there was just that much to get." The TimeoutExpired branch was returning a DownloadResult with no stdout, no stderr, no files_downloaded, and a generic "Download timed out after N seconds" message — even though subprocess.TimeoutExpired carries the partial output gallery-dl emitted before being killed. Now: - Capture e.stdout / e.stderr (coerced str if bytes; "" if None). - Count files_downloaded from partial stdout via _count_downloaded_files. - Surface a tail-of-stderr hint in error_message so the UI summary tells the operator at a glance whether it was "lots of content" (high count, clean stderr), "stuck retrying" (any count, 429-spam stderr), or "hung silent" (zero count, "no stderr output"). - Promote error_type to RATE_LIMITED when the partial stderr matches RATE_LIMIT_PATTERNS — gallery-dl spinning on retries through the whole 900s window is the timeout-shaped tail of a real rate limit, and the platform cooldown should kick in for the same reason. Existing test_download_timeout strengthened to also assert empty-partial case stays correctly TIMEOUT-classified with no preserved output. New test_download_timeout_preserves_partial_output_and_classifies covers the rich-partial-output → RATE_LIMITED promotion path. DownloadEvent.metadata already flows stdout/stderr/run_stats from DownloadResult via _phase3_persist — no UI change needed; the existing DownloadDetailModal will surface the captured output automatically once the build redeploys. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -641,13 +641,57 @@ class GalleryDLService:
|
||||
started_at=started_at, completed_at=completed_at,
|
||||
)
|
||||
|
||||
except subprocess.TimeoutExpired:
|
||||
except subprocess.TimeoutExpired as e:
|
||||
duration = time.time() - start_time
|
||||
log.error("Download timeout for %s/%s after %.1fs", artist_slug, platform, duration)
|
||||
# subprocess.run(text=True) makes these str if non-None, but the
|
||||
# caller may have raised TimeoutExpired manually with None or
|
||||
# bytes (tests do); coerce both cases to str.
|
||||
partial_stdout = e.stdout or ""
|
||||
partial_stderr = e.stderr or ""
|
||||
if isinstance(partial_stdout, bytes):
|
||||
partial_stdout = partial_stdout.decode("utf-8", "replace")
|
||||
if isinstance(partial_stderr, bytes):
|
||||
partial_stderr = partial_stderr.decode("utf-8", "replace")
|
||||
|
||||
files_so_far = self._count_downloaded_files(partial_stdout)
|
||||
written_so_far = [str(p) for p in self._written_paths(partial_stdout)]
|
||||
stderr_lines = partial_stderr.strip().splitlines()
|
||||
tail_hint = stderr_lines[-1] if stderr_lines else "no stderr output"
|
||||
|
||||
# If the partial output already shows a rate-limit pattern, the
|
||||
# timeout was almost certainly gallery-dl spinning on retries —
|
||||
# promote to RATE_LIMITED so _update_source_health stamps the
|
||||
# platform cooldown (same code path as a clean-exit rate limit).
|
||||
# Otherwise stay TIMEOUT and let the captured stdout/stderr +
|
||||
# files_so_far tell the operator whether it was "lots of
|
||||
# content" vs "stuck retrying" vs "hung silent".
|
||||
combined = (partial_stdout + "\n" + partial_stderr).lower()
|
||||
if any(p in combined for p in self.RATE_LIMIT_PATTERNS):
|
||||
error_type = ErrorType.RATE_LIMITED
|
||||
error_message = (
|
||||
f"Rate-limited and never completed within "
|
||||
f"{source_config.timeout}s ({files_so_far} files written)"
|
||||
)
|
||||
else:
|
||||
error_type = ErrorType.TIMEOUT
|
||||
error_message = (
|
||||
f"Download timed out after {source_config.timeout}s — "
|
||||
f"{files_so_far} file(s) written; last stderr: {tail_hint}"
|
||||
)
|
||||
|
||||
log.error(
|
||||
"Download timeout for %s/%s after %.1fs (%d files written, "
|
||||
"last stderr: %s)",
|
||||
artist_slug, platform, duration, files_so_far, tail_hint,
|
||||
)
|
||||
|
||||
return DownloadResult(
|
||||
success=False, url=url, artist_slug=artist_slug, platform=platform,
|
||||
error_type=ErrorType.TIMEOUT,
|
||||
error_message=f"Download timed out after {source_config.timeout} seconds",
|
||||
files_downloaded=files_so_far,
|
||||
written_paths=written_so_far,
|
||||
stdout=partial_stdout, stderr=partial_stderr,
|
||||
return_code=-1, # killed by timeout, no real exit code
|
||||
error_type=error_type, error_message=error_message,
|
||||
duration_seconds=duration,
|
||||
started_at=started_at,
|
||||
completed_at=datetime.now(UTC).isoformat(),
|
||||
|
||||
Reference in New Issue
Block a user