fix(download): align gallery-dl subprocess timeout to Celery soft limit

SourceConfig.timeout defaulted to 3600s (1 hour), but download_source's
Celery task has soft_time_limit=900s and hard time_limit=1200s. So
gallery-dl never hit its own subprocess.run timeout — Celery always
killed it first. The hard SIGKILL leaves no terminal flip on the
DownloadEvent, which then sat pending/running until the recovery sweep
flipped it to error at 30 min from start. From the operator's seat that
was a ~30–40 min "hang" on every retry of a broken source.

Pinning the default to 900s (matching Celery's soft_time_limit) lets
subprocess.run raise TimeoutExpired cleanly inside Celery's window, and
the existing `except subprocess.TimeoutExpired` branch
(gallery_dl.py:635) captures it as a clean error_type='timeout' with a
real message. The DownloadEvent flips to error in ~15 min instead of
waiting on the recovery sweep at 30.

Per-source bumps still live in source.config_overrides for legitimately
long first-syncs. The new _DEFAULT_GDL_TIMEOUT_SECONDS constant carries
the rationale and the Celery-soft-limit dependency in its comment.

Operator-flagged 2026-05-30 on 59-source strand pile retries.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-05-30 02:25:04 -04:00
parent 77e9859da3
commit d28db32012
+11 -2
View File
@@ -42,6 +42,15 @@ class ErrorType(StrEnum):
UNKNOWN_ERROR = "unknown_error"
# Pinned to download_source's Celery soft_time_limit (900s, see
# tasks/download.py:32). Anything larger and Celery kills the task before
# subprocess.run can raise TimeoutExpired — leaving the DownloadEvent
# stranded for the recovery sweep instead of capturing a clean timeout
# error. Per-source bumps live in source.config_overrides for legitimately
# long syncs. Operator-confirmed 2026-05-30 (~40-min hang investigation).
_DEFAULT_GDL_TIMEOUT_SECONDS = 900
@dataclass
class SourceConfig:
content_types: list[str] = field(default_factory=lambda: ["all"])
@@ -51,7 +60,7 @@ class SourceConfig:
filename_pattern: str | None = None
skip_existing: bool = True
save_metadata: bool = True
timeout: int = 3600
timeout: int = _DEFAULT_GDL_TIMEOUT_SECONDS
@classmethod
def from_dict(cls, data: dict) -> SourceConfig:
@@ -63,7 +72,7 @@ class SourceConfig:
filename_pattern=data.get("filename_pattern"),
skip_existing=data.get("skip_existing", True),
save_metadata=data.get("save_metadata", True),
timeout=data.get("timeout", 3600),
timeout=data.get("timeout", _DEFAULT_GDL_TIMEOUT_SECONDS),
)