fix(maint): raise recovery-sweep threshold for fetch_external_link (#883)
External file-host fetches run to a 60-min hard limit (time_limit=3600, per-fetch _FETCH_TIMEOUT=3000s), far longer than the recovery sweep's 5-min default. recover_stalled_task_runs was phantom-flagging healthy in-flight fetches as "RecoverySweep: no completion signal received within 5 min" before the task's own timeout/error handling could surface the real error (operator-flagged: target 414 swept at 6.6min). The sweep already has per-queue/per-task overrides for long tasks, but fetch_external_link was never added and its TaskRun records queue='default' (no queue override) despite external.* routing to download. Add a task-name override of 65 min (time_limit 60 + 5 buffer); task-name precedence makes it robust regardless of the recorded queue. No new internal timeout needed — the existing _FETCH_TIMEOUT + soft_time_limit + except-block log.exception already capture the real failure once the sweep stops preempting. Pinned tests: external-fetch override survives a 10-min row / flags a 70-min row on queue='default'; invariant guard asserts override >= hard time_limit. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
@@ -147,6 +147,14 @@ TASK_STUCK_THRESHOLD_MINUTES: dict[str, int] = {
|
||||
"backend.app.tasks.backup.restore_images_task": 420,
|
||||
# Library audit scans the full library — 2h hard limit.
|
||||
"backend.app.tasks.library_audit.scan_library_for_rule": 130,
|
||||
# External file-host fetches (mega/gdrive/film packs) legitimately run to
|
||||
# the task's 60-min hard limit (time_limit=3600; per-fetch _FETCH_TIMEOUT
|
||||
# is 50min). Its TaskRun records queue='default' (no queue override), so
|
||||
# without this it fell to the 5-min default and healthy in-flight fetches
|
||||
# were phantom-flagged 'RecoverySweep' before their own timeout/error could
|
||||
# surface (operator-flagged 2026-06-17, target 414 swept at 6.6min). A
|
||||
# task-name override is robust whatever queue the row records. 65 = 60 + 5.
|
||||
"backend.app.tasks.external.fetch_external_link": 65,
|
||||
}
|
||||
|
||||
|
||||
|
||||
Reference in New Issue
Block a user