feat(ops): graceful shutdown — worker stop-grace + Interpreter drain resilience
Deploys (docker SIGTERM→SIGKILL, default 10s) were killing Celery jobs mid-flight. Give in-flight work room to drain and make interrupted work resume cleanly instead of stalling. - docker-compose.yml: stop_grace_period per lane (web 30s / worker 90s / scheduler 60s / maintenance-long 180s / ml-worker 120s) so warm shutdown can actually drain before SIGKILL. - celery_app.py: task_reject_on_worker_lost=True — a task killed past the grace window is re-queued (safe: idempotent + chunked, recovery sweeps re-drive stragglers). - interpreter_client.py: map 429/5xx (502/503/504) → InterpreterUnavailable and parse Retry-After (delta-seconds or HTTP-date); a draining Interpreter behind a reverse proxy no longer raises an opaque HTTPError. - translation.py: thread retry_after out of _translate_batch; retranslate_posts resumes after the Retry-After hint (or 60s default, capped 900s) on an interrupt with _reset_done=True, self-terminating via the health gate. - tests: 429/5xx mapping + Retry-After parse; interrupt-resume + default backoff. No migration. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -70,6 +70,14 @@ def make_celery() -> Celery:
|
||||
},
|
||||
# Heavy ML tasks need fair dispatch — see ImageRepo's precedent.
|
||||
task_acks_late=True,
|
||||
# Deploy graceful-shutdown safety: with acks_late, a task killed because
|
||||
# it outran the container's stop-grace window (SIGKILL) is re-queued
|
||||
# rather than silently lost. Safe because our long tasks are idempotent +
|
||||
# chunked (translation per-post commit, downloads terminal-status, audits
|
||||
# chunk) and the 5-min recovery sweeps re-drive anything left non-terminal
|
||||
# — a re-run resumes cleanly and never corrupts. No redeliver-loop risk:
|
||||
# heavy GPU work is tombstoned via gpu_queue, not run inline in a worker.
|
||||
task_reject_on_worker_lost=True,
|
||||
worker_prefetch_multiplier=1,
|
||||
# Broker resilience (2026-06-24): a swarm overlay-network blip after a
|
||||
# redeploy left Redis healthy but transiently unreachable, and a worker
|
||||
|
||||
Reference in New Issue
Block a user