feat(ops): graceful shutdown — worker stop-grace + Interpreter drain resilience
Deploys (docker SIGTERM→SIGKILL, default 10s) were killing Celery jobs mid-flight. Give in-flight work room to drain and make interrupted work resume cleanly instead of stalling. - docker-compose.yml: stop_grace_period per lane (web 30s / worker 90s / scheduler 60s / maintenance-long 180s / ml-worker 120s) so warm shutdown can actually drain before SIGKILL. - celery_app.py: task_reject_on_worker_lost=True — a task killed past the grace window is re-queued (safe: idempotent + chunked, recovery sweeps re-drive stragglers). - interpreter_client.py: map 429/5xx (502/503/504) → InterpreterUnavailable and parse Retry-After (delta-seconds or HTTP-date); a draining Interpreter behind a reverse proxy no longer raises an opaque HTTPError. - translation.py: thread retry_after out of _translate_batch; retranslate_posts resumes after the Retry-After hint (or 60s default, capped 900s) on an interrupt with _reset_done=True, self-terminating via the health gate. - tests: 429/5xx mapping + Retry-After parse; interrupt-resume + default backoff. No migration. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -47,6 +47,14 @@ services:
|
||||
web:
|
||||
image: git.fabledsword.com/bvandeusen/fabledcurator:dev
|
||||
command: ["web"]
|
||||
# Graceful shutdown: give the container time to drain in-flight work on a
|
||||
# deploy (docker SIGTERMs, then SIGKILLs after this window — default is only
|
||||
# 10s, far too short for real jobs). Hypercorn/Celery both warm-shut-down on
|
||||
# SIGTERM; per-lane values sized to typical task length. Anything that still
|
||||
# outruns the window is re-queued (task_reject_on_worker_lost) and re-driven
|
||||
# by the 5-min recovery sweeps, so a kill never corrupts. web = short HTTP
|
||||
# requests + the occasional file download.
|
||||
stop_grace_period: 30s
|
||||
ports:
|
||||
- "${PORT:-8080}:8080"
|
||||
environment: &app_env
|
||||
@@ -77,6 +85,8 @@ services:
|
||||
worker:
|
||||
image: git.fabledsword.com/bvandeusen/fabledcurator:dev
|
||||
command: ["worker"]
|
||||
# Drain in-flight import/thumbnail/download tasks before SIGKILL on deploy.
|
||||
stop_grace_period: 90s
|
||||
environment:
|
||||
<<: *app_env
|
||||
CELERY_QUEUES: default,import,thumbnail,download
|
||||
@@ -93,6 +103,8 @@ services:
|
||||
scheduler:
|
||||
image: git.fabledsword.com/bvandeusen/fabledcurator:dev
|
||||
command: ["scheduler"]
|
||||
# Quick maintenance/scan lane + beat — short tasks, modest drain window.
|
||||
stop_grace_period: 60s
|
||||
environment:
|
||||
<<: *app_env
|
||||
CELERY_QUEUES: maintenance,scan
|
||||
@@ -110,6 +122,10 @@ services:
|
||||
maintenance-long:
|
||||
image: git.fabledsword.com/bvandeusen/fabledcurator:dev
|
||||
command: ["worker"]
|
||||
# Longest lane (DB backups, library audits, translation backfill) — give it
|
||||
# the most room to finish a chunk gracefully. Chunked + idempotent, so a job
|
||||
# that still outruns this resumes cleanly next run rather than corrupting.
|
||||
stop_grace_period: 180s
|
||||
environment:
|
||||
<<: *app_env
|
||||
CELERY_QUEUES: maintenance_long
|
||||
@@ -125,6 +141,8 @@ services:
|
||||
ml-worker:
|
||||
image: git.fabledsword.com/bvandeusen/fabledcurator-ml:dev
|
||||
command: ["ml-worker"]
|
||||
# A single GPU inference pass can run tens of seconds — let it finish.
|
||||
stop_grace_period: 120s
|
||||
environment:
|
||||
<<: *app_env
|
||||
volumes:
|
||||
|
||||
Reference in New Issue
Block a user