feat(ops): graceful shutdown — worker stop-grace + Interpreter drain resilience
CI / frontend-build (push) Successful in 19s
CI / lint (push) Failing after 2s
CI / backend-lint-and-test (push) Successful in 35s
CI / integration (push) Successful in 3m42s

Deploys (docker SIGTERM→SIGKILL, default 10s) were killing Celery jobs
mid-flight. Give in-flight work room to drain and make interrupted work
resume cleanly instead of stalling.

- docker-compose.yml: stop_grace_period per lane (web 30s / worker 90s /
  scheduler 60s / maintenance-long 180s / ml-worker 120s) so warm shutdown
  can actually drain before SIGKILL.
- celery_app.py: task_reject_on_worker_lost=True — a task killed past the
  grace window is re-queued (safe: idempotent + chunked, recovery sweeps
  re-drive stragglers).
- interpreter_client.py: map 429/5xx (502/503/504) → InterpreterUnavailable
  and parse Retry-After (delta-seconds or HTTP-date); a draining Interpreter
  behind a reverse proxy no longer raises an opaque HTTPError.
- translation.py: thread retry_after out of _translate_batch; retranslate_posts
  resumes after the Retry-After hint (or 60s default, capped 900s) on an
  interrupt with _reset_done=True, self-terminating via the health gate.
- tests: 429/5xx mapping + Retry-After parse; interrupt-resume + default backoff.

No migration.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-07-07 21:01:00 -04:00
parent 1f6d94f51d
commit c64261593d
6 changed files with 215 additions and 19 deletions
+18
View File
@@ -47,6 +47,14 @@ services:
web:
image: git.fabledsword.com/bvandeusen/fabledcurator:dev
command: ["web"]
# Graceful shutdown: give the container time to drain in-flight work on a
# deploy (docker SIGTERMs, then SIGKILLs after this window — default is only
# 10s, far too short for real jobs). Hypercorn/Celery both warm-shut-down on
# SIGTERM; per-lane values sized to typical task length. Anything that still
# outruns the window is re-queued (task_reject_on_worker_lost) and re-driven
# by the 5-min recovery sweeps, so a kill never corrupts. web = short HTTP
# requests + the occasional file download.
stop_grace_period: 30s
ports:
- "${PORT:-8080}:8080"
environment: &app_env
@@ -77,6 +85,8 @@ services:
worker:
image: git.fabledsword.com/bvandeusen/fabledcurator:dev
command: ["worker"]
# Drain in-flight import/thumbnail/download tasks before SIGKILL on deploy.
stop_grace_period: 90s
environment:
<<: *app_env
CELERY_QUEUES: default,import,thumbnail,download
@@ -93,6 +103,8 @@ services:
scheduler:
image: git.fabledsword.com/bvandeusen/fabledcurator:dev
command: ["scheduler"]
# Quick maintenance/scan lane + beat — short tasks, modest drain window.
stop_grace_period: 60s
environment:
<<: *app_env
CELERY_QUEUES: maintenance,scan
@@ -110,6 +122,10 @@ services:
maintenance-long:
image: git.fabledsword.com/bvandeusen/fabledcurator:dev
command: ["worker"]
# Longest lane (DB backups, library audits, translation backfill) — give it
# the most room to finish a chunk gracefully. Chunked + idempotent, so a job
# that still outruns this resumes cleanly next run rather than corrupting.
stop_grace_period: 180s
environment:
<<: *app_env
CELERY_QUEUES: maintenance_long
@@ -125,6 +141,8 @@ services:
ml-worker:
image: git.fabledsword.com/bvandeusen/fabledcurator-ml:dev
command: ["ml-worker"]
# A single GPU inference pass can run tens of seconds — let it finish.
stop_grace_period: 120s
environment:
<<: *app_env
volumes: