Graceful shutdown + resilience & Explore polish (m147) #203

Merged

bvandeusen merged 7 commits from dev into main 2026-07-07 22:53:34 -04:00

Conversation 0 Commits 7 Files Changed 16

+511 -38

bvandeusen commented 2026-07-07 22:53:28 -04:00

Owner

Copy Link

Copy Source

Brings two CI-green batches from dev to main. No migration.

Graceful shutdown (task #1365)

• docker-compose.yml: stop_grace_period per lane (web 30s / worker 90s / scheduler 60s / maintenance-long 180s / ml-worker 120s) so warm shutdown can drain before SIGKILL (was Docker's 10s default).
• celery_app.py: task_reject_on_worker_lost=True — a task killed past the grace window is re-queued (safe: idempotent + chunked + recovery sweeps).
• interpreter_client.py: 429/5xx (incl 502/503/504) → InterpreterUnavailable + Retry-After parsed; retranslate_posts resumes on interrupt after the hint.

Milestone 147 — resilience & Explore polish (tasks #1367–#1372)

• #6 Explore excludes wip-tagged art from the rabbit-hole (similar(exclude_wip=…)); gallery "similar" button unchanged (#1274).
• #4 Pooled requests.Session for Interpreter with connect-only retry.
• #5 Manual "Translate now" resumes after a drain (Retry-After backoff).
• #1 Container healthchecks (/api/health, celery-ping) + Swarm deploy: auto-rollback in docker-compose.yml.
• #2 Daily janitor for orphaned .part/.partial staging files.
• #3 Live re-translate progress in the Settings card (active/last_run from TaskRun + polling spinner).

NB: prod deploys from a separate stack file, so the compose healthcheck/stop_grace_period/deploy: bits are reference here — the prod stack file needs the same additions ported (tracked separately).

🤖 Generated with Claude Code

Brings two CI-green batches from dev to main. **No migration.** ## Graceful shutdown (task #1365) - `docker-compose.yml`: `stop_grace_period` per lane (web 30s / worker 90s / scheduler 60s / maintenance-long 180s / ml-worker 120s) so warm shutdown can drain before SIGKILL (was Docker's 10s default). - `celery_app.py`: `task_reject_on_worker_lost=True` — a task killed past the grace window is re-queued (safe: idempotent + chunked + recovery sweeps). - `interpreter_client.py`: 429/5xx (incl 502/503/504) → `InterpreterUnavailable` + `Retry-After` parsed; `retranslate_posts` resumes on interrupt after the hint. ## Milestone 147 — resilience & Explore polish (tasks #1367–#1372) - **#6** Explore excludes `wip`-tagged art from the rabbit-hole (`similar(exclude_wip=…)`); gallery "similar" button unchanged (#1274). - **#4** Pooled `requests.Session` for Interpreter with connect-only retry. - **#5** Manual "Translate now" resumes after a drain (Retry-After backoff). - **#1** Container healthchecks (`/api/health`, celery-ping) + Swarm `deploy:` auto-rollback in `docker-compose.yml`. - **#2** Daily janitor for orphaned `.part`/`.partial` staging files. - **#3** Live re-translate progress in the Settings card (`active`/`last_run` from TaskRun + polling spinner). NB: prod deploys from a **separate stack file**, so the compose healthcheck/`stop_grace_period`/`deploy:` bits are reference here — the prod stack file needs the same additions ported (tracked separately). 🤖 Generated with [Claude Code](https://claude.com/claude-code)

bvandeusen added 7 commits 2026-07-07 22:53:28 -04:00

feat(ops): graceful shutdown — worker stop-grace + Interpreter drain resilience ... c64261593d

Deploys (docker SIGTERM→SIGKILL, default 10s) were killing Celery jobs
mid-flight. Give in-flight work room to drain and make interrupted work
resume cleanly instead of stalling.

- docker-compose.yml: stop_grace_period per lane (web 30s / worker 90s /
  scheduler 60s / maintenance-long 180s / ml-worker 120s) so warm shutdown
  can actually drain before SIGKILL.
- celery_app.py: task_reject_on_worker_lost=True — a task killed past the
  grace window is re-queued (safe: idempotent + chunked, recovery sweeps
  re-drive stragglers).
- interpreter_client.py: map 429/5xx (502/503/504) → InterpreterUnavailable
  and parse Retry-After (delta-seconds or HTTP-date); a draining Interpreter
  behind a reverse proxy no longer raises an opaque HTTPError.
- translation.py: thread retry_after out of _translate_batch; retranslate_posts
  resumes after the Retry-After hint (or 60s default, capped 900s) on an
  interrupt with _reset_done=True, self-terminating via the health gate.
- tests: 429/5xx mapping + Retry-After parse; interrupt-resume + default backoff.

No migration.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

style(translation): use datetime.UTC alias (ruff UP017) ... d631ed023c

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

feat(explore): exclude WIP-tagged work from the Explore rabbit-hole ... f8105046dc

Explore's neighbour grid (/api/gallery/similar → gallery_service.similar) now
takes an Explore-only exclude_wip flag that drops `wip` system-tagged images
from the candidates, alongside the banner/editor presentation tags. The
gallery's own "similar" button is unchanged (keeps wip, #1274) — only the
Explore store passes exclude_wip=1. The anchor itself may still be a WIP; only
neighbours are filtered.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

feat(translation): pooled Interpreter session + manual sweep resumes after drain ... 9eae636047

- interpreter_client: shared requests.Session with a connect-only retry
  (connect=2, no status retries — we map 429/5xx ourselves) so a proxy reload
  is smoothed and the keep-alive connection is pooled across the sweep.
- translate_posts: on an interrupt (drain), re-enqueue after the Retry-After
  hint / default backoff instead of waiting for the daily beat; self-terminates
  via the health gate. Steady-state one-chunk-per-run on success is unchanged.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

feat(maintenance): daily janitor for orphaned .part/.partial staging files ... 0b78264d62

Downloads/imports stage into <name>.part / <name>.partial then os.replace() into
place, so a kill mid-write leaves a discardable temp — never a corrupt final.
cleanup_orphaned_temp_files sweeps ones left behind under the images root, only
older than 6h so an in-flight download's staging file is never removed. Daily beat.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

feat(deploy): container healthchecks + Swarm rolling-update auto-rollback ... 40cc11be5b

web gets a /api/health liveness check; workers a lenient celery-ping check. A
shared deploy policy (update_config order=start-first, failure_action=rollback,
monitor 90s; rollback_config; restart_policy) means a bad image that never goes
healthy is rolled back automatically instead of taking the service down. Ignored
by plain `docker compose up` (deploy: is swarm-only), so the dev override is
unaffected. Assumes prod deploys from this file via docker stack deploy.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

feat(translation): live re-translate progress in the Settings card ... 4371ddb7e7

translation/status now reports `active` (a translate/retranslate sweep is
running, from the TaskRun table) and `last_run` (the most recent finished run's
task + status). The Settings card polls live while a sweep runs, showing a
spinner + "Translating… N remaining" that ticks down, and flags a last run that
ended in error/timeout. No migration.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

bvandeusen merged commit f8b667604f into main 2026-07-07 22:53:34 -04:00

Sign in to join this conversation.

Reviewers

No Reviewers

Labels

No items

No Label

Milestone

No items

No Milestone

Projects

Clear projects

No project

Assignees

Clear assignees

bvandeusen

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: bvandeusen/FabledCurator#203