feat(b3): ml-worker becomes optional — embed-only role, decoupled GPU coordination, cpu-embed switch
CI / lint (push) Successful in 3s
CI / frontend-build (push) Successful in 20s
CI / backend-lint-and-test (push) Successful in 30s
CI / integration (push) Successful in 3m31s

The ml-worker's ONLY processing role is now the CPU whole-image embed fallback
(tag_and_embed renamed embed_image — Camie tagging was retired #1189 and the
name kept implying otherwise; videos were already handled agent-style: frame
sampling + mean-pool). Detection/cropping/CCIP stay GPU-agent-only, and their
completion is judged per-pipeline: ccip by gpu_job rows, siglip by concept
regions at the current model version — never by image_record.siglip_embedding.
A CPU embed therefore can NEVER close crop work for the agent (regression test
pins this; only the whole-image 'embed' job, the same artifact, is satisfied).

Making removal actually safe (operator will drop the container):
- GPU-queue coordination (enqueue_gpu_backfill, recover_orphaned_gpu_jobs,
  reprocess_gpu_jobs) moved verbatim to tasks/gpu_queue.py on the maintenance
  quick lane — it lived on the 'ml' queue only by module colocation, which made
  the ml-worker a hard dependency of the whole agent pipeline.
- New ml_settings.cpu_embed_enabled (migration 0074, default ON so agent-less
  installs keep working): OFF stops the four import hooks queueing embed work
  nothing will consume and no-ops the manual backfill; switch lives on the
  renamed 'CPU embedding backfill' card.
- NB heads training / auto-apply still run on the ml image (sklearn) — a stack
  that removes the container gives those up too.

Deploy note: in-flight messages under the old task names are dropped by the
new workers; the 60s orphan sweep + hourly backfill re-fire under the new
names immediately.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
This commit is contained in:
2026-07-02 16:53:08 -04:00
parent 7c19ad91ed
commit 19b962f1a7
20 changed files with 428 additions and 202 deletions
+5 -5
View File
@@ -30,7 +30,7 @@ async def test_enqueue_siglip_backfill_gates_on_concept_region(db):
# back-catalogue) and skips ones that already have one — and never double-
# enqueues an image that already has a pending siglip job.
from backend.app.models import MLSettings
from backend.app.tasks.ml import enqueue_gpu_backfill
from backend.app.tasks.gpu_queue import enqueue_gpu_backfill
cur = (await db.execute(
select(MLSettings.embedder_model_version).where(MLSettings.id == 1)
@@ -71,7 +71,7 @@ async def test_enqueue_embed_backfill_selects_stale_and_unembedded(db):
# stamped under a DIFFERENT model version (an operator swap); skip ones
# already at the current version.
from backend.app.models import MLSettings
from backend.app.tasks.ml import enqueue_gpu_backfill
from backend.app.tasks.gpu_queue import enqueue_gpu_backfill
cur = (await db.execute(
select(MLSettings.embedder_model_version).where(MLSettings.id == 1)
@@ -99,7 +99,7 @@ async def test_enqueue_embed_backfill_selects_stale_and_unembedded(db):
async def test_reprocess_resets_done_jobs_to_pending(db):
# Re-process (#1202): done/error jobs of a task go back to pending so the
# agent re-runs the whole library under the current pipeline.
from backend.app.tasks.ml import reprocess_gpu_jobs
from backend.app.tasks.gpu_queue import reprocess_gpu_jobs
img = await _img(db, "r1" * 32)
job = await GpuJobService(db).enqueue(img.id, "ccip")
@@ -274,7 +274,7 @@ async def test_backfill_skips_errored_images(db):
# An errored job is a TOMBSTONE for its (image, task): no backfill variant
# re-enqueues it — retry is deliberate-only (/retry_errors). Pre-fix, the
# hourly ccip run minted a fresh doomed job per bad file forever.
from backend.app.tasks.ml import enqueue_gpu_backfill
from backend.app.tasks.gpu_queue import enqueue_gpu_backfill
img = await _img(db, "f1" * 32)
svc = GpuJobService(db)
@@ -294,7 +294,7 @@ async def test_backfill_prunes_moot_error_tombstones(db):
# Loop-era duplicates: several error rows for one (image, task), all made
# moot by a later done row. The backfill's dedupe pass removes them, and
# the done row still blocks re-enqueue.
from backend.app.tasks.ml import enqueue_gpu_backfill
from backend.app.tasks.gpu_queue import enqueue_gpu_backfill
img = await _img(db, "f2" * 32)
for i in range(3):