feat(b3): ml-worker becomes optional — embed-only role, decoupled GPU coordination, cpu-embed switch
CI / lint (push) Successful in 3s
CI / frontend-build (push) Successful in 20s
CI / backend-lint-and-test (push) Successful in 30s
CI / integration (push) Successful in 3m31s

The ml-worker's ONLY processing role is now the CPU whole-image embed fallback
(tag_and_embed renamed embed_image — Camie tagging was retired #1189 and the
name kept implying otherwise; videos were already handled agent-style: frame
sampling + mean-pool). Detection/cropping/CCIP stay GPU-agent-only, and their
completion is judged per-pipeline: ccip by gpu_job rows, siglip by concept
regions at the current model version — never by image_record.siglip_embedding.
A CPU embed therefore can NEVER close crop work for the agent (regression test
pins this; only the whole-image 'embed' job, the same artifact, is satisfied).

Making removal actually safe (operator will drop the container):
- GPU-queue coordination (enqueue_gpu_backfill, recover_orphaned_gpu_jobs,
  reprocess_gpu_jobs) moved verbatim to tasks/gpu_queue.py on the maintenance
  quick lane — it lived on the 'ml' queue only by module colocation, which made
  the ml-worker a hard dependency of the whole agent pipeline.
- New ml_settings.cpu_embed_enabled (migration 0074, default ON so agent-less
  installs keep working): OFF stops the four import hooks queueing embed work
  nothing will consume and no-ops the manual backfill; switch lives on the
  renamed 'CPU embedding backfill' card.
- NB heads training / auto-apply still run on the ml image (sklearn) — a stack
  that removes the container gives those up too.

Deploy note: in-flight messages under the old task names are dropped by the
new workers; the 60s orphan sweep + hourly backfill re-fire under the new
names immediately.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
This commit is contained in:
2026-07-02 16:53:08 -04:00
parent 7c19ad91ed
commit 19b962f1a7
20 changed files with 428 additions and 202 deletions
+31 -1
View File
@@ -127,7 +127,7 @@ async def test_backfill_enqueues_then_is_idempotent(db):
await _img(db, "c" * 64)
await _img(db, "d" * 64)
await db.commit()
from backend.app.tasks.ml import enqueue_gpu_backfill
from backend.app.tasks.gpu_queue import enqueue_gpu_backfill
n = enqueue_gpu_backfill("ccip") # sync task, own session
assert n >= 2
@@ -260,3 +260,33 @@ async def test_errors_endpoint_reports_triage_view(client, db):
assert item["reason_class"] == "truncated_or_corrupt"
assert item["triage_status"] is None
assert item["image_url"].startswith("/images/")
@pytest.mark.asyncio
async def test_cpu_embed_never_blocks_gpu_crop_backfills(db):
"""B3 invariant (operator 2026-07-02): ccip (detect + character) and
siglip (concept crops) completion is judged per-pipeline — gpu_job rows and
image_region state — never inferred from image_record.siglip_embedding. So
an image the CPU fallback already embedded still gets both crop jobs; only
the whole-image 'embed' job (the SAME artifact the CPU path produces) is
satisfied by it."""
from backend.app.models import MLSettings
from backend.app.tasks.gpu_queue import enqueue_gpu_backfill
img = await _img(db, "7" * 64)
cur = (await db.execute(
select(MLSettings.embedder_model_version).where(MLSettings.id == 1)
)).scalar_one()
# As if the CPU fallback already embedded it under the current model.
img.siglip_embedding = [0.1] * 1152
img.siglip_model_version = cur
await db.commit()
assert enqueue_gpu_backfill("ccip") == 1 # crops still open
assert enqueue_gpu_backfill("siglip") == 1 # concept crops still open
assert enqueue_gpu_backfill("embed") == 0 # same artifact — already done
tasks = set((await db.execute(
select(GpuJob.task).where(GpuJob.image_record_id == img.id)
)).scalars().all())
assert tasks == {"ccip", "siglip"}