feat(b3): ml-worker becomes optional — embed-only role, decoupled GPU coordination, cpu-embed switch
The ml-worker's ONLY processing role is now the CPU whole-image embed fallback (tag_and_embed renamed embed_image — Camie tagging was retired #1189 and the name kept implying otherwise; videos were already handled agent-style: frame sampling + mean-pool). Detection/cropping/CCIP stay GPU-agent-only, and their completion is judged per-pipeline: ccip by gpu_job rows, siglip by concept regions at the current model version — never by image_record.siglip_embedding. A CPU embed therefore can NEVER close crop work for the agent (regression test pins this; only the whole-image 'embed' job, the same artifact, is satisfied). Making removal actually safe (operator will drop the container): - GPU-queue coordination (enqueue_gpu_backfill, recover_orphaned_gpu_jobs, reprocess_gpu_jobs) moved verbatim to tasks/gpu_queue.py on the maintenance quick lane — it lived on the 'ml' queue only by module colocation, which made the ml-worker a hard dependency of the whole agent pipeline. - New ml_settings.cpu_embed_enabled (migration 0074, default ON so agent-less installs keep working): OFF stops the four import hooks queueing embed work nothing will consume and no-ops the manual backfill; switch lives on the renamed 'CPU embedding backfill' card. - NB heads training / auto-apply still run on the ml image (sklearn) — a stack that removes the container gives those up too. Deploy note: in-flight messages under the old task names are dropped by the new workers; the 60s orphan sweep + hourly backfill re-fire under the new names immediately. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
This commit is contained in:
@@ -1,16 +1,33 @@
|
||||
<template>
|
||||
<MaintenanceTile
|
||||
icon="mdi-refresh"
|
||||
title="ML backfill"
|
||||
blurb="Compute SigLIP embeddings on images missing them."
|
||||
title="CPU embedding backfill"
|
||||
blurb="Whole-image embeddings without a GPU agent — the built-in fallback."
|
||||
:open="busy"
|
||||
>
|
||||
<p class="text-body-2 mb-3">
|
||||
Compute the SigLIP embedding for any image that doesn't have one yet
|
||||
(CPU). Safe to re-run. To re-embed under a NEW model, use the GPU
|
||||
agent's "Re-embed library" instead.
|
||||
Computes the whole-image SigLIP embedding for anything missing one —
|
||||
images directly, videos by sampling frames (the same approach as the
|
||||
GPU agent). Runs on the ml-worker's CPU, so search, similarity and
|
||||
head suggestions work <strong>without</strong> a GPU agent; new imports
|
||||
are embedded this way automatically. Detection, cropping and character
|
||||
(CCIP) embeddings are GPU-agent-only. Safe to re-run. To re-embed under
|
||||
a NEW model, use the GPU agent's "Re-embed library" instead.
|
||||
</p>
|
||||
<v-btn color="primary" rounded="pill" :loading="busy" @click="run">
|
||||
<v-switch
|
||||
v-model="enabled" color="accent" hide-details density="compact"
|
||||
:loading="saving" label="CPU embedding enabled"
|
||||
class="mb-1" @update:model-value="onToggle"
|
||||
/>
|
||||
<p class="fc-muted text-caption mb-3">
|
||||
Turn OFF if you run the GPU agent and removed the ml-worker container —
|
||||
imports then stop queueing CPU embed work nothing will consume (the
|
||||
daily GPU embed backfill covers those images instead).
|
||||
</p>
|
||||
<v-btn
|
||||
color="primary" rounded="pill" :loading="busy" :disabled="!enabled"
|
||||
@click="run"
|
||||
>
|
||||
<v-icon start>mdi-refresh</v-icon> Run backfill now
|
||||
</v-btn>
|
||||
<span v-if="done" class="ml-3 text-caption">Enqueued.</span>
|
||||
@@ -20,13 +37,40 @@
|
||||
|
||||
<script setup>
|
||||
import { toast } from '../../utils/toast.js'
|
||||
import { ref } from 'vue'
|
||||
import { onMounted, ref } from 'vue'
|
||||
import { useMLStore } from '../../stores/ml.js'
|
||||
import MaintenanceTile from '../common/MaintenanceTile.vue'
|
||||
import QueueStatusBar from './QueueStatusBar.vue'
|
||||
const store = useMLStore()
|
||||
const busy = ref(false)
|
||||
const done = ref(false)
|
||||
const enabled = ref(true)
|
||||
const saving = ref(false)
|
||||
onMounted(async () => {
|
||||
try {
|
||||
await store.loadSettings()
|
||||
if (store.settings?.cpu_embed_enabled != null) {
|
||||
enabled.value = store.settings.cpu_embed_enabled
|
||||
}
|
||||
} catch { /* non-fatal */ }
|
||||
})
|
||||
async function onToggle() {
|
||||
saving.value = true
|
||||
try {
|
||||
await store.patchSettings({ cpu_embed_enabled: enabled.value })
|
||||
toast({
|
||||
text: enabled.value
|
||||
? 'CPU embedding on — imports queue embeds for the ml-worker'
|
||||
: 'CPU embedding off — the GPU embed backfill owns whole-image embeds',
|
||||
type: 'success',
|
||||
})
|
||||
} catch (e) {
|
||||
toast({ text: `Could not save: ${e.message}`, type: 'error' })
|
||||
enabled.value = !enabled.value
|
||||
} finally {
|
||||
saving.value = false
|
||||
}
|
||||
}
|
||||
async function run() {
|
||||
busy.value = true
|
||||
try { await store.triggerBackfill(); done.value = true }
|
||||
@@ -34,3 +78,7 @@ async function run() {
|
||||
finally { busy.value = false }
|
||||
}
|
||||
</script>
|
||||
|
||||
<style scoped>
|
||||
.fc-muted { color: rgb(var(--v-theme-on-surface-variant)); }
|
||||
</style>
|
||||
|
||||
Reference in New Issue
Block a user