fix(migration): make 0045 DDL-only; backfill image_prediction via batched task (#768)
The inline INSERT…SELECT backfill in migration 0045 wrapped the table creation and a ~100 GB pass over image_record.tagger_predictions in one transaction: nothing committed until the end, it was unmonitorable, and an earlier MATERIALIZED-CTE form spilled the full 100 GB to temp on NFS. A deploy got stuck on it for ~2h with image_prediction never appearing. Split the concerns: - 0045 now creates ONLY the table + indexes (instant DDL → web boots). - New backend.app.tasks.admin.backfill_image_predictions_task copies the >= store-floor predictions from the JSON into image_prediction, batched by id window and committed per chunk: live progress, resumable (re-enqueues from the last committed id), idempotent (ON CONFLICT DO NOTHING). json_each stays in the DB executor streaming each window — no Python-side 100 GB load, no materialization. - POST /api/admin/maintenance/backfill-predictions + a Maintenance-tab card to trigger the one-time run after upgrading. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,50 @@
|
||||
<template>
|
||||
<!-- #768: one-time copy of stored tagger predictions from the
|
||||
image_record.tagger_predictions JSON into the normalized
|
||||
image_prediction table. Migration 0045 creates the empty table; this
|
||||
populates it for the existing library. -->
|
||||
<v-card>
|
||||
<v-card-title>Backfill normalized predictions</v-card-title>
|
||||
<v-card-text>
|
||||
<p class="text-body-2 mb-3">
|
||||
Copies each image's stored tagger predictions into the new
|
||||
<code>image_prediction</code> table (the source the suggestions and
|
||||
allowlist now read from). Run this <strong>once</strong> after the
|
||||
upgrade so existing images get their suggestions back — newly tagged
|
||||
images populate it automatically. Batched, resumable and idempotent;
|
||||
safe to run more than once and to leave running in the background.
|
||||
</p>
|
||||
<v-btn color="primary" rounded="pill" :loading="busy" @click="run">
|
||||
<v-icon start>mdi-database-import-outline</v-icon> Backfill predictions now
|
||||
</v-btn>
|
||||
<span v-if="queued" class="ml-3 text-caption text-success">Queued ✓</span>
|
||||
<QueueStatusBar queue="maintenance_long" queue-label="Maintenance (long)" />
|
||||
</v-card-text>
|
||||
</v-card>
|
||||
</template>
|
||||
|
||||
<script setup>
|
||||
import { ref } from 'vue'
|
||||
|
||||
import { useApi } from '../../composables/useApi.js'
|
||||
import { toast } from '../../utils/toast.js'
|
||||
import QueueStatusBar from './QueueStatusBar.vue'
|
||||
|
||||
const api = useApi()
|
||||
const busy = ref(false)
|
||||
const queued = ref(false)
|
||||
|
||||
async function run () {
|
||||
busy.value = true
|
||||
queued.value = false
|
||||
try {
|
||||
await api.post('/api/admin/maintenance/backfill-predictions')
|
||||
queued.value = true
|
||||
toast({ text: 'Prediction backfill queued', type: 'success' })
|
||||
} catch (e) {
|
||||
toast({ text: e?.body?.detail || e?.message || 'Failed to queue', type: 'error' })
|
||||
} finally {
|
||||
busy.value = false
|
||||
}
|
||||
}
|
||||
</script>
|
||||
@@ -12,6 +12,7 @@
|
||||
<ThumbnailBackfillCard />
|
||||
</div>
|
||||
<MLThresholdSliders class="mt-4" />
|
||||
<BackfillPredictionsCard class="mt-4" />
|
||||
<PrunePredictionsCard class="mt-4" />
|
||||
<AllowlistTable class="mt-4" />
|
||||
<AliasTable class="mt-4" />
|
||||
@@ -32,6 +33,7 @@ import MLBackfillCard from './MLBackfillCard.vue'
|
||||
import CentroidRecomputeCard from './CentroidRecomputeCard.vue'
|
||||
import ThumbnailBackfillCard from './ThumbnailBackfillCard.vue'
|
||||
import MLThresholdSliders from './MLThresholdSliders.vue'
|
||||
import BackfillPredictionsCard from './BackfillPredictionsCard.vue'
|
||||
import PrunePredictionsCard from './PrunePredictionsCard.vue'
|
||||
import AllowlistTable from './AllowlistTable.vue'
|
||||
import AliasTable from './AliasTable.vue'
|
||||
|
||||
Reference in New Issue
Block a user