fix(migration): make 0045 DDL-only; backfill image_prediction via batched task (#768)
CI / lint (push) Successful in 3s
CI / backend-lint-and-test (push) Successful in 35s
CI / frontend-build (push) Successful in 42s
CI / integration (push) Successful in 3m16s

The inline INSERT…SELECT backfill in migration 0045 wrapped the table
creation and a ~100 GB pass over image_record.tagger_predictions in one
transaction: nothing committed until the end, it was unmonitorable, and an
earlier MATERIALIZED-CTE form spilled the full 100 GB to temp on NFS. A
deploy got stuck on it for ~2h with image_prediction never appearing.

Split the concerns:
- 0045 now creates ONLY the table + indexes (instant DDL → web boots).
- New backend.app.tasks.admin.backfill_image_predictions_task copies the
  >= store-floor predictions from the JSON into image_prediction, batched by
  id window and committed per chunk: live progress, resumable (re-enqueues
  from the last committed id), idempotent (ON CONFLICT DO NOTHING). json_each
  stays in the DB executor streaming each window — no Python-side 100 GB load,
  no materialization.
- POST /api/admin/maintenance/backfill-predictions + a Maintenance-tab card
  to trigger the one-time run after upgrading.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-06-11 09:18:25 -04:00
parent e6d5f67f11
commit 65211a3f2f
6 changed files with 206 additions and 42 deletions
@@ -12,6 +12,7 @@
<ThumbnailBackfillCard />
</div>
<MLThresholdSliders class="mt-4" />
<BackfillPredictionsCard class="mt-4" />
<PrunePredictionsCard class="mt-4" />
<AllowlistTable class="mt-4" />
<AliasTable class="mt-4" />
@@ -32,6 +33,7 @@ import MLBackfillCard from './MLBackfillCard.vue'
import CentroidRecomputeCard from './CentroidRecomputeCard.vue'
import ThumbnailBackfillCard from './ThumbnailBackfillCard.vue'
import MLThresholdSliders from './MLThresholdSliders.vue'
import BackfillPredictionsCard from './BackfillPredictionsCard.vue'
import PrunePredictionsCard from './PrunePredictionsCard.vue'
import AllowlistTable from './AllowlistTable.vue'
import AliasTable from './AliasTable.vue'