fix(migration): make 0045 DDL-only; backfill image_prediction via batched task (#768)
The inline INSERT…SELECT backfill in migration 0045 wrapped the table creation and a ~100 GB pass over image_record.tagger_predictions in one transaction: nothing committed until the end, it was unmonitorable, and an earlier MATERIALIZED-CTE form spilled the full 100 GB to temp on NFS. A deploy got stuck on it for ~2h with image_prediction never appearing. Split the concerns: - 0045 now creates ONLY the table + indexes (instant DDL → web boots). - New backend.app.tasks.admin.backfill_image_predictions_task copies the >= store-floor predictions from the JSON into image_prediction, batched by id window and committed per chunk: live progress, resumable (re-enqueues from the last committed id), idempotent (ON CONFLICT DO NOTHING). json_each stays in the DB executor streaming each window — no Python-side 100 GB load, no materialization. - POST /api/admin/maintenance/backfill-predictions + a Maintenance-tab card to trigger the one-time run after upgrading. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -49,6 +49,13 @@ def test_prune_low_confidence_predictions_task_registered():
|
||||
)
|
||||
|
||||
|
||||
def test_backfill_image_predictions_task_registered():
|
||||
assert (
|
||||
"backend.app.tasks.admin.backfill_image_predictions_task"
|
||||
in celery.tasks
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_prune_low_confidence_predictions(db_sync, tmp_path):
|
||||
# #764: drop stored tagger predictions below the store floor (default
|
||||
|
||||
Reference in New Issue
Block a user