bvandeusen 75eab188c8 fix(migration): 0045 backfill filters to >= store floor (supersedes #764 prune) ...

The #764 in-place prune (rewrite tagger_predictions to >=0.70) is too slow on
100 GB of TOAST and fails at its soft limit (interrupts a query mid-flight ->
'another command is already in progress'). #768 supersedes it: extract only
the >=floor predictions into image_prediction via this set-based backfill,
then drop the column (step 3) — reading 100 GB once + writing ~840k small rows
beats rewriting 100 GB in place.

So this backfill no longer assumes the prune ran: it filters by
ml_settings.tagger_store_floor (default 0.70) itself, handling the full or
partially-pruned JSON identically.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

2026-06-10 19:37:38 -04:00

..

.gitkeep

feat: add SQLAlchemy declarative base, Alembic environment, and gitignore fix

2026-05-14 07:32:58 -04:00

0001_initial_unified_schema.py

feat: define unified schema (Artist, Source, Post, ImageRecord, etc.) and initial migration

2026-05-14 07:34:29 -04:00

0002_fc2a_tag_kinds_and_import_tasks.py

feat(fc2a): schema migration 0002 — tag kinds + fandom hierarchy + import lifecycle

2026-05-14 12:03:41 -04:00

0003_fc2b_ml_pipeline.py

feat(fc2b): schema migration 0003 — ML pipeline tables

2026-05-15 07:35:22 -04:00

0004_fc2c_i_tsm_system_rows.py

feat(fc2c-i): migration 0004 enables tsm_system_rows

2026-05-15 15:48:29 -04:00

0005_fc2c_iii_a_series_page.py

feat(series): SeriesPage model + 0005 migration (unique image, ordered pages)

2026-05-16 23:33:12 -04:00

0006_fc2d_phash_threshold.py

feat(phash): import_settings.phash_threshold (migration 0006) exposed + validated in API

2026-05-17 22:04:13 -04:00

0007_fc2d_post_metadata_fields.py

feat(post): add nullable description + attachment_count (migration 0007)

2026-05-18 09:38:43 -04:00

0008_fc2d_vii_c_artist_deconfliction.py

feat(provenance): image_record.artist_id + 0008 backfill/drop-artist-tags

2026-05-18 21:33:11 -04:00

0009_fc2d_iii_post_attachment.py

feat(attachments): PostAttachment model + 0009 (table + batch counter)

2026-05-19 11:10:53 -04:00

0010_fc3a_source_unique_artist_platform_url.py

feat(fc3a): migration 0010 — unique(source.artist_id, platform, url)

2026-05-20 12:53:34 -04:00

0011_fc3b_credential_schema_alignment.py

feat(fc3b): migration 0011 — credential schema aligned with GS wire fields

2026-05-20 18:32:52 -04:00

0012_fc3b_app_setting.py

feat(fc3b): migration 0012 + AppSetting model

2026-05-20 18:33:14 -04:00

0013_fc3c_download_event_metadata.py

feat(fc3c): migration 0013 — download_event.metadata + import_settings downloader fields

2026-05-20 20:36:19 -04:00

0014_fc3d_scheduling.py

fc3d: migration 0014 — source.consecutive_failures + import_settings scheduling knobs

2026-05-21 15:58:30 -04:00

0015_fc5_migration_run.py

fc5: migration 0015 + MigrationRun model (kind/status as String(32), API-validated)

2026-05-22 08:11:42 -04:00

0016_fc3i_task_run.py

fc3i: alembic 0016 — task_run table with single + composite indexes

2026-05-24 21:00:46 -04:00

0017_fc3h_backup_run.py

fc3h: alembic 0017 — backup_run table with indexes + partial-tag

2026-05-24 22:51:22 -04:00

0018_fc3h_backup_settings.py

fc3h: ImportSettings backup_* knobs + alembic 0018 (nightly-enabled, hour, keep-N per kind)

2026-05-24 22:52:00 -04:00

0019_import_batch_refreshed.py

feat(deep-scan): IR-parity port — refreshed status + counter, re-queue completed paths in deep mode, honest UX — Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

2026-05-25 22:16:26 -04:00

0020_library_audit_run.py

feat(fc-cleanup): migration 0020 + LibraryAuditRun model — Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

2026-05-26 01:17:42 -04:00

0021_image_provenance_unique.py

fix(importer): ImageProvenance (image_record_id, post_id) race-safe via savepoint + alembic 0021 UNIQUE — closes the SELECT-then-INSERT window that planted duplicates and broke .scalar_one_or_none() on every later deep-scan rederive (MultipleResultsFound). Migration dedupes existing rows (min(id) per pair); model gains __table_args__; gallery-filter test that seeded duplicates dropped.

2026-05-26 13:47:55 -04:00

0022_source_per_artist_platform.py

fix(migration-0022): pre-DELETE colliding image_provenance rows before the UPDATE post_id — same row-by-row UNIQUE pattern as the post-collision case, just one level deeper. When image X has provenance under both keep and drop, UPDATE drop→keep would fire uq_image_provenance_image_post on the row that'd collide with the existing (X, keep). Pre-delete those rows (their info is already represented by the keep-side provenance) before the UPDATE moves the rest.

2026-05-26 18:05:49 -04:00

0023_drop_meta_rating_tag_kinds.py

fix(extension+migration): MV3 CSP opt-out from upgrade-insecure-requests (v1.0.4) + alembic 0023 drops the ck_tag_fandom_requires_character check before the type swap

2026-05-26 21:57:38 -04:00

0024_backfill_post_title_from_description.py

fix(sidecar): synthesize post_title from content first-line when title is empty (subscribestar)

2026-05-27 14:44:09 -04:00

0025_fix_subscribestar_post_ids.py

fix(alembic 0025): qualify ambiguous post.id / post.source_id in fragment-group SELECT (post JOIN source — both have id)

2026-05-27 15:45:42 -04:00

0026_import_task_recovery_count_refetched.py

feat(import-resilience L1): poison-pill circuit breaker — cap stuck-task re-queues

2026-05-27 23:54:35 -04:00

0027_drop_migration_run.py

refactor(I5): remove one-and-done GS/IR migration tooling

2026-05-29 14:38:59 -04:00

0028_collapse_sidecar_synthetics_into_real_sources.py

fix(subscriptions): hide sidecar synthetic Sources + prefer real on lookup

2026-05-31 23:08:38 -04:00

0029_drop_artist_copyright_ml_thresholds.py

feat(modal): autofocus tag input, expand general suggestions, retire copyright/artist categories

2026-06-01 02:08:10 -04:00

0030_nullable_post_source_id_denorm_artist_id.py

fix(migration): use canonical fk_<table>_<col>_<ref> names per Base naming_convention

2026-06-01 14:23:18 -04:00

0031_source_backfill_runs_remaining.py

feat(download): tick/backfill modes + partial-success classifier (plan #544)

2026-06-01 18:23:28 -04:00

0032_source_error_type.py

fix(audit-g5d): surface ErrorType taxonomy on FailingSourcesCard

2026-06-02 17:54:44 -04:00

0033_suggestion_threshold_default_070.py

chore(ml): suggestion_threshold default 0.50 → 0.70

2026-06-02 18:38:12 -04:00

0034_artist_visit.py

feat(artist): "new since last visit" badge + banner

2026-06-03 15:27:11 -04:00

0035_image_record_effective_date.py

perf(gallery): materialize indexed effective_date sort key

2026-06-03 22:58:46 -04:00

0036_siglip_embedding_hnsw_index.py

feat(gallery): visual 'more like this' search (Phase 3 backend)

2026-06-04 08:47:43 -04:00

0037_patreon_seen_media.py

feat(patreon): seen-ledger table + model — ingester build step 2a (plan #697)

2026-06-05 19:26:57 -04:00

0038_patreon_failed_media.py

feat(patreon): dead-letter ledger for permanently-failing media — #705 step 2 (#7)

2026-06-06 00:04:24 -04:00

0039_library_audit_resume.py

fix(audit): chunk + self-resume library scans (stop the 2h queue-hog timeouts)

2026-06-07 00:08:19 -04:00

0040_series_chapters.py

feat(series): chapter layer over series_page — backend (FC-6.1)

2026-06-07 16:31:55 -04:00

0041_series_suggestions.py

feat(series): assisted-continuation matcher + suggestion queue — backend (FC-6.3)

2026-06-07 18:58:18 -04:00

0042_series_chapter_stated_part.py

feat(series): manage-view redesign — big pages, editable Part #, slide-over picker (FC-6.4)

2026-06-07 20:29:10 -04:00

0043_post_attachment_per_post_unique.py

fix(posts): link duplicate items to every post + prune bare shells

2026-06-08 19:28:33 -04:00

0044_ml_settings_tagger_store_floor.py

feat(ml): DB-backed tagger_store_floor (default 0.70), the ingest confidence floor

2026-06-10 13:50:30 -04:00

0045_image_prediction_table.py

fix(migration): 0045 backfill filters to >= store floor (supersedes #764 prune)

2026-06-10 19:37:38 -04:00