7 Commits

Include renames

Author	SHA1	Message	Date
bvandeusen	6959e1220c	Revert "db: collapse alembic 0001..0087 into one baseline" ... This reverts 2529b51. Not a retreat — a reordering, on the operator's call, and the better sequence. The squash's acceptance test (run 4971) found ~130 places where the ORM models do not describe the deployed schema (#3275), including a unique=True the database never had and two UNIQUE indexes that exist only in migrations. Collapsing now would have baked all of that into the one file a public installer starts from. So: fix the drift first as ordinary migrations on the intact chain, let the operator deploy so their database moves to the corrected head, and only then collapse. The baseline is then generated from reconciled models and reproduces a schema worth reproducing. Nothing is lost by reverting. The baseline was never deployed, and regenerating it after the fixes is strictly better than patching this copy — it will come out of autogenerate correct rather than needing the same hand-finishing twice.	2026-08-30 14:34:28 -04:00
bvandeusen	2529b516e6	db: collapse alembic 0001..0087 into one baseline (milestone 328 step 1) ... 87 revisions narrating this project's build-out become one file that creates the schema in a single step. They cost nothing at runtime — all 86 upgrade steps ran in 0.2s (note #3260) — so this is a presentation change, not a performance one: a new installer should not inherit our development history to stand up a database. Deleted: 87 revisions (6,052 lines), the 10 tests/test_migration_*.py files (483 lines) that asserted intermediate states and backfills which no longer exist, and backend/app/utils/artist_backfill.py — the only live module a migration imported, with no other consumer anywhere. That last one satisfies the operator's separate request to inline it into 0008 and delete the module; the squash removes both outright. THE REVISION ID IS "0087", NOT "0001", ON PURPOSE. It is the id of the last revision collapsed, so an existing database is already at head and `alembic upgrade head` does nothing. The alternative is `alembic stamp` against live data, and stamp validates NOTHING — it writes a version string whether or not the schema matches, so a wrong baseline surfaces later, via the next real migration, with no clean way back. This removes that operation rather than making it safe. Future revisions run from 0088. Four things are hand-written because SQLAlchemy metadata does not carry them, and none fail at generation time: 1. CREATE EXTENSION vector — the VECTOR columns cannot be created without it, so it is ordered first in upgrade(). 2. CREATE EXTENSION tsm_system_rows — surfaces only when the random sample query runs. 3. the HNSW index on image_record.siglip_embedding, raw SQL because create_index cannot express USING hnsw (... vector_cosine_ops). The quietest of the four: everything works, similarity search just stops using an index. 4. import pgvector.sqlalchemy.vector — autogenerate EMITS pgvector.sqlalchemy.vector.VECTOR references without importing it, so the generated file dies with NameError on first run. The candidate came out of CI (run 4967) as checksummed base64 rather than a plain cat, because run 4964's cat was truncated mid-line inside a column definition with the step still green — 29 tables instead of 42, and it looked entirely plausible. Verified here: 56,582 bytes, sha256 471acfca69c0…, 42 tables, 66 indexes, 42 drops. NOT YET PROVEN against the old chain. baseline.yml does that, and it is step 2's gate; this commit does not claim the schemas match.	2026-08-30 13:45:35 -04:00
bvandeusenandClaude Opus 4.8	bc6d43d3f2	refactor(ml): drop dead tagger/suggestion settings + columns (#1199) ... Hygiene follow-up to the Camie retirement (#1189) — these were left inert to bound that change; nothing reads them now. Migration 0068 drops: - ml_settings: tagger_store_floor, tagger_model_version, suggestion_threshold_ character/general (already dead pre-retirement — scoring uses per-head thresholds), video_min_tag_frames (only the deleted video-prediction aggregator used it). - image_record: tagger_model_version (no writer), centroid_scores (dead JSON cache, no reader). Also: ml_admin _EDITABLE/GET/_validate pruned (dropped the store-floor invariant + video_min_tag_frames check); MLThresholdSliders trimmed to a video-embedding card (interval + max frames only); importer no longer resets the dropped cols; download_models drops the Camie fetch; stale CASCADE comments in cleanup_service no longer name the removed tables. Tests updated. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa	2026-06-30 13:41:25 -04:00
bvandeusenandClaude Opus 4.8	485387ff0b	refactor(ml): retire the Camie tagger + allowlist bulk-apply (#1189) ... Heads + CCIP are the tag source and head auto-apply is the earned propagation. The Camie tagger ran only to feed the allowlist bulk-apply (its ImagePrediction rows had no other consumer), and the allowlist was a SECOND, un-earned auto-apply path firing in parallel with heads on every accept — exactly the un-earned spray the v2 pivot replaced. Retire both. Behavior change: accepting a suggestion now applies the tag to THAT image only (source='ml_accepted', a head-training positive) — it no longer allowlists + fans the tag across the library via Camie. Propagation is heads' earned auto-apply. (Loses instant cold-start propagation for booru-vocab tags; that was un-earned and bypassed the precision gate.) - tag_and_embed is now EMBED-ONLY (no Camie load/infer, no ImagePrediction writes); backfill enqueues it for images with no embedding. - Removed: services/ml/tagger.py, apply_allowlist_tags + helpers + daily beat + every enqueue caller (accept/alias/merge/per-image), api/allowlist.py + blueprint, ImagePrediction + TagAllowlist models/tables (migration 0067), AllowlistTable.vue + allowlist store, the accept coverage-projection payload. - AllowlistService gutted to accept/dismiss/undismiss/reject (the rejection store the rail still needs); accept returns nothing, API returns {accepted, tag_id}. - tag merge no longer repoints/triggers the allowlist; _keep_as_alias now keys on ML-applied image_tag sources (incl. head_auto) instead of the allowlist. - UI: MLBackfillCard relabelled to embedding-only; accept toast simplified; MaintenancePanel drops the allowlist tile. Left for a follow-up hygiene pass (now-inert, harmless): the dead settings columns (tagger_store_floor, tagger_model_version, suggestion_threshold_*, video_min_tag_frames), image_record.tagger_model_version, MLThresholdSliders trim, and the Camie model download in download_models.py. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa	2026-06-30 13:04:31 -04:00
bvandeusenandClaude Opus 4.8	3d77a38a25	refactor(ml): remove the dead per-tag centroid subsystem (#1189) ... The v2 pivot replaced per-tag SigLIP centroids with learned heads + CCIP. Centroids were still recomputed (on every tag merge + a daily beat) but NOTHING read them — suggestions come from heads+CCIP and apply_allowlist_tags applies via Camie predictions, not centroids. Pure dead wiring; remove it. Removed: CentroidService, recompute_centroid/recompute_centroids tasks, the daily beat, POST /api/ml/recompute-centroids, the recompute-on-merge trigger, the tag_reference_embedding table + model, the centroid_similarity_threshold + min_reference_images settings (migration 0066), the CentroidRecomputeCard + its store action + MaintenancePanel tile, and the centroid slider in MLThresholdSliders. _keep_as_alias drops its vestigial has-centroid branch (the allowlist branch already covers "could re-emit"); tag merge no longer clears a table that no longer exists. NOT touched (still live, parallel to heads): the Camie tagger, ImagePrediction, and the allowlist bulk-apply — accepting a suggestion still allowlists + applies it across the library. The tag-eval "centroid" baseline metric is unrelated (in-memory) and stays. (image_record.centroid_scores JSON column also remains — separate legacy field, its own micro-cleanup.) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa	2026-06-30 11:48:09 -04:00
bvandeusenandClaude Opus 4.8	3610ba495f	feat(ml): drop image_record.tagger_predictions — image_prediction is sole store (#768 step 3) ... Read cutover verified in prod (suggestions + allowlist read image_prediction; backfill complete at 908k rows / 51k images). Removes the old JSON column and everything that fed it: - ImageRecord.tagger_predictions column removed; migration 0046 DROPs it. tagger_model_version kept as the "tagged / current?" signal the backfill sweep reads (needs-tagging check switched to tagger_model_version IS NULL). - tag_and_embed no longer dual-writes the JSON — image_prediction is the only write path. - importer re-import reset drops the JSON line (image_prediction rows are already deleted on re-import). - Retired the one-time #768 backfill task + the #764 prune task, their admin endpoints, and their Maintenance cards (Backfill/PrunePredictionsCard). - Tests seed/assert via image_prediction; stale column refs removed. Disk reclaim is NOT automatic: DROP COLUMN is a catalog change. Run `VACUUM FULL image_record` off-hours afterward to return the ~100 GB to the OS so DB backups go small (#739). image_prediction (~90 MB) stays in pg_dump — it's the source of truth now. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-11 18:52:33 -04:00
bvandeusenandClaude Opus 4.7	906804140c	feat(fc2b): schema migration 0003 — ML pipeline tables ... Renames image_record.wd14_* -> tagger_* (we're on Camie now, not WD14). Adds tag_allowlist (auto-apply opt-in, per-tag confidence), tag_suggestion_rejection (per-image dismissals), tag_alias (composite (string, category) -> canonical tag, resolved at read time), tag_reference_embedding (per-tag SigLIP centroids), and the ml_settings singleton (per-category + centroid thresholds, model version pins). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-15 07:35:22 -04:00