4 Commits

Include renames

Author	SHA1	Message	Date
bvandeusenandClaude Opus 5	08418d54a3	db: index the seven unindexed FKs, drop the seven redundant ones (#3300, #3301) ... A structural sweep of the deployed schema, run AFTER 0088 got the models and the chain to exact agreement. That agreement is what 0088 achieved, and it is worth naming what it does not prove: a models-vs-chain diff shows the two describe the same schema, not that the schema is right. Everything here was wrong in BOTH. The one that matters: image_tag has PRIMARY KEY (image_record_id, tag_id) and no other index, so tag_id is unindexed. That is the gallery's tag filter (tag_query.py builds `image_tag.c.tag_id == tid`) and the ON DELETE CASCADE from tag, both scanning the largest table in the schema. Six more FKs were unindexed on smaller tables; presentation_review.tag_id also CASCADEs. Dropped, on the other side: ix_image_record_sha256 was an exact duplicate of the index uq_image_record_sha256 already builds — two btrees on the same column of the highest-insert-rate table. The other six are single-column indexes a later composite superseded without the narrow one being retired; a btree on (a,b) already serves lookups on a. 0088 deliberately taught the models to declare BOTH sha256 indexes so they would describe reality. This changes the reality instead, and the models change with it — otherwise the next baseline.yml run reintroduces exactly the drift 0088 removed. CONCURRENTLY throughout, so building the image_tag index does not hold an ACCESS EXCLUSIVE lock over every write for the duration. The cost is that the migration cannot run in a transaction and so is not atomic: every statement is IF NOT EXISTS / IF EXISTS, making a re-run after a partial failure safe. The docstring carries the query for finding an INVALID index left by an interrupted CONCURRENTLY build. What the sweep found clean, for the record: all 43 tables have a primary key; all 51 FKs declare an explicit ON DELETE, so none silently blocks a delete; the three enum CHECKs match the code that writes them (rule 36). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017QHszn9H8VBvx5Ke8x1hvw	2026-08-31 08:25:41 -04:00
bvandeusen	5e1996e77f	db: reconcile the models with the deployed schema (#3275) ... Milestone 328's acceptance test compared a database built by the real 0001..0087 chain against one built from the models, and found ~130 places where they disagree. This closes them. Almost all were the MODEL being wrong, so almost all of this is model edits with no DDL — the database already had these things, nothing in it changes, and no deploy is needed for this part: * 92 columns gained server_default. The models carried Python-side `default=` only, so the ORM filled the value and the column had no database default. Anything inserting outside the ORM behaved differently from production. * Eleven indexes that existed only in migrations are now declared: the three backup_run reporting indexes, the two date-ordered image_record browse indexes, import_task and presentation_review, and the three task_run history indexes. All use text() for their DESC ordering and postgresql_where for the partial one. * Two UNIQUE indexes that autogenerate silently proposed DROPPING, because neither is expressible as a UniqueConstraint: uq_tag_name_kind_fandom — an EXPRESSION index over (name, kind, COALESCE(fandom_id, 0)) uq_post_artist_external_id_null_source — PARTIAL, WHERE source_id IS NULL post.py already had a comment describing the second one. The comment was right; nothing declared it. * The two external_link enum CHECKs (host, status) — rule 36 territory, and absent from the model entirely. * Two indexes were named explicitly. A bare index=True generated ix_tag_alias_canonical_tag_id where the database has ix_tag_alias_canonical, so autogenerate proposed a drop+create of an index that was already there under another name. Same for tag_suggestion_rejection. Only ONE thing needed DDL, as 0088: tag.fandom_id is declared index=True but no migration ever created that index. Deliberately NOT here: image_record.sha256. The model says unique=True; 0001 created a plain index. Duplicates are possible today and the ORM believes otherwise. The fix depends on whether duplicates already exist — if they do, that is a dedupe decision, not a constraint — so it waits on an answer about live data. The real severity of #3275 is not the squash. It is that --autogenerate has been unsafe on this project: run against the old models it would have proposed dropping eleven indexes and two uniqueness guarantees.	2026-08-30 14:42:30 -04:00
bvandeusen	e77afe8295	feat(import-resilience L1): poison-pill circuit breaker — cap stuck-task re-queues ... Layer 1 of the import-task resilience work (operator-requested 2026-05-28). The recover_interrupted_tasks sweep re-queues rows stuck in 'processing' — correct for a worker crash, but without a cap a row that RELIABLY hard-crashes the worker (OOM/segfault/SIGKILL on a corrupt or oversized input) loops forever: re-queue → crash → re-queue, burning a worker slot every 5 min. A caught exception flips to terminal 'failed' and never enters this loop; only process-killing inputs do. - alembic 0026: import_task.recovery_count (int, default 0) + import_task.refetched (bool, default false — backs Layer 2). - recover_interrupted_tasks now runs a poison-pill UPDATE FIRST: stuck rows whose recovery_count has already reached MAX_RECOVERY_ATTEMPTS-1 are marked 'failed' with a diagnostic ("crashed or stalled the worker N times … likely a corrupt or oversized input … inspect/replace the file, then retry via /api/import/retry-failed") instead of re-queued. The re-queue pass then handles the remaining stuck rows and bumps recovery_count. Shared stuck_predicate (and_/or_) keeps the media-5min / archive-40min split. - MAX_RECOVERY_ATTEMPTS=3 (two recoveries then give up). The failed poison pill surfaces in the existing import-failures view with its file path, directly answering "help me identify them." Test test_recover_interrupted_poison_pill_caps_at_max pins both branches: a row at the cap is failed (not re-enqueued, diagnostic present), a row one short is re-queued + incremented.	2026-05-27 23:54:35 -04:00
bvandeusenandClaude Opus 4.7	9eb1614fbf	feat(fc2a): schema migration 0002 — tag kinds + fandom hierarchy + import lifecycle ... Adds Tag.kind enum (artist/character/fandom/general/series/archive/post/meta/rating), Tag.fandom_id FK with CHECK constraint (only valid for kind='character'), and a kind-aware uniqueness index so the same name can exist across kinds and the same character name can exist in different fandoms. Adds ImportBatch + ImportTask state-machine tables for scan tracking, plus a single-row ImportSettings table (CHECK id=1) holding the importer's filter knobs. Adds image_record.integrity_status column (defaults to 'unknown'); FC-2e populates this via the integrity verifier. Drops the unused tag.namespace column from FC-1 — superseded by kind. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-14 12:03:41 -04:00