8 Commits

Include renames

Author	SHA1	Message	Date
bvandeusenandClaude Opus 5	b979062dd7	db: rename the four double-prefixed CHECK constraints (#3275) ... Run 5026 got the models-vs-chain diff to 7 lines. Three findings, and one of them reverses an assumption I made in the previous commit. The doubled CHECK names are what the DATABASE has, not what the generator invented. base.py's convention is ck_%(table_name)s_%(constraint_name)s, which — unlike uq/fk/ix — applies even to a constraint that already has a name, so four migrations that passed an already-prefixed name got it prefixed twice: ck_import_settings_ck_import_settings_singleton ck_ml_settings_ck_ml_settings_singleton ck_post_ck_post_translation_override ck_tag_ck_tag_fandom_requires_character The workflow repair added last commit is still correct and still needed — autogenerate really does re-double a name on the round trip — but it was making the MODELS side clean against a chain that is dirty. The comment in ml_settings.py claiming its bare name "matches migration 0003" was simply false; 0003 produces the doubled form. Nothing reads a CHECK constraint by name, so this has never done harm. But it is precisely the development-era residue the collapsed baseline exists to leave behind, and a public schema should not ship it — so 0088 renames the deployed constraints and all six models now declare bare names. RENAME CONSTRAINT is catalog-only: no scan, no rewrite, no revalidation, which is why this is safe on post and tag. Guarded on pg_constraint scoped by conrelid, so it is a no-op on a database built from the models. ix_tag_fandom_id showed as a difference only because chain_ref was pinned to 0a5bbe8, which predates 0088 — the comparison was measuring the models against a chain missing the migration that closes the gap. chain_ref now defaults to blank, meaning "the chain in this ref". Pin it to a commit only after the collapse, when the tree no longer carries the revisions. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017QHszn9H8VBvx5Ke8x1hvw	2026-08-31 00:29:57 -04:00
bvandeusenandClaude Opus 5	573228b9da	db: finish reconciling the models with the deployed schema (#3275) ... Closes the residue the first reconciliation pass left, and corrects a factual error I put into the record. sha256 was NOT missing a uniqueness guarantee. I read `op.create_index("ix_image_record_sha256", ...)` at 0001 line 151 and concluded duplicates were possible, without reading line 149 two lines above it: sa.UniqueConstraint("sha256", name="uq_image_record_sha256"), Uniqueness has held since the initial schema. The database expresses it as a CONSTRAINT plus a separate non-unique lookup index; the model said `unique=True, index=True`, which is one UNIQUE index under a different name. Same guarantee, different objects — which is exactly why the two schemas did not line up. The model now declares both objects. No DDL. 0088's docstring, which repeated the claim, is corrected in place. Two real divergences, both the MODEL over-claiming: * source: uq_source_artist_platform_url (alembic 0010) was declared nowhere in the models — source.py had no __table_args__ at all — so autogenerate would have proposed DROPPING it. * head_metrics_snapshot.tag_id: model said NOT NULL, 0060 created it nullable. Left nullable; the FK already cascades. Seven constraints renamed to what the chain actually created, rather than what base.py's naming convention renders: uq_series_page_image, uq_series_chapter_anchor_page, fk_series_chapter_anchor_page, fk_image_record_artist_id, fk_image_provenance_from_attachment, and the two hand-shortened fk_tsr_* names from 0003. Float server_defaults now mirror their own migration, per column. The chain is MIXED: a plain string renders DEFAULT '0.90'::double precision, sa.text() renders DEFAULT 0.90, and the migrations used both. Seven columns take text(); the rest stay strings. Two literals also disagreed outright — process_{auto_apply,conflict}_threshold said 0.9/0.5 against the migration's 0.90/0.50. baseline.yml gains two things. A repair for a SECOND generator defect in the same class as the missing pgvector import: base.py's ck convention contains %(constraint_name)s, so it applies even to a NAMED CheckConstraint — autogenerate writes the already-rendered name into the migration and running it applies the convention again, yielding ck_ml_settings_ck_ml_settings_singleton. That is round-tripping damage, not a claim the models make, so it is undone rather than counted. And the diff now runs twice. Column ORDER differs permanently between a schema built by 87 ADD COLUMNs and one built in a single shot — the operator's database keeps chain order forever, a fresh install gets model order — so a check that failed on it could never pass. The second pass SORTS column lines within each CREATE TABLE instead of deleting them, which cannot hide a column present on one side only, or one whose type, nullability or default differs. Ordered diff is reported as information; the order-insensitive one is the verdict. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017QHszn9H8VBvx5Ke8x1hvw	2026-08-31 00:24:00 -04:00
bvandeusen	d044e93bdb	ci: repair autogenerate's missing pgvector import before applying (#3275) ... mode: models applies the raw autogenerated candidate, and it cannot run: sa.Column('weights', pgvector.sqlalchemy.vector.VECTOR(dim=1152), ...) NameError: name 'pgvector' is not defined Alembic emits the qualified reference without emitting the import. Observed on run 4988, which turns this from a thing I predicted by reading the candidate into a thing demonstrated by executing it. Repaired in the workflow rather than counted as a schema difference: the comparison asks whether the MODELS describe the schema, and this is a defect in the generator. The same fixup has to be applied by hand to any baseline generated this way, which is why it is item 4 on the collapsed baseline's hand-written list.	2026-08-30 14:47:29 -04:00
bvandeusen	ed2b1adc2e	ci: compare the schema the MODELS produce against the migrations (#3275) ... baseline.yml only ever compared migrations against migrations. The question #3275 exists because nobody had ever asked the other one: does a database built from the MODELS match the one the chain produces? `mode: models` answers it. It applies the candidate autogenerated from the models instead of this tree's revisions, and diffs that against the chain. A clean run means --autogenerate is trustworthy again, which it demonstrably has not been: against the pre-reconciliation models it would have proposed dropping eleven indexes and two uniqueness guarantees. The two extensions are created by hand in that mode. They are database objects rather than table metadata, so no model can carry them — their absence is outside what this comparison asks about, and silently tolerating it is correct rather than a filter that hides a defect. Also declares the HNSW index on the ImageRecord model. SQLAlchemy can express an hnsw access method with an operator class (postgresql_using + postgresql_ops), so there was never a reason for it to live only in 0036. That removes the last item from the list of things a generated baseline cannot reproduce, leaving only the two extensions.	2026-08-30 14:43:26 -04:00
bvandeusen	98b56330d0	ci: emit the chain schema dump for local reconciliation work (#3275) ... Reconciling the models against the deployed schema needs the actual pg_dump, not an inference from the unified diff. Parsing table context out of diff hunks drops every table whose CREATE TABLE line falls outside a hunk — it under-reported 81 columns across 13 tables when the real figure spans more, missing artist, gpu_job, download_event and external_link entirely. Same checksummed-base64 transport as the candidate baseline, for the same reason: a plain cat of a file this size was silently truncated mid-line by the runner on run 4964.	2026-08-30 14:35:02 -04:00
bvandeusen	8f1ac0c96a	ci: transport the candidate baseline as verifiable base64 ... Run 4964 passed the control (1121 normalised lines, schemas identical) but its candidate print was silently truncated. `cat` of the ~33KB generated file stopped mid-line inside sa.Column('mime', sa.String(length=128) and the runner carried straight on to the next traced command with the step still green. The captured text was 484 lines and 29 tables, and looked entirely plausible — which is exactly what makes it dangerous: a schema definition cut in half is still syntactically suggestive, and nothing in the log says it was cut. Now emitted as base64 at a fixed 120-column width, followed by a sha256, a byte count and a base64 line count. Short lines instead of long ones, and more importantly the receiving end can PROVE it got the whole file rather than trusting that it did. Also found in that output, and the reason the candidate could never have been committed as-is: it references pgvector.sqlalchemy.vector.VECTOR(dim=1152) for head_training_run.weights and image_record.siglip_embedding, but autogenerate does not add the corresponding import. The file would die with NameError on the first run. That is the fourth item on the list of things the generator cannot be trusted with, alongside the two CREATE EXTENSIONs and the HNSW index.	2026-08-30 13:40:15 -04:00
bvandeusen	5fd171a544	ci: fix two things the baseline control run found ... Run 4960 was the control — the chain compared against itself, which must come back clean before a clean diff after the squash means anything. It did its job and failed on both counts. 1. The harness is sound. Both dumps came back 1123 normalised lines and differed on EXACTLY two, the \restrict / \unrestrict pair that newer pg_dump emits to fence a dump against injection during restore. It is a fresh random nonce per invocation, so it differs by construction and is noise by definition. Now filtered — and the control is what licenses that filter: it was OBSERVED to be the only false positive rather than assumed to be one, which matters for a check whose whole value is that its normalisation does not hide a real difference. 2. The candidate-baseline step never ran. `if: github.event.inputs .generate == 'true'` on a `type: boolean` input silently evaluated false — no diagnostic, step skipped, job carried on. The same `github.event.inputs` typing quirk build.yml already works around for force_build. Rather than fight the input typing, the gate is now the tree itself: skip if alembic/versions holds one file. That is the real question anyway — there is nothing to generate once the chain is collapsed — and it cannot be silently wrong the way an unevaluated expression can. Worth noting what the control also proved incidentally: the two schemas were byte-identical across 1123 lines despite being built by separate alembic runs into separate databases, so pg_dump's object ordering is stable enough to diff directly and no sort normalisation is needed.	2026-08-30 13:35:26 -04:00
bvandeusen	62583791d8	ci: a workflow that proves a collapsed alembic chain matches the old one ... Milestone 328 step 1 needs a baseline generated from the models, and step 2 must not stamp the operator's live database until that baseline is proven to reproduce what the 87-revision chain produced. `alembic stamp` validates nothing, so an unproven baseline fails silently now and loudly later, on real data. There is no local Python environment and rules 10/12 point away from standing one up, so the comparison runs in CI, where a pgvector Postgres is already built from the chain on every integration run and nothing is at risk. It builds two databases and diffs their pg_dump --schema-only output: one from `alembic upgrade head` on the revisions read out of git at `chain_ref`, one from the current tree. Reading the chain from git via a worktree — rather than from the working tree — is what keeps this usable AFTER the old revisions are deleted, so it is the proof for step 1 and the pre-flight for step 2 rather than a one-shot script. Both sides use `alembic upgrade head`, never metadata.create_all, per rule 82 — and that rule's reasoning is exactly the hazard here. `create_all` emits plain CREATE TABLE and skips everything else, which is why the optional autogenerated candidate CANNOT be trusted as the answer. Three things in this schema are invisible to SQLAlchemy metadata: CREATE EXTENSION vector (0001) CREATE EXTENSION tsm_system_rows (0004) the HNSW index on image_record.siglip_embedding, raw SQL because alembic's create_index cannot express USING hnsw (...) (0036) plus any CHECK constraint or server_default a migration added without the model declaring it — 4 model files declare CheckConstraints against 6 migrations that touch them. The candidate is a starting point to hand finish; the diff is what proves nothing was missed. Results are printed to the job log rather than uploaded: ci-requirements records that this runner cannot do actions/upload-artifact@v4+, and the repo dropped the action entirely in 2026-05. Run it first with the chain still present, as a control — the diff compares the chain against itself and must come back clean. A clean diff after the squash only means something if the harness was shown to be capable of producing one beforehand. Temporary. Delete once the baseline is stamped.	2026-08-30 13:31:16 -04:00