Schema reconciliation + index hygiene, and the weekly base-image refresh #243

Merged
bvandeusen merged 17 commits from dev into main 2026-08-31 08:34:55 -04:00
Showing only changes of commit 5fd171a544 - Show all commits
+33 -17
View File
@@ -17,9 +17,10 @@
# still has them. That is what makes this the proof for step 1 and the
# pre-flight for step 2, rather than a one-shot script.
#
# `generate: true` additionally autogenerates a candidate baseline from the
# models and uploads it. That is a starting point, NOT the answer: autogenerate
# reads SQLAlchemy metadata, and three things in this schema do not live there —
# While the chain is still present it also autogenerates a candidate baseline
# from the models and prints it. That is a starting point, NOT the answer:
# autogenerate reads SQLAlchemy metadata, and three things here do not live
# there —
# * CREATE EXTENSION vector (0001)
# * CREATE EXTENSION tsm_system_rows (0004)
# * the HNSW index on image_record.siglip_embedding, which is raw SQL
@@ -36,10 +37,6 @@ on:
description: 'Commit/tag that still carries the full 0001..0087 chain'
type: string
default: '0a5bbe8'
generate:
description: 'Also autogenerate a candidate baseline from the models'
type: boolean
default: false
jobs:
compare:
@@ -111,13 +108,23 @@ jobs:
--no-owner --no-privileges -d fc_chain > chain.sql
wc -l chain.sql
# Optional: a candidate baseline, autogenerated from the models against an
# EMPTY database so every table shows up as a create. Uploaded for a human
# to finish — it will be missing the three raw-SQL items named at the top.
# A candidate baseline, autogenerated from the models against an EMPTY
# database so every table shows up as a create. Printed for a human to
# finish — it will be missing the three raw-SQL items named at the top.
#
# Gated on the TREE, not on a workflow input. A `type: boolean` input
# read back as `github.event.inputs.generate == 'true'` silently
# evaluated false on this runner (run 4960 skipped this step entirely
# with no diagnostic) — the same `github.event.inputs` typing quirk
# build.yml already works around. The file count is the real question
# anyway: there is nothing to generate once the chain is collapsed.
- name: Autogenerate a candidate baseline
if: ${{ github.event.inputs.generate == 'true' }}
run: |
set -eux
if [ "$(ls alembic/versions/*.py | wc -l)" -le 1 ]; then
echo "already collapsed — nothing to generate"
exit 0
fi
docker exec "$PG_CONTAINER" createdb -U fabledcurator fc_gen
# Hide the existing revisions so alembic sees an empty history and
# emits the whole schema rather than a delta.
@@ -153,16 +160,25 @@ jobs:
#
# pg_dump orders dumpable objects by name within type, not by creation
# order, so two schemas built by different routes are directly
# comparable. The only normalisation applied is dropping blank lines,
# comment lines and the alembic_version row-count noise — deliberately
# minimal, because a filter that hides a real difference is the one way
# this check passes when it should fail. Whatever is normalised is
# printed, so the filtering itself is reviewable.
# comparable. Normalisation is deliberately minimal, because a filter
# that hides a real difference is the one way this check passes when it
# should fail — blank lines, SQL comments, trailing whitespace, and:
#
# \restrict / \unrestrict — a per-invocation RANDOM NONCE that newer
# pg_dump emits to fence the dump against injection during restore. It
# differs on every run by construction, so it is noise by definition,
# not a schema difference. Measured on run 4960, the control: two dumps
# of the SAME schema came back 1123 lines each and differed on exactly
# these two lines and nothing else. That control is what licenses this
# filter — it was observed to be the only false positive, rather than
# assumed to be one.
- name: Diff
run: |
set -eu
norm() {
grep -vE '^\s*(--|$)' "$1" | sed 's/[[:space:]]*$//'
grep -vE '^\s*(--|$)' "$1" \
| grep -vE '^\\(un)?restrict ' \
| sed 's/[[:space:]]*$//'
}
norm chain.sql > a.txt
norm baseline.sql > b.txt