Compare commits

..
18 Commits
Author SHA1 Message Date
bvandeusen b14818303c Merge pull request 'Schema reconciliation + index hygiene, and the weekly base-image refresh' (#243) from dev into main
CI / lint (push) Successful in 3s
Build images / sign-extension (push) Successful in 4s
CI / extension-version (push) Successful in 2s
Build images / build-agent (push) Successful in 8s
Build images / build-ml (push) Successful in 14s
CI / frontend-build (push) Successful in 19s
Build images / build-web (push) Successful in 14s
extension / lint (push) Successful in 18s
CI / backend-lint-and-test (push) Successful in 30s
CI / integration (push) Successful in 3m54s
2026-08-31 08:34:55 -04:00
bvandeusenandClaude Opus 5 08418d54a3 db: index the seven unindexed FKs, drop the seven redundant ones (#3300, #3301)
Build images / build-ml (push) Successful in 32s
Build images / sign-extension (push) Successful in 4s
Build images / build-agent (push) Successful in 11s
Build images / build-web (push) Successful in 26s
CI / lint (push) Successful in 3s
CI / extension-version (push) Successful in 3s
CI / integration (push) Successful in 3m44s
CI / frontend-build (push) Successful in 20s
CI / backend-lint-and-test (push) Successful in 30s
extension / lint (pull_request) Successful in 24s
A structural sweep of the deployed schema, run AFTER 0088 got the models
and the chain to exact agreement. That agreement is what 0088 achieved,
and it is worth naming what it does not prove: a models-vs-chain diff
shows the two describe the same schema, not that the schema is right.
Everything here was wrong in BOTH.

The one that matters: image_tag has PRIMARY KEY (image_record_id, tag_id)
and no other index, so tag_id is unindexed. That is the gallery's tag
filter (tag_query.py builds `image_tag.c.tag_id == tid`) and the ON
DELETE CASCADE from tag, both scanning the largest table in the schema.
Six more FKs were unindexed on smaller tables; presentation_review.tag_id
also CASCADEs.

Dropped, on the other side: ix_image_record_sha256 was an exact duplicate
of the index uq_image_record_sha256 already builds — two btrees on the
same column of the highest-insert-rate table. The other six are
single-column indexes a later composite superseded without the narrow one
being retired; a btree on (a,b) already serves lookups on a.

0088 deliberately taught the models to declare BOTH sha256 indexes so
they would describe reality. This changes the reality instead, and the
models change with it — otherwise the next baseline.yml run reintroduces
exactly the drift 0088 removed.

CONCURRENTLY throughout, so building the image_tag index does not hold an
ACCESS EXCLUSIVE lock over every write for the duration. The cost is that
the migration cannot run in a transaction and so is not atomic: every
statement is IF NOT EXISTS / IF EXISTS, making a re-run after a partial
failure safe. The docstring carries the query for finding an INVALID
index left by an interrupted CONCURRENTLY build.

What the sweep found clean, for the record: all 43 tables have a primary
key; all 51 FKs declare an explicit ON DELETE, so none silently blocks a
delete; the three enum CHECKs match the code that writes them (rule 36).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017QHszn9H8VBvx5Ke8x1hvw
2026-08-31 08:25:41 -04:00
bvandeusenandClaude Opus 5 b1bd2531ad style: sort JSON first in three sqlalchemy import blocks
CI / lint (push) Successful in 4s
Build images / sign-extension (push) Successful in 4s
CI / extension-version (push) Successful in 4s
Build images / build-agent (push) Successful in 8s
CI / frontend-build (push) Successful in 27s
CI / backend-lint-and-test (push) Successful in 33s
Build images / build-web (push) Successful in 43s
Build images / build-ml (push) Successful in 51s
CI / integration (push) Successful in 3m47s
ruff's isort runs with order-by-type, which sorts ALL_CAPS names ahead of
CamelCase ones, so `JSON` belongs at the head of the list rather than
between `Integer` and `String`.

Two of these (backup_run.py, post.py) have been failing lint since
5e1996e — I did not check the push CI after that commit, only the
baseline workflow I had dispatched, so ci.yml has been red on dev across
5e1996e, ed2b1ad and d044e93.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017QHszn9H8VBvx5Ke8x1hvw
2026-08-31 00:38:46 -04:00
bvandeusenandClaude Opus 5 389afe2f7b db: the doubled CHECK list was six, not four (#3275)
CI / extension-version (push) Successful in 6s
CI / lint (push) Failing after 6s
Build images / sign-extension (push) Successful in 6s
Build images / build-agent (push) Successful in 11s
CI / backend-lint-and-test (push) Successful in 31s
CI / frontend-build (push) Successful in 22s
Build images / build-ml (push) Successful in 52s
CI / integration (push) Successful in 3m43s
Build images / build-web (push) Successful in 41s
Run 5029 confirmed the four renames landed and surfaced two I had missed:
external_link's host and status CHECKs are doubled the same way. They did
not show in run 5026's diff because BOTH sides produced the doubled form
back then — external_link.py pre-prefixed its names, so the models
matched the chain's mistake. Switching all six models to bare names is
what exposed the two the migration did not cover.

The list in the file now comes from matching ck_(\w+?)_ck_\1_ against the
chain's own pg_dump, rather than from reading migrations by eye. Reading
by eye is what missed these, in the same way it earlier missed a UNIQUE
constraint sitting two lines above the index being looked at.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017QHszn9H8VBvx5Ke8x1hvw
2026-08-31 00:33:50 -04:00
bvandeusenandClaude Opus 5 b979062dd7 db: rename the four double-prefixed CHECK constraints (#3275)
Build images / sign-extension (push) Successful in 4s
CI / lint (push) Failing after 5s
CI / extension-version (push) Successful in 5s
Build images / build-agent (push) Successful in 9s
CI / frontend-build (push) Successful in 20s
CI / backend-lint-and-test (push) Successful in 33s
Build images / build-ml (push) Successful in 44s
Build images / build-web (push) Successful in 41s
CI / integration (push) Successful in 3m52s
Run 5026 got the models-vs-chain diff to 7 lines. Three findings, and one
of them reverses an assumption I made in the previous commit.

The doubled CHECK names are what the DATABASE has, not what the generator
invented. base.py's convention is ck_%(table_name)s_%(constraint_name)s,
which — unlike uq/fk/ix — applies even to a constraint that already has a
name, so four migrations that passed an already-prefixed name got it
prefixed twice:

    ck_import_settings_ck_import_settings_singleton
    ck_ml_settings_ck_ml_settings_singleton
    ck_post_ck_post_translation_override
    ck_tag_ck_tag_fandom_requires_character

The workflow repair added last commit is still correct and still needed —
autogenerate really does re-double a name on the round trip — but it was
making the MODELS side clean against a chain that is dirty. The
comment in ml_settings.py claiming its bare name "matches migration 0003"
was simply false; 0003 produces the doubled form.

Nothing reads a CHECK constraint by name, so this has never done harm.
But it is precisely the development-era residue the collapsed baseline
exists to leave behind, and a public schema should not ship it — so 0088
renames the deployed constraints and all six models now declare bare
names. RENAME CONSTRAINT is catalog-only: no scan, no rewrite, no
revalidation, which is why this is safe on post and tag. Guarded on
pg_constraint scoped by conrelid, so it is a no-op on a database built
from the models.

ix_tag_fandom_id showed as a difference only because chain_ref was pinned
to 0a5bbe8, which predates 0088 — the comparison was measuring the models
against a chain missing the migration that closes the gap. chain_ref now
defaults to blank, meaning "the chain in this ref". Pin it to a commit
only after the collapse, when the tree no longer carries the revisions.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017QHszn9H8VBvx5Ke8x1hvw
2026-08-31 00:29:57 -04:00
bvandeusenandClaude Opus 5 573228b9da db: finish reconciling the models with the deployed schema (#3275)
CI / lint (push) Failing after 3s
Build images / sign-extension (push) Successful in 4s
CI / extension-version (push) Successful in 3s
Build images / build-agent (push) Successful in 9s
CI / frontend-build (push) Successful in 34s
Build images / build-ml (push) Successful in 53s
Build images / build-web (push) Successful in 44s
CI / integration (push) Successful in 4m5s
CI / backend-lint-and-test (push) Successful in 1m6s
Closes the residue the first reconciliation pass left, and corrects a
factual error I put into the record.

sha256 was NOT missing a uniqueness guarantee. I read
`op.create_index("ix_image_record_sha256", ...)` at 0001 line 151 and
concluded duplicates were possible, without reading line 149 two lines
above it:

    sa.UniqueConstraint("sha256", name="uq_image_record_sha256"),

Uniqueness has held since the initial schema. The database expresses it
as a CONSTRAINT plus a separate non-unique lookup index; the model said
`unique=True, index=True`, which is one UNIQUE index under a different
name. Same guarantee, different objects — which is exactly why the two
schemas did not line up. The model now declares both objects. No DDL.
0088's docstring, which repeated the claim, is corrected in place.

Two real divergences, both the MODEL over-claiming:

  * source: uq_source_artist_platform_url (alembic 0010) was declared
    nowhere in the models — source.py had no __table_args__ at all — so
    autogenerate would have proposed DROPPING it.
  * head_metrics_snapshot.tag_id: model said NOT NULL, 0060 created it
    nullable. Left nullable; the FK already cascades.

Seven constraints renamed to what the chain actually created, rather than
what base.py's naming convention renders: uq_series_page_image,
uq_series_chapter_anchor_page, fk_series_chapter_anchor_page,
fk_image_record_artist_id, fk_image_provenance_from_attachment, and the
two hand-shortened fk_tsr_* names from 0003.

Float server_defaults now mirror their own migration, per column. The
chain is MIXED: a plain string renders DEFAULT '0.90'::double precision,
sa.text() renders DEFAULT 0.90, and the migrations used both. Seven
columns take text(); the rest stay strings. Two literals also disagreed
outright — process_{auto_apply,conflict}_threshold said 0.9/0.5 against
the migration's 0.90/0.50.

baseline.yml gains two things. A repair for a SECOND generator defect in
the same class as the missing pgvector import: base.py's ck convention
contains %(constraint_name)s, so it applies even to a NAMED
CheckConstraint — autogenerate writes the already-rendered name into the
migration and running it applies the convention again, yielding
ck_ml_settings_ck_ml_settings_singleton. That is round-tripping damage,
not a claim the models make, so it is undone rather than counted.

And the diff now runs twice. Column ORDER differs permanently between a
schema built by 87 ADD COLUMNs and one built in a single shot — the
operator's database keeps chain order forever, a fresh install gets model
order — so a check that failed on it could never pass. The second pass
SORTS column lines within each CREATE TABLE instead of deleting them,
which cannot hide a column present on one side only, or one whose type,
nullability or default differs. Ordered diff is reported as information;
the order-insensitive one is the verdict.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017QHszn9H8VBvx5Ke8x1hvw
2026-08-31 00:24:00 -04:00
bvandeusen d044e93bdb ci: repair autogenerate's missing pgvector import before applying (#3275)
CI / lint (push) Failing after 3s
Build images / sign-extension (push) Successful in 3s
CI / extension-version (push) Successful in 3s
Build images / build-ml (push) Successful in 7s
Build images / build-agent (push) Successful in 8s
CI / frontend-build (push) Successful in 23s
CI / backend-lint-and-test (push) Successful in 38s
Build images / build-web (push) Successful in 13s
CI / integration (push) Successful in 3m52s
mode: models applies the raw autogenerated candidate, and it cannot run:

  sa.Column('weights', pgvector.sqlalchemy.vector.VECTOR(dim=1152), ...)
  NameError: name 'pgvector' is not defined

Alembic emits the qualified reference without emitting the import.
Observed on run 4988, which turns this from a thing I predicted by
reading the candidate into a thing demonstrated by executing it.

Repaired in the workflow rather than counted as a schema difference: the
comparison asks whether the MODELS describe the schema, and this is a
defect in the generator. The same fixup has to be applied by hand to any
baseline generated this way, which is why it is item 4 on the collapsed
baseline's hand-written list.
2026-08-30 14:47:29 -04:00
bvandeusen ed2b1adc2e ci: compare the schema the MODELS produce against the migrations (#3275)
Build images / sign-extension (push) Successful in 4s
CI / lint (push) Failing after 4s
CI / extension-version (push) Successful in 4s
Build images / build-agent (push) Successful in 11s
Build images / build-ml (push) Successful in 47s
CI / frontend-build (push) Successful in 22s
CI / backend-lint-and-test (push) Successful in 35s
Build images / build-web (push) Successful in 37s
CI / integration (push) Successful in 3m53s
baseline.yml only ever compared migrations against migrations. The
question #3275 exists because nobody had ever asked the other one: does
a database built from the MODELS match the one the chain produces?

`mode: models` answers it. It applies the candidate autogenerated from
the models instead of this tree's revisions, and diffs that against the
chain. A clean run means --autogenerate is trustworthy again, which it
demonstrably has not been: against the pre-reconciliation models it
would have proposed dropping eleven indexes and two uniqueness
guarantees.

The two extensions are created by hand in that mode. They are database
objects rather than table metadata, so no model can carry them — their
absence is outside what this comparison asks about, and silently
tolerating it is correct rather than a filter that hides a defect.

Also declares the HNSW index on the ImageRecord model. SQLAlchemy can
express an hnsw access method with an operator class
(postgresql_using + postgresql_ops), so there was never a reason for it
to live only in 0036. That removes the last item from the list of things
a generated baseline cannot reproduce, leaving only the two extensions.
2026-08-30 14:43:26 -04:00
bvandeusen 5e1996e77f db: reconcile the models with the deployed schema (#3275)
Build images / sign-extension (push) Successful in 4s
CI / lint (push) Failing after 2s
CI / extension-version (push) Successful in 2s
Build images / build-agent (push) Successful in 7s
CI / frontend-build (push) Successful in 27s
Build images / build-ml (push) Successful in 48s
CI / backend-lint-and-test (push) Successful in 1m7s
Build images / build-web (push) Successful in 40s
CI / integration (push) Successful in 4m1s
Milestone 328's acceptance test compared a database built by the real
0001..0087 chain against one built from the models, and found ~130
places where they disagree. This closes them.

Almost all were the MODEL being wrong, so almost all of this is model
edits with no DDL — the database already had these things, nothing in it
changes, and no deploy is needed for this part:

* 92 columns gained server_default. The models carried Python-side
  `default=` only, so the ORM filled the value and the column had no
  database default. Anything inserting outside the ORM behaved
  differently from production.

* Eleven indexes that existed only in migrations are now declared:
  the three backup_run reporting indexes, the two date-ordered
  image_record browse indexes, import_task and presentation_review,
  and the three task_run history indexes. All use text() for their DESC
  ordering and postgresql_where for the partial one.

* Two UNIQUE indexes that autogenerate silently proposed DROPPING,
  because neither is expressible as a UniqueConstraint:
    uq_tag_name_kind_fandom  — an EXPRESSION index over
                               (name, kind, COALESCE(fandom_id, 0))
    uq_post_artist_external_id_null_source — PARTIAL, WHERE source_id
                               IS NULL
  post.py already had a comment describing the second one. The comment
  was right; nothing declared it.

* The two external_link enum CHECKs (host, status) — rule 36 territory,
  and absent from the model entirely.

* Two indexes were named explicitly. A bare index=True generated
  ix_tag_alias_canonical_tag_id where the database has
  ix_tag_alias_canonical, so autogenerate proposed a drop+create of an
  index that was already there under another name. Same for
  tag_suggestion_rejection.

Only ONE thing needed DDL, as 0088: tag.fandom_id is declared
index=True but no migration ever created that index.

Deliberately NOT here: image_record.sha256. The model says unique=True;
0001 created a plain index. Duplicates are possible today and the ORM
believes otherwise. The fix depends on whether duplicates already exist
— if they do, that is a dedupe decision, not a constraint — so it waits
on an answer about live data.

The real severity of #3275 is not the squash. It is that --autogenerate
has been unsafe on this project: run against the old models it would
have proposed dropping eleven indexes and two uniqueness guarantees.
2026-08-30 14:42:30 -04:00
bvandeusen 98b56330d0 ci: emit the chain schema dump for local reconciliation work (#3275)
Build images / sign-extension (push) Successful in 4s
CI / lint (push) Successful in 4s
CI / extension-version (push) Successful in 5s
Build images / build-agent (push) Successful in 8s
CI / frontend-build (push) Successful in 26s
Build images / build-ml (push) Successful in 44s
CI / backend-lint-and-test (push) Successful in 43s
Build images / build-web (push) Successful in 36s
CI / integration (push) Successful in 3m55s
Reconciling the models against the deployed schema needs the actual
pg_dump, not an inference from the unified diff. Parsing table context
out of diff hunks drops every table whose CREATE TABLE line falls
outside a hunk — it under-reported 81 columns across 13 tables when the
real figure spans more, missing artist, gpu_job, download_event and
external_link entirely.

Same checksummed-base64 transport as the candidate baseline, for the
same reason: a plain cat of a file this size was silently truncated
mid-line by the runner on run 4964.
2026-08-30 14:35:02 -04:00
bvandeusen 6959e1220c Revert "db: collapse alembic 0001..0087 into one baseline"
This reverts 2529b51. Not a retreat — a reordering, on the operator's
call, and the better sequence.

The squash's acceptance test (run 4971) found ~130 places where the ORM
models do not describe the deployed schema (#3275), including a
unique=True the database never had and two UNIQUE indexes that exist
only in migrations. Collapsing now would have baked all of that into the
one file a public installer starts from.

So: fix the drift first as ordinary migrations on the intact chain, let
the operator deploy so their database moves to the corrected head, and
only then collapse. The baseline is then generated from reconciled
models and reproduces a schema worth reproducing.

Nothing is lost by reverting. The baseline was never deployed, and
regenerating it after the fixes is strictly better than patching this
copy — it will come out of autogenerate correct rather than needing the
same hand-finishing twice.
2026-08-30 14:34:28 -04:00
bvandeusen 2529b516e6 db: collapse alembic 0001..0087 into one baseline (milestone 328 step 1)
Build images / sign-extension (push) Successful in 4s
Build images / build-agent (push) Successful in 9s
CI / lint (push) Successful in 4s
CI / extension-version (push) Successful in 5s
CI / frontend-build (push) Successful in 24s
Build images / build-ml (push) Successful in 42s
CI / backend-lint-and-test (push) Successful in 53s
Build images / build-web (push) Successful in 33s
CI / integration (push) Failing after 3m47s
87 revisions narrating this project's build-out become one file that
creates the schema in a single step. They cost nothing at runtime — all
86 upgrade steps ran in 0.2s (note #3260) — so this is a presentation
change, not a performance one: a new installer should not inherit our
development history to stand up a database.

Deleted: 87 revisions (6,052 lines), the 10 tests/test_migration_*.py
files (483 lines) that asserted intermediate states and backfills which
no longer exist, and backend/app/utils/artist_backfill.py — the only
live module a migration imported, with no other consumer anywhere. That
last one satisfies the operator's separate request to inline it into
0008 and delete the module; the squash removes both outright.

THE REVISION ID IS "0087", NOT "0001", ON PURPOSE. It is the id of the
last revision collapsed, so an existing database is already at head and
`alembic upgrade head` does nothing. The alternative is `alembic stamp`
against live data, and stamp validates NOTHING — it writes a version
string whether or not the schema matches, so a wrong baseline surfaces
later, via the next real migration, with no clean way back. This removes
that operation rather than making it safe. Future revisions run from
0088.

Four things are hand-written because SQLAlchemy metadata does not carry
them, and none fail at generation time:

  1. CREATE EXTENSION vector          — the VECTOR columns cannot be
     created without it, so it is ordered first in upgrade().
  2. CREATE EXTENSION tsm_system_rows — surfaces only when the random
     sample query runs.
  3. the HNSW index on image_record.siglip_embedding, raw SQL because
     create_index cannot express USING hnsw (... vector_cosine_ops).
     The quietest of the four: everything works, similarity search just
     stops using an index.
  4. import pgvector.sqlalchemy.vector — autogenerate EMITS
     pgvector.sqlalchemy.vector.VECTOR references without importing it,
     so the generated file dies with NameError on first run.

The candidate came out of CI (run 4967) as checksummed base64 rather
than a plain cat, because run 4964's cat was truncated mid-line inside a
column definition with the step still green — 29 tables instead of 42,
and it looked entirely plausible. Verified here: 56,582 bytes,
sha256 471acfca69c0…, 42 tables, 66 indexes, 42 drops.

NOT YET PROVEN against the old chain. baseline.yml does that, and it is
step 2's gate; this commit does not claim the schemas match.
2026-08-30 13:45:35 -04:00
bvandeusen 8f1ac0c96a ci: transport the candidate baseline as verifiable base64
CI / integration (push) Successful in 3m48s
CI / lint (push) Successful in 4s
Build images / sign-extension (push) Successful in 5s
CI / extension-version (push) Successful in 5s
Build images / build-ml (push) Successful in 8s
Build images / build-agent (push) Successful in 9s
Build images / build-web (push) Successful in 7s
CI / frontend-build (push) Successful in 19s
CI / backend-lint-and-test (push) Successful in 45s
Run 4964 passed the control (1121 normalised lines, schemas identical)
but its candidate print was silently truncated. `cat` of the ~33KB
generated file stopped mid-line inside

  sa.Column('mime', sa.String(length=128)

and the runner carried straight on to the next traced command with the
step still green. The captured text was 484 lines and 29 tables, and
looked entirely plausible — which is exactly what makes it dangerous:
a schema definition cut in half is still syntactically suggestive, and
nothing in the log says it was cut.

Now emitted as base64 at a fixed 120-column width, followed by a
sha256, a byte count and a base64 line count. Short lines instead of
long ones, and more importantly the receiving end can PROVE it got the
whole file rather than trusting that it did.

Also found in that output, and the reason the candidate could never
have been committed as-is: it references

  pgvector.sqlalchemy.vector.VECTOR(dim=1152)

for head_training_run.weights and image_record.siglip_embedding, but
autogenerate does not add the corresponding import. The file would die
with NameError on the first run. That is the fourth item on the list of
things the generator cannot be trusted with, alongside the two CREATE
EXTENSIONs and the HNSW index.
2026-08-30 13:40:15 -04:00
bvandeusen 5fd171a544 ci: fix two things the baseline control run found
CI / lint (push) Successful in 3s
CI / extension-version (push) Successful in 4s
Build images / sign-extension (push) Successful in 4s
Build images / build-web (push) Successful in 6s
CI / integration (push) Successful in 3m52s
Build images / build-ml (push) Successful in 8s
Build images / build-agent (push) Successful in 8s
CI / frontend-build (push) Successful in 23s
CI / backend-lint-and-test (push) Successful in 32s
Run 4960 was the control — the chain compared against itself, which must
come back clean before a clean diff after the squash means anything. It
did its job and failed on both counts.

1. The harness is sound. Both dumps came back 1123 normalised lines and
   differed on EXACTLY two, the \restrict / \unrestrict pair that newer
   pg_dump emits to fence a dump against injection during restore. It is
   a fresh random nonce per invocation, so it differs by construction and
   is noise by definition. Now filtered — and the control is what
   licenses that filter: it was OBSERVED to be the only false positive
   rather than assumed to be one, which matters for a check whose whole
   value is that its normalisation does not hide a real difference.

2. The candidate-baseline step never ran. `if: github.event.inputs
   .generate == 'true'` on a `type: boolean` input silently evaluated
   false — no diagnostic, step skipped, job carried on. The same
   `github.event.inputs` typing quirk build.yml already works around for
   force_build.

   Rather than fight the input typing, the gate is now the tree itself:
   skip if alembic/versions holds one file. That is the real question
   anyway — there is nothing to generate once the chain is collapsed —
   and it cannot be silently wrong the way an unevaluated expression can.

Worth noting what the control also proved incidentally: the two schemas
were byte-identical across 1123 lines despite being built by separate
alembic runs into separate databases, so pg_dump's object ordering is
stable enough to diff directly and no sort normalisation is needed.
2026-08-30 13:35:26 -04:00
bvandeusen 62583791d8 ci: a workflow that proves a collapsed alembic chain matches the old one
Build images / sign-extension (push) Successful in 4s
CI / lint (push) Successful in 4s
CI / extension-version (push) Successful in 3s
Build images / build-agent (push) Successful in 9s
CI / frontend-build (push) Successful in 25s
Build images / build-ml (push) Successful in 7s
Build images / build-web (push) Successful in 8s
CI / backend-lint-and-test (push) Successful in 43s
CI / integration (push) Successful in 3m59s
Milestone 328 step 1 needs a baseline generated from the models, and
step 2 must not stamp the operator's live database until that baseline
is proven to reproduce what the 87-revision chain produced. `alembic
stamp` validates nothing, so an unproven baseline fails silently now and
loudly later, on real data.

There is no local Python environment and rules 10/12 point away from
standing one up, so the comparison runs in CI, where a pgvector Postgres
is already built from the chain on every integration run and nothing is
at risk.

It builds two databases and diffs their pg_dump --schema-only output:
one from `alembic upgrade head` on the revisions read out of git at
`chain_ref`, one from the current tree. Reading the chain from git via a
worktree — rather than from the working tree — is what keeps this usable
AFTER the old revisions are deleted, so it is the proof for step 1 and
the pre-flight for step 2 rather than a one-shot script.

Both sides use `alembic upgrade head`, never metadata.create_all, per
rule 82 — and that rule's reasoning is exactly the hazard here.
`create_all` emits plain CREATE TABLE and skips everything else, which is
why the optional autogenerated candidate CANNOT be trusted as the answer.
Three things in this schema are invisible to SQLAlchemy metadata:

  CREATE EXTENSION vector           (0001)
  CREATE EXTENSION tsm_system_rows  (0004)
  the HNSW index on image_record.siglip_embedding, raw SQL because
    alembic's create_index cannot express USING hnsw (...)   (0036)

plus any CHECK constraint or server_default a migration added without the
model declaring it — 4 model files declare CheckConstraints against 6
migrations that touch them. The candidate is a starting point to hand
finish; the diff is what proves nothing was missed.

Results are printed to the job log rather than uploaded: ci-requirements
records that this runner cannot do actions/upload-artifact@v4+, and the
repo dropped the action entirely in 2026-05.

Run it first with the chain still present, as a control — the diff
compares the chain against itself and must come back clean. A clean diff
after the squash only means something if the harness was shown to be
capable of producing one beforehand.

Temporary. Delete once the baseline is stamped.
2026-08-30 13:31:16 -04:00
bvandeusen 0a5bbe81dc docs: the scheduled refresh does NOT republish nothing (#3265)
Build images / sign-extension (push) Successful in 4s
CI / lint (push) Successful in 4s
CI / extension-version (push) Successful in 5s
Build images / build-agent (push) Successful in 8s
CI / integration (push) Successful in 3m58s
Build images / build-ml (push) Successful in 7s
Build images / build-web (push) Successful in 8s
CI / frontend-build (push) Successful in 20s
extension / lint (push) Successful in 27s
CI / backend-lint-and-test (push) Successful in 52s
Step 4 asserted that when the base has not moved the refresh is "a ~13s
no-op that republishes nothing", and that this no-op was the point. The
first half is false and was written without being tested.

Run 4934, the first real fire: every content step reported CACHED and
both bases resolved to unchanged pinned digests, yet all three :latest
tags took a new manifest digest anyway.

  fabledcurator        4ea5265ba017 -> 380e504de0fa
  fabledcurator-ml     6e7cfc0c09fd -> 6b2eefc301d8
  fabledcurator-agent  44920e0af1f3 -> 54accbeb52ed

buildkit mints a fresh image config per run, so identical layers get
republished under a new config blob. Storage cost is trivial; the cost
that matters is that a :latest digest change stops meaning "something is
different", and :c-<sha> is handed a new manifest to diverge from every
Sunday for no reason.

Corrects the workflow comment (x3) and ci-requirements.md to say what
actually happens. Filed as #3265 with the candidate fixes; the likely one
is a deterministic SOURCE_DATE_EPOCH off the value artifacts.sh already
derives, which would make "same source, same version" into "same source,
same bytes".

The rest of step 4 verified clean on the same run: the guard passed
(HEAD is main (499720d), `git checkout -B main`) — so this runner DOES
evaluate the env context inside `with:` — the tag list was :latest alone
with no :c-<sha>, and the repoint step correctly found nothing to write.
2026-08-30 12:57:06 -04:00
bvandeusen 6663e06aa6 ci: assert the scheduled refresh actually checked out main
CI / extension-version (push) Successful in 5s
CI / lint (push) Successful in 6s
Build images / build-ml (push) Successful in 9s
Build images / build-web (push) Successful in 6s
CI / frontend-build (push) Successful in 18s
extension / lint (push) Successful in 19s
Build images / sign-extension (push) Successful in 6s
Build images / build-agent (push) Successful in 11s
CI / backend-lint-and-test (push) Successful in 39s
CI / integration (push) Successful in 3m48s
BUILD_REF is read through the `env` context inside `with:`, which this
runner is not known to evaluate. `${{ steps.* }}` and `${{ secrets.* }}`
in `with:`/`env:` are proven here; `env` is not, and run 4915's checkout
log (`git checkout -B dev refs/remotes/origin/dev`) cannot tell an
honoured `refs/heads/dev` from an empty value falling back to the same
place — the two are indistinguishable on every path except the one that
matters.

If it does resolve empty, the weekly refresh checks out dev and pushes
its source to :latest, which is production. Every lane stays green and
the first symptom is production running code that was never merged.

So each of the four jobs now asserts its own checkout before doing
anything, gated on `github.event_name` — the `github` context is
demonstrably evaluated in `if:`, so the guard cannot be disabled by the
same uncertainty it covers. A red weekly job is an acceptable outcome;
shipping dev to production is not.
2026-08-29 22:57:41 -04:00
bvandeusen 63e0a423d7 ci: a weekly base-image refresh on the channel tags (milestone 326 step 4)
Build images / sign-extension (push) Successful in 4s
CI / extension-version (push) Successful in 3s
CI / lint (push) Successful in 3s
Build images / build-web (push) Successful in 6s
extension / lint (push) Successful in 20s
Build images / build-agent (push) Successful in 7s
Build images / build-ml (push) Successful in 8s
CI / frontend-build (push) Successful in 21s
CI / backend-lint-and-test (push) Successful in 31s
CI / integration (push) Successful in 3m50s
Skip-if-exists is keyed on our own source, so an artifact whose source
stops moving stops picking up base-image updates. `agent/` last changed
2026-07-17; every push since has correctly declined to rebuild it, which
also means it will serve that day's nvidia/cuda layers indefinitely.

A `schedule:` trigger, Sunday 06:00 UTC, away from CI-runner's Monday
security sweep so the two are never diagnosing each other.

#3154's blocking open question is dissolved rather than answered. It was
written when the identity was a `r-<revision>` TAG, and asked how the
next ordinary push could avoid repointing :latest back off the refresh.
Milestone 318 replaced that tag with a LABEL, and #3183 made the repoint
step exclude its source tag so the label stays readable. Excluding the
source is what also keeps a refresh from being undone: on the next main
push the reuse check hits, :latest is not rewritten, and the new :c-<sha>
is written FROM the refreshed :latest. To be verified by digest, not by
this argument.

Four decisions, each commented where it lives:

* It builds `main`, not the branch that triggered it. Forgejo registers a
  cron from the default branch — `dev` here — so a scheduled run arrives
  with github.ref on dev, and a refresh of :dev would be refreshing the
  one channel that is rebuilt constantly anyway. The ref is decided once
  in a top-level `env: BUILD_REF` that all four checkouts take. Deriving
  it per job would let the halves disagree: sign-extension would derive
  dev's extension version while build-web bundled main's, and the release
  download would 404 on a version that exists perfectly well.

* It publishes only the channel tag. :c-<sha> for main's HEAD already
  names the bytes that commit built; re-pushing it over refreshed layers
  would break the one tag rule 145 makes immutable, and it is the
  rollback unit — so the breakage would surface on the day somebody
  needed it. The repoint step needs no schedule case: the tag list is the
  channel tag alone, SOURCE is the only entry, it is excluded as always,
  and the step correctly does nothing.

* It bypasses reuse by construction, since it rebuilds the same source
  and fc.revision always matches. Checked in the reuse step beside
  force_build, so one decision still drives both the build and the
  repoint.

* `pull: true`, on the scheduled path only, is the actual mechanism. A
  moved base tag changes the FROM layer's cache key and everything above
  it rebuilds; an unmoved one is satisfied by the registry cache and the
  refresh is a ~13s no-op that republishes nothing. That no-op is the
  point — :latest should change when there is something new in it, not
  every Sunday. The known lag, left deliberately: an apt package update
  while the base tag stands still is not caught, and closing it needs
  no-cache: true, which buys weekly churn for it.
2026-08-29 22:53:48 -04:00
34 changed files with 1334 additions and 119 deletions
+370
View File
@@ -0,0 +1,370 @@
# TEMPORARY — milestone 328 steps 1-2. Delete once the baseline is stamped.
#
# Squashing 87 alembic revisions into one baseline has exactly one dangerous
# failure: the generated baseline does not reproduce the schema the chain
# produced, `alembic stamp` writes a version string anyway (it validates
# NOTHING), and the divergence surfaces on the next real migration against the
# operator's live data.
#
# So this workflow does the comparison in CI, where a pgvector Postgres already
# gets built from the chain on every integration run, and nothing is at risk.
# It answers one question: does `upgrade head` on the collapsed chain produce a
# byte-identical schema to `upgrade head` on the 87-revision chain?
#
# The chain is read from git rather than from the working tree, so this keeps
# working AFTER the old revisions are deleted — `chain_ref` names a commit that
# still has them. That is what makes this the proof for step 1 and the
# pre-flight for step 2, rather than a one-shot script.
#
# While the chain is still present it also autogenerates a candidate baseline
# from the models and prints it. That is a starting point, NOT the answer:
# autogenerate reads SQLAlchemy metadata, and three things here do not live
# there —
# * CREATE EXTENSION vector (0001)
# * CREATE EXTENSION tsm_system_rows (0004)
# * the HNSW index on image_record.siglip_embedding, which is raw SQL
# because alembic's create_index cannot express `USING hnsw (...)` (0036)
# plus any CHECK constraint or server_default that a migration added without
# the model declaring it. Those must be hand-added, and the diff below is what
# proves none were missed.
name: Alembic baseline
on:
workflow_dispatch:
inputs:
chain_ref:
description: 'Commit/tag carrying the full chain; blank = this ref (use a pinned commit only AFTER the collapse)'
type: string
default: ''
mode:
description: 'chain = compare against this tree''s migrations; models = compare against a schema built from the MODELS'
type: string
default: 'chain'
jobs:
compare:
runs-on: python-ci
container:
image: git.fabledsword.com/bvandeusen/ci-python:3.14
env:
DB_USER: fabledcurator
DB_PASSWORD: ci_integration
DB_PORT: "5432"
DB_NAME: fabledcurator_test
SECRET_KEY: ci_integration_placeholder
services:
postgres:
image: pgvector/pgvector:pg16
env:
POSTGRES_USER: fabledcurator
POSTGRES_PASSWORD: ci_integration
POSTGRES_DB: fabledcurator_test
options: >-
--health-cmd "pg_isready -U fabledcurator"
--health-interval 10s
--health-timeout 5s
--health-retries 10
steps:
- uses: actions/checkout@v4
with:
# Full history is the point: `chain_ref` is read out of git, so a
# shallow clone would not have the revisions to compare against.
fetch-depth: 0
- name: Resolve the Postgres service and install deps
run: |
set -eux
# Same service-IP dance as ci.yml's integration job; see the long
# comment there for why the job name must stay separator-free.
PG=$(docker ps --filter "name=compare" --filter "ancestor=pgvector/pgvector:pg16" -q | head -n1)
test -n "$PG"
PG_IP=$(docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' "$PG")
test -n "$PG_IP"
echo "PG_CONTAINER=$PG" >> "$GITHUB_ENV"
echo "DB_HOST=$PG_IP" >> "$GITHUB_ENV"
for i in $(seq 1 60); do
(echo > "/dev/tcp/$PG_IP/5432") >/dev/null 2>&1 && break
sleep 2
done
if command -v uv >/dev/null 2>&1; then
uv pip install --system -r requirements.txt
else
pip install -r requirements.txt
fi
# DB 1: the 87-revision chain, read out of git at `chain_ref`.
#
# A git worktree rather than a checkout, so the current tree — which is
# what we are testing — is left completely alone.
- name: Build the schema the OLD chain produces
env:
CHAIN_REF: ${{ github.event.inputs.chain_ref }}
THIS_SHA: ${{ github.sha }}
run: |
set -eux
docker exec "$PG_CONTAINER" createdb -U fabledcurator fc_chain
# Blank means "the chain in this ref", which is what you want while
# the chain is still intact — comparing the models against a PINNED
# older commit reports every migration written since as a difference.
# Pin it only after the collapse, when the tree no longer has them.
git worktree add /tmp/chain "${CHAIN_REF:-$THIS_SHA}"
ls /tmp/chain/alembic/versions/*.py | wc -l
cd /tmp/chain
DB_NAME=fc_chain alembic upgrade head
cd -
docker exec "$PG_CONTAINER" pg_dump -U fabledcurator --schema-only \
--no-owner --no-privileges -d fc_chain > chain.sql
wc -l chain.sql
# Emit the dump itself, checksummed, for local analysis. Reconciling
# the models against the deployed schema (#3275) needs the ACTUAL
# schema, not an inference from a diff — parsing table context out of
# unified-diff hunks drops every table whose CREATE TABLE line falls
# outside a hunk, which silently under-reports.
#
# base64 + sha256 for the same reason as the candidate: a plain cat
# of a file this size was truncated mid-line by the runner with the
# step still green (run 4964).
set +x
B64=$(base64 -w 120 chain.sql)
echo "===== BEGIN CHAIN SCHEMA (base64) ====="
echo "$B64"
echo "===== END CHAIN SCHEMA ====="
echo "chain-sha256: $(sha256sum chain.sql | cut -d' ' -f1)"
echo "chain-bytes: $(wc -c < chain.sql)"
set -x
# A candidate baseline, autogenerated from the models against an EMPTY
# database so every table shows up as a create. Printed for a human to
# finish — it will be missing the three raw-SQL items named at the top.
#
# Gated on the TREE, not on a workflow input. A `type: boolean` input
# read back as `github.event.inputs.generate == 'true'` silently
# evaluated false on this runner (run 4960 skipped this step entirely
# with no diagnostic) — the same `github.event.inputs` typing quirk
# build.yml already works around. The file count is the real question
# anyway: there is nothing to generate once the chain is collapsed.
- name: Autogenerate a candidate baseline
run: |
set -eux
if [ "$(ls alembic/versions/*.py | wc -l)" -le 1 ]; then
echo "already collapsed — nothing to generate"
exit 0
fi
docker exec "$PG_CONTAINER" createdb -U fabledcurator fc_gen
# Hide the existing revisions so alembic sees an empty history and
# emits the whole schema rather than a delta.
mkdir -p /tmp/versions_held
mv alembic/versions/*.py /tmp/versions_held/ 2>/dev/null || true
DB_NAME=fc_gen alembic revision --autogenerate -m "baseline" || true
# Printed rather than uploaded: ci-requirements.md records that this
# runner cannot do actions/upload-artifact@v4+, and the repo dropped
# the action entirely in 2026-05, so the job log is the retrieval
# channel actually proven here.
#
# base64, not the raw file. A plain `cat` of the ~33KB candidate was
# TRUNCATED MID-LINE by the runner on run 4964 — it stopped inside
# `sa.Column('mime', sa.String(length=128)` and carried straight on
# to the next traced command, with the step still green. A silent
# cut in the middle of a schema definition is the worst possible
# failure here, because the truncated text still looks like a
# plausible file.
#
# base64 at a fixed narrow width gives many short lines instead of
# few long ones, and — the actual point — a checksum and a line
# count that make truncation DETECTABLE rather than invisible.
set +x
F=$(ls alembic/versions/*.py | head -1)
B64=$(base64 -w 120 "$F")
echo "===== BEGIN CANDIDATE BASELINE (base64) ====="
echo "$B64"
echo "===== END CANDIDATE BASELINE ====="
echo "candidate-sha256: $(sha256sum "$F" | cut -d' ' -f1)"
echo "candidate-bytes: $(wc -c < "$F")"
echo "candidate-b64-lines: $(echo "$B64" | wc -l)"
set -x
mkdir -p /tmp/candidate
cp alembic/versions/*.py /tmp/candidate/
# Put the tree back exactly as it was; this job never mutates state.
rm -f alembic/versions/*.py
mv /tmp/versions_held/*.py alembic/versions/ 2>/dev/null || true
# DB 2: what the CURRENT tree produces.
#
# `mode: models` applies the candidate autogenerated from the MODELS
# instead, which is what answers "do the models describe the schema?" —
# the question #3275 exists because nobody had ever asked it. Under that
# mode a clean diff means autogenerate is trustworthy again.
#
# The two extensions are created by hand first. They are database
# objects, not table metadata, so no model can carry them and their
# absence is not a model defect — it is simply outside what this
# comparison is asking about.
- name: Build the schema the CURRENT tree produces
env:
MODE: ${{ github.event.inputs.mode }}
run: |
set -eux
docker exec "$PG_CONTAINER" createdb -U fabledcurator fc_base
if [ "${MODE:-chain}" = "models" ]; then
docker exec "$PG_CONTAINER" psql -U fabledcurator -d fc_base \
-c "CREATE EXTENSION IF NOT EXISTS vector" \
-c "CREATE EXTENSION IF NOT EXISTS tsm_system_rows"
mkdir -p /tmp/held
mv alembic/versions/*.py /tmp/held/
cp /tmp/candidate/*.py alembic/versions/
# Autogenerate EMITS pgvector.sqlalchemy.vector.VECTOR(...) without
# importing it, so the file it writes cannot run:
# NameError: name 'pgvector' is not defined
# Observed on run 4988, which is the proof rather than the theory.
# This is a defect in the GENERATOR, not in the models, so it is
# repaired here rather than counted as a schema difference — the
# comparison is about whether the models describe the schema.
sed -i '0,/^import sqlalchemy as sa$/s//import sqlalchemy as sa\nimport pgvector.sqlalchemy.vector/' alembic/versions/*.py
grep -n 'import pgvector' alembic/versions/*.py
# Second generator defect, same class as the missing import.
#
# base.py's naming convention includes %(constraint_name)s for ck,
# which — unlike uq/fk/ix — means the convention is applied even to
# a CheckConstraint that HAS a name. So a model declaring
# name="singleton" correctly becomes ck_ml_settings_singleton in
# the metadata. Autogenerate then writes that RENDERED name into
# the migration, and running the migration applies the convention a
# SECOND time: ck_ml_settings_ck_ml_settings_singleton.
#
# That is round-tripping damage done by the generator, not a claim
# the models make, so it is repaired here rather than counted as a
# schema difference. Undone by removing the ck_<table>_ prefix the
# convention will re-add — the exact inverse, and it only fires on
# a name that actually carries its own table's prefix.
python3 - alembic/versions/*.py <<'PYEOF'
import re, sys
table = None
for path in sys.argv[1:]:
out = []
for line in open(path):
m = re.search(r"op\.create_table\(\s*[\"']([A-Za-z0-9_]+)[\"']", line)
if m:
table = m.group(1)
if table and "CheckConstraint" in line:
prefix = f"ck_{table}_"
line = re.sub(
r"(name=[\"'])" + re.escape(prefix),
r"\1",
line,
)
out.append(line)
open(path, "w").writelines(out)
PYEOF
grep -n 'CheckConstraint' alembic/versions/*.py || true
ls alembic/versions/*.py
DB_NAME=fc_base alembic upgrade head
rm -f alembic/versions/*.py
mv /tmp/held/*.py alembic/versions/
else
ls alembic/versions/*.py | wc -l
DB_NAME=fc_base alembic upgrade head
fi
docker exec "$PG_CONTAINER" pg_dump -U fabledcurator --schema-only \
--no-owner --no-privileges -d fc_base > baseline.sql
wc -l baseline.sql
# The verdict.
#
# pg_dump orders dumpable objects by name within type, not by creation
# order, so two schemas built by different routes are directly
# comparable. Normalisation is deliberately minimal, because a filter
# that hides a real difference is the one way this check passes when it
# should fail — blank lines, SQL comments, trailing whitespace, and:
#
# \restrict / \unrestrict — a per-invocation RANDOM NONCE that newer
# pg_dump emits to fence the dump against injection during restore. It
# differs on every run by construction, so it is noise by definition,
# not a schema difference. Measured on run 4960, the control: two dumps
# of the SAME schema came back 1123 lines each and differed on exactly
# these two lines and nothing else. That control is what licenses this
# filter — it was observed to be the only false positive, rather than
# assumed to be one.
# Column ORDER inside a CREATE TABLE is compared separately from column
# CONTENT, and only content is fatal.
#
# A table built by 87 migrations has its columns in ADD COLUMN order; the
# same table built in one shot has them in declaration order. That is a
# real and permanent difference which no baseline can erase — the
# operator's existing database keeps chain order forever, a fresh install
# gets model order — so a check that fails on it would never pass and
# would teach nothing. FC reaches every column through the ORM by name,
# and `SELECT *` ordering is not depended on anywhere.
#
# So the second pass SORTS the column lines within each CREATE TABLE
# rather than DELETING them. That distinction is the whole point: sorting
# cannot hide a column that exists on one side only, or one whose type,
# nullability or default differs — those still land in the diff. A filter
# could have hidden all three.
#
# Both diffs are reported. The ordered one is informational; the
# order-insensitive one is the verdict.
- name: Diff
run: |
set -eu
norm() {
grep -vE '^\s*(--|$)' "$1" \
| grep -vE '^\\(un)?restrict ' \
| sed 's/[[:space:]]*$//'
}
norm chain.sql > a.txt
norm baseline.sql > b.txt
echo "normalised: chain=$(wc -l < a.txt) lines, current=$(wc -l < b.txt) lines"
sort_table_columns() {
python3 - "$1" <<'PYEOF'
import re, sys
lines = open(sys.argv[1]).read().splitlines()
out, block = [], None
for line in lines:
if block is not None:
# ');' on its own closes the CREATE TABLE body.
if line.strip() == ");":
out.extend(sorted(block))
out.append(line)
block = None
else:
# Drop the list comma before sorting. Only the LAST
# column lacks one, so keeping it would make every
# reordering look like a content change as well — the
# comma is punctuation, and carries no schema meaning.
block.append(line.rstrip().rstrip(","))
continue
out.append(line)
if re.match(r"CREATE TABLE .*\($", line):
block = []
if block is not None: # unterminated body: emit it rather than drop it
out.extend(block)
print("\n".join(out))
PYEOF
}
sort_table_columns a.txt > a.sorted.txt
sort_table_columns b.txt > b.sorted.txt
test "$(wc -l < a.sorted.txt)" = "$(wc -l < a.txt)"
test "$(wc -l < b.sorted.txt)" = "$(wc -l < b.txt)"
if diff -u a.txt b.txt > schema.diff; then
echo "ORDERED DIFF: identical, column order included."
else
echo "ORDERED DIFF: $(grep -cE '^[+-]' schema.diff) changed lines (informational):"
cat schema.diff
fi
echo
echo "================================================================"
echo
if diff -u a.sorted.txt b.sorted.txt > sorted.diff; then
echo "SCHEMAS MATCH — every difference above is column ORDER alone."
else
echo "SCHEMAS DIFFER — $(grep -cE '^[+-]' sorted.diff) changed lines that are NOT ordering:"
cat sorted.diff
echo
echo "The baseline is wrong, not the database. Do not stamp."
exit 1
fi
+271 -6
View File
@@ -45,6 +45,36 @@ on:
type: boolean type: boolean
default: false default: false
# The base-image refresh (milestone 326 step 4, #3154).
#
# Skip-if-exists is keyed on OUR source, so an artifact whose source stops
# moving stops picking up base-image updates. `agent/` last changed
# 2026-07-17; every push since has correctly declined to rebuild it, which
# also means it will serve that day's `nvidia/cuda` layers forever. Nothing
# is wrong until it has been unchanged for months, which is precisely why
# this is a calendar trigger and not a condition on the push path.
#
# Weekly, Sunday 06:00 UTC. Away from CI-runner's Monday security sweep so
# the two are never diagnosing each other, and on the quietest day so a
# surprise rebuild is not competing with a push.
schedule:
- cron: '0 6 * * 0'
# Which branch a run BUILDS, as opposed to which one triggered it.
#
# They are the same thing on every trigger but `schedule`. Forgejo registers a
# cron from the DEFAULT branch — `dev` here — so a scheduled run arrives with
# `github.ref` pointing at dev, and a refresh that rebuilt `:dev` would be
# refreshing the one channel that gets rebuilt constantly anyway. Production is
# `main` (rule 147), and `:latest` is the tag that goes stale.
#
# So the ref is decided once, here, and every checkout in the file takes it.
# Deriving it per job invites the two halves to disagree: sign-extension would
# derive dev's extension version while build-web bundled main's, and the
# release download would 404 on a version that exists perfectly well.
env:
BUILD_REF: ${{ github.event_name == 'schedule' && 'main' || github.ref }}
# Requires repo secret RELEASE_TOKEN — a Forgejo PAT with scopes: # Requires repo secret RELEASE_TOKEN — a Forgejo PAT with scopes:
# - write:package, read:package (for docker push to git.fabledsword.com) # - write:package, read:package (for docker push to git.fabledsword.com)
# - write:release (for ext-<version> release asset cache) # - write:release (for ext-<version> release asset cache)
@@ -88,12 +118,43 @@ jobs:
steps: steps:
- uses: actions/checkout@v4 - uses: actions/checkout@v4
with: with:
# Not the triggering ref — see the `env:` block at the top. On a
# scheduled refresh this is `main`; on everything else it is the ref
# that fired, so this is a no-op on every ordinary path.
ref: ${{ env.BUILD_REF }}
# Full history is load-bearing, not a convenience: the version this # Full history is load-bearing, not a convenience: the version this
# job signs is derived from the commit TIME of the newest packaged # job signs is derived from the commit TIME of the newest packaged
# extension change. A depth-1 clone sees one commit and derives a # extension change. A depth-1 clone sees one commit and derives a
# wrong, too-low value rather than failing (ci-requirements.md). # wrong, too-low value rather than failing (ci-requirements.md).
fetch-depth: 0 fetch-depth: 0
# BUILD_REF is what makes a scheduled run build `main` rather than the
# branch its cron fired from — and it is read through the `env` context
# inside `with:`, which this runner is NOT known to evaluate. If it does
# not, checkout silently falls back to the triggering ref and the weekly
# refresh publishes DEV's source to `:latest`, which is production.
# Every lane would stay green; the first sign of it would be production
# running code that was never merged.
#
# So assert the checkout instead of trusting the expression. A red
# weekly job is a fine outcome. Shipping dev to production is not.
#
# `if:` reads the `github` context, which the runner demonstrably does
# evaluate — this file already gates steps on it — so the guard cannot
# be disabled by the same uncertainty it exists to cover.
- name: Guard — a scheduled run must have checked out main
if: github.event_name == 'schedule'
run: |
set -eu
BRANCH=$(git rev-parse --abbrev-ref HEAD)
echo "schedule: HEAD is $BRANCH ($(git rev-parse --short HEAD))"
if [ "$BRANCH" != "main" ]; then
echo "schedule: expected main, got '$BRANCH'." >&2
echo "schedule: BUILD_REF was not honoured by the runner." >&2
echo "schedule: refusing to publish a channel tag from it." >&2
exit 1
fi
# The version is DERIVED, not read from the repo (milestone 271 step 4, # The version is DERIVED, not read from the repo (milestone 271 step 4,
# cut over 2026-08-27). `packaging.sh version` returns `YYYY.M.D.HHMM` # cut over 2026-08-27). `packaging.sh version` returns `YYYY.M.D.HHMM`
# UTC — the commit TIME of the newest change to a PACKAGED extension # UTC — the commit TIME of the newest change to a PACKAGED extension
@@ -364,12 +425,30 @@ jobs:
steps: steps:
- uses: actions/checkout@v4 - uses: actions/checkout@v4
with: with:
# Not the triggering ref — see the `env:` block at the top. On a
# scheduled refresh this is `main`; on everything else it is the ref
# that fired, so this is a no-op on every ordinary path.
ref: ${{ env.BUILD_REF }}
# Full history: this job RE-DERIVES the extension version rather than # Full history: this job RE-DERIVES the extension version rather than
# being handed it, and a depth-1 clone derives a wrong, too-low value # being handed it, and a depth-1 clone derives a wrong, too-low value
# rather than failing — which would 404 the download of a release # rather than failing — which would 404 the download of a release
# that exists perfectly well under its real name. # that exists perfectly well under its real name.
fetch-depth: 0 fetch-depth: 0
# See sign-extension's copy for why this guard exists.
- name: Guard — a scheduled run must have checked out main
if: github.event_name == 'schedule'
run: |
set -eu
BRANCH=$(git rev-parse --abbrev-ref HEAD)
echo "schedule: HEAD is $BRANCH ($(git rev-parse --short HEAD))"
if [ "$BRANCH" != "main" ]; then
echo "schedule: expected main, got '$BRANCH'." >&2
echo "schedule: BUILD_REF was not honoured by the runner." >&2
echo "schedule: refusing to publish a channel tag from it." >&2
exit 1
fi
# --- derived values, one line (milestone 313) ------------------------ # --- derived values, one line (milestone 313) ------------------------
# These stopped being shadow output at step 3. `revision` decides # These stopped being shadow output at step 3. `revision` decides
# whether the build below runs at all and `version` is what the image # whether the build below runs at all and `version` is what the image
@@ -429,8 +508,30 @@ jobs:
# everywhere). Operator-flagged 2026-06-01 after the first :c-<sha> # everywhere). Operator-flagged 2026-06-01 after the first :c-<sha>
# main-push build failed at this step. # main-push build failed at this step.
SHORT_SHA=$(printf '%s' "$GITHUB_SHA" | cut -c1-7) SHORT_SHA=$(printf '%s' "$GITHUB_SHA" | cut -c1-7)
# Mirrors build-web's tag list; see the comment there.
if [ "${GITHUB_REF##*/}" = "main" ]; then # A scheduled refresh publishes the CHANNEL and nothing else
# (#3154). :c-<sha> for main's HEAD already exists and names the
# bytes that commit actually built; re-pushing it over refreshed
# base layers would break the one tag rule 145 makes immutable —
# and it is the rollback unit, so the breakage would surface on the
# day somebody needed it.
#
# The accepted consequence: between a refresh and the next main
# push, :latest and :c-<sha> point at different manifests. That is
# the design, not drift. They RE-CONVERGE on that push — it hits
# reuse (a refresh does not move fc.revision, because it does not
# touch the source), and the repoint step then writes the new
# :c-<sha> from the refreshed :latest. So the rollback unit ends up
# naming the bytes production is actually running, which is the
# property that matters.
#
# Checked BEFORE the ref test, not after: a scheduled run's
# GITHUB_REF is the default branch (dev), so the main test would
# never fire on it.
if [ "${GITHUB_EVENT_NAME:-}" = "schedule" ]; then
echo "tags=git.fabledsword.com/bvandeusen/fabledcurator:latest" >> "$GITHUB_OUTPUT"
echo "channel=main" >> "$GITHUB_OUTPUT"
elif [ "${GITHUB_REF##*/}" = "main" ]; then
echo "tags=git.fabledsword.com/bvandeusen/fabledcurator:latest,git.fabledsword.com/bvandeusen/fabledcurator:c-${SHORT_SHA}" >> "$GITHUB_OUTPUT" echo "tags=git.fabledsword.com/bvandeusen/fabledcurator:latest,git.fabledsword.com/bvandeusen/fabledcurator:c-${SHORT_SHA}" >> "$GITHUB_OUTPUT"
echo "channel=main" >> "$GITHUB_OUTPUT" echo "channel=main" >> "$GITHUB_OUTPUT"
else else
@@ -524,6 +625,10 @@ jobs:
# this runner is known to evaluate. Read through env rather than # this runner is known to evaluate. Read through env rather than
# interpolated into the run block, same rule as release.yml's TAG. # interpolated into the run block, same rule as release.yml's TAG.
FORCE: ${{ github.event.inputs.force_build }} FORCE: ${{ github.event.inputs.force_build }}
# A scheduled refresh has to bypass reuse by construction: it
# rebuilds the SAME source, so fc.revision always matches and the
# check would skip every refresh there has ever been.
EVENT: ${{ github.event_name }}
run: | run: |
set -eu set -eu
DERIVED=$(sh scripts/artifacts.sh revision web) DERIVED=$(sh scripts/artifacts.sh revision web)
@@ -570,6 +675,9 @@ jobs:
if [ "${FORCE:-false}" = "true" ]; then if [ "${FORCE:-false}" = "true" ]; then
echo "hit=false" >> "$GITHUB_OUTPUT" echo "hit=false" >> "$GITHUB_OUTPUT"
echo "reuse: force_build set — building regardless" echo "reuse: force_build set — building regardless"
elif [ "${EVENT:-}" = "schedule" ]; then
echo "hit=false" >> "$GITHUB_OUTPUT"
echo "reuse: scheduled base refresh — building regardless"
elif [ -n "$PUBLISHED" ] && [ "$PUBLISHED" = "$DERIVED" ]; then elif [ -n "$PUBLISHED" ] && [ "$PUBLISHED" = "$DERIVED" ]; then
echo "hit=true" >> "$GITHUB_OUTPUT" echo "hit=true" >> "$GITHUB_OUTPUT"
echo "reuse: already published — skipping the build" echo "reuse: already published — skipping the build"
@@ -661,6 +769,37 @@ jobs:
context: . context: .
file: Dockerfile file: Dockerfile
push: true push: true
# Re-resolve the FROM references against the registry instead of
# trusting whatever digest the cache was built against. This is the
# whole mechanism of the scheduled refresh (#3154): if the base tag
# moved, the FROM layer's cache key changes, every layer above it
# invalidates, and the image genuinely rebuilds.
#
# MEASURED on the first real fire, run 4934 (#3265): when the base
# did NOT move, the build is ~13s and every content step reports
# CACHED — but the channel tag STILL gets a new manifest digest.
# buildkit mints a fresh image config each run, so identical layers
# are republished under a new config blob. All three images moved
# that way on 2026-08-30 with nothing whatsoever changed in them.
#
# So a refresh currently rewrites :latest every Sunday whether or
# not there is anything new in it, and :c-<sha> is handed a new
# manifest to diverge from on the same cadence. Layers are shared,
# so the storage cost is a config blob; the cost that matters is
# that a digest change no longer MEANS anything. Tracked in #3265 —
# the likely fix is a deterministic SOURCE_DATE_EPOCH, which would
# make "same source, same bytes" true and turn the no-op case into
# a genuine no-op.
#
# What `pull` does NOT catch either: a Debian package update inside
# the `apt-get install` layer while the base tag itself stands
# still. The official python/cuda images rebuild with those updates
# baked in, so this is a lag rather than a hole; closing it needs
# `no-cache: true`, which is a much larger version of the same
# churn #3265 is about.
#
# Only on the schedule. An ordinary push wants the cached base.
pull: ${{ github.event_name == 'schedule' }}
# ONE tag, the channel's. Every other tag is written by the step # ONE tag, the channel's. Every other tag is written by the step
# below, registry-side. buildx here pushes the first tag to the # below, registry-side. buildx here pushes the first tag to the
# registry and then re-pushes the rest through the DOCKER driver, # registry and then re-pushes the rest through the DOCKER driver,
@@ -783,6 +922,12 @@ jobs:
ARGS="$ARGS -t $t" ARGS="$ARGS -t $t"
done done
unset IFS unset IFS
#
# This is also the whole of the scheduled refresh's tag handling
# (#3154): a refresh's tag list is the channel tag alone, so SOURCE
# is the only entry, it gets excluded, and this step correctly does
# nothing. No `if:` on the step and no schedule special-case —
# excluding the source was already the right rule.
if [ -z "$ARGS" ]; then if [ -z "$ARGS" ]; then
echo "repoint: $SOURCE is the only tag for this channel and" echo "repoint: $SOURCE is the only tag for this channel and"
echo "repoint: already holds this revision — nothing to write." echo "repoint: already holds this revision — nothing to write."
@@ -799,6 +944,10 @@ jobs:
steps: steps:
- uses: actions/checkout@v4 - uses: actions/checkout@v4
with: with:
# Not the triggering ref — see the `env:` block at the top. On a
# scheduled refresh this is `main`; on everything else it is the ref
# that fired, so this is a no-op on every ordinary path.
ref: ${{ env.BUILD_REF }}
# Full history: this job derives its artifact's version from the # Full history: this job derives its artifact's version from the
# commit its shipped files last changed in (milestone 313). A # commit its shipped files last changed in (milestone 313). A
# depth-1 clone cannot see that commit — it either derives a wrong, # depth-1 clone cannot see that commit — it either derives a wrong,
@@ -806,6 +955,20 @@ jobs:
# the build would otherwise notice. # the build would otherwise notice.
fetch-depth: 0 fetch-depth: 0
# See sign-extension's copy for why this guard exists.
- name: Guard — a scheduled run must have checked out main
if: github.event_name == 'schedule'
run: |
set -eu
BRANCH=$(git rev-parse --abbrev-ref HEAD)
echo "schedule: HEAD is $BRANCH ($(git rev-parse --short HEAD))"
if [ "$BRANCH" != "main" ]; then
echo "schedule: expected main, got '$BRANCH'." >&2
echo "schedule: BUILD_REF was not honoured by the runner." >&2
echo "schedule: refusing to publish a channel tag from it." >&2
exit 1
fi
# --- derived values, one line (milestone 313) ------------------------ # --- derived values, one line (milestone 313) ------------------------
# These stopped being shadow output at step 3. `revision` decides # These stopped being shadow output at step 3. `revision` decides
# whether the build below runs at all and `version` is what the image # whether the build below runs at all and `version` is what the image
@@ -843,8 +1006,12 @@ jobs:
# everywhere). Operator-flagged 2026-06-01 after first :c-<sha> # everywhere). Operator-flagged 2026-06-01 after first :c-<sha>
# main-push build failed at this step. # main-push build failed at this step.
SHORT_SHA=$(printf '%s' "$GITHUB_SHA" | cut -c1-7) SHORT_SHA=$(printf '%s' "$GITHUB_SHA" | cut -c1-7)
# Mirrors build-web's tag list; see the comment there. # Mirrors build-web's tag list and its schedule handling; see
if [ "${GITHUB_REF##*/}" = "main" ]; then # the comments there.
if [ "${GITHUB_EVENT_NAME:-}" = "schedule" ]; then
echo "tags=git.fabledsword.com/bvandeusen/fabledcurator-ml:latest" >> "$GITHUB_OUTPUT"
echo "channel=main" >> "$GITHUB_OUTPUT"
elif [ "${GITHUB_REF##*/}" = "main" ]; then
echo "tags=git.fabledsword.com/bvandeusen/fabledcurator-ml:latest,git.fabledsword.com/bvandeusen/fabledcurator-ml:c-${SHORT_SHA}" >> "$GITHUB_OUTPUT" echo "tags=git.fabledsword.com/bvandeusen/fabledcurator-ml:latest,git.fabledsword.com/bvandeusen/fabledcurator-ml:c-${SHORT_SHA}" >> "$GITHUB_OUTPUT"
echo "channel=main" >> "$GITHUB_OUTPUT" echo "channel=main" >> "$GITHUB_OUTPUT"
else else
@@ -921,6 +1088,10 @@ jobs:
# this runner is known to evaluate. Read through env rather than # this runner is known to evaluate. Read through env rather than
# interpolated into the run block, same rule as release.yml's TAG. # interpolated into the run block, same rule as release.yml's TAG.
FORCE: ${{ github.event.inputs.force_build }} FORCE: ${{ github.event.inputs.force_build }}
# A scheduled refresh has to bypass reuse by construction: it
# rebuilds the SAME source, so fc.revision always matches and the
# check would skip every refresh there has ever been.
EVENT: ${{ github.event_name }}
run: | run: |
set -eu set -eu
DERIVED=$(sh scripts/artifacts.sh revision ml) DERIVED=$(sh scripts/artifacts.sh revision ml)
@@ -963,6 +1134,9 @@ jobs:
if [ "${FORCE:-false}" = "true" ]; then if [ "${FORCE:-false}" = "true" ]; then
echo "hit=false" >> "$GITHUB_OUTPUT" echo "hit=false" >> "$GITHUB_OUTPUT"
echo "reuse: force_build set — building regardless" echo "reuse: force_build set — building regardless"
elif [ "${EVENT:-}" = "schedule" ]; then
echo "hit=false" >> "$GITHUB_OUTPUT"
echo "reuse: scheduled base refresh — building regardless"
elif [ -n "$PUBLISHED" ] && [ "$PUBLISHED" = "$DERIVED" ]; then elif [ -n "$PUBLISHED" ] && [ "$PUBLISHED" = "$DERIVED" ]; then
echo "hit=true" >> "$GITHUB_OUTPUT" echo "hit=true" >> "$GITHUB_OUTPUT"
echo "reuse: already published — skipping the build" echo "reuse: already published — skipping the build"
@@ -978,6 +1152,37 @@ jobs:
context: . context: .
file: Dockerfile.ml file: Dockerfile.ml
push: true push: true
# Re-resolve the FROM references against the registry instead of
# trusting whatever digest the cache was built against. This is the
# whole mechanism of the scheduled refresh (#3154): if the base tag
# moved, the FROM layer's cache key changes, every layer above it
# invalidates, and the image genuinely rebuilds.
#
# MEASURED on the first real fire, run 4934 (#3265): when the base
# did NOT move, the build is ~13s and every content step reports
# CACHED — but the channel tag STILL gets a new manifest digest.
# buildkit mints a fresh image config each run, so identical layers
# are republished under a new config blob. All three images moved
# that way on 2026-08-30 with nothing whatsoever changed in them.
#
# So a refresh currently rewrites :latest every Sunday whether or
# not there is anything new in it, and :c-<sha> is handed a new
# manifest to diverge from on the same cadence. Layers are shared,
# so the storage cost is a config blob; the cost that matters is
# that a digest change no longer MEANS anything. Tracked in #3265 —
# the likely fix is a deterministic SOURCE_DATE_EPOCH, which would
# make "same source, same bytes" true and turn the no-op case into
# a genuine no-op.
#
# What `pull` does NOT catch either: a Debian package update inside
# the `apt-get install` layer while the base tag itself stands
# still. The official python/cuda images rebuild with those updates
# baked in, so this is a lag rather than a hole; closing it needs
# `no-cache: true`, which is a much larger version of the same
# churn #3265 is about.
#
# Only on the schedule. An ordinary push wants the cached base.
pull: ${{ github.event_name == 'schedule' }}
# ONE tag, the channel's. Every other tag is written by the step # ONE tag, the channel's. Every other tag is written by the step
# below, registry-side. buildx here pushes the first tag to the # below, registry-side. buildx here pushes the first tag to the
# registry and then re-pushes the rest through the DOCKER driver, # registry and then re-pushes the rest through the DOCKER driver,
@@ -1113,6 +1318,10 @@ jobs:
steps: steps:
- uses: actions/checkout@v4 - uses: actions/checkout@v4
with: with:
# Not the triggering ref — see the `env:` block at the top. On a
# scheduled refresh this is `main`; on everything else it is the ref
# that fired, so this is a no-op on every ordinary path.
ref: ${{ env.BUILD_REF }}
# Full history: this job derives its artifact's version from the # Full history: this job derives its artifact's version from the
# commit its shipped files last changed in (milestone 313). A # commit its shipped files last changed in (milestone 313). A
# depth-1 clone cannot see that commit — it either derives a wrong, # depth-1 clone cannot see that commit — it either derives a wrong,
@@ -1120,6 +1329,20 @@ jobs:
# the build would otherwise notice. # the build would otherwise notice.
fetch-depth: 0 fetch-depth: 0
# See sign-extension's copy for why this guard exists.
- name: Guard — a scheduled run must have checked out main
if: github.event_name == 'schedule'
run: |
set -eu
BRANCH=$(git rev-parse --abbrev-ref HEAD)
echo "schedule: HEAD is $BRANCH ($(git rev-parse --short HEAD))"
if [ "$BRANCH" != "main" ]; then
echo "schedule: expected main, got '$BRANCH'." >&2
echo "schedule: BUILD_REF was not honoured by the runner." >&2
echo "schedule: refusing to publish a channel tag from it." >&2
exit 1
fi
# --- derived values, one line (milestone 313) ------------------------ # --- derived values, one line (milestone 313) ------------------------
# These stopped being shadow output at step 3. `revision` decides # These stopped being shadow output at step 3. `revision` decides
# whether the build below runs at all and `version` is what the image # whether the build below runs at all and `version` is what the image
@@ -1152,8 +1375,12 @@ jobs:
id: tag id: tag
run: | run: |
SHORT_SHA=$(printf '%s' "$GITHUB_SHA" | cut -c1-7) SHORT_SHA=$(printf '%s' "$GITHUB_SHA" | cut -c1-7)
# Mirrors build-web's tag list; see the comment there. # Mirrors build-web's tag list and its schedule handling; see
if [ "${GITHUB_REF##*/}" = "main" ]; then # the comments there.
if [ "${GITHUB_EVENT_NAME:-}" = "schedule" ]; then
echo "tags=git.fabledsword.com/bvandeusen/fabledcurator-agent:latest" >> "$GITHUB_OUTPUT"
echo "channel=main" >> "$GITHUB_OUTPUT"
elif [ "${GITHUB_REF##*/}" = "main" ]; then
echo "tags=git.fabledsword.com/bvandeusen/fabledcurator-agent:latest,git.fabledsword.com/bvandeusen/fabledcurator-agent:c-${SHORT_SHA}" >> "$GITHUB_OUTPUT" echo "tags=git.fabledsword.com/bvandeusen/fabledcurator-agent:latest,git.fabledsword.com/bvandeusen/fabledcurator-agent:c-${SHORT_SHA}" >> "$GITHUB_OUTPUT"
echo "channel=main" >> "$GITHUB_OUTPUT" echo "channel=main" >> "$GITHUB_OUTPUT"
else else
@@ -1230,6 +1457,10 @@ jobs:
# this runner is known to evaluate. Read through env rather than # this runner is known to evaluate. Read through env rather than
# interpolated into the run block, same rule as release.yml's TAG. # interpolated into the run block, same rule as release.yml's TAG.
FORCE: ${{ github.event.inputs.force_build }} FORCE: ${{ github.event.inputs.force_build }}
# A scheduled refresh has to bypass reuse by construction: it
# rebuilds the SAME source, so fc.revision always matches and the
# check would skip every refresh there has ever been.
EVENT: ${{ github.event_name }}
run: | run: |
set -eu set -eu
DERIVED=$(sh scripts/artifacts.sh revision agent) DERIVED=$(sh scripts/artifacts.sh revision agent)
@@ -1272,6 +1503,9 @@ jobs:
if [ "${FORCE:-false}" = "true" ]; then if [ "${FORCE:-false}" = "true" ]; then
echo "hit=false" >> "$GITHUB_OUTPUT" echo "hit=false" >> "$GITHUB_OUTPUT"
echo "reuse: force_build set — building regardless" echo "reuse: force_build set — building regardless"
elif [ "${EVENT:-}" = "schedule" ]; then
echo "hit=false" >> "$GITHUB_OUTPUT"
echo "reuse: scheduled base refresh — building regardless"
elif [ -n "$PUBLISHED" ] && [ "$PUBLISHED" = "$DERIVED" ]; then elif [ -n "$PUBLISHED" ] && [ "$PUBLISHED" = "$DERIVED" ]; then
echo "hit=true" >> "$GITHUB_OUTPUT" echo "hit=true" >> "$GITHUB_OUTPUT"
echo "reuse: already published — skipping the build" echo "reuse: already published — skipping the build"
@@ -1287,6 +1521,37 @@ jobs:
context: agent context: agent
file: agent/Dockerfile file: agent/Dockerfile
push: true push: true
# Re-resolve the FROM references against the registry instead of
# trusting whatever digest the cache was built against. This is the
# whole mechanism of the scheduled refresh (#3154): if the base tag
# moved, the FROM layer's cache key changes, every layer above it
# invalidates, and the image genuinely rebuilds.
#
# MEASURED on the first real fire, run 4934 (#3265): when the base
# did NOT move, the build is ~13s and every content step reports
# CACHED — but the channel tag STILL gets a new manifest digest.
# buildkit mints a fresh image config each run, so identical layers
# are republished under a new config blob. All three images moved
# that way on 2026-08-30 with nothing whatsoever changed in them.
#
# So a refresh currently rewrites :latest every Sunday whether or
# not there is anything new in it, and :c-<sha> is handed a new
# manifest to diverge from on the same cadence. Layers are shared,
# so the storage cost is a config blob; the cost that matters is
# that a digest change no longer MEANS anything. Tracked in #3265 —
# the likely fix is a deterministic SOURCE_DATE_EPOCH, which would
# make "same source, same bytes" true and turn the no-op case into
# a genuine no-op.
#
# What `pull` does NOT catch either: a Debian package update inside
# the `apt-get install` layer while the base tag itself stands
# still. The official python/cuda images rebuild with those updates
# baked in, so this is a lag rather than a hole; closing it needs
# `no-cache: true`, which is a much larger version of the same
# churn #3265 is about.
#
# Only on the schedule. An ordinary push wants the cached base.
pull: ${{ github.event_name == 'schedule' }}
# ONE tag, the channel's. Every other tag is written by the step # ONE tag, the channel's. Every other tag is written by the step
# below, registry-side. buildx here pushes the first tag to the # below, registry-side. buildx here pushes the first tag to the
# registry and then re-pushes the rest through the DOCKER driver, # registry and then re-pushes the rest through the DOCKER driver,
@@ -0,0 +1,128 @@
"""Reconcile the database with what the models have always claimed (#3275).
Milestone 328 discovered ~130 places where the ORM models and the deployed
schema disagreed. Almost all of them were the MODEL being wrong — missing
`server_default`s, indexes and CHECK constraints that only ever existed in a
migration — and those are fixed in the model files with no DDL at all, because
the database already had them.
This migration carries the remainder — the two places where DDL is actually
needed, because the database is what is wrong.
`tag.fandom_id` is declared `index=True` on the model, but no migration ever
created that index. Every autogenerate run since would have proposed adding
it; nobody ran one, so the model and the database simply drifted apart and
stayed that way.
Deliberately NOT in this migration: anything about `image_record.sha256`. An
earlier draft of this file claimed sha256 was not unique in the database and
that duplicate rows were therefore possible. That was WRONG, and it was wrong
because it was read off `op.create_index("ix_image_record_sha256", ...)` at
0001 line 151 without reading line 149 two lines above it:
sa.UniqueConstraint("sha256", name="uq_image_record_sha256"),
Uniqueness has been enforced since the initial schema. The database simply
expresses it as a CONSTRAINT plus a separate non-unique lookup index, where
the model expressed it as one `unique=True, index=True` column — the same
guarantee built from different objects, which is why the two schemas did not
line up. The model now declares the constraint and the plain index separately,
so it describes what is actually there. No DDL is needed for it.
Also here: six CHECK constraints whose names carry their table prefix TWICE.
`base.py`'s naming convention is `ck_%(table_name)s_%(constraint_name)s`, and
unlike the uq/fk/ix entries it applies even to a constraint that already has a
name. Six migrations passed an already-prefixed name, so the convention
prefixed it again:
ck_external_link_ck_external_link_host
ck_external_link_ck_external_link_status
ck_import_settings_ck_import_settings_singleton
ck_ml_settings_ck_ml_settings_singleton
ck_post_ck_post_translation_override
ck_tag_ck_tag_fandom_requires_character
Nothing reads a CHECK constraint by name, so this has never done any harm —
but it is exactly the development-era residue the collapsed baseline exists to
leave behind, and a public schema should not ship it. The models now declare
bare names, which the convention renders into the single-prefix form; this
renames the deployed constraints to match.
RENAME CONSTRAINT is a catalog-only operation: no table scan, no rewrite, no
validation of existing rows. It takes a brief ACCESS EXCLUSIVE lock and
returns. That is why this is safe to do on `post` and `tag`, which are the two
large tables in the schema.
Revision ID: 0088
Revises: 0087
Create Date: 2026-08-30
"""
from typing import Sequence, Union
from alembic import op
revision: str = "0088"
down_revision: Union[str, None] = "0087"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
# (table, doubled name, single-prefix name)
#
# Six, not the four a first read of the migrations turned up. The list that
# settles it is the one extracted from the chain's pg_dump by matching
# `ck_(\w+?)_ck_\1_` — reading the migrations by eye missed external_link
# twice over, in the same way an earlier pass missed a UNIQUE constraint two
# lines above the index it was looking at (see the sha256 note above).
DOUBLED_CHECKS = (
("external_link", "ck_external_link_ck_external_link_host",
"ck_external_link_host"),
("external_link", "ck_external_link_ck_external_link_status",
"ck_external_link_status"),
("import_settings", "ck_import_settings_ck_import_settings_singleton",
"ck_import_settings_singleton"),
("ml_settings", "ck_ml_settings_ck_ml_settings_singleton",
"ck_ml_settings_singleton"),
("post", "ck_post_ck_post_translation_override",
"ck_post_translation_override"),
("tag", "ck_tag_ck_tag_fandom_requires_character",
"ck_tag_fandom_requires_character"),
)
def _rename_check(table: str, old: str, new: str) -> None:
# Guarded on pg_constraint rather than run bare: a database built from the
# models (a fresh install, or the CI integration schema) already has the
# single-prefix name, and this migration must be a no-op there rather than
# an error. Same reasoning as the CREATE INDEX IF NOT EXISTS below.
op.execute(
f"""
DO $$
BEGIN
IF EXISTS (
SELECT 1 FROM pg_constraint
WHERE conname = '{old}' AND conrelid = '{table}'::regclass
) THEN
ALTER TABLE {table} RENAME CONSTRAINT {old} TO {new};
END IF;
END $$;
"""
)
def upgrade() -> None:
# IF NOT EXISTS because the index is what the model already asks for: any
# database built from metadata rather than from this chain will have it,
# and this migration must be a no-op there rather than an error.
op.execute("CREATE INDEX IF NOT EXISTS ix_tag_fandom_id ON tag (fandom_id)")
for table, old, new in DOUBLED_CHECKS:
_rename_check(table, old, new)
def downgrade() -> None:
for table, old, new in DOUBLED_CHECKS:
_rename_check(table, new, old)
op.execute("DROP INDEX IF EXISTS ix_tag_fandom_id")
+120
View File
@@ -0,0 +1,120 @@
"""Index the seven unindexed FKs; drop the seven redundant indexes (#3300, #3301).
Found by a structural sweep of the deployed schema done AFTER 0088 brought the
models and the migration chain into exact agreement. That agreement is what
0088 achieved, and it is worth being precise about what it does NOT prove: a
models-vs-chain diff shows the two describe the same schema. It says nothing
about whether that schema is right. Everything here was wrong in BOTH, which is
exactly the class of problem the reconciliation could not see.
## Added: seven FK indexes
`image_tag.tag_id` is the one that matters. The table's only index is
PRIMARY KEY (image_record_id, tag_id), which leads with the wrong column for
the two hottest things done with it:
* the gallery's tag filter — services/tag_query.py builds
`image_tag.c.tag_id == tid` (and `.in_(tids)`) on every tag-scoped browse;
* ON DELETE CASCADE from `tag` — deleting or merging a tag makes Postgres
find that tag's rows before it can remove them.
Both had to scan the largest table in the schema. The other six are the same
shape on much smaller tables; `presentation_review.tag_id` is the notable one,
since it also CASCADEs.
## Dropped: seven redundant indexes
`ix_image_record_sha256` was an exact duplicate. A UNIQUE constraint builds its
own index, so `uq_image_record_sha256` already covered the column and
`image_record` carried two btrees on `sha256` — on the highest-insert-rate
table in the system.
The other six are single-column indexes that a later composite superseded
without the narrow one being retired. A btree on (a, b) already serves lookups
on `a`, so each was pure write amplification. `task_run` and `backup_run` are
append-heavy operational logs, which is where that cost lands hardest.
Note for anyone reading 0088 next to this: 0088 deliberately taught the models
to declare BOTH sha256 indexes, so they would describe reality. That was right.
This migration changes the reality instead, and the models change with it.
## CONCURRENTLY, and why this migration has no transaction
`CREATE INDEX` takes an ACCESS EXCLUSIVE lock for the whole build, which on
`image_tag` means stalling every write for as long as it takes. CONCURRENTLY
builds without blocking writers, at the cost of two table passes and an
inability to run inside a transaction — hence `autocommit_block()`.
The consequence to know about: this migration is NOT atomic. If it fails
partway, the work already done stays done. Every statement is therefore written
IF NOT EXISTS / IF EXISTS so that re-running it after a failure is safe rather
than an error.
A failed CONCURRENTLY build also leaves an INVALID index behind — it is not
used by the planner and not repaired automatically. Find them with:
SELECT c.relname FROM pg_index i
JOIN pg_class c ON c.oid = i.indexrelid
WHERE NOT i.indisvalid;
Drop what that returns and re-run; nothing else is needed.
Revision ID: 0089
Revises: 0088
Create Date: 2026-08-31
"""
from typing import Sequence, Union
from alembic import op
revision: str = "0089"
down_revision: Union[str, None] = "0088"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
# (index name, table, column) — names match what the models render under
# base.py's naming convention, so autogenerate stays quiet after this.
MISSING_FK_INDEXES = (
("ix_image_tag_tag_id", "image_tag", "tag_id"),
("ix_presentation_review_tag_id", "presentation_review", "tag_id"),
("ix_presentation_review_conflict_tag_id", "presentation_review", "conflict_tag_id"),
("ix_import_task_result_image_id", "import_task", "result_image_id"),
("ix_external_link_attachment_id", "external_link", "attachment_id"),
("ix_character_prototype_region_id", "character_prototype", "region_id"),
("ix_backup_run_restored_from_id", "backup_run", "restored_from_id"),
)
# (index name, table, column) — redundant; the second element of each pair in
# the docstring is what still covers the column after the drop.
REDUNDANT_INDEXES = (
("ix_image_record_sha256", "image_record", "sha256"),
("ix_backup_run_kind", "backup_run", "kind"),
("ix_backup_run_status", "backup_run", "status"),
("ix_task_run_queue", "task_run", "queue"),
("ix_task_run_status", "task_run", "status"),
("ix_task_run_task_name", "task_run", "task_name"),
("ix_external_link_post_id", "external_link", "post_id"),
)
def upgrade() -> None:
with op.get_context().autocommit_block():
for name, table, column in MISSING_FK_INDEXES:
op.execute(
f"CREATE INDEX CONCURRENTLY IF NOT EXISTS {name} "
f"ON {table} ({column})"
)
for name, _table, _column in REDUNDANT_INDEXES:
op.execute(f"DROP INDEX CONCURRENTLY IF EXISTS {name}")
def downgrade() -> None:
with op.get_context().autocommit_block():
for name, table, column in REDUNDANT_INDEXES:
op.execute(
f"CREATE INDEX CONCURRENTLY IF NOT EXISTS {name} "
f"ON {table} ({column})"
)
for name, _table, _column in MISSING_FK_INDEXES:
op.execute(f"DROP INDEX CONCURRENTLY IF EXISTS {name}")
+2 -2
View File
@@ -27,10 +27,10 @@ class Artist(Base):
notes: Mapped[str | None] = mapped_column(Text, nullable=True) notes: Mapped[str | None] = mapped_column(Text, nullable=True)
# True once a Source is attached; flips false if all sources removed. # True once a Source is attached; flips false if all sources removed.
is_subscription: Mapped[bool] = mapped_column(Boolean, nullable=False, default=False) is_subscription: Mapped[bool] = mapped_column(Boolean, nullable=False, default=False, server_default="false")
# Per-artist scheduling overrides; null means "use global default". # Per-artist scheduling overrides; null means "use global default".
auto_check: Mapped[bool] = mapped_column(Boolean, nullable=False, default=True) auto_check: Mapped[bool] = mapped_column(Boolean, nullable=False, default=True, server_default="true")
check_interval_seconds: Mapped[int | None] = mapped_column(Integer, nullable=True) check_interval_seconds: Mapped[int | None] = mapped_column(Integer, nullable=True)
created_at: Mapped[datetime] = mapped_column( created_at: Mapped[datetime] = mapped_column(
+17 -4
View File
@@ -20,7 +20,7 @@ feedback_check_existing_enums):
from datetime import datetime from datetime import datetime
from sqlalchemy import JSON, BigInteger, DateTime, ForeignKey, Integer, String, Text from sqlalchemy import JSON, BigInteger, DateTime, ForeignKey, Index, Integer, String, Text, text
from sqlalchemy.orm import Mapped, mapped_column from sqlalchemy.orm import Mapped, mapped_column
from .base import Base from .base import Base
@@ -29,10 +29,21 @@ from .base import Base
class BackupRun(Base): class BackupRun(Base):
__tablename__ = "backup_run" __tablename__ = "backup_run"
__table_args__ = (
# alembic 0017: reporting indexes, never declared on the model (#3275).
Index("ix_backup_run_kind_started", "kind", text("started_at DESC")),
Index("ix_backup_run_status_finished", "status", text("finished_at DESC")),
Index("ix_backup_run_tag_partial", "tag", postgresql_where=text("tag IS NOT NULL")),
)
id: Mapped[int] = mapped_column(Integer, primary_key=True) id: Mapped[int] = mapped_column(Integer, primary_key=True)
kind: Mapped[str] = mapped_column(String(16), nullable=False, index=True) # No index=True: ix_backup_run_kind_started (above) already leads with
# `kind`, so a single-column index on it was pure write cost (#3301).
kind: Mapped[str] = mapped_column(String(16), nullable=False)
status: Mapped[str] = mapped_column( status: Mapped[str] = mapped_column(
String(16), nullable=False, default="pending", index=True, # No index=True — ix_backup_run_status_finished leads with `status`.
String(16), nullable=False, default="pending",
server_default="pending",
) )
tag: Mapped[str | None] = mapped_column(String(64), nullable=True, index=True) tag: Mapped[str | None] = mapped_column(String(64), nullable=True, index=True)
triggered_by: Mapped[str] = mapped_column(String(32), nullable=False) triggered_by: Mapped[str] = mapped_column(String(32), nullable=False)
@@ -49,7 +60,9 @@ class BackupRun(Base):
manifest: Mapped[dict] = mapped_column( manifest: Mapped[dict] = mapped_column(
JSON, nullable=False, default=dict, server_default="{}", JSON, nullable=False, default=dict, server_default="{}",
) )
# Self-referential FK, unindexed until 0089 (#3300): SET NULL has to find
# the rows pointing at a deleted run before it can null them.
restored_from_id: Mapped[int | None] = mapped_column( restored_from_id: Mapped[int | None] = mapped_column(
ForeignKey("backup_run.id", ondelete="SET NULL"), ForeignKey("backup_run.id", ondelete="SET NULL"),
nullable=True, nullable=True, index=True,
) )
+3 -1
View File
@@ -40,8 +40,10 @@ class CharacterPrototype(Base):
) )
# Provenance: the region this vector was copied from. SET NULL so pruning a # Provenance: the region this vector was copied from. SET NULL so pruning a
# region doesn't delete the prototype mid-cycle (the next refresh reconciles). # region doesn't delete the prototype mid-cycle (the next refresh reconciles).
# index=True added in 0089 — the FK was unindexed (#3300).
region_id: Mapped[int | None] = mapped_column( region_id: Mapped[int | None] = mapped_column(
ForeignKey("image_region.id", ondelete="SET NULL"), nullable=True ForeignKey("image_region.id", ondelete="SET NULL"), nullable=True,
index=True,
) )
+2 -2
View File
@@ -25,8 +25,8 @@ class DownloadEvent(Base):
DateTime(timezone=True), nullable=False, server_default=func.now() DateTime(timezone=True), nullable=False, server_default=func.now()
) )
finished_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True) finished_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
bytes_downloaded: Mapped[int] = mapped_column(BigInteger, nullable=False, default=0) bytes_downloaded: Mapped[int] = mapped_column(BigInteger, nullable=False, default=0, server_default="0")
files_count: Mapped[int] = mapped_column(Integer, nullable=False, default=0) files_count: Mapped[int] = mapped_column(Integer, nullable=False, default=0, server_default="0")
error: Mapped[str | None] = mapped_column(Text, nullable=True) error: Mapped[str | None] = mapped_column(Text, nullable=True)
metadata_: Mapped[dict] = mapped_column( metadata_: Mapped[dict] = mapped_column(
"metadata", JSONB, nullable=False, default=dict, "metadata", JSONB, nullable=False, default=dict,
+20 -1
View File
@@ -16,6 +16,7 @@ doesn't delete the link record).
from datetime import datetime from datetime import datetime
from sqlalchemy import ( from sqlalchemy import (
CheckConstraint,
DateTime, DateTime,
Float, Float,
ForeignKey, ForeignKey,
@@ -38,15 +39,33 @@ STATUSES = ("pending", "downloading", "downloaded", "failed", "skipped", "dead")
class ExternalLink(Base): class ExternalLink(Base):
__tablename__ = "external_link" __tablename__ = "external_link"
__table_args__ = ( __table_args__ = (
# alembic 0028 enum CHECKs. Rule 36 territory: a new host or status value
# needs its constraint swapped in the same migration (#3275).
CheckConstraint(
"host IN ('mega', 'gdrive', 'mediafire', 'dropbox', 'pixeldrain')",
# Bare name: Base.metadata's naming convention prepends
# ck_<table>_. Pre-prefixing it here doubles the prefix — see
# alembic 0088, which renames the four constraints that shipped
# that way (#3275).
name="host",
),
CheckConstraint(
"status IN ('pending', 'downloading', 'downloaded', 'failed', 'skipped', 'dead')",
name="status",
),
# One row per (post, url). The full url (incl. #fragment) is the identity # One row per (post, url). The full url (incl. #fragment) is the identity
# — the same file linked twice in a post collapses to one row. # — the same file linked twice in a post collapses to one row.
Index("uq_external_link_post_url", "post_id", "url", unique=True), Index("uq_external_link_post_url", "post_id", "url", unique=True),
Index("ix_external_link_status", "status"), Index("ix_external_link_status", "status"),
# Unindexed FK (#3300).
Index("ix_external_link_attachment_id", "attachment_id"),
) )
id: Mapped[int] = mapped_column(Integer, primary_key=True) id: Mapped[int] = mapped_column(Integer, primary_key=True)
# No index=True: uq_external_link_post_url (post_id, url) already leads
# with post_id (#3301).
post_id: Mapped[int] = mapped_column( post_id: Mapped[int] = mapped_column(
ForeignKey("post.id", ondelete="CASCADE"), nullable=False, index=True ForeignKey("post.id", ondelete="CASCADE"), nullable=False
) )
artist_id: Mapped[int | None] = mapped_column( artist_id: Mapped[int | None] = mapped_column(
ForeignKey("artist.id", ondelete="SET NULL"), nullable=True, index=True ForeignKey("artist.id", ondelete="SET NULL"), nullable=True, index=True
+3 -2
View File
@@ -50,7 +50,8 @@ class GpuJob(Base):
# What to compute, e.g. 'ccip' (detect figures + CCIP-embed) or 'siglip_region'. # What to compute, e.g. 'ccip' (detect figures + CCIP-embed) or 'siglip_region'.
task: Mapped[str] = mapped_column(String(32), nullable=False) task: Mapped[str] = mapped_column(String(32), nullable=False)
status: Mapped[str] = mapped_column( status: Mapped[str] = mapped_column(
String(16), nullable=False, default="pending", index=True String(16), nullable=False, default="pending", index=True,
server_default="pending",
) )
# pending | leased | done | error # pending | leased | done | error
lease_token: Mapped[str | None] = mapped_column(String(64), nullable=True) lease_token: Mapped[str | None] = mapped_column(String(64), nullable=True)
@@ -60,7 +61,7 @@ class GpuJob(Base):
lease_expires_at: Mapped[datetime | None] = mapped_column( lease_expires_at: Mapped[datetime | None] = mapped_column(
DateTime(timezone=True), nullable=True DateTime(timezone=True), nullable=True
) )
attempts: Mapped[int] = mapped_column(Integer, nullable=False, default=0) attempts: Mapped[int] = mapped_column(Integer, nullable=False, default=0, server_default="0")
error: Mapped[str | None] = mapped_column(Text, nullable=True) error: Mapped[str | None] = mapped_column(Text, nullable=True)
# Triage verdict for an ERRORED job (#125): NULL = not yet probed; # Triage verdict for an ERRORED job (#125): NULL = not yet probed;
# 'defect' = the integrity probe says the FILE itself is bad (surfaced for # 'defect' = the integrity probe says the FILE itself is bad (surfaced for
+3 -2
View File
@@ -24,10 +24,11 @@ class HeadAutoApplyRun(Base):
id: Mapped[int] = mapped_column(Integer, primary_key=True) id: Mapped[int] = mapped_column(Integer, primary_key=True)
# dry_run=True is a PREVIEW: scores + counts what WOULD apply, writes nothing # dry_run=True is a PREVIEW: scores + counts what WOULD apply, writes nothing
# (preview/apply parity, rule 93). # (preview/apply parity, rule 93).
dry_run: Mapped[bool] = mapped_column(Boolean, nullable=False, default=False) dry_run: Mapped[bool] = mapped_column(Boolean, nullable=False, default=False, server_default="false")
params: Mapped[dict[str, Any]] = mapped_column(JSONB, nullable=False) params: Mapped[dict[str, Any]] = mapped_column(JSONB, nullable=False)
status: Mapped[str] = mapped_column( status: Mapped[str] = mapped_column(
String(16), nullable=False, default="running", index=True String(16), nullable=False, default="running", index=True,
server_default="running",
) )
# running | ready | error # running | ready | error
started_at: Mapped[datetime] = mapped_column( started_at: Mapped[datetime] = mapped_column(
+2 -2
View File
@@ -24,9 +24,9 @@ class HeadMetric(Base):
ForeignKey("tag.id", ondelete="CASCADE"), primary_key=True ForeignKey("tag.id", ondelete="CASCADE"), primary_key=True
) )
# An auto-applied (source='head_auto') tag the operator later REMOVED. # An auto-applied (source='head_auto') tag the operator later REMOVED.
n_misfires: Mapped[int] = mapped_column(Integer, nullable=False, default=0) n_misfires: Mapped[int] = mapped_column(Integer, nullable=False, default=0, server_default="0")
# A tag with a head that the operator added by HAND (the head missed it). # A tag with a head that the operator added by HAND (the head missed it).
n_underfires: Mapped[int] = mapped_column(Integer, nullable=False, default=0) n_underfires: Mapped[int] = mapped_column(Integer, nullable=False, default=0, server_default="0")
updated_at: Mapped[datetime] = mapped_column( updated_at: Mapped[datetime] = mapped_column(
DateTime(timezone=True), nullable=False, server_default=func.now() DateTime(timezone=True), nullable=False, server_default=func.now()
) )
+11 -5
View File
@@ -19,8 +19,14 @@ class HeadMetricsSnapshot(Base):
__tablename__ = "head_metrics_snapshot" __tablename__ = "head_metrics_snapshot"
id: Mapped[int] = mapped_column(Integer, primary_key=True) id: Mapped[int] = mapped_column(Integer, primary_key=True)
tag_id: Mapped[int] = mapped_column( # Nullable, matching alembic 0060, which declared this column without
ForeignKey("tag.id", ondelete="CASCADE"), index=True # `nullable=False`. The model had it as `Mapped[int]` — NOT NULL — which
# was simply never true of the database (#3275). Left nullable rather than
# tightened: a snapshot of a tag that is later hard-deleted is a row worth
# keeping, and the FK is ON DELETE CASCADE, so tightening it would only
# change behaviour, not correct a bug.
tag_id: Mapped[int | None] = mapped_column(
ForeignKey("tag.id", ondelete="CASCADE"), nullable=True, index=True
) )
# Denormalized so a snapshot stays readable even if the tag is later renamed. # Denormalized so a snapshot stays readable even if the tag is later renamed.
name: Mapped[str] = mapped_column(String(255), nullable=False) name: Mapped[str] = mapped_column(String(255), nullable=False)
@@ -28,9 +34,9 @@ class HeadMetricsSnapshot(Base):
DateTime(timezone=True), nullable=False, server_default=func.now(), index=True DateTime(timezone=True), nullable=False, server_default=func.now(), index=True
) )
# Current count of source='head_auto' applications still standing. # Current count of source='head_auto' applications still standing.
n_auto_applied: Mapped[int] = mapped_column(Integer, nullable=False, default=0) n_auto_applied: Mapped[int] = mapped_column(Integer, nullable=False, default=0, server_default="0")
n_misfires: Mapped[int] = mapped_column(Integer, nullable=False, default=0) n_misfires: Mapped[int] = mapped_column(Integer, nullable=False, default=0, server_default="0")
n_underfires: Mapped[int] = mapped_column(Integer, nullable=False, default=0) n_underfires: Mapped[int] = mapped_column(Integer, nullable=False, default=0, server_default="0")
# The head's measured quality at snapshot time (null if no head exists). # The head's measured quality at snapshot time (null if no head exists).
ap: Mapped[float | None] = mapped_column(Float, nullable=True) ap: Mapped[float | None] = mapped_column(Float, nullable=True)
precision_cv: Mapped[float | None] = mapped_column(Float, nullable=True) precision_cv: Mapped[float | None] = mapped_column(Float, nullable=True)
+2 -1
View File
@@ -24,7 +24,8 @@ class HeadTrainingRun(Base):
# Training parameters: {min_positives, neg_ratio, precision_target, ...}. # Training parameters: {min_positives, neg_ratio, precision_target, ...}.
params: Mapped[dict[str, Any]] = mapped_column(JSONB, nullable=False) params: Mapped[dict[str, Any]] = mapped_column(JSONB, nullable=False)
status: Mapped[str] = mapped_column( status: Mapped[str] = mapped_column(
String(16), nullable=False, default="running", index=True String(16), nullable=False, default="running", index=True,
server_default="running",
) )
# running | ready | error # running | ready | error
started_at: Mapped[datetime] = mapped_column( started_at: Mapped[datetime] = mapped_column(
+8 -1
View File
@@ -47,8 +47,15 @@ class ImageProvenance(Base):
# attachment on the post. NULL for loose downloads and pre-backfill rows. # attachment on the post. NULL for loose downloads and pre-backfill rows.
# SET NULL so deleting the archive attachment never destroys the (image, # SET NULL so deleting the archive attachment never destroys the (image,
# post) edge — it just forgets which archive it came from. # post) edge — it just forgets which archive it came from.
# FK named explicitly: the convention renders this
# `fk_image_provenance_from_attachment_id_post_attachment`, but alembic
# 0055 created it as `fk_image_provenance_from_attachment` (#3275).
from_attachment_id: Mapped[int | None] = mapped_column( from_attachment_id: Mapped[int | None] = mapped_column(
ForeignKey("post_attachment.id", ondelete="SET NULL"), ForeignKey(
"post_attachment.id",
ondelete="SET NULL",
name="fk_image_provenance_from_attachment",
),
nullable=True, index=True, nullable=True, index=True,
) )
captured_metadata: Mapped[dict | None] = mapped_column(JSON, nullable=True) captured_metadata: Mapped[dict | None] = mapped_column(JSON, nullable=True)
+41 -3
View File
@@ -14,10 +14,13 @@ from sqlalchemy import (
Enum, Enum,
Float, Float,
ForeignKey, ForeignKey,
Index,
Integer, Integer,
String, String,
Text, Text,
UniqueConstraint,
func, func,
text,
) )
from sqlalchemy.orm import Mapped, mapped_column from sqlalchemy.orm import Mapped, mapped_column
@@ -29,11 +32,38 @@ ORIGIN_CHOICES = ("downloaded", "imported_filesystem", "uploaded")
class ImageRecord(Base): class ImageRecord(Base):
__tablename__ = "image_record" __tablename__ = "image_record"
__table_args__ = (
# alembic 0001. The database enforces sha256 uniqueness with a
# CONSTRAINT and carries a SEPARATE non-unique btree index; the model
# said `unique=True, index=True`, which collapses both into a single
# UNIQUE index under a different name. Same guarantee either way, but
# not the same objects, so autogenerate saw a drop and an add (#3275).
UniqueConstraint("sha256", name="uq_image_record_sha256"),
# alembic 0036, and the last thing in this schema that lived only in a
# migration. SQLAlchemy CAN express an hnsw index with an operator
# class, so there is no reason for it to be invisible to the models —
# and its absence was the quietest failure of the lot: everything
# works, similarity search just silently stops using an index.
Index(
"ix_image_record_siglip_hnsw",
"siglip_embedding",
postgresql_using="hnsw",
postgresql_ops={"siglip_embedding": "vector_cosine_ops"},
),
# alembic 0035/0071: the date-ordered browse indexes (#3275).
Index("ix_image_record_effective_date", text("effective_date DESC"), text("id DESC")),
Index("ix_image_record_earliest_post_date", text("earliest_post_date DESC"), text("id DESC")),
)
id: Mapped[int] = mapped_column(Integer, primary_key=True) id: Mapped[int] = mapped_column(Integer, primary_key=True)
# On-disk identity # On-disk identity
path: Mapped[str] = mapped_column(Text, nullable=False, unique=True) path: Mapped[str] = mapped_column(Text, nullable=False, unique=True)
sha256: Mapped[str] = mapped_column(String(64), nullable=False, unique=True, index=True) # Neither unique= nor index=: uq_image_record_sha256 in __table_args__
# above creates its own index, and the separate ix_image_record_sha256
# that 0001 also built was an exact duplicate of it — dropped in 0089
# (#3301). Lookups by sha256 use the constraint's index.
sha256: Mapped[str] = mapped_column(String(64), nullable=False)
phash: Mapped[str | None] = mapped_column(String(32), nullable=True, index=True) phash: Mapped[str | None] = mapped_column(String(32), nullable=True, index=True)
size_bytes: Mapped[int] = mapped_column(BigInteger, nullable=False) size_bytes: Mapped[int] = mapped_column(BigInteger, nullable=False)
mime: Mapped[str] = mapped_column(String(64), nullable=False) mime: Mapped[str] = mapped_column(String(64), nullable=False)
@@ -47,7 +77,8 @@ class ImageRecord(Base):
# Integrity verification status. FC-2e populates this; FC-2a leaves rows at 'unknown'. # Integrity verification status. FC-2e populates this; FC-2a leaves rows at 'unknown'.
# Values: 'unknown' (default), 'ok', 'corrupt', 'failed_verification'. # Values: 'unknown' (default), 'ok', 'corrupt', 'failed_verification'.
integrity_status: Mapped[str] = mapped_column( integrity_status: Mapped[str] = mapped_column(
String(24), nullable=False, default="unknown", index=True String(24), nullable=False, default="unknown", index=True,
server_default="unknown",
) )
# Thumbnail (populated by FC-2) # Thumbnail (populated by FC-2)
@@ -72,8 +103,15 @@ class ImageRecord(Base):
) )
# FC-2d-vii-c: canonical per-image artist (the single source of truth # FC-2d-vii-c: canonical per-image artist (the single source of truth
# for attribution; provenance posts remain lineage detail). # for attribution; provenance posts remain lineage detail).
# FK named explicitly: the naming convention renders this
# `fk_image_record_artist_id_artist`, but alembic 0008 created it as
# `fk_image_record_artist_id` (#3275).
artist_id: Mapped[int | None] = mapped_column( artist_id: Mapped[int | None] = mapped_column(
ForeignKey("artist.id", ondelete="SET NULL"), nullable=True, index=True ForeignKey(
"artist.id", ondelete="SET NULL", name="fk_image_record_artist_id"
),
nullable=True,
index=True,
) )
# ML fields (populated by the ml-worker / GPU agent). 1152 = SigLIP-so400m # ML fields (populated by the ml-worker / GPU agent). 1152 = SigLIP-so400m
+7 -7
View File
@@ -21,17 +21,17 @@ class ImportBatch(Base):
) )
finished_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True) finished_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
total_files: Mapped[int] = mapped_column(Integer, nullable=False, default=0) total_files: Mapped[int] = mapped_column(Integer, nullable=False, default=0, server_default="0")
imported: Mapped[int] = mapped_column(Integer, nullable=False, default=0) imported: Mapped[int] = mapped_column(Integer, nullable=False, default=0, server_default="0")
skipped: Mapped[int] = mapped_column(Integer, nullable=False, default=0) skipped: Mapped[int] = mapped_column(Integer, nullable=False, default=0, server_default="0")
failed: Mapped[int] = mapped_column(Integer, nullable=False, default=0) failed: Mapped[int] = mapped_column(Integer, nullable=False, default=0, server_default="0")
attachments: Mapped[int] = mapped_column(Integer, nullable=False, default=0) attachments: Mapped[int] = mapped_column(Integer, nullable=False, default=0, server_default="0")
# Deep-scan only: count of already-imported files whose sidecar metadata # Deep-scan only: count of already-imported files whose sidecar metadata
# got re-applied this run (post/source/provenance upsert). Stays 0 on # got re-applied this run (post/source/provenance upsert). Stays 0 on
# quick-scan batches. See `Importer.import_one(deep_scan=True)`. # quick-scan batches. See `Importer.import_one(deep_scan=True)`.
refreshed: Mapped[int] = mapped_column(Integer, nullable=False, default=0) refreshed: Mapped[int] = mapped_column(Integer, nullable=False, default=0, server_default="0")
status: Mapped[str] = mapped_column(String(16), nullable=False, default="running", index=True) status: Mapped[str] = mapped_column(String(16), nullable=False, default="running", index=True, server_default="running")
# running | complete | cancelled # running | complete | cancelled
tasks = relationship("ImportTask", back_populates="batch", cascade="all, delete-orphan") tasks = relationship("ImportTask", back_populates="batch", cascade="all, delete-orphan")
+42 -16
View File
@@ -4,7 +4,15 @@ Enforced as a single row via a CHECK (id = 1) constraint. The application
always SELECTs id=1 and never inserts/deletes after the initial migration. always SELECTs id=1 and never inserts/deletes after the initial migration.
""" """
from sqlalchemy import Boolean, CheckConstraint, Float, Integer, Text, select from sqlalchemy import (
Boolean,
CheckConstraint,
Float,
Integer,
Text,
select,
text,
)
from sqlalchemy.orm import Mapped, mapped_column from sqlalchemy.orm import Mapped, mapped_column
from .base import Base from .base import Base
@@ -14,63 +22,79 @@ class ImportSettings(Base):
__tablename__ = "import_settings" __tablename__ = "import_settings"
# Bare constraint name — Base.metadata's naming convention applies the # Bare constraint name — Base.metadata's naming convention applies the
# ck_<table>_<name> prefix, producing the final ck_import_settings_singleton. # ck_<table>_<name> prefix, producing the final ck_import_settings_singleton.
# Bare name — Base.metadata's naming convention prepends ck_<table>_,
# producing ck_import_settings_singleton. The chain shipped the DOUBLED
# ck_import_settings_ck_import_settings_singleton, because the migration
# pre-prefixed the name and the convention prefixed it again; alembic
# 0088 renames it to what this line has always produced (#3275).
__table_args__ = (CheckConstraint("id = 1", name="singleton"),) __table_args__ = (CheckConstraint("id = 1", name="singleton"),)
id: Mapped[int] = mapped_column(Integer, primary_key=True) id: Mapped[int] = mapped_column(Integer, primary_key=True)
import_scan_path: Mapped[str] = mapped_column(Text, nullable=False, default="/import") import_scan_path: Mapped[str] = mapped_column(Text, nullable=False, default="/import", server_default="/import")
min_width: Mapped[int] = mapped_column(Integer, nullable=False, default=0) min_width: Mapped[int] = mapped_column(Integer, nullable=False, default=0, server_default="0")
min_height: Mapped[int] = mapped_column(Integer, nullable=False, default=0) min_height: Mapped[int] = mapped_column(Integer, nullable=False, default=0, server_default="0")
skip_transparent: Mapped[bool] = mapped_column(Boolean, nullable=False, default=False) skip_transparent: Mapped[bool] = mapped_column(Boolean, nullable=False, default=False, server_default="false")
transparency_threshold: Mapped[float] = mapped_column(Float, nullable=False, default=0.9) transparency_threshold: Mapped[float] = mapped_column(Float, nullable=False, default=0.9, server_default="0.9")
skip_single_color: Mapped[bool] = mapped_column(Boolean, nullable=False, default=False) skip_single_color: Mapped[bool] = mapped_column(Boolean, nullable=False, default=False, server_default="false")
single_color_threshold: Mapped[float] = mapped_column(Float, nullable=False, default=0.95) single_color_threshold: Mapped[float] = mapped_column(Float, nullable=False, default=0.95, server_default="0.95")
single_color_tolerance: Mapped[int] = mapped_column(Integer, nullable=False, default=30) single_color_tolerance: Mapped[int] = mapped_column(Integer, nullable=False, default=30, server_default="30")
phash_threshold: Mapped[int] = mapped_column(Integer, nullable=False, default=10) phash_threshold: Mapped[int] = mapped_column(Integer, nullable=False, default=10, server_default="10")
# FC-3c downloader knobs # FC-3c downloader knobs
download_rate_limit_seconds: Mapped[float] = mapped_column( download_rate_limit_seconds: Mapped[float] = mapped_column(
Float, nullable=False, default=3.0 Float, nullable=False, default=3.0,
server_default="3",
) )
download_validate_files: Mapped[bool] = mapped_column( download_validate_files: Mapped[bool] = mapped_column(
Boolean, nullable=False, default=True Boolean, nullable=False, default=True,
server_default="true",
) )
# FC-3d scheduling knobs # FC-3d scheduling knobs
download_schedule_default_seconds: Mapped[int] = mapped_column( download_schedule_default_seconds: Mapped[int] = mapped_column(
Integer, nullable=False, default=28800 Integer, nullable=False, default=28800,
server_default="28800",
) )
download_event_retention_days: Mapped[int] = mapped_column( download_event_retention_days: Mapped[int] = mapped_column(
Integer, nullable=False, default=90 Integer, nullable=False, default=90,
server_default="90",
) )
download_failure_warning_threshold: Mapped[int] = mapped_column( download_failure_warning_threshold: Mapped[int] = mapped_column(
Integer, nullable=False, default=5 Integer, nullable=False, default=5,
server_default="5",
) )
# FC-3h backup knobs. # FC-3h backup knobs.
backup_db_nightly_enabled: Mapped[bool] = mapped_column( backup_db_nightly_enabled: Mapped[bool] = mapped_column(
Boolean, nullable=False, default=False, Boolean, nullable=False, default=False,
server_default="false",
) )
backup_db_nightly_hour_utc: Mapped[int] = mapped_column( backup_db_nightly_hour_utc: Mapped[int] = mapped_column(
Integer, nullable=False, default=3, Integer, nullable=False, default=3,
server_default="3",
) )
backup_db_keep_last_n: Mapped[int] = mapped_column( backup_db_keep_last_n: Mapped[int] = mapped_column(
Integer, nullable=False, default=14, Integer, nullable=False, default=14,
server_default="14",
) )
backup_images_keep_last_n: Mapped[int] = mapped_column( backup_images_keep_last_n: Mapped[int] = mapped_column(
Integer, nullable=False, default=3, Integer, nullable=False, default=3,
server_default="3",
) )
# FC-6.3 series continuation matcher. enabled gates the rescan; threshold is # FC-6.3 series continuation matcher. enabled gates the rescan; threshold is
# the weighted-score cut-off (0..1) above which a pending suggestion is made. # the weighted-score cut-off (0..1) above which a pending suggestion is made.
series_suggest_enabled: Mapped[bool] = mapped_column( series_suggest_enabled: Mapped[bool] = mapped_column(
Boolean, nullable=False, default=True, Boolean, nullable=False, default=True,
server_default="true",
) )
series_suggest_threshold: Mapped[float] = mapped_column( series_suggest_threshold: Mapped[float] = mapped_column(
Float, nullable=False, default=0.5, Float, nullable=False, default=0.5,
server_default="0.5",
) )
# #830 off-platform file-host downloads — per-host enable lever (default on, # #830 off-platform file-host downloads — per-host enable lever (default on,
@@ -113,7 +137,9 @@ class ImportSettings(Base):
# English (e.g. "… WIP Part 1") as a European language at ~0.86. CJK stays # English (e.g. "… WIP Part 1") as a European language at ~0.86. CJK stays
# trusted regardless (script-detected). Per-post overrides handle the misses. # trusted regardless (script-detected). Per-post overrides handle the misses.
translation_min_confidence: Mapped[float] = mapped_column( translation_min_confidence: Mapped[float] = mapped_column(
Float, nullable=False, default=0.9, server_default="0.9", # text() because alembic 0084 used sa.text(); see ml_settings for why
# the form matters and why it is per-column (#3275).
Float, nullable=False, default=0.9, server_default=text("0.9"),
) )
# Title-based WIP auto-tagging (task #1458). When a freshly-imported post's # Title-based WIP auto-tagging (task #1458). When a freshly-imported post's
+11 -3
View File
@@ -13,10 +13,12 @@ from sqlalchemy import (
Boolean, Boolean,
DateTime, DateTime,
ForeignKey, ForeignKey,
Index,
Integer, Integer,
String, String,
Text, Text,
func, func,
text,
) )
from sqlalchemy.orm import Mapped, mapped_column, relationship from sqlalchemy.orm import Mapped, mapped_column, relationship
@@ -26,6 +28,12 @@ from .base import Base
class ImportTask(Base): class ImportTask(Base):
__tablename__ = "import_task" __tablename__ = "import_task"
__table_args__ = (
Index("ix_import_task_created_at_desc", text("created_at DESC")),
# Unindexed FK (#3300).
Index("ix_import_task_result_image_id", "result_image_id"),
)
id: Mapped[int] = mapped_column(Integer, primary_key=True) id: Mapped[int] = mapped_column(Integer, primary_key=True)
batch_id: Mapped[int] = mapped_column( batch_id: Mapped[int] = mapped_column(
ForeignKey("import_batch.id", ondelete="CASCADE"), nullable=False, index=True ForeignKey("import_batch.id", ondelete="CASCADE"), nullable=False, index=True
@@ -33,14 +41,14 @@ class ImportTask(Base):
source_path: Mapped[str] = mapped_column(Text, nullable=False) source_path: Mapped[str] = mapped_column(Text, nullable=False)
task_type: Mapped[str] = mapped_column(String(16), nullable=False) # media|archive task_type: Mapped[str] = mapped_column(String(16), nullable=False) # media|archive
status: Mapped[str] = mapped_column(String(16), nullable=False, default="pending", index=True) status: Mapped[str] = mapped_column(String(16), nullable=False, default="pending", index=True, server_default="pending")
# Poison-pill circuit breaker (alembic 0026). recovery_count tracks # Poison-pill circuit breaker (alembic 0026). recovery_count tracks
# how many times the stuck-task sweep has re-queued this row; after # how many times the stuck-task sweep has re-queued this row; after
# the cap it's failed with a diagnostic instead of looping. refetched # the cap it's failed with a diagnostic instead of looping. refetched
# bounds the one-shot re-download remediation to a single attempt. # bounds the one-shot re-download remediation to a single attempt.
recovery_count: Mapped[int] = mapped_column(Integer, nullable=False, default=0) recovery_count: Mapped[int] = mapped_column(Integer, nullable=False, default=0, server_default="0")
refetched: Mapped[bool] = mapped_column(Boolean, nullable=False, default=False) refetched: Mapped[bool] = mapped_column(Boolean, nullable=False, default=False, server_default="false")
result_image_id: Mapped[int | None] = mapped_column( result_image_id: Mapped[int | None] = mapped_column(
ForeignKey("image_record.id", ondelete="SET NULL"), nullable=True ForeignKey("image_record.id", ondelete="SET NULL"), nullable=True
+8 -5
View File
@@ -8,7 +8,7 @@ reads it and routes through cleanup_service.delete_images.
from datetime import datetime from datetime import datetime
from typing import Any from typing import Any
from sqlalchemy import DateTime, Integer, String, Text, func from sqlalchemy import DateTime, Integer, String, Text, func, text
from sqlalchemy.dialects.postgresql import JSONB from sqlalchemy.dialects.postgresql import JSONB
from sqlalchemy.orm import Mapped, mapped_column from sqlalchemy.orm import Mapped, mapped_column
@@ -23,6 +23,7 @@ class LibraryAuditRun(Base):
params: Mapped[dict[str, Any]] = mapped_column(JSONB, nullable=False) params: Mapped[dict[str, Any]] = mapped_column(JSONB, nullable=False)
status: Mapped[str] = mapped_column( status: Mapped[str] = mapped_column(
String(16), nullable=False, default="running", index=True, String(16), nullable=False, default="running", index=True,
server_default="running",
) )
# running | ready | applied | cancelled | error # running | ready | applied | cancelled | error
started_at: Mapped[datetime] = mapped_column( started_at: Mapped[datetime] = mapped_column(
@@ -31,14 +32,16 @@ class LibraryAuditRun(Base):
finished_at: Mapped[datetime | None] = mapped_column( finished_at: Mapped[datetime | None] = mapped_column(
DateTime(timezone=True), nullable=True, DateTime(timezone=True), nullable=True,
) )
scanned_count: Mapped[int] = mapped_column(Integer, nullable=False, default=0) scanned_count: Mapped[int] = mapped_column(Integer, nullable=False, default=0, server_default="0")
matched_count: Mapped[int] = mapped_column(Integer, nullable=False, default=0) matched_count: Mapped[int] = mapped_column(Integer, nullable=False, default=0, server_default="0")
matched_ids: Mapped[list[int]] = mapped_column(JSONB, nullable=False, default=list) matched_ids: Mapped[list[int]] = mapped_column(
JSONB, nullable=False, default=list, server_default=text("'[]'::jsonb")
)
error: Mapped[str | None] = mapped_column(Text, nullable=True) error: Mapped[str | None] = mapped_column(Text, nullable=True)
# Chunked-scan state (alembic 0039): keyset cursor the next chunk resumes # Chunked-scan state (alembic 0039): keyset cursor the next chunk resumes
# from, and the last time a chunk made progress (so the recovery sweep can # from, and the last time a chunk made progress (so the recovery sweep can
# tell a progressing multi-chunk audit from a stuck one). # tell a progressing multi-chunk audit from a stuck one).
resume_after_id: Mapped[int] = mapped_column(Integer, nullable=False, default=0) resume_after_id: Mapped[int] = mapped_column(Integer, nullable=False, default=0, server_default="0")
last_progress_at: Mapped[datetime | None] = mapped_column( last_progress_at: Mapped[datetime | None] = mapped_column(
DateTime(timezone=True), nullable=True, DateTime(timezone=True), nullable=True,
) )
+74 -32
View File
@@ -11,6 +11,7 @@ from sqlalchemy import (
String, String,
func, func,
select, select,
text,
) )
from sqlalchemy.orm import Mapped, mapped_column from sqlalchemy.orm import Mapped, mapped_column
@@ -20,7 +21,10 @@ from .base import Base
class MLSettings(Base): class MLSettings(Base):
__tablename__ = "ml_settings" __tablename__ = "ml_settings"
# Bare name — Base.metadata's naming convention prepends ck_<table>_, # Bare name — Base.metadata's naming convention prepends ck_<table>_,
# producing the final ck_ml_settings_singleton (matches migration 0003). # producing ck_ml_settings_singleton. The chain shipped the DOUBLED
# ck_ml_settings_ck_ml_settings_singleton, because the migration
# pre-prefixed the name and the convention prefixed it again; alembic
# 0088 renames it to what this line has always produced (#3275).
__table_args__ = (CheckConstraint("id = 1", name="singleton"),) __table_args__ = (CheckConstraint("id = 1", name="singleton"),)
id: Mapped[int] = mapped_column(Integer, primary_key=True) id: Mapped[int] = mapped_column(Integer, primary_key=True)
@@ -31,17 +35,20 @@ class MLSettings(Base):
# queueing embed work nothing will consume (the daily GPU 'embed' backfill # queueing embed work nothing will consume (the daily GPU 'embed' backfill
# covers those images instead). # covers those images instead).
cpu_embed_enabled: Mapped[bool] = mapped_column( cpu_embed_enabled: Mapped[bool] = mapped_column(
Boolean, nullable=False, default=True Boolean, nullable=False, default=True,
server_default="true",
) )
# Video embedding (#747). Sample one frame every N seconds (fixed CADENCE, not # Video embedding (#747). Sample one frame every N seconds (fixed CADENCE, not
# a fixed count) so coverage reflects real screen time regardless of length; # a fixed count) so coverage reflects real screen time regardless of length;
# cap the total so a long video can't explode into hundreds of embeds. The # cap the total so a long video can't explode into hundreds of embeds. The
# per-frame SigLIP embeddings are mean-pooled. Operator-tunable. # per-frame SigLIP embeddings are mean-pooled. Operator-tunable.
video_frame_interval_seconds: Mapped[float] = mapped_column( video_frame_interval_seconds: Mapped[float] = mapped_column(
Float, nullable=False, default=4.0 Float, nullable=False, default=4.0,
server_default="4",
) )
video_max_frames: Mapped[int] = mapped_column( video_max_frames: Mapped[int] = mapped_column(
Integer, nullable=False, default=64 Integer, nullable=False, default=64,
server_default="64",
) )
# Tagging-v2 head training (#114). The head is the suggestion source that # Tagging-v2 head training (#114). The head is the suggestion source that
# LEARNS from the operator's tags (replacing Camie + centroid). A concept # LEARNS from the operator's tags (replacing Camie + centroid). A concept
@@ -49,10 +56,12 @@ class MLSettings(Base):
# head_auto_apply_precision is the precision bar a head must clear (at some # head_auto_apply_precision is the precision bar a head must clear (at some
# operating point) to "graduate" into earned auto-apply. Operator-tunable. # operating point) to "graduate" into earned auto-apply. Operator-tunable.
head_min_positives: Mapped[int] = mapped_column( head_min_positives: Mapped[int] = mapped_column(
Integer, nullable=False, default=8 Integer, nullable=False, default=8,
server_default="8",
) )
head_auto_apply_precision: Mapped[float] = mapped_column( head_auto_apply_precision: Mapped[float] = mapped_column(
Float, nullable=False, default=0.97 Float, nullable=False, default=0.97,
server_default="0.97",
) )
# Earned auto-apply (#114). A graduated head fires (tags images without a # Earned auto-apply (#114). A graduated head fires (tags images without a
# human) when this master switch is on AND the head has at least # human) when this master switch is on AND the head has at least
@@ -61,29 +70,34 @@ class MLSettings(Base):
# default (operator-asked 2026-06-29: opt-OUT, not opt-in); the support + # default (operator-asked 2026-06-29: opt-OUT, not opt-in); the support +
# measured-precision gates keep it safe, and every auto-tag is reversible. # measured-precision gates keep it safe, and every auto-tag is reversible.
head_auto_apply_enabled: Mapped[bool] = mapped_column( head_auto_apply_enabled: Mapped[bool] = mapped_column(
Boolean, nullable=False, default=True Boolean, nullable=False, default=True,
server_default="true",
) )
head_auto_apply_min_positives: Mapped[int] = mapped_column( head_auto_apply_min_positives: Mapped[int] = mapped_column(
# Support floor raised 30→50 (operator-asked 2026-07-06): a head needs # Support floor raised 30→50 (operator-asked 2026-07-06): a head needs
# more human labels before it may fire without a human. # more human labels before it may fire without a human.
Integer, nullable=False, default=50 Integer, nullable=False, default=50,
server_default="30",
) )
# CCIP character-match cosine cut (#114). 0.85 default — the v1 flat 0.75 # CCIP character-match cosine cut (#114). 0.85 default — the v1 flat 0.75
# over-fired (high-reference characters matched a scatter of images); 0.85 # over-fired (high-reference characters matched a scatter of images); 0.85
# keeps the confident single-character matches. Tunable from the agent card. # keeps the confident single-character matches. Tunable from the agent card.
ccip_match_threshold: Mapped[float] = mapped_column( ccip_match_threshold: Mapped[float] = mapped_column(
Float, nullable=False, default=0.85 Float, nullable=False, default=0.85,
server_default="0.85",
) )
# CCIP auto-apply (#114). Confident matches (>= ccip_auto_apply_threshold, # CCIP auto-apply (#114). Confident matches (>= ccip_auto_apply_threshold,
# above the suggest cut) auto-tag on a daily sweep. ON by default (opt-out); # above the suggest cut) auto-tag on a daily sweep. ON by default (opt-out);
# single-character references + the high bar keep it safe, every tag reversible. # single-character references + the high bar keep it safe, every tag reversible.
ccip_auto_apply_enabled: Mapped[bool] = mapped_column( ccip_auto_apply_enabled: Mapped[bool] = mapped_column(
Boolean, nullable=False, default=True Boolean, nullable=False, default=True,
server_default="true",
) )
ccip_auto_apply_threshold: Mapped[float] = mapped_column( ccip_auto_apply_threshold: Mapped[float] = mapped_column(
# Raised 0.92→0.95 (operator-asked 2026-07-06) so only very confident # Raised 0.92→0.95 (operator-asked 2026-07-06) so only very confident
# character matches auto-tag. # character matches auto-tag.
Float, nullable=False, default=0.95 Float, nullable=False, default=0.95,
server_default="0.92",
) )
# -- Presentation chrome auto-hide (#141) ------------------------------- # -- Presentation chrome auto-hide (#141) -------------------------------
# `banner` (chrome — clusters on UI, not content) auto-applies on the sweep # `banner` (chrome — clusters on UI, not content) auto-applies on the sweep
@@ -95,13 +109,21 @@ class MLSettings(Base):
# (opt-out); every auto-tag is reversible. NOTE (#1464): `wip` + `editor # (opt-out); every auto-tag is reversible. NOTE (#1464): `wip` + `editor
# screenshot` are no longer chrome — they went to the PROCESS path below. # screenshot` are no longer chrome — they went to the PROCESS path below.
presentation_auto_apply_enabled: Mapped[bool] = mapped_column( presentation_auto_apply_enabled: Mapped[bool] = mapped_column(
Boolean, nullable=False, default=True Boolean, nullable=False, default=True,
server_default="true",
) )
presentation_auto_apply_threshold: Mapped[float] = mapped_column( presentation_auto_apply_threshold: Mapped[float] = mapped_column(
Float, nullable=False, default=0.90 Float, nullable=False, default=0.90,
# text(), not a string, because alembic 0082 used sa.text(): a bare
# string renders DEFAULT '0.90'::double precision while text() renders
# DEFAULT 0.90, and the chain is MIXED — some migrations used one,
# some the other. Same value, different stored expression, so each
# column here mirrors whichever form its own migration used (#3275).
server_default=text("0.90"),
) )
presentation_conflict_threshold: Mapped[float] = mapped_column( presentation_conflict_threshold: Mapped[float] = mapped_column(
Float, nullable=False, default=0.50 Float, nullable=False, default=0.50,
server_default=text("0.50"),
) )
# -- Process auto-apply (#1464) ---------------------------------------- # -- Process auto-apply (#1464) ----------------------------------------
# `wip` / `editor screenshot` are PROCESS art — unfinished pieces + program # `wip` / `editor screenshot` are PROCESS art — unfinished pieces + program
@@ -115,24 +137,29 @@ class MLSettings(Base):
# (PresentationReview, mode='process') rather than silently marked. OFF by # (PresentationReview, mode='process') rather than silently marked. OFF by
# default — a new whole-library auto-tagger is opt-in; every auto-tag reversible. # default — a new whole-library auto-tagger is opt-in; every auto-tag reversible.
process_auto_apply_enabled: Mapped[bool] = mapped_column( process_auto_apply_enabled: Mapped[bool] = mapped_column(
Boolean, nullable=False, default=False Boolean, nullable=False, default=False,
server_default="false",
) )
process_auto_apply_threshold: Mapped[float] = mapped_column( process_auto_apply_threshold: Mapped[float] = mapped_column(
Float, nullable=False, default=0.90 Float, nullable=False, default=0.90,
server_default="0.90",
) )
process_conflict_threshold: Mapped[float] = mapped_column( process_conflict_threshold: Mapped[float] = mapped_column(
Float, nullable=False, default=0.50 Float, nullable=False, default=0.50,
server_default="0.50",
) )
# Default = SigLIP 2 (so400m, 512px) for new installs (migration 0069); # Default = SigLIP 2 (so400m, 512px) for new installs (migration 0069);
# existing libraries keep their stored value until the operator re-embeds. # existing libraries keep their stored value until the operator re-embeds.
embedder_model_version: Mapped[str] = mapped_column( embedder_model_version: Mapped[str] = mapped_column(
String(128), nullable=False, default="siglip2-so400m-patch16-512" String(128), nullable=False, default="siglip2-so400m-patch16-512",
server_default="siglip2-so400m-patch16-512",
) )
# The HF model NAME the embedder loads (server CPU embed + announced to the # The HF model NAME the embedder loads (server CPU embed + announced to the
# GPU agent in the lease). Operator-settable so the embedder is a choice, not # GPU agent in the lease). Operator-settable so the embedder is a choice, not
# a hardcode (#1190): set name + version together, then re-embed + retrain. # a hardcode (#1190): set name + version together, then re-embed + retrain.
embedder_model_name: Mapped[str] = mapped_column( embedder_model_name: Mapped[str] = mapped_column(
String(128), nullable=False, default="google/siglip2-so400m-patch16-512" String(128), nullable=False, default="google/siglip2-so400m-patch16-512",
server_default="google/siglip2-so400m-patch16-512",
) )
# -- Crop proposers / detectors (#1202, #134) -------------------------- # -- Crop proposers / detectors (#1202, #134) --------------------------
# WHERE-to-crop YOLO detectors feeding the crop→SigLIP bag + CCIP. Config # WHERE-to-crop YOLO detectors feeding the crop→SigLIP bag + CCIP. Config
@@ -145,20 +172,24 @@ class MLSettings(Base):
# person: general COCO figure detector for Western/realistic art the anime # person: general COCO figure detector for Western/realistic art the anime
# person-detector misses → NMS-merged with imgutils → CCIP + concept. # person-detector misses → NMS-merged with imgutils → CCIP + concept.
detector_person_enabled: Mapped[bool] = mapped_column( detector_person_enabled: Mapped[bool] = mapped_column(
Boolean, nullable=False, default=True Boolean, nullable=False, default=True,
server_default="true",
) )
detector_person_weights: Mapped[str] = mapped_column( detector_person_weights: Mapped[str] = mapped_column(
String(512), nullable=False, default="yolo11n.pt" String(512), nullable=False, default="yolo11n.pt",
server_default="yolo11n.pt",
) )
detector_person_conf: Mapped[float] = mapped_column( detector_person_conf: Mapped[float] = mapped_column(
Float, nullable=False, default=0.35 Float, nullable=False, default=0.35,
server_default=text("0.35"),
) )
# anatomy: booru_yolo anime/furry/NSFW torso components → concept crops. # anatomy: booru_yolo anime/furry/NSFW torso components → concept crops.
# Default = yolov11m_aa22 (26 classes, best mAP50-95 0.96), committed in the # Default = yolov11m_aa22 (26 classes, best mAP50-95 0.96), committed in the
# upstream repo so the URL resolves. License UNSTATED — fine for a private # upstream repo so the URL resolves. License UNSTATED — fine for a private
# homelab (operator accepted #1202). # homelab (operator accepted #1202).
detector_anatomy_enabled: Mapped[bool] = mapped_column( detector_anatomy_enabled: Mapped[bool] = mapped_column(
Boolean, nullable=False, default=True Boolean, nullable=False, default=True,
server_default="true",
) )
detector_anatomy_weights: Mapped[str] = mapped_column( detector_anatomy_weights: Mapped[str] = mapped_column(
String(512), nullable=False, String(512), nullable=False,
@@ -166,37 +197,47 @@ class MLSettings(Base):
"https://github.com/aperveyev/booru_yolo/raw/main/models/" "https://github.com/aperveyev/booru_yolo/raw/main/models/"
"yolov11m_aa22.pt" "yolov11m_aa22.pt"
), ),
server_default="https://github.com/aperveyev/booru_yolo/raw/main/models/yolov11m_aa22.pt",
) )
detector_anatomy_conf: Mapped[float] = mapped_column( detector_anatomy_conf: Mapped[float] = mapped_column(
Float, nullable=False, default=0.30 Float, nullable=False, default=0.30,
server_default=text("0.30"),
) )
# panel: comic page → panel regions → concept crops (Apache-2.0, YOLOv12x). # panel: comic page → panel regions → concept crops (Apache-2.0, YOLOv12x).
detector_panel_enabled: Mapped[bool] = mapped_column( detector_panel_enabled: Mapped[bool] = mapped_column(
Boolean, nullable=False, default=True Boolean, nullable=False, default=True,
server_default="true",
) )
detector_panel_weights: Mapped[str] = mapped_column( detector_panel_weights: Mapped[str] = mapped_column(
String(512), nullable=False, String(512), nullable=False,
default="mosesb/best-comic-panel-detection::best.pt", default="mosesb/best-comic-panel-detection::best.pt",
server_default="mosesb/best-comic-panel-detection::best.pt",
) )
detector_panel_conf: Mapped[float] = mapped_column( detector_panel_conf: Mapped[float] = mapped_column(
Float, nullable=False, default=0.30 Float, nullable=False, default=0.30,
server_default=text("0.30"),
) )
# Per-frame caps bound the crop→embed explosion; max_regions is the hard # Per-frame caps bound the crop→embed explosion; max_regions is the hard
# per-job backstop; dedupe_iou drops near-duplicate crops before the embed. # per-job backstop; dedupe_iou drops near-duplicate crops before the embed.
detector_max_figures: Mapped[int] = mapped_column( detector_max_figures: Mapped[int] = mapped_column(
Integer, nullable=False, default=8 Integer, nullable=False, default=8,
server_default="8",
) )
detector_max_components: Mapped[int] = mapped_column( detector_max_components: Mapped[int] = mapped_column(
Integer, nullable=False, default=8 Integer, nullable=False, default=8,
server_default="8",
) )
detector_max_panels: Mapped[int] = mapped_column( detector_max_panels: Mapped[int] = mapped_column(
Integer, nullable=False, default=8 Integer, nullable=False, default=8,
server_default="8",
) )
detector_max_regions: Mapped[int] = mapped_column( detector_max_regions: Mapped[int] = mapped_column(
Integer, nullable=False, default=128 Integer, nullable=False, default=128,
server_default="128",
) )
detector_dedupe_iou: Mapped[float] = mapped_column( detector_dedupe_iou: Mapped[float] = mapped_column(
Float, nullable=False, default=0.85 Float, nullable=False, default=0.85,
server_default=text("0.85"),
) )
# -- CCIP character prototypes (#1317) --------------------------------- # -- CCIP character prototypes (#1317) ---------------------------------
# The per-character reference set is precomputed + refreshed INCREMENTALLY # The per-character reference set is precomputed + refreshed INCREMENTALLY
@@ -208,7 +249,8 @@ class MLSettings(Base):
String(128), nullable=True String(128), nullable=True
) )
ccip_prototype_cap: Mapped[int] = mapped_column( ccip_prototype_cap: Mapped[int] = mapped_column(
Integer, nullable=False, default=64 Integer, nullable=False, default=64,
server_default="64",
) )
updated_at: Mapped[datetime] = mapped_column( updated_at: Mapped[datetime] = mapped_column(
DateTime(timezone=True), nullable=False, server_default=func.now() DateTime(timezone=True), nullable=False, server_default=func.now()
+1 -1
View File
@@ -35,7 +35,7 @@ class PatreonFailedMedia(Base):
ForeignKey("source.id", ondelete="CASCADE"), nullable=False, index=True ForeignKey("source.id", ondelete="CASCADE"), nullable=False, index=True
) )
filehash: Mapped[str] = mapped_column(String(128), nullable=False) filehash: Mapped[str] = mapped_column(String(128), nullable=False)
attempts: Mapped[int] = mapped_column(Integer, nullable=False, default=1) attempts: Mapped[int] = mapped_column(Integer, nullable=False, default=1, server_default="1")
last_error: Mapped[str | None] = mapped_column(Text, nullable=True) last_error: Mapped[str | None] = mapped_column(Text, nullable=True)
first_failed_at: Mapped[datetime] = mapped_column( first_failed_at: Mapped[datetime] = mapped_column(
DateTime(timezone=True), nullable=False, server_default=func.now() DateTime(timezone=True), nullable=False, server_default=func.now()
+1 -1
View File
@@ -35,7 +35,7 @@ class PixivFailedMedia(Base):
ForeignKey("source.id", ondelete="CASCADE"), nullable=False, index=True ForeignKey("source.id", ondelete="CASCADE"), nullable=False, index=True
) )
filehash: Mapped[str] = mapped_column(String(128), nullable=False) filehash: Mapped[str] = mapped_column(String(128), nullable=False)
attempts: Mapped[int] = mapped_column(Integer, nullable=False, default=1) attempts: Mapped[int] = mapped_column(Integer, nullable=False, default=1, server_default="1")
last_error: Mapped[str | None] = mapped_column(Text, nullable=True) last_error: Mapped[str | None] = mapped_column(Text, nullable=True)
first_failed_at: Mapped[datetime] = mapped_column( first_failed_at: Mapped[datetime] = mapped_column(
DateTime(timezone=True), nullable=False, server_default=func.now() DateTime(timezone=True), nullable=False, server_default=func.now()
+11 -1
View File
@@ -13,11 +13,13 @@ from sqlalchemy import (
CheckConstraint, CheckConstraint,
DateTime, DateTime,
ForeignKey, ForeignKey,
Index,
Integer, Integer,
String, String,
Text, Text,
UniqueConstraint, UniqueConstraint,
func, func,
text,
) )
from sqlalchemy.orm import Mapped, mapped_column from sqlalchemy.orm import Mapped, mapped_column
@@ -27,6 +29,10 @@ from .base import Base
class Post(Base): class Post(Base):
__tablename__ = "post" __tablename__ = "post"
__table_args__ = ( __table_args__ = (
# alembic 0030. The comment above described this index; nothing declared
# it, so autogenerate proposed dropping it (#3275).
Index("uq_post_artist_external_id_null_source", "artist_id", "external_post_id",
unique=True, postgresql_where=text("source_id IS NULL")),
# Source-bound dedup. Postgres treats NULL != NULL so rows # Source-bound dedup. Postgres treats NULL != NULL so rows
# with source_id IS NULL aren't deduped by this constraint; # with source_id IS NULL aren't deduped by this constraint;
# the partial unique index `uq_post_artist_external_id_null_source` # the partial unique index `uq_post_artist_external_id_null_source`
@@ -35,7 +41,11 @@ class Post(Base):
UniqueConstraint("source_id", "external_post_id", name="uq_post_source_external_id"), UniqueConstraint("source_id", "external_post_id", name="uq_post_source_external_id"),
CheckConstraint( CheckConstraint(
"translation_override IN ('auto', 'force', 'original')", "translation_override IN ('auto', 'force', 'original')",
name="ck_post_translation_override", # Bare name: Base.metadata's naming convention prepends
# ck_<table>_. Pre-prefixing it here doubles the prefix — see
# alembic 0088, which renames the four constraints that shipped
# that way (#3275).
name="translation_override",
), ),
) )
+9 -1
View File
@@ -11,7 +11,7 @@ are pruned by retention.
from datetime import datetime from datetime import datetime
from sqlalchemy import DateTime, Float, ForeignKey, String, func from sqlalchemy import DateTime, Float, ForeignKey, Index, String, func
from sqlalchemy.orm import Mapped, mapped_column from sqlalchemy.orm import Mapped, mapped_column
from .base import Base from .base import Base
@@ -20,6 +20,14 @@ from .base import Base
class PresentationReview(Base): class PresentationReview(Base):
__tablename__ = "presentation_review" __tablename__ = "presentation_review"
__table_args__ = (
Index("ix_presentation_review_resolved_at", "resolved_at"),
# Both FKs to tag were unindexed (#3300); tag_id CASCADEs, so a tag
# delete had to scan this table to find its rows.
Index("ix_presentation_review_tag_id", "tag_id"),
Index("ix_presentation_review_conflict_tag_id", "conflict_tag_id"),
)
image_record_id: Mapped[int] = mapped_column( image_record_id: Mapped[int] = mapped_column(
ForeignKey("image_record.id", ondelete="CASCADE"), primary_key=True ForeignKey("image_record.id", ondelete="CASCADE"), primary_key=True
) )
+22 -3
View File
@@ -16,7 +16,14 @@ title is the optional chapter name; stated_part is the optional operator-facing
from datetime import datetime from datetime import datetime
from sqlalchemy import DateTime, ForeignKey, Integer, Text, func from sqlalchemy import (
DateTime,
ForeignKey,
Integer,
Text,
UniqueConstraint,
func,
)
from sqlalchemy.orm import Mapped, mapped_column from sqlalchemy.orm import Mapped, mapped_column
from .base import Base from .base import Base
@@ -25,14 +32,26 @@ from .base import Base
class SeriesChapter(Base): class SeriesChapter(Base):
__tablename__ = "series_chapter" __tablename__ = "series_chapter"
__table_args__ = (
# alembic 0047 named the UNIQUE `uq_series_chapter_anchor_page`, not
# the `uq_series_chapter_anchor_page_id` a bare `unique=True` would
# render (#3275).
UniqueConstraint("anchor_page_id", name="uq_series_chapter_anchor_page"),
)
id: Mapped[int] = mapped_column(Integer, primary_key=True) id: Mapped[int] = mapped_column(Integer, primary_key=True)
series_tag_id: Mapped[int] = mapped_column( series_tag_id: Mapped[int] = mapped_column(
ForeignKey("tag.id", ondelete="CASCADE"), nullable=False, index=True ForeignKey("tag.id", ondelete="CASCADE"), nullable=False, index=True
) )
# Both the UNIQUE (above) and the FK carry the names 0047 gave them; the
# convention would render the FK `fk_series_chapter_anchor_page_id_series_page`.
anchor_page_id: Mapped[int] = mapped_column( anchor_page_id: Mapped[int] = mapped_column(
ForeignKey("series_page.id", ondelete="CASCADE"), ForeignKey(
"series_page.id",
ondelete="CASCADE",
name="fk_series_chapter_anchor_page",
),
nullable=False, nullable=False,
unique=True,
) )
title: Mapped[str | None] = mapped_column(Text, nullable=True) title: Mapped[str | None] = mapped_column(Text, nullable=True)
stated_part: Mapped[int | None] = mapped_column(Integer, nullable=True) stated_part: Mapped[int | None] = mapped_column(Integer, nullable=True)
+17 -2
View File
@@ -14,7 +14,14 @@ number parsed from the source post, nullable when unknown.
from datetime import datetime from datetime import datetime
from sqlalchemy import DateTime, ForeignKey, Integer, String, func from sqlalchemy import (
DateTime,
ForeignKey,
Integer,
String,
UniqueConstraint,
func,
)
from sqlalchemy.orm import Mapped, mapped_column from sqlalchemy.orm import Mapped, mapped_column
from .base import Base from .base import Base
@@ -23,14 +30,22 @@ from .base import Base
class SeriesPage(Base): class SeriesPage(Base):
__tablename__ = "series_page" __tablename__ = "series_page"
__table_args__ = (
# alembic 0005 named this `uq_series_page_image`; a bare `unique=True`
# on the column renders `uq_series_page_image_id` under the naming
# convention, which is a different object from the one the database
# has (#3275).
UniqueConstraint("image_id", name="uq_series_page_image"),
)
id: Mapped[int] = mapped_column(Integer, primary_key=True) id: Mapped[int] = mapped_column(Integer, primary_key=True)
series_tag_id: Mapped[int] = mapped_column( series_tag_id: Mapped[int] = mapped_column(
ForeignKey("tag.id", ondelete="CASCADE"), nullable=False, index=True ForeignKey("tag.id", ondelete="CASCADE"), nullable=False, index=True
) )
# UNIQUE lives in __table_args__ above, under the name 0005 gave it.
image_id: Mapped[int] = mapped_column( image_id: Mapped[int] = mapped_column(
ForeignKey("image_record.id", ondelete="CASCADE"), ForeignKey("image_record.id", ondelete="CASCADE"),
nullable=False, nullable=False,
unique=True,
) )
# 'placed' = in the series-global run (page_number set); 'pending' = staged # 'placed' = in the series-global run (page_number set); 'pending' = staged
# from a post awaiting the operator's sort (page_number NULL). (#789 P2) # from a post awaiting the operator's sort (page_number NULL). (#789 P2)
+26 -3
View File
@@ -5,7 +5,16 @@ Multiple sources per artist support creators with cross-platform presence.
from datetime import datetime from datetime import datetime
from sqlalchemy import JSON, Boolean, DateTime, ForeignKey, Integer, String, Text from sqlalchemy import (
JSON,
Boolean,
DateTime,
ForeignKey,
Integer,
String,
Text,
UniqueConstraint,
)
from sqlalchemy.orm import Mapped, mapped_column, relationship from sqlalchemy.orm import Mapped, mapped_column, relationship
from .base import Base from .base import Base
@@ -14,13 +23,27 @@ from .base import Base
class Source(Base): class Source(Base):
__tablename__ = "source" __tablename__ = "source"
__table_args__ = (
# alembic 0010. One row per (artist, platform, url): re-adding a source
# the artist already has is an update, not a second row. The model had
# never declared it (#3275), so autogenerate would have proposed
# DROPPING it — the guarantee existed only in the migration chain.
#
# Named explicitly because the naming convention would render this
# `uq_source_artist_id` (uq keys off column_0_name), which is both
# wrong about the shape and not what the database actually has.
UniqueConstraint(
"artist_id", "platform", "url", name="uq_source_artist_platform_url"
),
)
id: Mapped[int] = mapped_column(Integer, primary_key=True) id: Mapped[int] = mapped_column(Integer, primary_key=True)
artist_id: Mapped[int] = mapped_column( artist_id: Mapped[int] = mapped_column(
ForeignKey("artist.id", ondelete="CASCADE"), nullable=False, index=True ForeignKey("artist.id", ondelete="CASCADE"), nullable=False, index=True
) )
platform: Mapped[str] = mapped_column(String(64), nullable=False) platform: Mapped[str] = mapped_column(String(64), nullable=False)
url: Mapped[str] = mapped_column(Text, nullable=False) url: Mapped[str] = mapped_column(Text, nullable=False)
enabled: Mapped[bool] = mapped_column(Boolean, nullable=False, default=True) enabled: Mapped[bool] = mapped_column(Boolean, nullable=False, default=True, server_default="true")
config_overrides: Mapped[dict | None] = mapped_column(JSON, nullable=True) config_overrides: Mapped[dict | None] = mapped_column(JSON, nullable=True)
@@ -32,7 +55,7 @@ class Source(Base):
# by _update_source_health alongside last_error; cleared on 'ok'. # by _update_source_health alongside last_error; cleared on 'ok'.
error_type: Mapped[str | None] = mapped_column(String(32), nullable=True, index=True) error_type: Mapped[str | None] = mapped_column(String(32), nullable=True, index=True)
check_interval_override: Mapped[int | None] = mapped_column(Integer, nullable=True) check_interval_override: Mapped[int | None] = mapped_column(Integer, nullable=True)
consecutive_failures: Mapped[int] = mapped_column(Integer, nullable=False, default=0) consecutive_failures: Mapped[int] = mapped_column(Integer, nullable=False, default=0, server_default="0")
# alembic 0031: sticky deep-scan budget. When > 0, the next N download # alembic 0031: sticky deep-scan budget. When > 0, the next N download
# runs use gallery-dl's full-walk config (skip: True + 1800s timeout); # runs use gallery-dl's full-walk config (skip: True + 1800s timeout);
@@ -34,7 +34,7 @@ class SubscribeStarFailedMedia(Base):
ForeignKey("source.id", ondelete="CASCADE"), nullable=False, index=True ForeignKey("source.id", ondelete="CASCADE"), nullable=False, index=True
) )
filehash: Mapped[str] = mapped_column(String(128), nullable=False) filehash: Mapped[str] = mapped_column(String(128), nullable=False)
attempts: Mapped[int] = mapped_column(Integer, nullable=False, default=1) attempts: Mapped[int] = mapped_column(Integer, nullable=False, default=1, server_default="1")
last_error: Mapped[str | None] = mapped_column(Text, nullable=True) last_error: Mapped[str | None] = mapped_column(Text, nullable=True)
first_failed_at: Mapped[datetime] = mapped_column( first_failed_at: Mapped[datetime] = mapped_column(
DateTime(timezone=True), nullable=False, server_default=func.now() DateTime(timezone=True), nullable=False, server_default=func.now()
+19 -2
View File
@@ -15,11 +15,13 @@ from sqlalchemy import (
Column, Column,
DateTime, DateTime,
ForeignKey, ForeignKey,
Index,
Integer, Integer,
String, String,
Table, Table,
false, false,
func, func,
text,
) )
from sqlalchemy import ( from sqlalchemy import (
Enum as SQLEnum, Enum as SQLEnum,
@@ -67,17 +69,31 @@ image_tag = Table(
primary_key=True, primary_key=True,
), ),
Column("tag_id", ForeignKey("tag.id", ondelete="CASCADE"), primary_key=True), Column("tag_id", ForeignKey("tag.id", ondelete="CASCADE"), primary_key=True),
Column("source", String(32), nullable=False, default="manual"), Column("source", String(32), nullable=False, default="manual", server_default="manual"),
Column("created_at", DateTime(timezone=True), nullable=False, server_default=func.now()), Column("created_at", DateTime(timezone=True), nullable=False, server_default=func.now()),
# The PK is (image_record_id, tag_id), which leads with the WRONG column
# for the two things that matter most here (#3300): the gallery's tag
# filter (tag_query.py builds `image_tag.c.tag_id == tid`) and the
# ON DELETE CASCADE from tag, which has to find a tag's rows to remove
# them. Without this index both scan the largest table in the schema.
Index("ix_image_tag_tag_id", "tag_id"),
) )
class Tag(Base): class Tag(Base):
__tablename__ = "tag" __tablename__ = "tag"
__table_args__ = ( __table_args__ = (
# alembic 0002. An EXPRESSION index — COALESCE cannot be expressed as a
# UniqueConstraint, which is why it only ever existed in a migration (#3275).
Index("uq_tag_name_kind_fandom", "name", "kind", text("COALESCE(fandom_id, 0)"),
unique=True),
CheckConstraint( CheckConstraint(
"(fandom_id IS NULL) OR (kind = 'character')", "(fandom_id IS NULL) OR (kind = 'character')",
name="ck_tag_fandom_requires_character", # Bare name: Base.metadata's naming convention prepends
# ck_<table>_. Pre-prefixing it here doubles the prefix — see
# alembic 0088, which renames the four constraints that shipped
# that way (#3275).
name="fandom_requires_character",
), ),
) )
@@ -87,6 +103,7 @@ class Tag(Base):
SQLEnum(TagKind, name="tag_kind", values_callable=lambda e: [m.value for m in e]), SQLEnum(TagKind, name="tag_kind", values_callable=lambda e: [m.value for m in e]),
nullable=False, nullable=False,
default=TagKind.general, default=TagKind.general,
server_default="general",
) )
fandom_id: Mapped[int | None] = mapped_column( fandom_id: Mapped[int | None] = mapped_column(
ForeignKey("tag.id", ondelete="SET NULL"), nullable=True, index=True ForeignKey("tag.id", ondelete="SET NULL"), nullable=True, index=True
+9 -2
View File
@@ -5,7 +5,7 @@ in image_prediction stay unmolested.
from datetime import datetime from datetime import datetime
from sqlalchemy import DateTime, ForeignKey, String, func from sqlalchemy import DateTime, ForeignKey, Index, String, func
from sqlalchemy.orm import Mapped, mapped_column from sqlalchemy.orm import Mapped, mapped_column
from .base import Base from .base import Base
@@ -14,10 +14,17 @@ from .base import Base
class TagAlias(Base): class TagAlias(Base):
__tablename__ = "tag_alias" __tablename__ = "tag_alias"
__table_args__ = (
# Named explicitly: the database calls this ix_tag_alias_canonical, while
# a bare index=True on the column would generate ix_tag_alias_canonical_tag_id
# and silently propose a drop+create on the next autogenerate (#3275).
Index("ix_tag_alias_canonical", "canonical_tag_id"),
)
alias_string: Mapped[str] = mapped_column(String(255), primary_key=True) alias_string: Mapped[str] = mapped_column(String(255), primary_key=True)
alias_category: Mapped[str] = mapped_column(String(32), primary_key=True) alias_category: Mapped[str] = mapped_column(String(32), primary_key=True)
canonical_tag_id: Mapped[int] = mapped_column( canonical_tag_id: Mapped[int] = mapped_column(
ForeignKey("tag.id", ondelete="CASCADE"), nullable=False, index=True ForeignKey("tag.id", ondelete="CASCADE"), nullable=False
) )
created_at: Mapped[datetime] = mapped_column( created_at: Mapped[datetime] = mapped_column(
DateTime(timezone=True), nullable=False, server_default=func.now() DateTime(timezone=True), nullable=False, server_default=func.now()
+16 -3
View File
@@ -5,7 +5,7 @@ Prevents re-suggestion AND prevents allowlist auto-apply on that image.
from datetime import datetime from datetime import datetime
from sqlalchemy import DateTime, ForeignKey, func from sqlalchemy import DateTime, ForeignKey, Index, func
from sqlalchemy.orm import Mapped, mapped_column from sqlalchemy.orm import Mapped, mapped_column
from .base import Base from .base import Base
@@ -14,11 +14,24 @@ from .base import Base
class TagSuggestionRejection(Base): class TagSuggestionRejection(Base):
__tablename__ = "tag_suggestion_rejection" __tablename__ = "tag_suggestion_rejection"
__table_args__ = (
# Named explicitly; see tag_alias for why (#3275).
Index("ix_tag_suggestion_rejection_tag", "tag_id"),
)
# Both FKs named explicitly. alembic 0003 used a hand-shortened `tsr`
# prefix; the convention would render the full table name (#3275).
image_record_id: Mapped[int] = mapped_column( image_record_id: Mapped[int] = mapped_column(
ForeignKey("image_record.id", ondelete="CASCADE"), primary_key=True ForeignKey(
"image_record.id",
ondelete="CASCADE",
name="fk_tsr_image_record_id_image_record",
),
primary_key=True,
) )
tag_id: Mapped[int] = mapped_column( tag_id: Mapped[int] = mapped_column(
ForeignKey("tag.id", ondelete="CASCADE"), primary_key=True, index=True ForeignKey("tag.id", ondelete="CASCADE", name="fk_tsr_tag_id_tag"),
primary_key=True,
) )
rejected_at: Mapped[datetime] = mapped_column( rejected_at: Mapped[datetime] = mapped_column(
DateTime(timezone=True), nullable=False, server_default=func.now() DateTime(timezone=True), nullable=False, server_default=func.now()
+15 -4
View File
@@ -15,7 +15,7 @@ backend.app.tasks.maintenance.recover_stalled_task_runs (Beat 5 min).
from datetime import datetime from datetime import datetime
from sqlalchemy import DateTime, Integer, String, Text from sqlalchemy import DateTime, Index, Integer, String, Text, text
from sqlalchemy.orm import Mapped, mapped_column from sqlalchemy.orm import Mapped, mapped_column
from .base import Base from .base import Base
@@ -24,12 +24,21 @@ from .base import Base
class TaskRun(Base): class TaskRun(Base):
__tablename__ = "task_run" __tablename__ = "task_run"
__table_args__ = (
# alembic 0016: the three task-history indexes (#3275).
Index("ix_task_run_name_started", "task_name", text("started_at DESC")),
Index("ix_task_run_queue_started", "queue", text("started_at DESC")),
Index("ix_task_run_status_started", "status", text("started_at DESC")),
)
id: Mapped[int] = mapped_column(Integer, primary_key=True) id: Mapped[int] = mapped_column(Integer, primary_key=True)
celery_task_id: Mapped[str] = mapped_column( celery_task_id: Mapped[str] = mapped_column(
String(64), nullable=False, index=True, String(64), nullable=False, index=True,
) )
queue: Mapped[str] = mapped_column(String(32), nullable=False, index=True) # Neither carries index=True: ix_task_run_queue_started and
task_name: Mapped[str] = mapped_column(String(128), nullable=False, index=True) # ix_task_run_name_started already lead with these columns (#3301).
queue: Mapped[str] = mapped_column(String(32), nullable=False)
task_name: Mapped[str] = mapped_column(String(128), nullable=False)
target_id: Mapped[int | None] = mapped_column(Integer, nullable=True) target_id: Mapped[int | None] = mapped_column(Integer, nullable=True)
started_at: Mapped[datetime] = mapped_column( started_at: Mapped[datetime] = mapped_column(
DateTime(timezone=True), nullable=False, index=True, DateTime(timezone=True), nullable=False, index=True,
@@ -39,7 +48,9 @@ class TaskRun(Base):
) )
duration_ms: Mapped[int | None] = mapped_column(Integer, nullable=True) duration_ms: Mapped[int | None] = mapped_column(Integer, nullable=True)
status: Mapped[str] = mapped_column( status: Mapped[str] = mapped_column(
String(16), nullable=False, default="running", index=True, # No index=True — ix_task_run_status_started leads with `status`.
String(16), nullable=False, default="running",
server_default="running",
) )
error_type: Mapped[str | None] = mapped_column(String(128), nullable=True) error_type: Mapped[str | None] = mapped_column(String(128), nullable=True)
error_message: Mapped[str | None] = mapped_column(Text, nullable=True) error_message: Mapped[str | None] = mapped_column(Text, nullable=True)
+42
View File
@@ -167,6 +167,48 @@ per `docs/process.md`'s "add deps to the image when used by >1 project".
`github.event.inputs` into an env var rather than interpolated into a run `github.event.inputs` into an env var rather than interpolated into a run
block, and it is checked inside the reuse step so that one decision drives block, and it is checked inside the reuse step so that one decision drives
both the build and the repoint. both the build and the repoint.
- **A weekly `schedule` rebuilds all three images against fresh base layers**
(Sunday 06:00 UTC, milestone 326 step 4, #3154). Skip-if-exists is keyed on
OUR source, so an artifact whose source stops moving stops picking up base
updates — `agent/` has not changed since 2026-07-17 and would otherwise serve
that day's `nvidia/cuda` layers forever. Four things make it work:
- It **builds `main`, not the branch that triggered it.** Forgejo registers a
cron from the DEFAULT branch (`dev` here), so a scheduled run arrives with
`github.ref` on dev. The ref is decided once in a top-level `env:
BUILD_REF` that every checkout in the file takes, rather than per job —
otherwise `sign-extension` would derive dev's extension version while
`build-web` bundled main's, and the release download would 404 on a version
that exists perfectly well. Every job then ASSERTS its checkout is `main`
before doing anything, because `env` inside `with:` is not a context this
runner is known to evaluate — if it silently resolved to empty, checkout
would fall back to the triggering ref and the refresh would publish dev's
source to `:latest` with every lane green.
- It **publishes only `:latest`.** `:c-<sha>` for main's HEAD already names
the bytes that commit built; re-pushing it over refreshed layers would
break the one tag rule 145 makes immutable, and it is the rollback unit.
The repoint step needs no schedule case for this — the tag list is the
channel tag alone, so SOURCE is the only entry, it is excluded as always,
and the step correctly does nothing.
- **`:latest` and `:c-<sha>` therefore diverge between a refresh and the next
`main` push, by design.** They re-converge on that push: it hits reuse (a
refresh does not move `fc.revision`, because it does not touch the source),
and the repoint writes the NEW `:c-<sha>` from the refreshed `:latest`. The
push path needed no change for this, because the repoint already excluded
the source tag — the same rule that keeps the label readable also keeps a
refresh from being undone.
- **`pull: true` on the scheduled path only** is the mechanism: a moved base
tag changes the `FROM` layer's cache key and everything above it rebuilds.
**It does not currently make the unmoved case free.** Measured on the first
real fire (run 4934, 2026-08-30): every content step reported `CACHED` and
the bases resolved to unchanged digests, yet all three `:latest` tags got a
NEW manifest digest, because buildkit mints a fresh image config per run and
republishes identical layers under it. So `:latest` is rewritten weekly
whether or not anything changed, and `:c-<sha>` is handed a new manifest to
diverge from on the same cadence — a digest change stops meaning anything.
Tracked as #3265; the likely fix is a deterministic `SOURCE_DATE_EPOCH`.
Separately not caught: a Debian package update inside the `apt-get install`
layer while the base tag stands still — a lag rather than a hole, since the
official python/cuda images rebuild with those updates baked in.
- **`FC_CHANNEL` and `FC_VERSION` are build args, not runtime settings.** - **`FC_CHANNEL` and `FC_VERSION` are build args, not runtime settings.**
`build.yml` passes them to the web image only — the ml and agent images have `build.yml` passes them to the web image only — the ml and agent images have
nothing to report them to. `/api/health` returns both, the foot of Settings nothing to report them to. `/api/health` returns both, the foot of Settings