Compare commits

...
Author SHA1 Message Date
bvandeusen 2529b516e6 db: collapse alembic 0001..0087 into one baseline (milestone 328 step 1)
Build images / sign-extension (push) Successful in 4s
Build images / build-agent (push) Successful in 9s
CI / lint (push) Successful in 4s
CI / extension-version (push) Successful in 5s
CI / frontend-build (push) Successful in 24s
Build images / build-ml (push) Successful in 42s
CI / backend-lint-and-test (push) Successful in 53s
Build images / build-web (push) Successful in 33s
CI / integration (push) Failing after 3m47s
87 revisions narrating this project's build-out become one file that
creates the schema in a single step. They cost nothing at runtime — all
86 upgrade steps ran in 0.2s (note #3260) — so this is a presentation
change, not a performance one: a new installer should not inherit our
development history to stand up a database.

Deleted: 87 revisions (6,052 lines), the 10 tests/test_migration_*.py
files (483 lines) that asserted intermediate states and backfills which
no longer exist, and backend/app/utils/artist_backfill.py — the only
live module a migration imported, with no other consumer anywhere. That
last one satisfies the operator's separate request to inline it into
0008 and delete the module; the squash removes both outright.

THE REVISION ID IS "0087", NOT "0001", ON PURPOSE. It is the id of the
last revision collapsed, so an existing database is already at head and
`alembic upgrade head` does nothing. The alternative is `alembic stamp`
against live data, and stamp validates NOTHING — it writes a version
string whether or not the schema matches, so a wrong baseline surfaces
later, via the next real migration, with no clean way back. This removes
that operation rather than making it safe. Future revisions run from
0088.

Four things are hand-written because SQLAlchemy metadata does not carry
them, and none fail at generation time:

  1. CREATE EXTENSION vector          — the VECTOR columns cannot be
     created without it, so it is ordered first in upgrade().
  2. CREATE EXTENSION tsm_system_rows — surfaces only when the random
     sample query runs.
  3. the HNSW index on image_record.siglip_embedding, raw SQL because
     create_index cannot express USING hnsw (... vector_cosine_ops).
     The quietest of the four: everything works, similarity search just
     stops using an index.
  4. import pgvector.sqlalchemy.vector — autogenerate EMITS
     pgvector.sqlalchemy.vector.VECTOR references without importing it,
     so the generated file dies with NameError on first run.

The candidate came out of CI (run 4967) as checksummed base64 rather
than a plain cat, because run 4964's cat was truncated mid-line inside a
column definition with the step still green — 29 tables instead of 42,
and it looked entirely plausible. Verified here: 56,582 bytes,
sha256 471acfca69c0…, 42 tables, 66 indexes, 42 drops.

NOT YET PROVEN against the old chain. baseline.yml does that, and it is
step 2's gate; this commit does not claim the schemas match.
2026-08-30 13:45:35 -04:00
bvandeusen 8f1ac0c96a ci: transport the candidate baseline as verifiable base64
CI / integration (push) Successful in 3m48s
CI / lint (push) Successful in 4s
Build images / sign-extension (push) Successful in 5s
CI / extension-version (push) Successful in 5s
Build images / build-ml (push) Successful in 8s
Build images / build-agent (push) Successful in 9s
Build images / build-web (push) Successful in 7s
CI / frontend-build (push) Successful in 19s
CI / backend-lint-and-test (push) Successful in 45s
Run 4964 passed the control (1121 normalised lines, schemas identical)
but its candidate print was silently truncated. `cat` of the ~33KB
generated file stopped mid-line inside

  sa.Column('mime', sa.String(length=128)

and the runner carried straight on to the next traced command with the
step still green. The captured text was 484 lines and 29 tables, and
looked entirely plausible — which is exactly what makes it dangerous:
a schema definition cut in half is still syntactically suggestive, and
nothing in the log says it was cut.

Now emitted as base64 at a fixed 120-column width, followed by a
sha256, a byte count and a base64 line count. Short lines instead of
long ones, and more importantly the receiving end can PROVE it got the
whole file rather than trusting that it did.

Also found in that output, and the reason the candidate could never
have been committed as-is: it references

  pgvector.sqlalchemy.vector.VECTOR(dim=1152)

for head_training_run.weights and image_record.siglip_embedding, but
autogenerate does not add the corresponding import. The file would die
with NameError on the first run. That is the fourth item on the list of
things the generator cannot be trusted with, alongside the two CREATE
EXTENSIONs and the HNSW index.
2026-08-30 13:40:15 -04:00
bvandeusen 5fd171a544 ci: fix two things the baseline control run found
CI / lint (push) Successful in 3s
CI / extension-version (push) Successful in 4s
Build images / sign-extension (push) Successful in 4s
Build images / build-web (push) Successful in 6s
CI / integration (push) Successful in 3m52s
Build images / build-ml (push) Successful in 8s
Build images / build-agent (push) Successful in 8s
CI / frontend-build (push) Successful in 23s
CI / backend-lint-and-test (push) Successful in 32s
Run 4960 was the control — the chain compared against itself, which must
come back clean before a clean diff after the squash means anything. It
did its job and failed on both counts.

1. The harness is sound. Both dumps came back 1123 normalised lines and
   differed on EXACTLY two, the \restrict / \unrestrict pair that newer
   pg_dump emits to fence a dump against injection during restore. It is
   a fresh random nonce per invocation, so it differs by construction and
   is noise by definition. Now filtered — and the control is what
   licenses that filter: it was OBSERVED to be the only false positive
   rather than assumed to be one, which matters for a check whose whole
   value is that its normalisation does not hide a real difference.

2. The candidate-baseline step never ran. `if: github.event.inputs
   .generate == 'true'` on a `type: boolean` input silently evaluated
   false — no diagnostic, step skipped, job carried on. The same
   `github.event.inputs` typing quirk build.yml already works around for
   force_build.

   Rather than fight the input typing, the gate is now the tree itself:
   skip if alembic/versions holds one file. That is the real question
   anyway — there is nothing to generate once the chain is collapsed —
   and it cannot be silently wrong the way an unevaluated expression can.

Worth noting what the control also proved incidentally: the two schemas
were byte-identical across 1123 lines despite being built by separate
alembic runs into separate databases, so pg_dump's object ordering is
stable enough to diff directly and no sort normalisation is needed.
2026-08-30 13:35:26 -04:00
bvandeusen 62583791d8 ci: a workflow that proves a collapsed alembic chain matches the old one
Build images / sign-extension (push) Successful in 4s
CI / lint (push) Successful in 4s
CI / extension-version (push) Successful in 3s
Build images / build-agent (push) Successful in 9s
CI / frontend-build (push) Successful in 25s
Build images / build-ml (push) Successful in 7s
Build images / build-web (push) Successful in 8s
CI / backend-lint-and-test (push) Successful in 43s
CI / integration (push) Successful in 3m59s
Milestone 328 step 1 needs a baseline generated from the models, and
step 2 must not stamp the operator's live database until that baseline
is proven to reproduce what the 87-revision chain produced. `alembic
stamp` validates nothing, so an unproven baseline fails silently now and
loudly later, on real data.

There is no local Python environment and rules 10/12 point away from
standing one up, so the comparison runs in CI, where a pgvector Postgres
is already built from the chain on every integration run and nothing is
at risk.

It builds two databases and diffs their pg_dump --schema-only output:
one from `alembic upgrade head` on the revisions read out of git at
`chain_ref`, one from the current tree. Reading the chain from git via a
worktree — rather than from the working tree — is what keeps this usable
AFTER the old revisions are deleted, so it is the proof for step 1 and
the pre-flight for step 2 rather than a one-shot script.

Both sides use `alembic upgrade head`, never metadata.create_all, per
rule 82 — and that rule's reasoning is exactly the hazard here.
`create_all` emits plain CREATE TABLE and skips everything else, which is
why the optional autogenerated candidate CANNOT be trusted as the answer.
Three things in this schema are invisible to SQLAlchemy metadata:

  CREATE EXTENSION vector           (0001)
  CREATE EXTENSION tsm_system_rows  (0004)
  the HNSW index on image_record.siglip_embedding, raw SQL because
    alembic's create_index cannot express USING hnsw (...)   (0036)

plus any CHECK constraint or server_default a migration added without the
model declaring it — 4 model files declare CheckConstraints against 6
migrations that touch them. The candidate is a starting point to hand
finish; the diff is what proves nothing was missed.

Results are printed to the job log rather than uploaded: ci-requirements
records that this runner cannot do actions/upload-artifact@v4+, and the
repo dropped the action entirely in 2026-05.

Run it first with the chain still present, as a control — the diff
compares the chain against itself and must come back clean. A clean diff
after the squash only means something if the harness was shown to be
capable of producing one beforehand.

Temporary. Delete once the baseline is stamped.
2026-08-30 13:31:16 -04:00
bvandeusen 0a5bbe81dc docs: the scheduled refresh does NOT republish nothing (#3265)
Build images / sign-extension (push) Successful in 4s
CI / lint (push) Successful in 4s
CI / extension-version (push) Successful in 5s
Build images / build-agent (push) Successful in 8s
CI / integration (push) Successful in 3m58s
Build images / build-ml (push) Successful in 7s
Build images / build-web (push) Successful in 8s
CI / frontend-build (push) Successful in 20s
extension / lint (push) Successful in 27s
CI / backend-lint-and-test (push) Successful in 52s
Step 4 asserted that when the base has not moved the refresh is "a ~13s
no-op that republishes nothing", and that this no-op was the point. The
first half is false and was written without being tested.

Run 4934, the first real fire: every content step reported CACHED and
both bases resolved to unchanged pinned digests, yet all three :latest
tags took a new manifest digest anyway.

  fabledcurator        4ea5265ba017 -> 380e504de0fa
  fabledcurator-ml     6e7cfc0c09fd -> 6b2eefc301d8
  fabledcurator-agent  44920e0af1f3 -> 54accbeb52ed

buildkit mints a fresh image config per run, so identical layers get
republished under a new config blob. Storage cost is trivial; the cost
that matters is that a :latest digest change stops meaning "something is
different", and :c-<sha> is handed a new manifest to diverge from every
Sunday for no reason.

Corrects the workflow comment (x3) and ci-requirements.md to say what
actually happens. Filed as #3265 with the candidate fixes; the likely one
is a deterministic SOURCE_DATE_EPOCH off the value artifacts.sh already
derives, which would make "same source, same version" into "same source,
same bytes".

The rest of step 4 verified clean on the same run: the guard passed
(HEAD is main (499720d), `git checkout -B main`) — so this runner DOES
evaluate the env context inside `with:` — the tag list was :latest alone
with no :c-<sha>, and the repoint step correctly found nothing to write.
2026-08-30 12:57:06 -04:00
bvandeusen 6663e06aa6 ci: assert the scheduled refresh actually checked out main
CI / extension-version (push) Successful in 5s
CI / lint (push) Successful in 6s
Build images / build-ml (push) Successful in 9s
Build images / build-web (push) Successful in 6s
CI / frontend-build (push) Successful in 18s
extension / lint (push) Successful in 19s
Build images / sign-extension (push) Successful in 6s
Build images / build-agent (push) Successful in 11s
CI / backend-lint-and-test (push) Successful in 39s
CI / integration (push) Successful in 3m48s
BUILD_REF is read through the `env` context inside `with:`, which this
runner is not known to evaluate. `${{ steps.* }}` and `${{ secrets.* }}`
in `with:`/`env:` are proven here; `env` is not, and run 4915's checkout
log (`git checkout -B dev refs/remotes/origin/dev`) cannot tell an
honoured `refs/heads/dev` from an empty value falling back to the same
place — the two are indistinguishable on every path except the one that
matters.

If it does resolve empty, the weekly refresh checks out dev and pushes
its source to :latest, which is production. Every lane stays green and
the first symptom is production running code that was never merged.

So each of the four jobs now asserts its own checkout before doing
anything, gated on `github.event_name` — the `github` context is
demonstrably evaluated in `if:`, so the guard cannot be disabled by the
same uncertainty it covers. A red weekly job is an acceptable outcome;
shipping dev to production is not.
2026-08-29 22:57:41 -04:00
bvandeusen 63e0a423d7 ci: a weekly base-image refresh on the channel tags (milestone 326 step 4)
Build images / sign-extension (push) Successful in 4s
CI / extension-version (push) Successful in 3s
CI / lint (push) Successful in 3s
Build images / build-web (push) Successful in 6s
extension / lint (push) Successful in 20s
Build images / build-agent (push) Successful in 7s
Build images / build-ml (push) Successful in 8s
CI / frontend-build (push) Successful in 21s
CI / backend-lint-and-test (push) Successful in 31s
CI / integration (push) Successful in 3m50s
Skip-if-exists is keyed on our own source, so an artifact whose source
stops moving stops picking up base-image updates. `agent/` last changed
2026-07-17; every push since has correctly declined to rebuild it, which
also means it will serve that day's nvidia/cuda layers indefinitely.

A `schedule:` trigger, Sunday 06:00 UTC, away from CI-runner's Monday
security sweep so the two are never diagnosing each other.

#3154's blocking open question is dissolved rather than answered. It was
written when the identity was a `r-<revision>` TAG, and asked how the
next ordinary push could avoid repointing :latest back off the refresh.
Milestone 318 replaced that tag with a LABEL, and #3183 made the repoint
step exclude its source tag so the label stays readable. Excluding the
source is what also keeps a refresh from being undone: on the next main
push the reuse check hits, :latest is not rewritten, and the new :c-<sha>
is written FROM the refreshed :latest. To be verified by digest, not by
this argument.

Four decisions, each commented where it lives:

* It builds `main`, not the branch that triggered it. Forgejo registers a
  cron from the default branch — `dev` here — so a scheduled run arrives
  with github.ref on dev, and a refresh of :dev would be refreshing the
  one channel that is rebuilt constantly anyway. The ref is decided once
  in a top-level `env: BUILD_REF` that all four checkouts take. Deriving
  it per job would let the halves disagree: sign-extension would derive
  dev's extension version while build-web bundled main's, and the release
  download would 404 on a version that exists perfectly well.

* It publishes only the channel tag. :c-<sha> for main's HEAD already
  names the bytes that commit built; re-pushing it over refreshed layers
  would break the one tag rule 145 makes immutable, and it is the
  rollback unit — so the breakage would surface on the day somebody
  needed it. The repoint step needs no schedule case: the tag list is the
  channel tag alone, SOURCE is the only entry, it is excluded as always,
  and the step correctly does nothing.

* It bypasses reuse by construction, since it rebuilds the same source
  and fc.revision always matches. Checked in the reuse step beside
  force_build, so one decision still drives both the build and the
  repoint.

* `pull: true`, on the scheduled path only, is the actual mechanism. A
  moved base tag changes the FROM layer's cache key and everything above
  it rebuilds; an unmoved one is satisfied by the registry cache and the
  refresh is a ~13s no-op that republishes nothing. That no-op is the
  point — :latest should change when there is something new in it, not
  every Sunday. The known lag, left deliberately: an apt package update
  while the base tag stands still is not caught, and closing it needs
  no-cache: true, which buys weekly churn for it.
2026-08-29 22:53:48 -04:00
bvandeusenandClaude Opus 5 5e72076298 ci: registry-backed layer cache for all three images (milestone 326 step 2)
Build images / sign-extension (push) Successful in 4s
CI / lint (push) Successful in 5s
CI / extension-version (push) Successful in 5s
Build images / build-ml (push) Successful in 8s
Build images / build-agent (push) Successful in 9s
extension / lint (push) Successful in 20s
Build images / build-web (push) Successful in 6s
CI / frontend-build (push) Successful in 17s
CI / backend-lint-and-test (push) Successful in 33s
CI / integration (push) Successful in 3m44s
extension / lint (pull_request) Successful in 29s
`cache-from`/`cache-to` on `<image>:buildcache`, `mode=max`, on all three
build steps. Closes the half of the driver change that step 1 left open.

Step 1 measured worse, not better, and that was expected but is worth stating
with numbers. Run 4896, first builds after moving to `docker-container`:

    build-web    3m44s   (cold baseline 2m23s)
    build-ml     3m49s   (cold baseline 3m20s)
    build-agent  11m12s  (cold baseline 9m26s)

The container driver gets a FRESH buildkit instance per job, so it has no
local layer store to fall back on — where the old docker driver at least
reused whatever the runner's dockerd happened to hold. That is why a registry
cache is the only cache this driver can have, and why step 1 on its own is a
regression rather than a win.

`mode=max` so intermediate stages cache too. The agent's two ~150s pip layers
and web's frontend-builder stage are the entire cost, and a min-mode cache
would drop exactly those.

A `:buildcache` tag is not the withdrawn tag scheme returning. Rule 145
narrowed against names NOTHING reads; this one is read by every build that
runs, is one moving ref per image rather than one per build, holds cache blobs
rather than a shippable artifact, and is overwritten in place rather than
accumulating. Closer to `:dev` than to the `:2026.8.28` tags 318 deleted — and
said so in the workflow, so it is not "cleaned up" by a later reader.

Expect the next build to be slower again, once: it is still cold AND now pays
the cache export. The measurement that matters is the one after that.

Scribe #3114.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-29 18:56:13 -04:00
bvandeusenandClaude Opus 5 e21c9fdd34 ci: a force_build escape hatch for the path skip-if-exists hides (326 step 3)
CI / lint (push) Successful in 3s
CI / extension-version (push) Successful in 3s
Build images / sign-extension (push) Successful in 3s
Build images / build-ml (push) Successful in 7s
Build images / build-agent (push) Successful in 6s
CI / frontend-build (push) Successful in 20s
CI / backend-lint-and-test (push) Successful in 29s
Build images / build-web (push) Successful in 7s
extension / lint (push) Successful in 18s
CI / integration (push) Successful in 3m47s
`workflow_dispatch` with a `force_build` boolean, honoured inside each of the
three reuse steps.

It exists because skip-if-exists made its own build path untestable. `agent/`
has not changed since 2026-07-17, so the agent build has correctly declined to
run on every push since — which also means #3190, whose whole symptom lives on
that path, cannot be reproduced on demand. Editing build.yml does not force a
build either, and that is deliberate: the workflow is not shipped bytes, so it
is in no artifact's path set, and putting it in one would re-version every
artifact for a comment change.

That is also why this lands before step 2 rather than after. Step 1 moved the
builds onto a container driver and turned attestations off; the claim that
`fc.revision` still reads back cannot be checked until something actually
builds under that driver. Run 4887 confirmed only the cheaper half — `Set up
buildx` succeeded on all three jobs, so the buildkit sibling container does
start against the mounted socket.

Details worth keeping:

* FORCE is checked in the reuse step, not in the build step's `if:`. The
  repoint step keys off `hit` too, and a force that bypassed only the build
  would leave the two disagreeing about what had happened.
* `github.event.inputs`, not the `inputs` context — release.yml already uses
  that form and it is the one this runner is known to evaluate. Read through
  env rather than interpolated into the run block, same as release.yml's TAG.
* One input, not one per artifact. Three booleans is an interface nobody
  remembers.

Scribe #3252, #3249.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-29 18:38:29 -04:00
bvandeusenandClaude Opus 5 6b3ec98fa8 ci: build on a real buildx driver, attestations off (milestone 326 step 1)
CI / lint (push) Successful in 3s
Build images / sign-extension (push) Successful in 3s
CI / extension-version (push) Successful in 4s
Build images / build-ml (push) Successful in 33s
CI / backend-lint-and-test (push) Successful in 31s
CI / frontend-build (push) Successful in 29s
Build images / build-agent (push) Successful in 34s
Build images / build-web (push) Successful in 7s
extension / lint (push) Successful in 24s
CI / integration (push) Successful in 3m52s
Adds `docker/setup-buildx-action@v3` to build-web, build-ml and build-agent,
and sets `provenance: false` / `sbom: false` on all three build-push steps.

Two open issues share one root, which is why this is one change:

* #3114 — the agent rebuilds a ~6.3 GB CUDA + torch image whenever the
  runner's local cache is cold, 9m26s against 7s warm. The default `docker`
  driver cannot export a registry cache at all, so the fix is unavailable
  until the driver moves. The cache itself is step 2, deliberately not here.
* #3190 — build-agent goes red AFTER a successful push, `No such image` from
  the local daemon. The leading candidate is the docker driver resolving
  image metadata against a local store a registry-direct push never filled.

The attestation flags are the load-bearing part. On the default driver they
were no-ops; on the container driver, build-push-action@v5 defaults provenance
to TRUE when pushing. Provenance attaches an attestation manifest, that makes
the pushed tag a manifest INDEX, and `.Image.Config.Labels` does not resolve
through an index — so the `fc.revision` label the reuse check reads off the
channel tag would come back `<none>` on every push. Every image would rebuild
forever, every lane would stay green, and the only symptom would be the bill.
Same failure as #3183, through a different door; note #3127 §4 records the
same shape for `platforms:`.

Unverified until CI says otherwise: these jobs run INSIDE a container against
a mounted docker socket, so the buildkit container is a sibling of the job
container rather than a child. That works over a socket mount and has never
been tried on this runner.

The gate is the SECOND dev push, not this one. The images currently published
were built by the old driver, so one `reuse: NOTE ... no readable fc.revision`
is expected now; what must not appear is a second one.

Scribe #3249.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-29 18:36:35 -04:00
bvandeusenandClaude Opus 5 1a941e900b test: encode the extension's AMO rendering exception (milestone 318 step 8)
Build images / sign-extension (push) Successful in 4s
Build images / build-agent (push) Successful in 4s
Build images / build-ml (push) Successful in 5s
CI / lint (push) Successful in 2s
Build images / build-web (push) Successful in 3s
CI / extension-version (push) Successful in 3s
CI / frontend-build (push) Successful in 22s
CI / backend-lint-and-test (push) Successful in 31s
CI / integration (push) Successful in 3m40s
extension / lint (pull_request) Successful in 21s
Missed in 2e01242. This module pinned zero-padded `YYYY.MM.DD.HHMM` for all
four artifacts, which is the family shape and was right until the extension
acquired a documented reason not to use it. Both assertions failed exactly as
written, on the value they were written to catch.

Rather than exempt the extension, the exception is pinned to the constraint
that justifies it:

* `test_version_is_zero_padded_calver` now covers the three padded artifacts.
* A new sibling covers the unpadded one against **AMO's own grammar** —
  `2026.08.29.0201` fails it, so a regression to padding fires immediately.
  Matching only `YYYY.M.D.HHMM` would not: on a date with no leading zeros the
  two renderings are the same string, so a padding regression would sit unseen
  until the first single-digit month, and surface as a burned AMO version
  rather than a red lane.
* `test_version_and_revision_describe_the_same_commit` compares NUMBERS, per
  rule 148's own definition of comparison — so one assertion covers both
  renderings and says the real thing: whatever the padding, the extension must
  denote exactly the value its own commit stamps. Exact-string equality is
  still asserted for everything not in AMO_UNPADDED, so the exception cannot
  quietly spread.

Scribe #3138.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-29 13:46:18 -04:00
bvandeusenandClaude Opus 5 2e01242381 feat(extension): derive the version as unpadded CalVer (milestone 318 step 8)
Build images / build-ml (push) Successful in 4s
CI / lint (push) Successful in 4s
Build images / build-agent (push) Successful in 5s
CI / extension-version (push) Successful in 4s
CI / frontend-build (push) Successful in 22s
extension / lint (push) Successful in 22s
CI / backend-lint-and-test (push) Failing after 33s
Build images / sign-extension (push) Successful in 2m24s
Build images / build-web (push) Successful in 2m38s
CI / integration (push) Successful in 5m15s
`1.0.<minutes since 2020>` -> `YYYY.M.D.HHMM` UTC, from the commit time of
the newest change to a packaged extension file. Same clock and same commit as
before; readable instead of opaque, and the same value the rest of the family
derives.

The hold on this step was two questions about AMO, and Mozilla's own docs
answer both:

    ^(0|[1-9][0-9]{0,8})([.](0|[1-9][0-9]{0,8})){0,3}$

  1. four all-numeric segments -> ACCEPTED ({0,3} more after the first).
  2. leading zeros              -> REJECTED. A segment is the single digit
     `0` or starts 1-9, so `08` and `0201` are refused. MDN says it in prose
     too: "Non-zero numbers must not include a leading zero."

So the documented fallback applies, extension only: the same numbers rendered
without the family's zero-padding. `2026.08.29.0201` and `2026.8.29.201` are
one value in two renderings — rule 148 defines comparison as numeric per
segment, under which they are equal — so nothing already published is
reordered, and left-padding each segment recovers the family string exactly.
HHMM stays one segment because AMO allows at most four.

The transition is safe in the other direction too: 2026 > 1, so every CalVer
outranks every published 1.0.x. build.yml's downgrade guard confirms it.

Also in scope:

* MAJOR.MINOR is gone. `cmd_major_minor`, `cmd_patch` and VERSION_EPOCH go
  with it, the committed version in manifest.json / package.json is now
  wholly inert, and ci.yml's MAJOR.MINOR-agreement check is retired rather
  than left running beside a fact that stopped existing (rule 22).
* ci.yml's `extension-version` lane now asserts Mozilla's regex verbatim
  instead of a loose `^[0-9]+(\.[0-9]+)*$` — which would have passed the
  padded shape. It also asserts YYYY.M.D.HHMM, because AMO would accept a
  regression to `1.0.<minutes>` while that orders below everything signed
  since. Checking here is the point: AMO 409s on re-signing, so a version it
  rejects is burned and cannot be reused.
* `artifacts.sh version extension` delegates to packaging.sh, so the two
  cannot answer differently. The direction matches the existing one —
  artifacts.sh already asks packaging.sh for the extension's path set.

#3156 is what makes this commit safe to make: packaging.sh is in web's path
set, so the web revision moves with the extension version and build-web
rebuilds instead of republishing an image bundling the previous XPI.

Scribe #3138.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-29 13:43:30 -04:00
bvandeusen 41f2bec3af fix(ci): the build pushes one tag; the rest are written registry-side (#3190)
CI / lint (push) Successful in 3s
Build images / sign-extension (push) Successful in 4s
Build images / build-ml (push) Successful in 4s
CI / extension-version (push) Successful in 3s
Build images / build-web (push) Successful in 6s
Build images / build-agent (push) Successful in 5s
CI / frontend-build (push) Successful in 23s
extension / lint (push) Successful in 28s
CI / backend-lint-and-test (push) Successful in 46s
CI / integration (push) Successful in 3m56s
buildx on this runner pushes the first tag to the registry and then
re-pushes the remaining ones through the DOCKER driver, reading them out
of a local image store that a registry-direct build never populated:

  #27 pushing …/fabledcurator:latest           DONE 15.8s
  #28 pushing …/fabledcurator:c-0e15c44 with docker
  #28 ERROR: tag does not exist: …:c-0e15c44

It is intermittent — build-ml made the identical two-tag push seconds
later in the same run and succeeded — and the consequence is worse than
the red job suggests. `:latest` had already published, so production was
correct while the immutable rollback tag rule 145 requires of every main
push simply did not exist. Nothing else would ever have noticed: a
missing :c-<sha> has no consumer that fails, so it surfaces at the moment
somebody needs to roll back, which is the worst time to learn a rollback
target was never written.

So the build now pushes exactly one ref — the channel's — and the
existing repoint step, which already excluded the source tag and already
ran on every reuse, now runs on the build path too and owns every other
tag. `imagetools create` is a registry-side manifest copy: no local
daemon, nothing that can be absent. This adds no new code path; it puts
the build case onto the one that was already proven.

Chosen over the alternative of asserting each tag resolves after the
build, which would have made the failure loud without making it rarer.

The cost, accepted: `imagetools create` wraps its source in an index, so
:c-<sha> is an index rather than a plain image and fc.revision does not
resolve through it. Nothing reads that label off :c-<sha> — the reuse
check only ever inspects the CHANNEL tag — and the index names the same
manifest, so a pull is byte-identical. The reuse path already produced
:c-<sha> this way; this only makes it uniform.

`build_tags` goes with it — the tag list now has exactly one consumer.
2026-08-29 00:38:26 -04:00
bvandeusen d38585ed94 docs: true up the tag scheme against what the pipeline publishes (318 step 9)
CI / lint (push) Successful in 3s
Build images / sign-extension (push) Successful in 4s
Build images / build-ml (push) Successful in 4s
CI / extension-version (push) Successful in 3s
Build images / build-agent (push) Successful in 4s
Build images / build-web (push) Successful in 5s
CI / frontend-build (push) Successful in 22s
extension / lint (push) Successful in 30s
CI / backend-lint-and-test (push) Successful in 48s
CI / integration (push) Successful in 3m57s
extension / lint (pull_request) Successful in 28s
A doc describing a tag scheme the pipeline stopped using is worse than no
doc — it is a confident wrong answer, and #3159 named it as the drift that
had just bitten rule 148.

build.yml (each edit ×3, once per build job):

* The derived-values block claimed three values including a `tag`
  subcommand that no longer exists, and said "on dev the date tag is
  computed nowhere else". There are two values and no date tag.
* The BASE-IMAGE FRESHNESS paragraph argued from a pin that no longer
  exists — "a date tag has to keep serving the bytes it served
  (fabledcurator:2026.7.17 still resolves to July's image)". Milestone 318
  removed the argument's premise rather than answering it: with no version
  tags there is no immutable name a refresh could contradict, and rule 145
  already allows a moving tag to be republished. Deleted the argument,
  kept a line saying why there is nothing to argue about (#3159's phrasing:
  the workflow becomes obviously correct instead of carefully correct).
* The repoint comment said ":c-<sha> and the date pin". There is no date pin.
* "its set is too narrow — the direction that serves stale bytes on a pin"
  now names the failure that actually exists: the reuse check hits and the
  channel serves a web image bundling the previous XPI (#3156).

README gains a "Versions and tags" section — the three tags, why there is
no fourth, and the fact that with no registry name carrying it, the
instance's own report is the only answer to "which build is this?". Also
fixes two stale claims: three workflows (four since release.yml) and "AMO
signing runs on main only", which has been false since milestone 271
step 6 signed on both channels.

ci-requirements.md was missing milestones 313 and 318 entirely: no
artifacts.sh, no label-keyed reuse, no FC_VERSION. Added those, plus the
#3156/#3202 membership test (can changing this file change the published
bytes — not is it copied in), and corrected packaging.sh's "two consumers"
to three now that artifacts.sh reads it.

docker-compose.yml needed nothing — it names `:dev` directly and never
offered a pin target.

Note #3127 and rule 145's relation notes are updated in Scribe.
2026-08-29 00:09:52 -04:00
bvandeusen a3071a7549 fix(ci): the file that decides FC_VERSION now moves it (#3202)
CI / lint (push) Successful in 3s
Build images / sign-extension (push) Successful in 4s
CI / extension-version (push) Successful in 4s
CI / frontend-build (push) Successful in 29s
CI / backend-lint-and-test (push) Successful in 1m5s
Build images / build-ml (push) Successful in 5s
Build images / build-agent (push) Successful in 6s
Build images / build-web (push) Successful in 3m15s
CI / integration (push) Successful in 4m29s
`scripts/artifacts.sh` decides both values the web image carries — the
`fc.revision` label the reuse check compares and the `FC_VERSION` baked into
the image — and was in no artifact's path set. So a change to `cmd_version`
alone left every revision untouched, the reuse check hit, the build was
skipped, and the published image went on reporting the OLD version format,
indefinitely, until some unrelated commit forced a rebuild. Nothing goes red;
the footer just shows a well-formed string of the wrong shape.

Milestone 318 step 5 is the worked instance:

  b3989d0 -> rev=fb2c4d5b80be  ver=2026.8.28.1249
  5771fd5 -> rev=fb2c4d5b80be  ver=2026.08.28.1249
  bce894b -> rev=bce894ba2499  ver=2026.08.28.2208

Same revision across the zero-pad commit, so web's build was skipped. It cost
nothing only by timing: FC_VERSION did not exist until step 6 landed one
commit later.

Web only, and that is the interesting part. Every artifact stamps a revision,
but only web also stamps a version. A revision-only artifact needs no entry
here, because changing how a revision is COMPUTED changes the derived value,
which then disagrees with the label on the published image and forces a
rebuild — the mechanism self-corrects, since it compares against a string
stamped into a real artifact. Nothing compares a version to anything. That
asymmetry is why this was invisible and is now written down in both files.

Named as a file rather than `scripts`: release_notes.py sits beside it and
only reads derived values, so it decides nothing and must not re-version web.

This is #3156 one level up — packaging.sh excluded from the version it
derives — so the guard is generalised rather than duplicated: one DERIVERS
table naming each deriver and the artifacts whose identity it decides. The
too-wide test gains a note saying where the line is, since "copied into no
image" no longer settles it on its own.
2026-08-28 22:01:58 -04:00
bvandeusen b6b9fd8287 ci: a release publishes a changelog, not an image (318 step 7)
CI / extension-version (push) Successful in 3s
CI / lint (push) Successful in 3s
Build images / build-ml (push) Successful in 4s
extension / lint (push) Successful in 25s
CI / backend-lint-and-test (push) Successful in 30s
CI / integration (push) Successful in 3m52s
Build images / sign-extension (push) Successful in 4s
Build images / build-agent (push) Successful in 5s
Build images / build-web (push) Successful in 4s
CI / frontend-build (push) Successful in 18s
Step 2 took the build consequence away from a `v*` tag — `main` has already
built and published the commit by the time anyone tags it, and rebuilding
would re-push `:c-<sha>`, which rule 145 forbids even when the source
matches. That left the tag with nothing to do at all.

This is the job it has instead. Step 6 put the derived version in the
Settings footer, so an operator can say WHICH build they are running; this
says what is in it that was not in the one they ran last month. Both halves
of one question (note #3127 §5).

The previous release is found by walking ANCESTRY, not by sorting a list.
That is load-bearing here specifically: rule 148 moved the tag shape from
`v26.05.22.0` to `v2026.08.28.2208`, and lexicographically `v2026...` sorts
BEFORE `v26...` — the third character is `0` against `6`. A sorted
implementation would reach back past every new-shape tag to the newest
old-shape one and publish months of commits as "changes since", looking
entirely correct while doing it. `git describe --exclude` is immune to the
shape change, and reachability is the more honest question anyway.

The publisher GETs and PATCHes rather than POSTing and recovering the id
from a 409 — note #3127 §6.7, which is ThoughtSync #2182's bug. A `v*` tag
is created once so the conflict path is rare, but "rare" is how that one
survived to be found somewhere else.

Cross-checks are reported on the release, not enforced. The tag is already
pushed by the time this runs, so failing would leave the operator with a tag,
no release, and a red lane to explain it — while the release is still the
useful object. It says so at the top when the tag names a version the web
image does not report, or when the commit is not on `main` and the `:c-`
rollback refs it lists were never published.

Nothing runs on a schedule and nothing auto-tags on merge. Release tags are
bookmarks (note #3127 §0); FC went twelve weeks without one and nothing was
wrong.

Also here:
- `scripts/` joins the ruff lane. release_notes.py runs only on a tag push,
  so a syntax error there would otherwise surface at the one moment nobody
  wants to be debugging a workflow.
- version.spec.js reads the workflow directory instead of listing three
  files by hand. Its own comment says the assertion should survive consumers
  coming and going; the hardcoded list was the part that could not, and
  release.yml would have joined the directory without joining the check.

Tests build a synthetic history spanning the tag-shape change rather than
leaning on this repo's tags, so the span assertion holds whether or not a
checkout brought the tags along — a span test that quietly skips is worse
than one that fails.
2026-08-28 20:57:25 -04:00
bvandeusen bce894ba24 feat(settings): the instance reports which build it is (318 step 6)
CI / lint (push) Successful in 4s
Build images / sign-extension (push) Successful in 4s
CI / extension-version (push) Successful in 4s
Build images / build-agent (push) Successful in 4s
CI / frontend-build (push) Successful in 21s
extension / lint (push) Successful in 25s
CI / backend-lint-and-test (push) Successful in 32s
Build images / build-ml (push) Successful in 2m50s
Build images / build-web (push) Successful in 2m49s
CI / integration (push) Successful in 3m52s
A dim line at the foot of Settings: `FabledCurator 2026.08.28.1249 · dev`.

This is no longer a convenience. Milestone 318 stopped publishing version
image tags, so an instance's own report is the ONLY answer to "which build is
this?" — there is no registry name left to check it against. Note #3127 §5
says it directly: a wrong answer here has no second source to contradict it.

Three states, kept distinct because collapsing any two of them lies:

  not asked yet         render nothing
  asked, no version     render "unknown"
  asked, has a version  render it

A blank footer reads as "no version", which is a different claim from "I
cannot say". And a failed health call deliberately does NOT mark the build
loaded — a network blip says nothing about the image, and presenting it as
"unknown" would look like a defective build.

Carried on /api/health rather than a new route: it answers at the same cost
(two module constants, no I/O) and TopNav already fetches it app-wide, so a
separate endpoint would mean a second request for two strings.

Both fields are OMITTED when unset rather than sent empty. Absence already
means "cannot say" — an image predating the field says exactly that by not
having the key — so a second spelling would make every reader special-case
it. The pre-existing test asserting the body is EXACTLY {"status": "ok"} is
what keeps a well-meaning `or ""` default from creeping in.

FC_CHANNEL now has one definition. It was read from the environment in
extension.py and would have been read again here; the new build_info module
holds both, and extension.py binds it as a module-level name so existing
tests monkeypatch it exactly as before. Separate from config.py on purpose:
those are operator settings meant to be changed, these describe the artifact.

Channel sits beside the version, never inside it (rule 149), asserted from
both ends. A `-dev` suffix would read as a 0 segment to the extension's
parseInt comparator and make every dev build compare equal — #2993 exactly.

Not hidden, per the operator and §7: the JS bundle and asset hashes
fingerprint the build anyway, and "I'm on 2026.08.28.1249" is the single most
useful line in a bug report.
2026-08-28 18:08:31 -04:00
bvandeusen 5771fd5770 build: zero-pad the derived version to YYYY.MM.DD.HHMM (318 step 5)
Build images / sign-extension (push) Successful in 4s
CI / lint (push) Successful in 4s
Build images / build-agent (push) Successful in 5s
Build images / build-web (push) Successful in 4s
CI / integration (push) Successful in 3m50s
CI / extension-version (push) Successful in 5s
Build images / build-ml (push) Successful in 4s
CI / frontend-build (push) Successful in 17s
CI / backend-lint-and-test (push) Successful in 32s
`2026.8.28.1249` becomes `2026.08.28.1249`. Note #3127 §1 and rule 148 both
specify the padded form.

The old reasoning was that each segment should read as a plain integer, and
it never held — comparison strips leading zeros on parse anyway, which the
same paragraph said. What stripping actually bought was this project emitting
`2026.8.28.1432` while a sibling emitted `2026.08.28.1432`: two shapes one
character apart, which is the hard kind of difference to notice. Two
obviously different formats would be safer than two nearly identical ones,
and identical is safer still.

Nothing already published is reordered: comparison is numeric per dot-segment,
so `08` and `8` are equal.

strip0 goes, and with it three of the four git calls per version — git's
format-local takes the whole format string, and splitting it into pieces only
ever existed to strip the padding between them.

It also fixes a real edge the old helper mangled. A commit at 03:22 UTC
derived `322` for its HHMM field, silently turning a four-digit field into
three; it now derives `0322`. Verified against a real commit rather than
reasoned about.

Checked before relying on it, since step 8 feeds this to Firefox: the
extension's comparator is `parseInt(n, 10)` with an explicit radix, so `08`
reads as 8 and there is no octal hazard (rule 150).

Two tests added. One pins the padded shape — the only thing keeping the
family's projects emitting one string is an assertion that they do. The other
asserts version and revision describe the same commit: they are derived
independently, and a divergence would mean an instance naming one commit
while carrying another's bytes, which is unfalsifiable from outside because
both values still look well-formed.
2026-08-28 16:35:26 -04:00
bvandeusen b3989d0224 fix(ci): the repoint was destroying the label it depends on
Build images / sign-extension (push) Successful in 3s
Build images / build-ml (push) Successful in 3s
CI / lint (push) Successful in 3s
CI / extension-version (push) Successful in 2s
CI / frontend-build (push) Successful in 20s
Build images / build-web (push) Successful in 3s
extension / lint (push) Successful in 19s
CI / backend-lint-and-test (push) Successful in 42s
Build images / build-agent (push) Failing after 2m33s
CI / integration (push) Successful in 3m44s
extension / lint (pull_request) Successful in 25s
Reuse worked exactly once per artifact, then every later push rebuilt at full
price. Nothing failed and nothing went red — the savings simply evaporated.

`imagetools create` wraps its source manifest in an INDEX. The repoint step
passed the channel tag as both source and target, so after any reuse :dev
stopped being a plain image, `.Image.Config.Labels` no longer resolved
through it, and fc.revision read as absent on the next push.

Observed across three runs rather than reasoned about: run 4749 read
fc.revision=a7e626a67a79 off fabledcurator-ml:dev and skipped the build; run
4751 read <none> off the same tag and rebuilt. The only thing to touch it in
between was 4749's own repoint. The agent hit in 4751 precisely because its
:dev had last been written by a real build, not by a repoint — which is the
control case.

Milestone 313's r-<rev> design was immune without anyone noticing why: the
source (the identity tag) was never one of the targets. Step 3 made the
channel tag both, and inherited a bug the earlier shape had avoided by
accident.

Fix: exclude the source from the target list, so the channel tag is only ever
written by a real build and stays a plain readable image. On dev that leaves
nothing to do, which is correct — the hit already established that :dev
points at the right content. On main it leaves :c-<sha>, which rule 145
requires of every main push whether or not a build ran.

Also added a note the reuse step prints when a channel tag exists but carries
no readable label. That is expected exactly once per artifact during the
migration; if it appears on every push, the tag is being index-wrapped again
and reuse is dead. This class of failure — correct behaviour, quietly worth
less than it reads — is the third one this milestone has turned up, and it is
the one that does not announce itself.
2026-08-28 14:55:49 -04:00
bvandeusen cd0b0ff04a ci: publish :latest + :c-<sha> on main, :dev on dev — nothing else (318 step 4)
Build images / sign-extension (push) Successful in 4s
CI / lint (push) Successful in 4s
CI / extension-version (push) Successful in 4s
Build images / build-agent (push) Successful in 6s
CI / frontend-build (push) Successful in 19s
extension / lint (push) Successful in 22s
CI / backend-lint-and-test (push) Successful in 30s
Build images / build-web (push) Successful in 1m57s
Build images / build-ml (push) Successful in 2m35s
CI / integration (push) Successful in 3m46s
The narrowing itself. Rule 145, narrowed 2026-08-28 once it was verified that
nothing pins: "a third name for the same thing is upkeep for a model we do
not run."

Gone:

  :2026.8.28  the per-artifact date tag from milestone 313 step 3. It shipped,
              it was verified on both branches, and its premise is gone. This
              is a withdrawal, not a correction — nothing about it was wrong.
  :main       a second moving name for whatever :latest already pointed at,
              justified by nothing but symmetry with :dev.

Kept, and note the inversion: :c-<sha> on main. Milestone 313 step 6 planned
to retire it on "haven't used, not important" — amended rule 145 makes it the
rollback unit and, with version tags gone, the only immutable identifier
left. #3139 is cancelled as superseded rather than quietly dropped.

No retention job is built. One already runs (#3157): a nightly Gitea
package_cleanup_rule with keep_count=10. Two consequences worth writing down.
Its keep_pattern protects any dotted-numeric tag, so the date tags this
commit stops minting would have been exempt from retention and accumulated
without bound — silently, in the direction of unbounded growth. That trap is
now dissolved rather than needing the pattern change #3139 planned. And
:c-<sha> IS prunable under that pattern, so the honest statement is that it
is the rollback unit for roughly the last 10 builds, not for every main
commit ever.

cmd_tag goes with its last consumer rather than being left as a dead
subcommand for a later step to notice. The report step logs version and
revision, which is the same information — the date is the first three fields
of the version.

Docs needed no change: every consumer already tracks a moving tag
(docker-compose.yml on :dev, agent/README.md on :latest), which is the
evidence the whole stance rests on.
2026-08-28 14:45:11 -04:00
bvandeusen 454eb3f973 fix(tests): artifact identity tests talk to artifacts.sh, not to a sibling
CI / lint (push) Successful in 4s
Build images / sign-extension (push) Successful in 4s
Build images / build-ml (push) Successful in 5s
CI / extension-version (push) Successful in 3s
CI / frontend-build (push) Successful in 22s
Build images / build-web (push) Successful in 4s
CI / backend-lint-and-test (push) Successful in 33s
Build images / build-agent (push) Successful in 3m50s
CI / integration (push) Successful in 3m51s
Run 4746 failed lint and pytest on the same cause: `from test_artifact_paths
import ROOT, declared_paths`. No other test module in this repo imports
another, so that was a new convention introduced for no gain — and the wrong
one, since `tests/` is a package and the bare name does not resolve.

Everything now goes through `artifacts.sh`, which is the interface build.yml
actually calls. The tests exercise the contract rather than a Python
re-implementation of it, and the duplicate `declared_paths` helper is gone
rather than copied.

Two real defects found while fixing it:

The newest commit is now computed by committer TIME, matching what
artifacts.sh means. It was `git log -1`, whose default order is
reverse-chronological only within topological constraints — so on a merged
history it can name a different commit than the newest timestamp does. Both
agree on this repo today (verified across all four artifacts), which is
exactly what makes it a flake waiting for the branch shape that separates
them.

The third test asserted the same invariant as the first in different words.
Removed rather than left as apparent coverage.
2026-08-28 14:42:42 -04:00
bvandeusen 7e065fed70 ci: key the reuse check on an image label, not a tag (318 step 3)
CI / extension-version (push) Successful in 4s
CI / lint (push) Failing after 4s
Build images / sign-extension (push) Successful in 4s
CI / backend-lint-and-test (push) Failing after 13s
CI / frontend-build (push) Successful in 20s
extension / lint (push) Successful in 22s
CI / integration (push) Failing after 2m24s
Build images / build-web (push) Successful in 2m44s
Build images / build-ml (push) Successful in 3m13s
Build images / build-agent (push) Successful in 8m56s
The shadow (dee93fa, run 4732) answered the gate: `imagetools inspect
--format` reads `.Image.Config.Labels` against this registry on buildx
v0.36.1. So the reuse check now asks the moving channel tag whether the image
it already points at carries this commit's `fc.revision`, and the r-<rev>
identity tags stop being published.

Three things this removes rather than manages:

A name minted per build that one thing read. Rule 145's narrowing is aimed
exactly there — "a third name for the same thing is upkeep for a model we do
not run."

The -main/-dev qualifier, and the CHANNELLED list behind it. Which tag you
inspect IS the channel, so the distinction has nowhere to live. cmd_identity
goes with it.

A silent expiry nobody wrote down. r-<rev> matches no branch of the
registry's keep_pattern (#3157), so identity tags were prunable past the
newest 10 — a pruned one costs a rebuild, in the safe direction and entirely
invisibly. A label rides inside a tag that has to exist anyway.

It also dissolves #3154 instead of deferring it: a scheduled base refresh
rebuilds :latest with the same revision label, the next unrelated push sees a
match and skips, and the refreshed base survives. Under the tag scheme that
push repointed :latest back to the older base.

The measured detail that shapes the code: a missing label returns an EMPTY
STRING and exits 0. Branching on the exit code would read "no label yet" as
success and skip a build that was needed. So it compares values, and every
uncertain case — absent label, unreachable tag, older image — lands as empty,
never equals a 12-char revision, and falls through to a build.

Reading the specific key matters too. The map carries the base image's labels,
and org.opencontainers.image.version sits right beside ours reading 24.04 on
the agent — a plausible-looking wrong answer.

Expect every artifact to rebuild once on this push: nothing carries a label
yet and it cannot be backfilled, since the reuse path copies a manifest and
config labels are not manifest annotations. One rebuild per artifact, ever,
self-healing after.

test_artifact_identity.py is rewritten around what is now load-bearing. The
CHANNELLED drift test had nothing left to guard; in its place the revision is
asked of git directly, so the file fails if the derivation ever stops being
"the commit this artifact's own shipped files last changed in".
2026-08-28 14:37:36 -04:00
bvandeusen dee93faa37 ci: shadow the label-based reuse key before trusting it (318 step 3)
CI / lint (push) Successful in 4s
Build images / sign-extension (push) Successful in 5s
CI / extension-version (push) Successful in 5s
Build images / build-ml (push) Successful in 6s
Build images / build-agent (push) Successful in 7s
Build images / build-web (push) Successful in 6s
CI / frontend-build (push) Successful in 18s
extension / lint (push) Successful in 22s
CI / backend-lint-and-test (push) Successful in 29s
CI / integration (push) Successful in 3m50s
Additive and inert. Every build now stamps `fc.revision` as an image label,
and a diagnostic step tries to read it back off the channel tag. Nothing
downstream reads either yet — the reuse check still keys on the r-<rev> tag.

The question step 3 has to answer before committing: can the reuse check be
keyed on a LABEL read off the channel tag instead of on a separate tag? If
it can, nothing is minted per build, the -main/-dev identity split
disappears with CHANNELLED, and #3154's base-refresh clobber stops existing
rather than needing a workaround. It also closes a defect found in what
already shipped: r-<rev> tags do not match the registry's keep_pattern
(#3157), so they are prunable, and a pruned identity tag silently costs a
rebuild. A label cannot be pruned — it rides inside a tag that must exist.

Three probes rather than one, because `--format` templating over .Image
varies by buildx version and one round trip per guess is a poor trade.

Two questions, answered at different times, and conflating them would
abandon a working design:

1. Do the template mechanics work here? Answered on the first run — if the
   probes print a labels map, even an empty one, `--format` resolves on this
   buildx and this registry. That is the real gate.
2. Does a value round-trip? Only once an artifact genuinely rebuilds. This
   push touches build.yml, which is in no path set, so all three artifacts
   reuse and stamp nothing. Expect no values this run.

That second point is the migration path rather than a defect: under the real
mechanism "no label" reads as a MISS, so each artifact rebuilds once and is
stamped from then on — one extra build per artifact, ever, in the safe
direction. `imagetools create` cannot backfill it, since it copies a
manifest and config labels are not manifest annotations.
2026-08-28 14:00:09 -04:00
bvandeusen d9aa5aa832 ci: a release names a build, it does not make one (milestone 318 step 2)
Build images / sign-extension (push) Successful in 4s
CI / lint (push) Successful in 3s
CI / extension-version (push) Successful in 4s
Build images / build-ml (push) Successful in 4s
Build images / build-agent (push) Successful in 5s
Build images / build-web (push) Successful in 4s
CI / frontend-build (push) Successful in 20s
extension / lint (push) Successful in 26s
CI / backend-lint-and-test (push) Successful in 31s
CI / integration (push) Successful in 3m50s
Removes the `v*` image-build trigger. A release tag names a commit `main`
already built and published; rebuilding it produced the same source under
the same names and RE-PUSHED `:c-<sha>` — which rule 145 forbids even when
the bytes match, because image configs carry timestamps and "same source"
does not mean "same manifest". The tag build was publishing nothing new and
violating an immutability rule to do it.

Three constructs go with it, all of which existed only to manage that build:

The 10-minute XPI polling loop. A release cut fired the tag build and the
main build together; the tag build skipped sign-extension and raced straight
to the download, losing every time (operator-flagged 2026-05-27 after
v26.05.27.0). Polling was the fix for a build that should not have run. It is
now a single fetch, and a 404 means the two jobs derived different versions
for one commit — which should fail loudly rather than be slept through.

The IS_TAG_PUSH carve-out from milestone 313 step 4, which stopped a tag
build from claiming an identity tag it would have re-pointed at fresh bytes.

build-web's `always() && (success || skipped)` gate, which existed to let a
SKIPPED sign-extension through on a tag push while still blocking a failed
one. sign-extension now always runs, so a plain `needs` gives exactly the
wanted behaviour. Its own branch condition goes too: main and dev are the
only triggers, so a gate naming both matched everything, and a condition
that is always true reads as though some path avoids it.

Releases still happen — rule 148 is untouched and tags are still cut on
explicit request per rule 2. They stop building images and start carrying a
changelog (step 7).

Net 115 lines deleted, 73 added, most of that comments explaining races that
can no longer occur. Nothing in the repo referenced the tag build, so no doc
changes were needed; the broader doc pass is step 9.
2026-08-28 13:49:45 -04:00
bvandeusen fb2c4d5b80 fix(extension): packaging.sh decides the version, so it must move the version
CI / lint (push) Successful in 3s
CI / extension-version (push) Successful in 4s
Build images / build-ml (push) Successful in 4s
Build images / build-agent (push) Successful in 5s
CI / frontend-build (push) Successful in 18s
extension / lint (push) Successful in 18s
CI / backend-lint-and-test (push) Successful in 30s
Build images / sign-extension (push) Successful in 1m41s
Build images / build-web (push) Successful in 11s
CI / integration (push) Successful in 3m42s
extension / lint (pull_request) Successful in 21s
`packaging.sh pathspec` excluded `extension/scripts/**` — the same list
web-ext ignores. But the two lists answer different questions, and this is
the one place they disagree.

packaging.sh is not packaged into the XPI. It does decide the version
string, and build.yml stamps that string into the manifest.json that IS
packaged. Changing how the version is computed therefore changes the shipped
bytes, and the derivation has to see it.

Harmless while every push rebuilt the web image. Step 4 made the rebuild
conditional on the derived revision moving, which turns it into a silent
failure: a packaging.sh change yields a NEW version, so sign-extension misses
its ext-<version> cache and signs — while build-web sees an unmoved revision,
reuses the published image and ships the OLD XPI. One orphaned AMO signature,
and an instance serving code the registry calls current. Found while checking
the ground under step 5, which changes the version format and is exactly the
commit that would have hit it.

Split the list rather than widening the shared one: NOT_VERSION_RELEVANT
drives the pathspec, NOT_PACKAGED_TRACKED still drives web-ext's ignore
list, and scripts/ stays out of the XPI. The two directions are not
symmetric, which is why the version list is the narrower one — too wide
costs a re-sign and a rebuild for a change that ships nothing new, too
narrow serves stale bytes and says nothing.

No version churn: the last packaging.sh commit predates the current
extension revision, so the derived version is unchanged at 1.0.3500147 and
web's revision stays a7e626a67a.

Both suites now pin the disagreement from their own side, because the
tempting fix for either half is to make the lists one again — version.spec.js
asserts the pathspec does NOT exclude scripts while the ignore list still
does, and test_artifact_paths.py asserts packaging.sh is inside the extension
and web path sets.
2026-08-28 08:49:06 -04:00
bvandeusen 609bc82acc ci: reuse the published image instead of rebuilding it (milestone 313 step 4)
Build images / sign-extension (push) Successful in 4s
CI / extension-version (push) Successful in 4s
CI / lint (push) Successful in 4s
Build images / build-ml (push) Successful in 7s
Build images / build-agent (push) Successful in 7s
Build images / build-web (push) Successful in 7s
CI / frontend-build (push) Successful in 17s
extension / lint (push) Successful in 20s
CI / backend-lint-and-test (push) Successful in 29s
CI / integration (push) Successful in 3m43s
Before building, each job asks the registry whether this artifact's content
is already published. On a hit it skips the build entirely and repoints the
channel and date tags at the existing manifest with `imagetools create` —
registry-side, no layer transfer, seconds. This is the step that stops a
push touching only `agent/` from rebuilding web and ml, and stops a merge to
main rebuilding what dev already built.

The question is asked with a new `artifacts.sh identity`, not with the date
tag: the date tag is day-precise and last-one-wins, so two different builds
share it and it cannot answer "is this content published?". The commit sha
would move on every push and never hit, which is the redundant rebuild being
removed. The revision does both jobs — content-unique, and stable across
pushes that did not touch the artifact.

Identity is channel-qualified for web and only for web, because web is the
only image that takes a build-arg: FC_CHANNEL is baked in and reported by
/api/extension/manifest, so its dev and main builds of one revision are
genuinely different images. ml and agent take none, which is what lets a
merge reuse dev's build rather than rebuilding the agent's CUDA image to
produce bytes that already exist. tests/test_artifact_identity.py reads the
Dockerfiles and fails if that list drifts from the ARG declarations, in
either direction — collapsing the channels ships an instance that reports
the wrong one, and splitting them needlessly rebuilds every merge.

Failure direction is deliberate: an inspect that errors for any reason reads
as a miss and the build runs. Only a real 200 skips one.

A tag-push never claims the identity. It rebuilds a revision main already
published, and image configs are not bit-reproducible, so re-pushing
r-<rev> would point an immutable tag at fresh bytes — rule 145's exact
prohibition. It publishes only its own :v... label and otherwise reuses.

Base-image freshness, decided rather than left implicit: an artifact whose
source stops moving stops picking up base updates under its pinned tag. That
is what a pin means, and rule 145 already says the refresh belongs on the
moving tag instead. Filed as #3154 rather than folded in here, because the
naive version regresses :latest on the next unrelated push.

ci.yml's backend lane gains fetch-depth: 0 — the new tests derive real
revisions, and on a depth-1 clone that derivation returns the tip sha or
fails, so the lane would go green while asserting nothing.

The three build jobs' shadow steps are renamed and re-commented: those
values stopped being informational at step 3, and a step captioned "nothing
reads this" beside steps that do is worse than no caption.
2026-08-28 08:24:58 -04:00
bvandeusen 7a20c55441 ci: publish a per-artifact date tag on main builds (milestone 313 step 3)
CI / lint (push) Successful in 3s
Build images / sign-extension (push) Successful in 4s
CI / extension-version (push) Successful in 3s
Build images / build-ml (push) Successful in 6s
Build images / build-agent (push) Successful in 6s
Build images / build-web (push) Successful in 5s
extension / lint (push) Successful in 17s
CI / frontend-build (push) Successful in 17s
CI / backend-lint-and-test (push) Successful in 29s
CI / integration (push) Successful in 3m44s
Each main build now also publishes <image>:YYYY.M.D — the date of the
commit that artifact's shipped files last changed in. Purely additive;
:main, :latest, :c-<sha> and the dev path are untouched, so reverting this
commit reverts the behaviour.

    main push -> :main, :latest, :c-<sha>, :2026.8.27
    dev push  -> :dev

Per artifact, so an image whose files did not change keeps the tag it
already had. On this commit the agent reads 2026.7.17 while web and ml
read 2026.8.27 — six weeks apart, from one push. Step 4 turns that into
not rebuilding it.

Day precision, and a second main build the same day replaces the first.
Operator's call, and the reasoning is theirs: same-day work is not
something worth pinning. A rollback goes to a day, not to the fourth merge
of a Tuesday afternoon. It also makes retention mean "the last N days"
rather than "the last N pushes".

CALVER is computed inside the main branch rather than at the top of the
step, and hard-fails when empty. There is no `set -e` here, so an
unconditional assignment that failed would have left it empty and published
the tag `fabledcurator:` — an invalid name, from a step that still reported
success. It is also simply unused on the dev and tag paths.

Fixed a stale comment while in this block rather than leaving it for step
7: it claimed release tags carry "no `.N` per family release-posture rule".
Rule 148 was amended on 2026-08-24 to REQUIRE the suffix, after the ban
caused a same-day tag to be retargeted and a release deleted to make room
(note 2813). FC's own tags already carry suffixes; only the comment was
asserting the superseded rule.

Verified before pushing: the derivation holds across 200 commits of real
history — a derived revision always touches its own path set, the version
never decreases along any parent->child edge, and web tracks all three
extension-only commits in the log. That last one is the direction that
would serve stale bytes on a pin.
2026-08-27 21:56:52 -04:00
bvandeusen 0c43fa3eb2 ci: shadow the per-artifact derived versions (milestone 313 step 2)
CI / lint (push) Successful in 4s
CI / extension-version (push) Successful in 3s
Build images / sign-extension (push) Successful in 4s
Build images / build-ml (push) Successful in 7s
Build images / build-agent (push) Successful in 7s
Build images / build-web (push) Successful in 6s
CI / frontend-build (push) Successful in 21s
extension / lint (push) Successful in 22s
CI / backend-lint-and-test (push) Successful in 30s
CI / integration (push) Successful in 3m44s
Every build job now logs the tag, version and revision its artifact would
get. Nothing reads them; no `set -e`, and each derivation falls back to
UNAVAILABLE, so a broken script cannot fail a build. Same discipline as
milestone 271 step 2, which is what made that cutover safe to do in one
commit.

Also fixes a landmine the plan named but had not checked: build-ml and
build-agent were checking out at depth 1. Both now use fetch-depth: 0.

That mattered more than it looks. A depth-1 clone sees one commit, so
`git log HEAD -- <shipped paths>` either returns that commit's timestamp —
plausible, and wrong — or returns nothing. For build-ml on this push it
would have returned today's date, because HEAD touches backend/, and
nothing downstream would have questioned it. For build-agent it would have
returned nothing at all, since no single commit here touches agent/, and
artifacts.sh exits non-zero rather than guessing. One direction is silent
and one is loud; only the loud one was ever going to get noticed.

What to read from the shadow lines over the next few pushes, in order of
how badly each would bite:

  * a push touching the extension must move BOTH the extension and web,
    because build-web bakes the XPI in. If web does not move, its path set
    is too narrow and a pinned web image will serve an extension it does
    not name.
  * a push touching only agent/ must leave web and ml still. If they move,
    their sets are too wide and they will rebuild for changes they do not
    ship.
  * a docs-only push must move nothing.
  * dev and main must derive the same values for the same source.

Step 3 only lets these values name a tag once those hold.
2026-08-27 21:41:14 -04:00
bvandeusen cf06c81db9 build: one definition per artifact of what it ships (milestone 313 step 1)
Build images / sign-extension (push) Successful in 4s
CI / lint (push) Successful in 4s
CI / extension-version (push) Successful in 5s
CI / frontend-build (push) Successful in 41s
CI / backend-lint-and-test (push) Successful in 2m4s
CI / integration (push) Successful in 4m15s
Build images / build-web (push) Successful in 4m50s
Build images / build-ml (push) Successful in 5m43s
Build images / build-agent (push) Successful in 10m45s
scripts/artifacts.sh generalises what packaging.sh established for the
extension: four published artifacts, four path sets, four independent
versions derived from the newest commit touching each set.

Measured on this commit, and this is the point of the whole thing:

  web        tag=2026.8.27  version=2026.8.27.1547  rev=a7e626a
  ml         tag=2026.8.27  version=2026.8.27.1547  rev=a7e626a
  agent      tag=2026.7.17  version=2026.7.17.1657  rev=57e5243
  extension  tag=2026.8.27  version=2026.8.27.1547  rev=a7e626a

The agent is six weeks behind because agent/fc_agent has not changed since
57e5243. Today it rebuilds and re-tags on every push regardless; from step
4 it will not.

Three outputs, because they answer different questions and conflating them
is how this goes wrong:

  tag       YYYY.M.D        the published image tag. Day precision, per the
                            operator: same-day work is not worth pinning, so
                            a second build that day replaces the first.
  version   YYYY.M.D.HHMM   the ordering key. The extension needs this and
                            cannot use `tag`: Firefox compares it to decide
                            whether an update exists, so two same-day builds
                            must be distinguishable or the second hits the
                            ext-<version> cache and ships stale bytes. That
                            is issue #2397's failure mode exactly.
  revision  <sha>           content identity. Because `tag` is only
                            day-precise, "does this tag already exist" cannot
                            decide whether a build can be skipped — two
                            different builds legitimately share a tag. Step 4
                            keys on this instead.

Path sets read from the Dockerfiles rather than guessed. Notable calls:

  - web includes the extension's packaged set, because build.yml bakes the
    signed XPI into frontend/public/extension/ before the docker build. Miss
    that and :latest serves a NEW extension under an unchanged web version.
  - web excludes frontend/test: vite builds from src/, index.html and
    public/, so a spec change lands in the builder layer but never in dist.
  - agent is agent/Dockerfile + agent/requirements.txt + agent/fc_agent,
    NOT agent/. README.md, ruff.toml and docker-compose.yml sit in that
    directory and never reach the image.
  - every set includes its own Dockerfile and requirements: a base-image
    bump changes the artifact as surely as a source edit does.
  - the extension's set is read from packaging.sh, not restated. One
    definition, per #2397.

tests/test_artifact_paths.py guards both directions of being wrong, since
both are silent. Too narrow — a COPY'd file missing from the set — means the
version does not move when the content does, and a pin serves stale bytes.
Too wide means re-versioning for a change the artifact does not ship. The
test parses each Dockerfile's COPY lines and compares them against the
declaration, so adding a COPY without updating the set fails the lane.

No workflow reads any of this yet. Step 2 shadows it.
2026-08-27 21:36:17 -04:00
bvandeusen 0db38cc111 ci: log in to the registry with the docker CLI, not docker/login-action
Build images / sign-extension (push) Successful in 4s
CI / frontend-build (push) Successful in 28s
CI / backend-lint-and-test (push) Successful in 32s
CI / lint (push) Successful in 6s
CI / extension-version (push) Successful in 6s
extension / lint (push) Successful in 45s
Build images / build-web (push) Successful in 2m21s
Build images / build-ml (push) Successful in 2m59s
CI / integration (push) Successful in 4m9s
Build images / build-agent (push) Failing after 10m58s
extension / lint (pull_request) Successful in 27s
build-ml failed at the login step twice on a7e626a, five seconds in, with
MODULE_NOT_FOUND on the action's own dist/index.js. Not the token — the
secret resolved to *** and the action never ran far enough to use it.

The cause is a race in act_runner's shared action cache, not corruption.
A remote action is cached at one /root/.cache/act/<hash> per runner, and
build-web, build-ml and build-agent all start in the same second and all
want docker/login-action@v3. One job re-clones that directory — emptying
and repopulating it — while another walks it to copy into its container,
and the walker lstat()s a file that just vanished. The two failures named
DIFFERENT missing files, eslint.config.mjs then jest.config.ts, which is
what rules out a bad cache entry and points at the race: a dangling entry
would name the same file every time.

Re-running does not help, because the re-run starts the three jobs
simultaneously again. It reproduced immediately.

Dropping the action removes FC from that race for this step. Logging in is
one command, the docker CLI is already in the CI image per
ci-requirements.md, and the same reasoning as family rule 5 applies: a
marketplace action buys nothing when the tool is baked into the image the
workflow already selected. Password on stdin, never as an argument — an
argument lands in the process table and draws docker's own deprecation
warning.

This narrows the exposure rather than closing it. All three jobs also
share docker/build-push-action@v5 and can race on it the same way; that
one has not lost yet, and replacing it means hand-rolling buildx
invocation including the build-args and provenance handling, which is a
bigger change than this failure justifies. Recorded on #3118.

Live consequence being cleared: fabledcurator-ml:dev was left a commit
behind fabledcurator:dev, which is the stale-pairing trap the trigger
comment on 239b1ed warns about.
2026-08-27 12:08:46 -04:00
bvandeusen a7e626a67a feat(extension): report the channel beside the version (step 7)
CI / lint (push) Successful in 4s
CI / extension-version (push) Successful in 5s
CI / frontend-build (push) Successful in 20s
CI / backend-lint-and-test (push) Successful in 32s
extension / lint (push) Successful in 28s
CI / integration (push) Successful in 3m52s
Build images / sign-extension (push) Successful in 4s
Build images / build-ml (push) Failing after 5s
Build images / build-agent (push) Successful in 13s
Build images / build-web (push) Successful in 2m4s
Closes the half of the ask the signing work didn't: a way to tell a dev
build from a main one. FC_CHANNEL is baked into the web image at build
time and /api/extension/manifest reports it as its own key, next to
version — the popup banner, the toolbar tooltip and the Settings card all
name it.

Beside the version, never inside it. A `1.0.3499884-dev` suffix is the
obvious shortcut and it is the exact failure this design comes from:
versionIsNewer parses each dotted segment with parseInt, so a suffixed
segment reads as 0, every dev build compares equal to every other, and
"no update available" stops being distinguishable from "I cannot read this
version". The comparator already degrades rather than discarding (rule
150), which is a reason not to NEED the suffix, not a licence to add one.
Two tests hold the line — one backend, asserting version and channel are
separate keys; one frontend, asserting the rendered version text stays the
bare derived number.

Optional on the read side, and absent rather than defaulted. An image
built before this field says nothing by not having the key; an image built
without a channel now says nothing the same way, so there is one absence
to handle instead of a second spelling of "unknown". Every reader drops
the label entirely when it is missing and reads exactly as it did before.
Reported verbatim rather than validated against {dev, main}: if an image
declares something else, showing what it claims helps whoever is debugging
more than dropping it would.

FC_CHANNEL is declared LAST in the Dockerfile. An ARG invalidates every
layer below it, and this is the one value that differs between the dev and
main builds of identical source — earlier, and the two channels could
never share a cached pip install. A tag push counts as main: a vYY.MM.DD
tag is cut from main, so that image is a main-channel artifact wearing an
immutable name.

No channel switcher, deliberately. background.js:34 already records that
Firefox's static update_url cannot apply, because every FC instance is a
different host — so the extension asks its configured backend, and the
channel IS the instance it points at. Switching is repointing apiUrl and
reinstalling from that host. A separate setting would contradict each
server build shipping its own extension.

This commit touches packaged extension files, so it moves the derived
version and will sign a new one via AMO — the first push to exercise the
extension-changed path from dev end to end.
2026-08-27 11:47:30 -04:00
bvandeusen fe48e77821 ci(extension): retire the manual-bump guard, true up the docs (step 5)
CI / lint (push) Successful in 3s
Build images / sign-extension (push) Successful in 4s
CI / extension-version (push) Successful in 3s
Build images / build-agent (push) Successful in 8s
CI / frontend-build (push) Successful in 22s
extension / lint (push) Successful in 25s
CI / backend-lint-and-test (push) Successful in 29s
Build images / build-web (push) Successful in 1m57s
Build images / build-ml (push) Successful in 2m38s
CI / integration (push) Successful in 3m50s
The guard asked whether a packaged extension file changed without the
version moving. Since step 4 nobody moves the version by hand, so it was
checking a fact that had stopped existing — and it was not merely dead
weight: it would have failed the lane on every real extension change,
demanding a bump that decides nothing. Removed rather than left running
beside the new mechanism (rule 22).

What replaces it is thinner and true. The extension-version lane now
asserts the derivation resolves on this commit, that the derived value is
the plain dotted-numeric shape AMO accepts, and that MAJOR.MINOR agrees
between manifest.json and package.json. MAJOR.MINOR is the one part still
hand-set, and packaging.sh reads it from manifest.json ALONE, so a
divergence ships a version package.json disagrees with. The lane keeps
fetch-depth: 0 — checking that the derivation survives a real checkout is
half its remaining value.

Deliberately not checked there: that the derived value beats what is
already signed. That guard belongs in build.yml, where it compares against
the real ext-* releases. Comparing against origin/main in a lane would be
wrong, because dev legitimately derives a LOWER value whenever main is
ahead on the extension, and a lane that fails for being behind is a lane
people learn to ignore.

packaging.sh is down to two consumers from three. version.spec.js's
"ci.yml derives its pathspec" test would have gone red on that, so it is
rewritten to assert the property rather than the consumer: no workflow
inlines an :(exclude)extension/ literal, across all three. That keeps the
#2397 anti-regression value while surviving consumers coming and going.
A second test pins build.yml to packaging.sh version and fails if it goes
back to grepping the committed value — which is not a style regression but
the #3092 bug itself. build.yml joins extension.yml's trigger paths, since
the suite now asserts against it.

The lockstep test narrows from the whole version string to MAJOR.MINOR.
The committed patch numbers are inert now; asserting on them would fail
for a difference that changes nothing.

Docs. extension/README.md's Release section described extension.yml
signing on main and committing the XPI into frontend/public/ — untrue
since 2026-05-25, and it told the reader to hand-bump both files, which is
now exactly the wrong instruction. Rewritten, with a Versioning section
that says plainly that editing the patch number does nothing and why the
key is commit time rather than a count. ci-requirements.md drops the third
packaging.sh consumer and names every job that needs full history. Root
README no longer claims the extension is signed on main only.
2026-08-27 11:29:44 -04:00
127 changed files with 4541 additions and 6896 deletions
+213
View File
@@ -0,0 +1,213 @@
# TEMPORARY — milestone 328 steps 1-2. Delete once the baseline is stamped.
#
# Squashing 87 alembic revisions into one baseline has exactly one dangerous
# failure: the generated baseline does not reproduce the schema the chain
# produced, `alembic stamp` writes a version string anyway (it validates
# NOTHING), and the divergence surfaces on the next real migration against the
# operator's live data.
#
# So this workflow does the comparison in CI, where a pgvector Postgres already
# gets built from the chain on every integration run, and nothing is at risk.
# It answers one question: does `upgrade head` on the collapsed chain produce a
# byte-identical schema to `upgrade head` on the 87-revision chain?
#
# The chain is read from git rather than from the working tree, so this keeps
# working AFTER the old revisions are deleted — `chain_ref` names a commit that
# still has them. That is what makes this the proof for step 1 and the
# pre-flight for step 2, rather than a one-shot script.
#
# While the chain is still present it also autogenerates a candidate baseline
# from the models and prints it. That is a starting point, NOT the answer:
# autogenerate reads SQLAlchemy metadata, and three things here do not live
# there —
# * CREATE EXTENSION vector (0001)
# * CREATE EXTENSION tsm_system_rows (0004)
# * the HNSW index on image_record.siglip_embedding, which is raw SQL
# because alembic's create_index cannot express `USING hnsw (...)` (0036)
# plus any CHECK constraint or server_default that a migration added without
# the model declaring it. Those must be hand-added, and the diff below is what
# proves none were missed.
name: Alembic baseline
on:
workflow_dispatch:
inputs:
chain_ref:
description: 'Commit/tag that still carries the full 0001..0087 chain'
type: string
default: '0a5bbe8'
jobs:
compare:
runs-on: python-ci
container:
image: git.fabledsword.com/bvandeusen/ci-python:3.14
env:
DB_USER: fabledcurator
DB_PASSWORD: ci_integration
DB_PORT: "5432"
DB_NAME: fabledcurator_test
SECRET_KEY: ci_integration_placeholder
services:
postgres:
image: pgvector/pgvector:pg16
env:
POSTGRES_USER: fabledcurator
POSTGRES_PASSWORD: ci_integration
POSTGRES_DB: fabledcurator_test
options: >-
--health-cmd "pg_isready -U fabledcurator"
--health-interval 10s
--health-timeout 5s
--health-retries 10
steps:
- uses: actions/checkout@v4
with:
# Full history is the point: `chain_ref` is read out of git, so a
# shallow clone would not have the revisions to compare against.
fetch-depth: 0
- name: Resolve the Postgres service and install deps
run: |
set -eux
# Same service-IP dance as ci.yml's integration job; see the long
# comment there for why the job name must stay separator-free.
PG=$(docker ps --filter "name=compare" --filter "ancestor=pgvector/pgvector:pg16" -q | head -n1)
test -n "$PG"
PG_IP=$(docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' "$PG")
test -n "$PG_IP"
echo "PG_CONTAINER=$PG" >> "$GITHUB_ENV"
echo "DB_HOST=$PG_IP" >> "$GITHUB_ENV"
for i in $(seq 1 60); do
(echo > "/dev/tcp/$PG_IP/5432") >/dev/null 2>&1 && break
sleep 2
done
if command -v uv >/dev/null 2>&1; then
uv pip install --system -r requirements.txt
else
pip install -r requirements.txt
fi
# DB 1: the 87-revision chain, read out of git at `chain_ref`.
#
# A git worktree rather than a checkout, so the current tree — which is
# what we are testing — is left completely alone.
- name: Build the schema the OLD chain produces
env:
CHAIN_REF: ${{ github.event.inputs.chain_ref }}
run: |
set -eux
docker exec "$PG_CONTAINER" createdb -U fabledcurator fc_chain
git worktree add /tmp/chain "$CHAIN_REF"
ls /tmp/chain/alembic/versions/*.py | wc -l
cd /tmp/chain
DB_NAME=fc_chain alembic upgrade head
cd -
docker exec "$PG_CONTAINER" pg_dump -U fabledcurator --schema-only \
--no-owner --no-privileges -d fc_chain > chain.sql
wc -l chain.sql
# A candidate baseline, autogenerated from the models against an EMPTY
# database so every table shows up as a create. Printed for a human to
# finish — it will be missing the three raw-SQL items named at the top.
#
# Gated on the TREE, not on a workflow input. A `type: boolean` input
# read back as `github.event.inputs.generate == 'true'` silently
# evaluated false on this runner (run 4960 skipped this step entirely
# with no diagnostic) — the same `github.event.inputs` typing quirk
# build.yml already works around. The file count is the real question
# anyway: there is nothing to generate once the chain is collapsed.
- name: Autogenerate a candidate baseline
run: |
set -eux
if [ "$(ls alembic/versions/*.py | wc -l)" -le 1 ]; then
echo "already collapsed — nothing to generate"
exit 0
fi
docker exec "$PG_CONTAINER" createdb -U fabledcurator fc_gen
# Hide the existing revisions so alembic sees an empty history and
# emits the whole schema rather than a delta.
mkdir -p /tmp/versions_held
mv alembic/versions/*.py /tmp/versions_held/ 2>/dev/null || true
DB_NAME=fc_gen alembic revision --autogenerate -m "baseline" || true
# Printed rather than uploaded: ci-requirements.md records that this
# runner cannot do actions/upload-artifact@v4+, and the repo dropped
# the action entirely in 2026-05, so the job log is the retrieval
# channel actually proven here.
#
# base64, not the raw file. A plain `cat` of the ~33KB candidate was
# TRUNCATED MID-LINE by the runner on run 4964 — it stopped inside
# `sa.Column('mime', sa.String(length=128)` and carried straight on
# to the next traced command, with the step still green. A silent
# cut in the middle of a schema definition is the worst possible
# failure here, because the truncated text still looks like a
# plausible file.
#
# base64 at a fixed narrow width gives many short lines instead of
# few long ones, and — the actual point — a checksum and a line
# count that make truncation DETECTABLE rather than invisible.
set +x
F=$(ls alembic/versions/*.py | head -1)
B64=$(base64 -w 120 "$F")
echo "===== BEGIN CANDIDATE BASELINE (base64) ====="
echo "$B64"
echo "===== END CANDIDATE BASELINE ====="
echo "candidate-sha256: $(sha256sum "$F" | cut -d' ' -f1)"
echo "candidate-bytes: $(wc -c < "$F")"
echo "candidate-b64-lines: $(echo "$B64" | wc -l)"
set -x
# Put the tree back exactly as it was; this job never mutates state.
rm -f alembic/versions/*.py
mv /tmp/versions_held/*.py alembic/versions/ 2>/dev/null || true
# DB 2: whatever the CURRENT tree's alembic/versions produces. Before the
# squash that is the same 87 revisions and the diff is trivially clean —
# which is worth running once as a control, so a clean diff after the
# squash means something.
- name: Build the schema the CURRENT tree produces
run: |
set -eux
docker exec "$PG_CONTAINER" createdb -U fabledcurator fc_base
ls alembic/versions/*.py | wc -l
DB_NAME=fc_base alembic upgrade head
docker exec "$PG_CONTAINER" pg_dump -U fabledcurator --schema-only \
--no-owner --no-privileges -d fc_base > baseline.sql
wc -l baseline.sql
# The verdict.
#
# pg_dump orders dumpable objects by name within type, not by creation
# order, so two schemas built by different routes are directly
# comparable. Normalisation is deliberately minimal, because a filter
# that hides a real difference is the one way this check passes when it
# should fail — blank lines, SQL comments, trailing whitespace, and:
#
# \restrict / \unrestrict — a per-invocation RANDOM NONCE that newer
# pg_dump emits to fence the dump against injection during restore. It
# differs on every run by construction, so it is noise by definition,
# not a schema difference. Measured on run 4960, the control: two dumps
# of the SAME schema came back 1123 lines each and differed on exactly
# these two lines and nothing else. That control is what licenses this
# filter — it was observed to be the only false positive, rather than
# assumed to be one.
- name: Diff
run: |
set -eu
norm() {
grep -vE '^\s*(--|$)' "$1" \
| grep -vE '^\\(un)?restrict ' \
| sed 's/[[:space:]]*$//'
}
norm chain.sql > a.txt
norm baseline.sql > b.txt
echo "normalised: chain=$(wc -l < a.txt) lines, current=$(wc -l < b.txt) lines"
if diff -u a.txt b.txt > schema.diff; then
echo "SCHEMAS IDENTICAL — the collapsed chain reproduces the old one."
else
echo "SCHEMAS DIFFER — $(grep -cE '^[+-]' schema.diff) changed lines:"
cat schema.diff
echo
echo "The baseline is wrong, not the database. Do not stamp."
exit 1
fi
+1272 -126
View File
File diff suppressed because it is too large Load Diff
+68 -93
View File
@@ -2,7 +2,7 @@ name: CI
# CI lanes per FabledRulebook/forgejo.md "CI philosophy":
# - lint: ruff only, no dep install — fast-fail for the common lint bounce.
# - extension-version: guards the extension publish path (see the job).
# - extension-version: the derived version resolves and is a shape AMO takes.
# - backend-lint-and-test: `pytest -m "not integration"`, no service containers.
# - frontend-build: vitest unit + vite build.
# - integration: pgvector + redis service containers; alembic + `pytest -m integration`.
@@ -35,25 +35,42 @@ jobs:
- name: Ruff lint
# agent/ included so the GPU-agent is linted before its image is built
# (build.yml only `docker build`s it — this is where it gets checked).
run: ruff check backend/ tests/ alembic/ agent/
# scripts/ likewise: release_notes.py runs only on a tag push, so a
# syntax or import error there would otherwise surface at the one
# moment nobody wants to debug a workflow.
run: ruff check backend/ tests/ alembic/ agent/ scripts/
- name: Agent syntax check
# The agent's runtime deps (torch/transformers/ultralytics) aren't in the
# CI image, so we can't import it — but compileall parses every module,
# catching syntax errors before the image build.
run: python -m compileall -q agent/fc_agent
# Guards the extension publish path, which has no self-correcting behavior.
# The extension version is DERIVED, not hand-maintained (milestone 271 step
# 4): build.yml computes it from the commit TIME of the newest packaged
# extension change and stamps it into manifest.json / package.json at build
# time. The guard that used to live here — "packaged files changed but nobody
# bumped the version" — was therefore checking a fact that had stopped
# existing. Worse than useless: it would have failed this lane on every real
# extension change, demanding a bump that decides nothing. Retired 2026-08-27
# rather than left running beside the new mechanism (rule 22).
#
# build.yml's sign-extension job keys its AMO-signing cache purely on the
# version string in extension/package.json: if an `ext-<version>` Forgejo
# release already carries an XPI, signing is SKIPPED and that old signed XPI
# is what build-web bakes into `:latest`. Nothing in that path inspects
# whether extension/ actually changed — so a forgotten version bump ships a
# stale extension on a fully green build, silently. (AMO can't help: it 409s
# on re-signing a version, which is exactly why the cache exists.)
# Two things are still worth asserting, and this is the only lane that can:
# the extension.yml suite runs on node:24-slim, which is exactly why
# version.spec.js sticks to packaging.sh's git-free subcommands.
# 1. the derivation actually resolves on this commit
# 2. the derived string is one AMO will accept, checked against Mozilla's
# own published grammar rather than a loose "digits and dots"
#
# This job makes that case loud, on the dev push, instead of invisible at
# merge-to-main. It is pure git + text work — no deps, no services.
# The MAJOR.MINOR-agreement check that used to be (2) is gone with milestone
# 318 step 8: the committed version no longer seeds anything, so there is no
# hand-set part left for the two files to disagree about.
#
# Deliberately NOT checked here: that the derived value beats what has already
# been signed. That guard belongs in build.yml, where it compares against the
# real ext-* releases. Comparing against origin/main here would be wrong —
# dev legitimately derives a LOWER value whenever main is ahead on the
# extension, and a lane that fails for being behind is a lane people learn to
# ignore.
extension-version:
runs-on: python-ci
container:
@@ -61,97 +78,48 @@ jobs:
steps:
- uses: actions/checkout@v4
with:
# Full history: the check diffs against the push's `before` SHA (or
# the PR base), which a depth-1 clone wouldn't contain.
# The derivation needs real history: a depth-1 clone sees one commit
# and produces a wrong, too-low value RATHER THAN FAILING. Checking
# that here is half the point of the lane.
fetch-depth: 0
- name: Extension version guard
env:
BEFORE: ${{ github.event.before }}
PR_BASE: ${{ github.event.pull_request.base.sha }}
- name: Extension version derives cleanly
run: |
set -eu
# busybox sh on the act_runner — no bashisms (family rule).
ver() { grep -E '"version"' "$1" | head -1 | sed -E 's/.*"version"[[:space:]]*:[[:space:]]*"([^"]+)".*/\1/'; }
PKG=$(ver extension/package.json)
MAN=$(ver extension/manifest.json)
test -n "$PKG" || { echo "ERROR: no version found in extension/package.json"; exit 1; }
test -n "$MAN" || { echo "ERROR: no version found in extension/manifest.json"; exit 1; }
VERSION=$(sh extension/scripts/packaging.sh version)
echo "derived: $VERSION"
# (1) Unconditional: the two version strings must agree. `web-ext sign`
# reads manifest.json (package.json sits in --ignore-files and isn't
# even inside the XPI), so AMO signs MAN and Firefox installs MAN.
# build.yml keys its cache, release tag, XPI filename — and therefore
# the version /api/extension/manifest reports to the update prompt —
# on PKG. Divergence either hard-fails at AMO or ships a mislabelled
# XPI whose update prompt lies about what's installed.
if [ "$MAN" != "$PKG" ]; then
echo "ERROR: extension version mismatch."
echo " extension/manifest.json = $MAN <- what AMO signs / Firefox installs"
echo " extension/package.json = $PKG <- what CI caches, names, and reports"
echo "Set both to the same value."
# Mozilla's published grammar for AMO, transcribed verbatim from
# MDN's manifest.json/version page:
#
# ^(0|[1-9][0-9]{0,8})([.](0|[1-9][0-9]{0,8})){0,3}$
#
# Not the looser `^[0-9]+(\.[0-9]+)*$` this lane used to carry. That
# one passes `2026.08.29.0201`, which AMO REJECTS — a segment must be
# the single digit 0 or start 1-9 — and it also passes five segments,
# where AMO allows four. Both would surface as a failed sign with the
# version already burned: AMO 409s on re-signing, so a rejected value
# cannot be reclaimed and cannot be reused. This lane is the cheap
# place to find out. (#3138, milestone 318 step 8.)
if ! echo "$VERSION" | grep -qE '^(0|[1-9][0-9]{0,8})(\.(0|[1-9][0-9]{0,8})){0,3}$'; then
echo "ERROR: derived version '$VERSION' is not a version AMO accepts."
echo "AMO's grammar: ^(0|[1-9][0-9]{0,8})([.](0|[1-9][0-9]{0,8})){0,3}$"
echo "Most likely cause: a zero-padded segment (08, 0201). The rest"
echo "of the family pads; the extension must not — see packaging.sh."
exit 1
fi
# (2) If the SHIPPED extension changed, the version must have moved.
#
# Compare against MAIN, not against the previous push. The publish
# decision is made at merge-to-main against whatever ext-<version>
# already exists, so "differs from main" is the question that matters.
# Diffing against the previous dev push instead would demand a fresh
# bump on every iteration — push, tweak the extension again, and CI
# would insist on a second bump that buys nothing, inflating the
# version for no reason. On a main push there is no "main to compare
# to" yet, so fall back to that push's own before-SHA.
if [ "${GITHUB_REF##*/}" = "main" ]; then
BASE="${BEFORE:-}"
else
BASE=$(git rev-parse --verify -q origin/main 2>/dev/null || git rev-parse --verify -q main 2>/dev/null || echo "")
# PR base is the fallback when main isn't in the clone at all.
[ -n "$BASE" ] || BASE="${PR_BASE:-}"
fi
case "$BASE" in
''|0000000000000000000000000000000000000000)
echo "No usable base ref (no main in clone / first push) — skipping the bump check."
echo "OK: extension version $PKG"
exit 0
;;
esac
if ! git cat-file -e "$BASE^{commit}" 2>/dev/null; then
echo "Base commit $BASE not in this clone — skipping the bump check."
echo "OK: extension version $PKG"
exit 0
fi
# The exclusion list is NOT written out here — it comes from
# extension/scripts/packaging.sh, the one definition of what ships,
# shared with web-ext's --ignore-files and the derived-version patch
# count. Three hand-kept copies of that fact is how #2397 happened.
#
# `set -f` is required around the substitution: without it the shell
# globs `test/**` against the working tree and silently narrows it.
set -f
CHANGED=$(git diff --name-only "$BASE" HEAD -- extension/ $(sh extension/scripts/packaging.sh pathspec))
set +f
if [ -z "$CHANGED" ]; then
echo "No packaged extension files changed since $BASE — nothing to guard."
echo "OK: extension version $PKG"
exit 0
fi
echo "Packaged extension files changed since $BASE:"
echo "$CHANGED" | sed 's/^/ /'
PKG_OLD=$(git show "$BASE:extension/package.json" 2>/dev/null | grep -E '"version"' | head -1 | sed -E 's/.*"version"[[:space:]]*:[[:space:]]*"([^"]+)".*/\1/')
if [ -z "$PKG_OLD" ]; then
echo "Could not read the base version — skipping the bump check."
echo "OK: extension version $PKG"
exit 0
fi
if [ "$PKG_OLD" = "$PKG" ]; then
echo "ERROR: packaged extension files changed but the version is still $PKG."
echo "build.yml would find the existing ext-$PKG release, skip AMO signing,"
echo "and bake the OLD signed XPI into :latest — a green build shipping stale code."
echo "Bump the version in BOTH extension/package.json and extension/manifest.json."
# ...and the shape this project actually derives. AMO would happily
# take `1.0.3500147` too, so the grammar check alone would not notice
# a regression to the pre-318 shape — which orders BELOW everything
# signed since, and is unrecoverable once Firefox has the higher one.
if ! echo "$VERSION" | grep -qE '^20[0-9][0-9]\.[0-9]{1,2}\.[0-9]{1,2}\.[0-9]{1,4}$'; then
echo "ERROR: derived version '$VERSION' is not YYYY.M.D.HHMM."
echo "Rule 148's CalVer is what build.yml signs; the old"
echo "1.0.<minutes> shape would order below every ext-2026.* release."
exit 1
fi
echo "OK: extension version $PKG_OLD -> $PKG"
echo "OK: derived version $VERSION"
backend-lint-and-test:
runs-on: python-ci
@@ -164,6 +132,13 @@ jobs:
SECRET_KEY: ci_unit_test_placeholder
steps:
- uses: actions/checkout@v4
with:
# Full history for tests/test_artifact_identity.py, which derives
# each artifact's revision to check the identity scheme. On a
# depth-1 clone that derivation either fails or returns the tip sha
# — so the lane would go green while asserting nothing, which is
# the one outcome worse than a red one.
fetch-depth: 0
# Cache step removed 2026-05-26: act_runner's cache backend has been
# broken on this homelab runner since 2026-05-15 (first as request-
+8 -3
View File
@@ -10,15 +10,20 @@ on:
paths:
- 'extension/**'
- '.forgejo/workflows/extension.yml'
# test/version.spec.js asserts ci.yml's extension-version guard never
# ignores a file web-ext actually packages, so a ci.yml-only edit can
# break this suite and must trigger it.
# test/version.spec.js asserts things ABOUT the other two workflows —
# that neither inlines the packaged-file set, and that build.yml derives
# the shipped version rather than reading it out of the repo. A
# workflow-only edit can therefore break this suite, so it has to trigger
# it. build.yml joined the list at milestone 271 step 5, when the spec
# started asserting against it.
- '.forgejo/workflows/ci.yml'
- '.forgejo/workflows/build.yml'
pull_request:
branches: [main]
paths:
- 'extension/**'
- '.forgejo/workflows/ci.yml'
- '.forgejo/workflows/build.yml'
workflow_dispatch:
jobs:
+80
View File
@@ -0,0 +1,80 @@
name: Release
# A `v*` tag publishes a changelog. It does NOT build anything.
#
# Milestone 318 step 2 removed the tag trigger from build.yml: by the time
# anyone tags a commit, `main` has already built and published it, and a
# rebuild would re-push `:c-<sha>` — which rule 145 forbids even when the
# source matches, since image configs carry timestamps and "same source" does
# not mean "same manifest". That left the tag with no consequence at all.
#
# This is the consequence it has instead. Step 6 put the derived version in the
# Settings footer, so an operator can say WHICH build they are running; this
# says what is IN it that was not in the one they ran last month. Both halves
# of one question (note #3127 §5).
#
# Nothing here runs on a schedule and nothing auto-tags on merge. Release tags
# are bookmarks — cut one when you will want to point at that day by name,
# otherwise don't (note #3127 §0). FC went twelve weeks between v26.06.04.0 and
# the next one and nothing was wrong. A schedule would turn an optional
# bookmark back into ceremony, which is the thing this milestone is removing.
#
# Cutting the tag is an explicit operator action under rule 2 ("`main` — never
# without explicit request", which since 2026-08-28 covers PR, merge and tag
# alike). This lane only decides what happens once they do.
#
# Requires repo secret RELEASE_TOKEN with the `write:release` scope — the same
# PAT build.yml uses for the ext-<version> XPI asset cache.
on:
push:
tags: ['v*']
# So a release body can be regenerated after the fact — the publisher PATCHes
# an existing release rather than falling through on a conflict, so re-running
# this on a tag rewrites the body instead of silently keeping the first one
# (note #3127 §6.7).
workflow_dispatch:
inputs:
tag:
description: 'Tag to (re)publish notes for'
required: true
jobs:
changelog:
runs-on: python-ci
container:
image: git.fabledsword.com/bvandeusen/ci-python:3.14
steps:
- uses: actions/checkout@v4
with:
# Load-bearing twice over: the previous release is found by walking
# ancestry back through the tag graph, and the cross-check against
# the derived web version calls artifacts.sh, which reads commit
# times. A shallow clone would find no previous tag and emit the
# entire history as the changelog — plausible-looking and wrong.
fetch-depth: 0
ref: ${{ github.event.inputs.tag || github.ref }}
# The `:c-<sha>` rollback refs are only real if `main` built this commit.
# The script checks that against origin/main and downgrades the claim to
# "unverified" when it cannot resolve one; fetching it here means that
# downgrade stays an actual signal instead of firing on every release.
- name: Make main's history resolvable
run: git fetch --no-tags --quiet origin +main:refs/remotes/origin/main || true
# TAG goes through the environment, not through `${{ }}` inside the
# run block. The value is operator-supplied, and an expression expanded
# into a shell line is expanded BEFORE the shell sees it — there is no
# quoting that makes that safe. On a tag push it is empty and the script
# falls back to GITHUB_REF.
- name: Publish the derived changelog
env:
RELEASE_TOKEN: ${{ secrets.RELEASE_TOKEN }}
TAG: ${{ github.event.inputs.tag }}
run: |
set -eu
if [ -n "${TAG:-}" ]; then
python3 scripts/release_notes.py "$TAG"
else
python3 scripts/release_notes.py
fi
+23
View File
@@ -47,6 +47,29 @@ RUN chmod +x entrypoint.sh
COPY --from=frontend-builder /build/dist ./frontend/dist
# Which channel this image belongs to — `dev` or `main` (milestone 271 step 7).
# build.yml passes it; /api/extension/manifest reports it beside the version so
# an operator can tell which channel an install came from without the channel
# ever touching the version string.
#
# Empty by default, deliberately: a locally-built image then reports NO channel
# rather than claiming to be one, and the manifest omits the field entirely —
# indistinguishable from an image built before the field existed, which is
# exactly the shape every reader already has to handle.
#
# Declared LAST on purpose. An ARG/ENV invalidates every layer below it, and
# these are the values that differ between builds of otherwise identical
# source — put them any earlier and the two channels could never share a
# cached pip install.
#
# FC_VERSION is what the instance reports about itself in the UI. Since
# milestone 318 stopped publishing version image tags, that self-report is
# the only answer to "which build is this?" — nothing else names it.
ARG FC_CHANNEL=""
ENV FC_CHANNEL=${FC_CHANNEL}
ARG FC_VERSION=""
ENV FC_VERSION=${FC_VERSION}
EXPOSE 8080
ENTRYPOINT ["./entrypoint.sh"]
+39 -5
View File
@@ -10,6 +10,35 @@ In production. `main` is continuously deployed — every merge to `main` builds
and publishes `:latest` images, so whatever is on `main` is what is running.
Day-to-day work happens on `dev`, which publishes `:dev` images.
## Versions and tags
Three image tags exist, and no others:
| Tag | Branch | Meaning |
| --- | --- | --- |
| `:latest` | `main` | Production. Moves on every merge. |
| `:c-<sha>` | `main` | Immutable — the rollback unit, all three images together. |
| `:dev` | `dev` | The rolling test channel. Moves on every push. |
There are deliberately **no version tags**. Nothing pins one, and a per-build
name nobody reads is upkeep for a model FC does not run (family rule 145; the
reasoning is note #3127 §5). Rolling back is `docker pull …:c-<sha>`.
Each artifact still has a version, derived rather than chosen: the commit time
of the newest change to that artifact's *own* shipped files, as
`YYYY.MM.DD.HHMM` UTC (rule 148). Four artifacts, four independent versions —
a push touching only `agent/` re-versions the agent and leaves web and ml
alone, and CI skips the builds whose content did not move.
Because no registry name carries it, the running instance's own report is the
only answer to "which build is this?". The foot of Settings shows
`FabledCurator 2026.08.29.0201 · dev`, and `/api/health` returns the same two
fields.
Release tags are optional bookmarks — FC went twelve weeks without one and
nothing was wrong. Pushing `v<version>` publishes a Forgejo release listing the
commits since the previous tag; it builds no image.
## What's in here
Five deployable pieces, built by `.forgejo/workflows/build.yml`:
@@ -19,7 +48,7 @@ Five deployable pieces, built by `.forgejo/workflows/build.yml`:
| **Web / workers** | `Dockerfile` | `fabledcurator` | Quart API + the built Vue SPA in one image. `entrypoint.sh` picks the role: `web`, `worker`, `scheduler`. The `maintenance-long` service is a second `worker` pinned to the long-running maintenance queue. |
| **ML worker** | `Dockerfile.ml` | `fabledcurator-ml` | Same app, plus `requirements-ml.txt` — tagging and embedding models that run in-container. |
| **GPU agent** | `agent/Dockerfile` | `fabledcurator-agent` | Optional desktop-GPU worker (`agent/`). Leases jobs over **HTTP only** — never touches the database or Redis. Run it for a burst, stop it to reclaim the card. See `agent/README.md`. |
| **Firefox extension** | `extension/` | signed XPI | MV3 extension: pushes platform session cookies into FC and adds a creator as a Source in one click. AMO-signed on `main` only, then bundled into the web image and served from Settings → Maintenance. See `extension/README.md`. |
| **Firefox extension** | `extension/` | signed XPI | MV3 extension: pushes platform session cookies into FC and adds a creator as a Source in one click. AMO-signed on both `dev` and `main` (one signature per extension change, shared by the two channels), bundled into that channel's web image and served from Settings → Maintenance. See `extension/README.md`. |
| **Data** | — | `pgvector/pgvector:pg16`, `redis:7-alpine` | Postgres with pgvector for embeddings; Redis as the Celery broker. |
## Quick start
@@ -52,9 +81,10 @@ FabledCurator is designed to run inside a self-hosted homelab environment over p
## CI / Forgejo setup
Three workflows: `ci.yml` (lint, extension-version guard, backend unit tests,
Four workflows: `ci.yml` (lint, extension-version check, backend unit tests,
frontend build, integration), `extension.yml` (extension lint, vitest, XPI
content verification), and `build.yml` (sign + publish).
content verification), `build.yml` (sign + publish), and `release.yml`, which
runs only on a `v*` tag and publishes a changelog without building anything.
**The toolchain each job runs in is its `container.image`, not its `runs-on`
label.** `runs-on: python-ci` only schedules the job onto a runner; every job
@@ -71,8 +101,12 @@ The repo expects one secret:
Generate at https://git.fabledsword.com/user/settings/applications. The injected `GITHUB_TOKEN` cannot be used because it lacks `write:package`.
AMO signing additionally needs `MOZILLA_AMO_JWT_KEY` / `MOZILLA_AMO_JWT_SECRET`; it runs on
`main` only and is cached per version, since AMO rejects a re-signed version.
AMO signing additionally needs `MOZILLA_AMO_JWT_KEY` / `MOZILLA_AMO_JWT_SECRET`.
It runs on **both** channels and is cached per version: because the version is
derived from commit time, `dev` and `main` derive the same number for the same
source, so `main` finds `dev`'s signature already cached and makes no second AMO
call. That cache is why signing must be one-shot — AMO rejects a re-signed
version.
## License
@@ -1,277 +0,0 @@
"""initial unified schema
Revision ID: 0001
Revises:
Create Date: 2026-05-13
"""
from typing import Sequence, Union
import sqlalchemy as sa
from alembic import op
from pgvector.sqlalchemy import Vector
revision: str = "0001"
down_revision: Union[str, None] = None
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
op.execute("CREATE EXTENSION IF NOT EXISTS vector")
op.create_table(
"artist",
sa.Column("id", sa.Integer(), nullable=False),
sa.Column("name", sa.String(length=255), nullable=False),
sa.Column("slug", sa.String(length=255), nullable=False),
sa.Column("notes", sa.Text(), nullable=True),
sa.Column("is_subscription", sa.Boolean(), nullable=False, server_default=sa.false()),
sa.Column("auto_check", sa.Boolean(), nullable=False, server_default=sa.true()),
sa.Column("check_interval_seconds", sa.Integer(), nullable=True),
sa.Column(
"created_at",
sa.DateTime(timezone=True),
nullable=False,
server_default=sa.func.now(),
),
sa.PrimaryKeyConstraint("id", name="pk_artist"),
sa.UniqueConstraint("name", name="uq_artist_name"),
sa.UniqueConstraint("slug", name="uq_artist_slug"),
)
op.create_table(
"source",
sa.Column("id", sa.Integer(), nullable=False),
sa.Column("artist_id", sa.Integer(), nullable=False),
sa.Column("platform", sa.String(length=64), nullable=False),
sa.Column("url", sa.Text(), nullable=False),
sa.Column("enabled", sa.Boolean(), nullable=False, server_default=sa.true()),
sa.Column("config_overrides", sa.JSON(), nullable=True),
sa.Column("last_checked_at", sa.DateTime(timezone=True), nullable=True),
sa.Column("last_error", sa.Text(), nullable=True),
sa.Column("check_interval_override", sa.Integer(), nullable=True),
sa.ForeignKeyConstraint(
["artist_id"], ["artist.id"], name="fk_source_artist_id_artist", ondelete="CASCADE"
),
sa.PrimaryKeyConstraint("id", name="pk_source"),
)
op.create_index("ix_source_artist_id", "source", ["artist_id"])
op.create_table(
"credential",
sa.Column("id", sa.Integer(), nullable=False),
sa.Column("platform", sa.String(length=64), nullable=False),
sa.Column("kind", sa.String(length=32), nullable=False),
sa.Column("encrypted_blob", sa.LargeBinary(), nullable=False),
sa.Column("status", sa.String(length=32), nullable=False, server_default="active"),
sa.Column(
"captured_at",
sa.DateTime(timezone=True),
nullable=False,
server_default=sa.func.now(),
),
sa.Column("expires_at", sa.DateTime(timezone=True), nullable=True),
sa.PrimaryKeyConstraint("id", name="pk_credential"),
sa.UniqueConstraint("platform", name="uq_credential_platform"),
)
op.create_table(
"post",
sa.Column("id", sa.Integer(), nullable=False),
sa.Column("source_id", sa.Integer(), nullable=False),
sa.Column("external_post_id", sa.String(length=128), nullable=False),
sa.Column("post_url", sa.Text(), nullable=True),
sa.Column("post_title", sa.Text(), nullable=True),
sa.Column("post_date", sa.DateTime(timezone=True), nullable=True),
sa.Column("raw_metadata", sa.JSON(), nullable=True),
sa.Column(
"downloaded_at",
sa.DateTime(timezone=True),
nullable=False,
server_default=sa.func.now(),
),
sa.ForeignKeyConstraint(
["source_id"], ["source.id"], name="fk_post_source_id_source", ondelete="CASCADE"
),
sa.PrimaryKeyConstraint("id", name="pk_post"),
sa.UniqueConstraint("source_id", "external_post_id", name="uq_post_source_external_id"),
)
op.create_index("ix_post_source_id", "post", ["source_id"])
op.create_table(
"image_record",
sa.Column("id", sa.Integer(), nullable=False),
sa.Column("path", sa.Text(), nullable=False),
sa.Column("sha256", sa.String(length=64), nullable=False),
sa.Column("phash", sa.String(length=32), nullable=True),
sa.Column("size_bytes", sa.BigInteger(), nullable=False),
sa.Column("mime", sa.String(length=64), nullable=False),
sa.Column("width", sa.Integer(), nullable=True),
sa.Column("height", sa.Integer(), nullable=True),
sa.Column("thumbnail_path", sa.Text(), nullable=True),
sa.Column(
"origin",
sa.Enum(
"downloaded",
"imported_filesystem",
"uploaded",
name="origin_enum",
),
nullable=False,
),
sa.Column("primary_post_id", sa.Integer(), nullable=True),
sa.Column("wd14_predictions", sa.JSON(), nullable=True),
sa.Column("wd14_model_version", sa.String(length=128), nullable=True),
sa.Column("siglip_embedding", Vector(1152), nullable=True),
sa.Column("siglip_model_version", sa.String(length=128), nullable=True),
sa.Column("centroid_scores", sa.JSON(), nullable=True),
sa.Column(
"created_at",
sa.DateTime(timezone=True),
nullable=False,
server_default=sa.func.now(),
),
sa.Column(
"updated_at",
sa.DateTime(timezone=True),
nullable=False,
server_default=sa.func.now(),
),
sa.ForeignKeyConstraint(
["primary_post_id"],
["post.id"],
name="fk_image_record_primary_post_id_post",
ondelete="SET NULL",
),
sa.PrimaryKeyConstraint("id", name="pk_image_record"),
sa.UniqueConstraint("path", name="uq_image_record_path"),
sa.UniqueConstraint("sha256", name="uq_image_record_sha256"),
)
op.create_index("ix_image_record_sha256", "image_record", ["sha256"])
op.create_index("ix_image_record_phash", "image_record", ["phash"])
op.create_index("ix_image_record_primary_post_id", "image_record", ["primary_post_id"])
op.create_table(
"image_provenance",
sa.Column("id", sa.Integer(), nullable=False),
sa.Column("image_record_id", sa.Integer(), nullable=False),
sa.Column("post_id", sa.Integer(), nullable=False),
sa.Column("source_id", sa.Integer(), nullable=False),
sa.Column("captured_metadata", sa.JSON(), nullable=True),
sa.Column(
"captured_at",
sa.DateTime(timezone=True),
nullable=False,
server_default=sa.func.now(),
),
sa.ForeignKeyConstraint(
["image_record_id"],
["image_record.id"],
name="fk_image_provenance_image_record_id_image_record",
ondelete="CASCADE",
),
sa.ForeignKeyConstraint(
["post_id"],
["post.id"],
name="fk_image_provenance_post_id_post",
ondelete="CASCADE",
),
sa.ForeignKeyConstraint(
["source_id"],
["source.id"],
name="fk_image_provenance_source_id_source",
ondelete="CASCADE",
),
sa.PrimaryKeyConstraint("id", name="pk_image_provenance"),
)
op.create_index("ix_image_provenance_image_record_id", "image_provenance", ["image_record_id"])
op.create_index("ix_image_provenance_post_id", "image_provenance", ["post_id"])
op.create_index("ix_image_provenance_source_id", "image_provenance", ["source_id"])
op.create_table(
"tag",
sa.Column("id", sa.Integer(), nullable=False),
sa.Column("name", sa.String(length=255), nullable=False),
sa.Column("namespace", sa.String(length=64), nullable=True),
sa.Column(
"created_at",
sa.DateTime(timezone=True),
nullable=False,
server_default=sa.func.now(),
),
sa.PrimaryKeyConstraint("id", name="pk_tag"),
sa.UniqueConstraint("name", name="uq_tag_name"),
)
op.create_index("ix_tag_name", "tag", ["name"])
op.create_index("ix_tag_namespace", "tag", ["namespace"])
op.create_table(
"image_tag",
sa.Column("image_record_id", sa.Integer(), nullable=False),
sa.Column("tag_id", sa.Integer(), nullable=False),
sa.Column("source", sa.String(length=32), nullable=False, server_default="manual"),
sa.Column(
"created_at",
sa.DateTime(timezone=True),
nullable=False,
server_default=sa.func.now(),
),
sa.ForeignKeyConstraint(
["image_record_id"],
["image_record.id"],
name="fk_image_tag_image_record_id_image_record",
ondelete="CASCADE",
),
sa.ForeignKeyConstraint(
["tag_id"], ["tag.id"], name="fk_image_tag_tag_id_tag", ondelete="CASCADE"
),
sa.PrimaryKeyConstraint("image_record_id", "tag_id", name="pk_image_tag"),
)
op.create_table(
"download_event",
sa.Column("id", sa.Integer(), nullable=False),
sa.Column("source_id", sa.Integer(), nullable=False),
sa.Column("post_id", sa.Integer(), nullable=True),
sa.Column("status", sa.String(length=32), nullable=False),
sa.Column(
"started_at",
sa.DateTime(timezone=True),
nullable=False,
server_default=sa.func.now(),
),
sa.Column("finished_at", sa.DateTime(timezone=True), nullable=True),
sa.Column("bytes_downloaded", sa.BigInteger(), nullable=False, server_default="0"),
sa.Column("files_count", sa.Integer(), nullable=False, server_default="0"),
sa.Column("error", sa.Text(), nullable=True),
sa.ForeignKeyConstraint(
["source_id"],
["source.id"],
name="fk_download_event_source_id_source",
ondelete="CASCADE",
),
sa.ForeignKeyConstraint(
["post_id"],
["post.id"],
name="fk_download_event_post_id_post",
ondelete="SET NULL",
),
sa.PrimaryKeyConstraint("id", name="pk_download_event"),
)
op.create_index("ix_download_event_source_id", "download_event", ["source_id"])
op.create_index("ix_download_event_post_id", "download_event", ["post_id"])
def downgrade() -> None:
op.drop_table("download_event")
op.drop_table("image_tag")
op.drop_table("tag")
op.drop_table("image_provenance")
op.drop_table("image_record")
op.execute("DROP TYPE IF EXISTS origin_enum")
op.drop_table("post")
op.drop_table("credential")
op.drop_table("source")
op.drop_table("artist")
op.execute("DROP EXTENSION IF EXISTS vector")
@@ -1,208 +0,0 @@
"""fc2a: tag kinds, import_task, import_batch, integrity_status
Revision ID: 0002
Revises: 0001
Create Date: 2026-05-14
"""
from typing import Sequence, Union
import sqlalchemy as sa
from alembic import op
revision: str = "0002"
down_revision: Union[str, None] = "0001"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
TAG_KINDS = (
"artist",
"character",
"fandom",
"general",
"series",
"archive",
"post",
"meta",
"rating",
)
def upgrade() -> None:
# --- Tag kind enum + fandom_id ---
tag_kind = sa.Enum(*TAG_KINDS, name="tag_kind")
tag_kind.create(op.get_bind(), checkfirst=True)
op.add_column(
"tag",
sa.Column("kind", tag_kind, nullable=False, server_default="general"),
)
op.add_column(
"tag",
sa.Column("fandom_id", sa.Integer(), nullable=True),
)
op.create_foreign_key(
"fk_tag_fandom_id_tag",
"tag",
"tag",
["fandom_id"],
["id"],
ondelete="SET NULL",
)
# Drop the old global uniqueness on name; add kind+fandom-aware uniqueness.
op.drop_constraint("uq_tag_name", "tag", type_="unique")
op.drop_index("ix_tag_name", table_name="tag")
op.execute(
"""
CREATE UNIQUE INDEX uq_tag_name_kind_fandom
ON tag (name, kind, COALESCE(fandom_id, 0))
"""
)
# CHECK: fandom_id is only allowed for character kind.
op.create_check_constraint(
"ck_tag_fandom_requires_character",
"tag",
"(fandom_id IS NULL) OR (kind = 'character')",
)
# Drop the old namespace column — superseded by kind.
op.drop_index("ix_tag_namespace", table_name="tag")
op.drop_column("tag", "namespace")
# --- ImportBatch ---
op.create_table(
"import_batch",
sa.Column("id", sa.Integer(), nullable=False),
sa.Column("triggered_by", sa.String(length=32), nullable=False),
sa.Column("source_path", sa.Text(), nullable=False),
sa.Column("scan_mode", sa.String(length=16), nullable=False),
sa.Column(
"started_at",
sa.DateTime(timezone=True),
nullable=False,
server_default=sa.func.now(),
),
sa.Column("finished_at", sa.DateTime(timezone=True), nullable=True),
sa.Column("total_files", sa.Integer(), nullable=False, server_default="0"),
sa.Column("imported", sa.Integer(), nullable=False, server_default="0"),
sa.Column("skipped", sa.Integer(), nullable=False, server_default="0"),
sa.Column("failed", sa.Integer(), nullable=False, server_default="0"),
sa.Column("status", sa.String(length=16), nullable=False, server_default="running"),
sa.PrimaryKeyConstraint("id", name="pk_import_batch"),
)
op.create_index("ix_import_batch_status", "import_batch", ["status"])
# --- ImportTask ---
op.create_table(
"import_task",
sa.Column("id", sa.Integer(), nullable=False),
sa.Column("batch_id", sa.Integer(), nullable=False),
sa.Column("source_path", sa.Text(), nullable=False),
sa.Column("task_type", sa.String(length=16), nullable=False),
sa.Column("status", sa.String(length=16), nullable=False, server_default="pending"),
sa.Column("result_image_id", sa.Integer(), nullable=True),
sa.Column("error", sa.Text(), nullable=True),
sa.Column("size_bytes", sa.BigInteger(), nullable=True),
sa.Column(
"created_at",
sa.DateTime(timezone=True),
nullable=False,
server_default=sa.func.now(),
),
sa.Column("started_at", sa.DateTime(timezone=True), nullable=True),
sa.Column("finished_at", sa.DateTime(timezone=True), nullable=True),
sa.ForeignKeyConstraint(
["batch_id"],
["import_batch.id"],
name="fk_import_task_batch_id_import_batch",
ondelete="CASCADE",
),
sa.ForeignKeyConstraint(
["result_image_id"],
["image_record.id"],
name="fk_import_task_result_image_id_image_record",
ondelete="SET NULL",
),
sa.PrimaryKeyConstraint("id", name="pk_import_task"),
)
op.create_index("ix_import_task_batch_id", "import_task", ["batch_id"])
op.create_index("ix_import_task_status", "import_task", ["status"])
op.create_index(
"ix_import_task_created_at_desc",
"import_task",
[sa.text("created_at DESC")],
)
# --- ImportSettings (single-row table) ---
op.create_table(
"import_settings",
sa.Column("id", sa.Integer(), nullable=False),
sa.Column("import_scan_path", sa.Text(), nullable=False, server_default="/import"),
sa.Column("min_width", sa.Integer(), nullable=False, server_default="0"),
sa.Column("min_height", sa.Integer(), nullable=False, server_default="0"),
sa.Column(
"skip_transparent", sa.Boolean(), nullable=False, server_default=sa.false()
),
sa.Column(
"transparency_threshold",
sa.Float(),
nullable=False,
server_default="0.9",
),
sa.Column(
"skip_single_color", sa.Boolean(), nullable=False, server_default=sa.false()
),
sa.Column(
"single_color_threshold",
sa.Float(),
nullable=False,
server_default="0.95",
),
sa.Column("single_color_tolerance", sa.Integer(), nullable=False, server_default="30"),
sa.PrimaryKeyConstraint("id", name="pk_import_settings"),
sa.CheckConstraint("id = 1", name="ck_import_settings_singleton"),
)
# Seed the single row immediately so callers can always SELECT id=1.
op.execute("INSERT INTO import_settings (id) VALUES (1)")
# --- ImageRecord additions ---
op.add_column(
"image_record",
sa.Column(
"integrity_status",
sa.String(length=24),
nullable=False,
server_default="unknown",
),
)
op.create_index(
"ix_image_record_integrity_status",
"image_record",
["integrity_status"],
)
def downgrade() -> None:
op.drop_index("ix_image_record_integrity_status", table_name="image_record")
op.drop_column("image_record", "integrity_status")
op.drop_table("import_settings")
op.drop_index("ix_import_task_created_at_desc", table_name="import_task")
op.drop_index("ix_import_task_status", table_name="import_task")
op.drop_index("ix_import_task_batch_id", table_name="import_task")
op.drop_table("import_task")
op.drop_index("ix_import_batch_status", table_name="import_batch")
op.drop_table("import_batch")
op.drop_constraint("ck_tag_fandom_requires_character", "tag", type_="check")
op.execute("DROP INDEX uq_tag_name_kind_fandom")
op.add_column("tag", sa.Column("namespace", sa.String(length=64), nullable=True))
op.create_index("ix_tag_namespace", "tag", ["namespace"])
op.create_index("ix_tag_name", "tag", ["name"], unique=False)
op.create_unique_constraint("uq_tag_name", "tag", ["name"])
op.drop_constraint("fk_tag_fandom_id_tag", "tag", type_="foreignkey")
op.drop_column("tag", "fandom_id")
op.drop_column("tag", "kind")
sa.Enum(name="tag_kind").drop(op.get_bind(), checkfirst=True)
-172
View File
@@ -1,172 +0,0 @@
"""fc2b: ML pipeline — allowlist, aliases, centroids, ml_settings
Revision ID: 0003
Revises: 0002
Create Date: 2026-05-15
"""
from typing import Sequence, Union
import sqlalchemy as sa
from alembic import op
from pgvector.sqlalchemy import Vector
revision: str = "0003"
down_revision: Union[str, None] = "0002"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
# 3.1 rename wd14_* -> tagger_*
op.alter_column("image_record", "wd14_predictions", new_column_name="tagger_predictions")
op.alter_column(
"image_record", "wd14_model_version", new_column_name="tagger_model_version"
)
# 3.2 tag_allowlist
op.create_table(
"tag_allowlist",
sa.Column("tag_id", sa.Integer(), nullable=False),
sa.Column(
"min_confidence", sa.Float(), nullable=False, server_default="0.95"
),
sa.Column(
"added_at", sa.DateTime(timezone=True), nullable=False,
server_default=sa.func.now(),
),
sa.ForeignKeyConstraint(
["tag_id"], ["tag.id"], name="fk_tag_allowlist_tag_id_tag",
ondelete="CASCADE",
),
sa.PrimaryKeyConstraint("tag_id", name="pk_tag_allowlist"),
sa.CheckConstraint(
"min_confidence > 0 AND min_confidence <= 1",
name="ck_tag_allowlist_confidence_range",
),
)
# 3.3 tag_suggestion_rejection
op.create_table(
"tag_suggestion_rejection",
sa.Column("image_record_id", sa.Integer(), nullable=False),
sa.Column("tag_id", sa.Integer(), nullable=False),
sa.Column(
"rejected_at", sa.DateTime(timezone=True), nullable=False,
server_default=sa.func.now(),
),
sa.ForeignKeyConstraint(
["image_record_id"], ["image_record.id"],
name="fk_tsr_image_record_id_image_record", ondelete="CASCADE",
),
sa.ForeignKeyConstraint(
["tag_id"], ["tag.id"], name="fk_tsr_tag_id_tag", ondelete="CASCADE",
),
sa.PrimaryKeyConstraint(
"image_record_id", "tag_id", name="pk_tag_suggestion_rejection"
),
)
op.create_index(
"ix_tag_suggestion_rejection_tag", "tag_suggestion_rejection", ["tag_id"]
)
# 3.4 tag_alias
op.create_table(
"tag_alias",
sa.Column("alias_string", sa.String(length=255), nullable=False),
sa.Column("alias_category", sa.String(length=32), nullable=False),
sa.Column("canonical_tag_id", sa.Integer(), nullable=False),
sa.Column(
"created_at", sa.DateTime(timezone=True), nullable=False,
server_default=sa.func.now(),
),
sa.ForeignKeyConstraint(
["canonical_tag_id"], ["tag.id"],
name="fk_tag_alias_canonical_tag_id_tag", ondelete="CASCADE",
),
sa.PrimaryKeyConstraint(
"alias_string", "alias_category", name="pk_tag_alias"
),
)
op.create_index("ix_tag_alias_canonical", "tag_alias", ["canonical_tag_id"])
# 3.5 tag_reference_embedding (centroids)
op.create_table(
"tag_reference_embedding",
sa.Column("tag_id", sa.Integer(), nullable=False),
sa.Column("embedding", Vector(1152), nullable=False),
sa.Column("reference_count", sa.Integer(), nullable=False),
sa.Column("model_version", sa.String(length=128), nullable=False),
sa.Column(
"updated_at", sa.DateTime(timezone=True), nullable=False,
server_default=sa.func.now(),
),
sa.ForeignKeyConstraint(
["tag_id"], ["tag.id"],
name="fk_tag_reference_embedding_tag_id_tag", ondelete="CASCADE",
),
sa.PrimaryKeyConstraint("tag_id", name="pk_tag_reference_embedding"),
)
# 3.6 ml_settings singleton
op.create_table(
"ml_settings",
sa.Column("id", sa.Integer(), nullable=False),
sa.Column(
"suggestion_threshold_artist", sa.Float(), nullable=False,
server_default="0.30",
),
sa.Column(
"suggestion_threshold_character", sa.Float(), nullable=False,
server_default="0.50",
),
sa.Column(
"suggestion_threshold_copyright", sa.Float(), nullable=False,
server_default="0.50",
),
sa.Column(
"suggestion_threshold_general", sa.Float(), nullable=False,
server_default="0.95",
),
sa.Column(
"centroid_similarity_threshold", sa.Float(), nullable=False,
server_default="0.55",
),
sa.Column(
"min_reference_images", sa.Integer(), nullable=False,
server_default="5",
),
sa.Column(
"tagger_model_version", sa.String(length=128), nullable=False,
server_default="camie-tagger-v2",
),
sa.Column(
"embedder_model_version", sa.String(length=128), nullable=False,
server_default="siglip-so400m-patch14-384",
),
sa.Column(
"updated_at", sa.DateTime(timezone=True), nullable=False,
server_default=sa.func.now(),
),
sa.PrimaryKeyConstraint("id", name="pk_ml_settings"),
sa.CheckConstraint("id = 1", name="ck_ml_settings_singleton"),
)
op.execute("INSERT INTO ml_settings (id) VALUES (1)")
def downgrade() -> None:
op.drop_table("ml_settings")
op.drop_table("tag_reference_embedding")
op.drop_index("ix_tag_alias_canonical", table_name="tag_alias")
op.drop_table("tag_alias")
op.drop_index(
"ix_tag_suggestion_rejection_tag", table_name="tag_suggestion_rejection"
)
op.drop_table("tag_suggestion_rejection")
op.drop_table("tag_allowlist")
op.alter_column(
"image_record", "tagger_model_version", new_column_name="wd14_model_version"
)
op.alter_column(
"image_record", "tagger_predictions", new_column_name="wd14_predictions"
)
@@ -1,23 +0,0 @@
"""fc2c-i: enable tsm_system_rows for scalable random sampling
Revision ID: 0004
Revises: 0003
Create Date: 2026-05-15
"""
from typing import Sequence, Union
from alembic import op
revision: str = "0004"
down_revision: Union[str, None] = "0003"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
op.execute("CREATE EXTENSION IF NOT EXISTS tsm_system_rows")
def downgrade() -> None:
op.execute("DROP EXTENSION IF EXISTS tsm_system_rows")
@@ -1,50 +0,0 @@
"""fc2c-iii-a: series_page ordered membership
Revision ID: 0005
Revises: 0004
Create Date: 2026-05-16
"""
from typing import Sequence, Union
import sqlalchemy as sa
from alembic import op
revision: str = "0005"
down_revision: Union[str, None] = "0004"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
op.create_table(
"series_page",
sa.Column("id", sa.Integer(), nullable=False),
sa.Column("series_tag_id", sa.Integer(), nullable=False),
sa.Column("image_id", sa.Integer(), nullable=False),
sa.Column("page_number", sa.Integer(), nullable=False),
sa.Column(
"created_at", sa.DateTime(timezone=True),
nullable=False, server_default=sa.func.now(),
),
sa.Column(
"updated_at", sa.DateTime(timezone=True),
nullable=False, server_default=sa.func.now(),
),
sa.ForeignKeyConstraint(
["series_tag_id"], ["tag.id"], ondelete="CASCADE"
),
sa.ForeignKeyConstraint(
["image_id"], ["image_record.id"], ondelete="CASCADE"
),
sa.PrimaryKeyConstraint("id"),
sa.UniqueConstraint("image_id", name="uq_series_page_image"),
)
op.create_index(
"ix_series_page_series_tag_id", "series_page", ["series_tag_id"]
)
def downgrade() -> None:
op.drop_index("ix_series_page_series_tag_id", table_name="series_page")
op.drop_table("series_page")
@@ -1,30 +0,0 @@
"""fc2d: import_settings.phash_threshold
Revision ID: 0006
Revises: 0005
Create Date: 2026-05-17
"""
from typing import Sequence, Union
import sqlalchemy as sa
from alembic import op
revision: str = "0006"
down_revision: Union[str, None] = "0005"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
op.add_column(
"import_settings",
sa.Column(
"phash_threshold", sa.Integer(),
nullable=False, server_default="10",
),
)
def downgrade() -> None:
op.drop_column("import_settings", "phash_threshold")
@@ -1,31 +0,0 @@
"""fc2d-iv: post.description + post.attachment_count
Revision ID: 0007
Revises: 0006
Create Date: 2026-05-18
"""
from typing import Sequence, Union
import sqlalchemy as sa
from alembic import op
revision: str = "0007"
down_revision: Union[str, None] = "0006"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
op.add_column(
"post", sa.Column("description", sa.Text(), nullable=True)
)
op.add_column(
"post",
sa.Column("attachment_count", sa.Integer(), nullable=True),
)
def downgrade() -> None:
op.drop_column("post", "attachment_count")
op.drop_column("post", "description")
@@ -1,52 +0,0 @@
"""fc2d-vii-c: image_record.artist_id + backfill + drop artist tags
Revision ID: 0008
Revises: 0007
Create Date: 2026-05-18
Internal forward-correctness migration (the big legacy-import migration
stays deferred). downgrade() does NOT recreate deleted artist tags;
downgrade is dev-only and the data is reconstructable by re-import.
"""
from typing import Sequence, Union
import sqlalchemy as sa
from alembic import op
from backend.app.utils.artist_backfill import (
BACKFILL_PRIMARY_SQL,
BACKFILL_PROVENANCE_SQL,
BACKFILL_TAG_SQL,
DELETE_ARTIST_TAGS_SQL,
)
revision: str = "0008"
down_revision: Union[str, None] = "0007"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
op.add_column(
"image_record",
sa.Column("artist_id", sa.Integer(), nullable=True),
)
op.create_foreign_key(
"fk_image_record_artist_id", "image_record", "artist",
["artist_id"], ["id"], ondelete="SET NULL",
)
op.create_index(
"ix_image_record_artist_id", "image_record", ["artist_id"],
)
op.execute(BACKFILL_PRIMARY_SQL)
op.execute(BACKFILL_PROVENANCE_SQL)
op.execute(BACKFILL_TAG_SQL)
op.execute(DELETE_ARTIST_TAGS_SQL)
def downgrade() -> None:
op.drop_index("ix_image_record_artist_id", table_name="image_record")
op.drop_constraint(
"fk_image_record_artist_id", "image_record", type_="foreignkey"
)
op.drop_column("image_record", "artist_id")
@@ -1,68 +0,0 @@
"""fc2d-iii: post_attachment + import_batch.attachments
Revision ID: 0009
Revises: 0008
Create Date: 2026-05-19
Internal forward-correctness migration (big legacy-import migration
stays deferred). No backfill — no attachments exist yet.
"""
from typing import Sequence, Union
import sqlalchemy as sa
from alembic import op
revision: str = "0009"
down_revision: Union[str, None] = "0008"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
op.create_table(
"post_attachment",
sa.Column("id", sa.Integer(), primary_key=True),
sa.Column(
"post_id", sa.Integer(),
sa.ForeignKey("post.id", ondelete="SET NULL"), nullable=True,
),
sa.Column(
"artist_id", sa.Integer(),
sa.ForeignKey("artist.id", ondelete="SET NULL"), nullable=True,
),
sa.Column("sha256", sa.String(64), nullable=False),
sa.Column("path", sa.Text(), nullable=False),
sa.Column("original_filename", sa.Text(), nullable=False),
sa.Column("ext", sa.String(32), nullable=False),
sa.Column("mime", sa.String(128), nullable=True),
sa.Column("size_bytes", sa.BigInteger(), nullable=False),
sa.Column(
"captured_at", sa.DateTime(timezone=True),
server_default=sa.func.now(), nullable=False,
),
)
op.create_index(
"ix_post_attachment_sha256", "post_attachment", ["sha256"],
unique=True,
)
op.create_index(
"ix_post_attachment_post_id", "post_attachment", ["post_id"],
)
op.create_index(
"ix_post_attachment_artist_id", "post_attachment", ["artist_id"],
)
op.add_column(
"import_batch",
sa.Column(
"attachments", sa.Integer(), nullable=False,
server_default="0",
),
)
def downgrade() -> None:
op.drop_column("import_batch", "attachments")
op.drop_index("ix_post_attachment_artist_id", table_name="post_attachment")
op.drop_index("ix_post_attachment_post_id", table_name="post_attachment")
op.drop_index("ix_post_attachment_sha256", table_name="post_attachment")
op.drop_table("post_attachment")
@@ -1,32 +0,0 @@
"""fc3a: unique(source.artist_id, source.platform, source.url)
Revision ID: 0010
Revises: 0009
Create Date: 2026-05-20
Enforces FC-3a's dedup invariant at the DB level. No backfill — no
existing rows are expected to collide; if they do the migration will
fail loudly (intended).
"""
from typing import Sequence, Union
from alembic import op
revision: str = "0010"
down_revision: Union[str, None] = "0009"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
op.create_unique_constraint(
"uq_source_artist_platform_url",
"source",
["artist_id", "platform", "url"],
)
def downgrade() -> None:
op.drop_constraint(
"uq_source_artist_platform_url", "source", type_="unique"
)
@@ -1,41 +0,0 @@
"""fc3b: rename credential.kind -> credential_type, drop status, add last_verified
Revision ID: 0011
Revises: 0010
Create Date: 2026-05-20
Aligns the credential table with the GallerySubscriber wire-field names
so the existing browser extension can POST to FC unmodified. Greenfield —
no rows exist in production yet, so no data preservation logic is
needed; the rename uses ALTER COLUMN rather than copy-then-drop.
"""
from typing import Sequence, Union
import sqlalchemy as sa
from alembic import op
revision: str = "0011"
down_revision: Union[str, None] = "0010"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
op.alter_column("credential", "kind", new_column_name="credential_type")
op.drop_column("credential", "status")
op.add_column(
"credential",
sa.Column("last_verified", sa.DateTime(timezone=True), nullable=True),
)
def downgrade() -> None:
op.drop_column("credential", "last_verified")
op.add_column(
"credential",
sa.Column(
"status", sa.String(length=32), nullable=False,
server_default="active",
),
)
op.alter_column("credential", "credential_type", new_column_name="kind")
-36
View File
@@ -1,36 +0,0 @@
"""fc3b: app_setting key/value table
Revision ID: 0012
Revises: 0011
Create Date: 2026-05-20
A simple key/value table for small app settings that don't fit
ImportSettings. Initially seeds only `extension_api_key` (done in
create_app on first boot — not in the migration, to keep it
deterministic and independent of randomness).
"""
from typing import Sequence, Union
import sqlalchemy as sa
from alembic import op
revision: str = "0012"
down_revision: Union[str, None] = "0011"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
op.create_table(
"app_setting",
sa.Column("key", sa.String(length=64), primary_key=True),
sa.Column("value", sa.Text(), nullable=False),
sa.Column(
"updated_at", sa.DateTime(timezone=True),
nullable=False, server_default=sa.func.now(),
),
)
def downgrade() -> None:
op.drop_table("app_setting")
@@ -1,52 +0,0 @@
"""fc3c: download_event.metadata + import_settings downloader fields
Revision ID: 0013
Revises: 0012
Create Date: 2026-05-20
Additive only. download_event.metadata is the rich JSONB blob FC-3c
populates per run (run_stats, stdout/stderr, quarantined paths, import
summary). import_settings gains two operator-tunable downloader knobs:
download_rate_limit_seconds (gallery-dl extractor.sleep) and
download_validate_files (toggle the magic-byte validator).
"""
from typing import Sequence, Union
import sqlalchemy as sa
from alembic import op
from sqlalchemy.dialects import postgresql
revision: str = "0013"
down_revision: Union[str, None] = "0012"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
op.add_column(
"download_event",
sa.Column(
"metadata", postgresql.JSONB,
nullable=False, server_default=sa.text("'{}'::jsonb"),
),
)
op.add_column(
"import_settings",
sa.Column(
"download_rate_limit_seconds", sa.Float(),
nullable=False, server_default="3.0",
),
)
op.add_column(
"import_settings",
sa.Column(
"download_validate_files", sa.Boolean(),
nullable=False, server_default=sa.true(),
),
)
def downgrade() -> None:
op.drop_column("import_settings", "download_validate_files")
op.drop_column("import_settings", "download_rate_limit_seconds")
op.drop_column("download_event", "metadata")
-58
View File
@@ -1,58 +0,0 @@
"""fc3d: scheduling + source health columns
Revision ID: 0014
Revises: 0013
Create Date: 2026-05-21
Additive only. source.consecutive_failures (default 0, DownloadService
finalize hook owns the writes). import_settings gains the three
scheduling knobs (global default interval, event retention, failure
warning threshold).
"""
from typing import Sequence, Union
import sqlalchemy as sa
from alembic import op
revision: str = "0014"
down_revision: Union[str, None] = "0013"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
op.add_column(
"source",
sa.Column(
"consecutive_failures", sa.Integer(),
nullable=False, server_default="0",
),
)
op.add_column(
"import_settings",
sa.Column(
"download_schedule_default_seconds", sa.Integer(),
nullable=False, server_default="28800",
),
)
op.add_column(
"import_settings",
sa.Column(
"download_event_retention_days", sa.Integer(),
nullable=False, server_default="90",
),
)
op.add_column(
"import_settings",
sa.Column(
"download_failure_warning_threshold", sa.Integer(),
nullable=False, server_default="5",
),
)
def downgrade() -> None:
op.drop_column("import_settings", "download_failure_warning_threshold")
op.drop_column("import_settings", "download_event_retention_days")
op.drop_column("import_settings", "download_schedule_default_seconds")
op.drop_column("source", "consecutive_failures")
@@ -1,51 +0,0 @@
"""fc5: migration_run table
Revision ID: 0015
Revises: 0014
Create Date: 2026-05-22
Additive only. New table tracks each invocation of the FC-5 migration
tooling (backup, gs, ir, ml_queue, verify, rollback). kind/status are
plain String(32) — values validated at the API layer per the spec, not
a Postgres ENUM (so adding kinds later doesn't need a schema migration).
"""
from typing import Sequence, Union
import sqlalchemy as sa
from alembic import op
from sqlalchemy.dialects import postgresql
revision: str = "0015"
down_revision: Union[str, None] = "0014"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
op.create_table(
"migration_run",
sa.Column("id", sa.Integer(), primary_key=True),
sa.Column("kind", sa.String(32), nullable=False, index=True),
sa.Column("status", sa.String(32), nullable=False, index=True),
sa.Column(
"dry_run", sa.Boolean(), nullable=False, server_default=sa.false(),
),
sa.Column(
"started_at", sa.DateTime(timezone=True),
nullable=False, server_default=sa.func.now(),
),
sa.Column("finished_at", sa.DateTime(timezone=True), nullable=True),
sa.Column(
"counts", postgresql.JSONB,
nullable=False, server_default=sa.text("'{}'::jsonb"),
),
sa.Column("error", sa.Text(), nullable=True),
sa.Column(
"metadata", postgresql.JSONB,
nullable=False, server_default=sa.text("'{}'::jsonb"),
),
)
def downgrade() -> None:
op.drop_table("migration_run")
-86
View File
@@ -1,86 +0,0 @@
"""fc3i: task_run table
Revision ID: 0016
Revises: 0015
Create Date: 2026-05-24
Additive only. New table records every Celery task attempt via signal
handlers (backend.app.celery_signals). Status is plain String(16) not
Postgres ENUM (per feedback_check_existing_enums: ENUM columns hard-
fail at INSERT, String columns extend cleanly).
Composite indexes anticipate the three dashboard panes:
- (queue, started_at desc) — per-lane recent activity
- (status, started_at desc) — recent failures pane
- (task_name, started_at desc) — drill-down by task
Indexed columns get individual indexes via `index=True` on the model;
the composites below cover the multi-column lookups.
"""
from typing import Sequence, Union
import sqlalchemy as sa
from alembic import op
revision: str = "0016"
down_revision: Union[str, None] = "0015"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
op.create_table(
"task_run",
sa.Column("id", sa.Integer(), primary_key=True),
sa.Column("celery_task_id", sa.String(length=64), nullable=False),
sa.Column("queue", sa.String(length=32), nullable=False),
sa.Column("task_name", sa.String(length=128), nullable=False),
sa.Column("target_id", sa.Integer(), nullable=True),
sa.Column("started_at", sa.DateTime(timezone=True), nullable=False),
sa.Column("finished_at", sa.DateTime(timezone=True), nullable=True),
sa.Column("duration_ms", sa.Integer(), nullable=True),
sa.Column(
"status", sa.String(length=16), nullable=False,
server_default="running",
),
sa.Column("error_type", sa.String(length=128), nullable=True),
sa.Column("error_message", sa.Text(), nullable=True),
sa.Column("retry_count", sa.Integer(), nullable=True),
sa.Column("worker_hostname", sa.String(length=128), nullable=True),
sa.Column("args_summary", sa.String(length=255), nullable=True),
)
# Single-column indexes (matches Mapped[...].index=True on model).
op.create_index("ix_task_run_celery_task_id", "task_run", ["celery_task_id"])
op.create_index("ix_task_run_queue", "task_run", ["queue"])
op.create_index("ix_task_run_task_name", "task_run", ["task_name"])
op.create_index("ix_task_run_started_at", "task_run", ["started_at"])
op.create_index("ix_task_run_finished_at", "task_run", ["finished_at"])
op.create_index("ix_task_run_status", "task_run", ["status"])
# Composite indexes for dashboard query patterns.
op.create_index(
"ix_task_run_queue_started",
"task_run", ["queue", sa.text("started_at DESC")],
)
op.create_index(
"ix_task_run_status_started",
"task_run", ["status", sa.text("started_at DESC")],
)
op.create_index(
"ix_task_run_name_started",
"task_run", ["task_name", sa.text("started_at DESC")],
)
def downgrade() -> None:
op.drop_index("ix_task_run_name_started", table_name="task_run")
op.drop_index("ix_task_run_status_started", table_name="task_run")
op.drop_index("ix_task_run_queue_started", table_name="task_run")
op.drop_index("ix_task_run_status", table_name="task_run")
op.drop_index("ix_task_run_finished_at", table_name="task_run")
op.drop_index("ix_task_run_started_at", table_name="task_run")
op.drop_index("ix_task_run_task_name", table_name="task_run")
op.drop_index("ix_task_run_queue", table_name="task_run")
op.drop_index("ix_task_run_celery_task_id", table_name="task_run")
op.drop_table("task_run")
-82
View File
@@ -1,82 +0,0 @@
"""fc3h: backup_run table
Revision ID: 0017
Revises: 0016
Create Date: 2026-05-24
Additive. New table records every backup/restore attempt with artifact
metadata. Lifecycle tracking lives in task_run from FC-3i; this is
artifact-only (paths, sizes, tag, restore lineage).
"""
from typing import Sequence, Union
import sqlalchemy as sa
from alembic import op
revision: str = "0017"
down_revision: Union[str, None] = "0016"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
op.create_table(
"backup_run",
sa.Column("id", sa.Integer(), primary_key=True),
sa.Column("kind", sa.String(length=16), nullable=False),
sa.Column(
"status", sa.String(length=16), nullable=False,
server_default="pending",
),
sa.Column("tag", sa.String(length=64), nullable=True),
sa.Column("triggered_by", sa.String(length=32), nullable=False),
sa.Column("started_at", sa.DateTime(timezone=True), nullable=False),
sa.Column("finished_at", sa.DateTime(timezone=True), nullable=True),
sa.Column("sql_path", sa.Text(), nullable=True),
sa.Column("tar_path", sa.Text(), nullable=True),
sa.Column("size_bytes", sa.BigInteger(), nullable=True),
sa.Column("error", sa.Text(), nullable=True),
sa.Column(
"manifest", sa.JSON(), nullable=False, server_default="{}",
),
sa.Column(
"restored_from_id", sa.Integer(),
sa.ForeignKey("backup_run.id", ondelete="SET NULL"),
nullable=True,
),
)
# Single-column indexes (matches Mapped[...].index=True).
op.create_index("ix_backup_run_kind", "backup_run", ["kind"])
op.create_index("ix_backup_run_status", "backup_run", ["status"])
op.create_index("ix_backup_run_tag", "backup_run", ["tag"])
op.create_index("ix_backup_run_started_at", "backup_run", ["started_at"])
op.create_index("ix_backup_run_finished_at", "backup_run", ["finished_at"])
# Composite indexes for dashboard query patterns.
op.create_index(
"ix_backup_run_kind_started",
"backup_run", ["kind", sa.text("started_at DESC")],
)
op.create_index(
"ix_backup_run_status_finished",
"backup_run", ["status", sa.text("finished_at DESC")],
)
# Partial index: only tagged rows participate in retention-exempt query.
op.create_index(
"ix_backup_run_tag_partial",
"backup_run", ["tag"],
postgresql_where=sa.text("tag IS NOT NULL"),
)
def downgrade() -> None:
op.drop_index("ix_backup_run_tag_partial", table_name="backup_run")
op.drop_index("ix_backup_run_status_finished", table_name="backup_run")
op.drop_index("ix_backup_run_kind_started", table_name="backup_run")
op.drop_index("ix_backup_run_finished_at", table_name="backup_run")
op.drop_index("ix_backup_run_started_at", table_name="backup_run")
op.drop_index("ix_backup_run_tag", table_name="backup_run")
op.drop_index("ix_backup_run_status", table_name="backup_run")
op.drop_index("ix_backup_run_kind", table_name="backup_run")
op.drop_table("backup_run")
@@ -1,62 +0,0 @@
"""fc3h: backup_* knobs on import_settings
Revision ID: 0018
Revises: 0017
Create Date: 2026-05-24
Adds four columns to the singleton import_settings row:
- backup_db_nightly_enabled (default False — opt-in)
- backup_db_nightly_hour_utc (default 3)
- backup_db_keep_last_n (default 14)
- backup_images_keep_last_n (default 3)
server_default ensures the singleton row is backfilled in place
without an UPDATE statement.
"""
from typing import Sequence, Union
import sqlalchemy as sa
from alembic import op
revision: str = "0018"
down_revision: Union[str, None] = "0017"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
op.add_column(
"import_settings",
sa.Column(
"backup_db_nightly_enabled", sa.Boolean(),
nullable=False, server_default=sa.false(),
),
)
op.add_column(
"import_settings",
sa.Column(
"backup_db_nightly_hour_utc", sa.Integer(),
nullable=False, server_default="3",
),
)
op.add_column(
"import_settings",
sa.Column(
"backup_db_keep_last_n", sa.Integer(),
nullable=False, server_default="14",
),
)
op.add_column(
"import_settings",
sa.Column(
"backup_images_keep_last_n", sa.Integer(),
nullable=False, server_default="3",
),
)
def downgrade() -> None:
op.drop_column("import_settings", "backup_images_keep_last_n")
op.drop_column("import_settings", "backup_db_keep_last_n")
op.drop_column("import_settings", "backup_db_nightly_hour_utc")
op.drop_column("import_settings", "backup_db_nightly_enabled")
@@ -1,38 +0,0 @@
"""import_batch.refreshed counter for deep-scan sidecar re-application
Revision ID: 0019
Revises: 0018
Create Date: 2026-05-25
Adds a `refreshed` counter to `import_batch`, mirroring the existing
`imported`/`skipped`/`failed`/`attachments` columns. Deep scan now
re-applies sidecar metadata to already-imported files (the IR feature
that didn't make the FC port the first time); a "refreshed" outcome
increments this counter so the UI can surface "X new, Y refreshed"
instead of the misleading "Scan complete — no new files" message.
server_default=0 backfills existing rows in place — no UPDATE needed.
"""
from typing import Sequence, Union
import sqlalchemy as sa
from alembic import op
revision: str = "0019"
down_revision: Union[str, None] = "0018"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
op.add_column(
"import_batch",
sa.Column(
"refreshed", sa.Integer(),
nullable=False, server_default=sa.text("0"),
),
)
def downgrade() -> None:
op.drop_column("import_batch", "refreshed")
@@ -1,65 +0,0 @@
"""fc-cleanup: library_audit_run table for async transparency/single_color audits
Revision ID: 0020
Revises: 0019
Create Date: 2026-05-26
The table backs the async audit lifecycle: rule + params snapshot, status
state machine ('running''ready''applied'/'cancelled'/'error'), and
the matched_ids JSONB array that the apply step deletes. Capped at 50k IDs
per row by the scan task (oversize = rule too aggressive, operator narrows
before re-running).
"""
from typing import Sequence, Union
import sqlalchemy as sa
from alembic import op
from sqlalchemy.dialects import postgresql
revision: str = "0020"
down_revision: Union[str, None] = "0019"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
op.create_table(
"library_audit_run",
sa.Column("id", sa.Integer(), primary_key=True),
sa.Column("rule", sa.String(32), nullable=False),
sa.Column("params", postgresql.JSONB(astext_type=sa.Text()), nullable=False),
sa.Column(
"status", sa.String(16),
nullable=False, server_default="running",
),
sa.Column(
"started_at", sa.DateTime(timezone=True),
nullable=False, server_default=sa.func.now(),
),
sa.Column("finished_at", sa.DateTime(timezone=True), nullable=True),
sa.Column(
"scanned_count", sa.Integer(),
nullable=False, server_default="0",
),
sa.Column(
"matched_count", sa.Integer(),
nullable=False, server_default="0",
),
sa.Column(
"matched_ids", postgresql.JSONB(astext_type=sa.Text()),
nullable=False, server_default=sa.text("'[]'::jsonb"),
),
sa.Column("error", sa.Text(), nullable=True),
)
op.create_index(
"ix_library_audit_run_rule", "library_audit_run", ["rule"],
)
op.create_index(
"ix_library_audit_run_status", "library_audit_run", ["status"],
)
def downgrade() -> None:
op.drop_index("ix_library_audit_run_status", table_name="library_audit_run")
op.drop_index("ix_library_audit_run_rule", table_name="library_audit_run")
op.drop_table("library_audit_run")
@@ -1,54 +0,0 @@
"""provenance-race: dedupe + UNIQUE(image_record_id, post_id) on image_provenance
Revision ID: 0021
Revises: 0020
Create Date: 2026-05-26
Closes the race in Importer._apply_sidecar's existence-check + INSERT pattern.
Two workers writing for the same (image, post) pair both saw no existing row
and both inserted, leaving duplicates that then broke .scalar_one_or_none()
on every subsequent deep-scan rederive against those images
(MultipleResultsFound). Most plausibly seeded when the 5-min recovery sweep
re-enqueued a still-running long-import task and the second worker collided
with the first inside _apply_sidecar.
Migration steps:
1. DELETE all but min(id) per (image_record_id, post_id) pair. Operator's
DB had 2 affected pairs at write-time; harmless no-op if zero.
2. Add UNIQUE constraint so the importer's new savepoint+IntegrityError
recovery path can trip on collision and re-select, mirroring
uq_source_artist_platform_url and uq_post_source_external_id.
"""
from typing import Sequence, Union
from alembic import op
revision: str = "0021"
down_revision: Union[str, None] = "0020"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
op.execute(
"""
DELETE FROM image_provenance ip1
USING image_provenance ip2
WHERE ip1.image_record_id = ip2.image_record_id
AND ip1.post_id = ip2.post_id
AND ip1.id > ip2.id
"""
)
op.create_unique_constraint(
"uq_image_provenance_image_post",
"image_provenance",
["image_record_id", "post_id"],
)
def downgrade() -> None:
op.drop_constraint(
"uq_image_provenance_image_post",
"image_provenance",
type_="unique",
)
@@ -1,223 +0,0 @@
"""source-collapse: one Source per (artist, platform) — consolidate junk per-post Sources
Revision ID: 0022
Revises: 0021
Create Date: 2026-05-26
Closes the operator-flagged 2026-05-26 issue where the filesystem importer
called _find_or_create_source(url=sd.post_url), creating one Source row per
imported post URL. Operator's Atole artist had 406 Source rows where there
should have been 1 (the /cw/Atole subscription Source).
Source represents a subscription feed (one per artist+platform — the
gallery-dl URL polled by the FC-3 downloader). Posts hang off it. The
filesystem importer was misusing Source as a per-post key.
Migration steps per (artist_id, platform) group with >1 Source:
1. Pick canonical — prefer a URL NOT matching '/posts/<id>$' (real
campaign URL like /cw/Atole); else min(id).
2. PRE-merge any Posts under non-canonical sources whose
external_post_id ALREADY exists under the canonical source. (Same
gallery-dl post imported via two different sidecar paths can plant
two Post rows with identical external_post_id under different
Sources for the same artist.) Repoint ImageProvenance +
ImageRecord.primary_post_id to the canonical-side Post, dedupe
ImageProvenance against alembic 0021's uq, then delete the
non-canonical-side Post. This MUST happen before step 3 — Postgres
fires uq_post_source_external_id row-by-row during the bulk UPDATE
and the merge-after-reparent ordering 500s on first collision
(operator-hit during v26.05.26.1 deploy, 2026-05-26).
3. Reparent remaining Posts onto canonical (no collisions possible now).
4. Reparent ImageProvenance.source_id off the non-canonical sources.
5. Delete the orphan Source rows.
6. If the canonical Source's URL still looks like a per-post URL (no
campaign URL existed among candidates), rewrite it to
'sidecar:<platform>:<artist_slug>' so the artist detail page shows
something readable.
"""
from typing import Sequence, Union
from alembic import op
from sqlalchemy import text
revision: str = "0022"
down_revision: Union[str, None] = "0021"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
_POST_URL_RE = r"/posts/[^/]+$"
def upgrade() -> None:
conn = op.get_bind()
# Find (artist_id, platform) groups with > 1 Source row.
groups = conn.execute(text("""
SELECT artist_id, platform
FROM source
GROUP BY artist_id, platform
HAVING COUNT(*) > 1
""")).fetchall()
for artist_id, platform in groups:
rows = conn.execute(
text("""
SELECT id, url FROM source
WHERE artist_id = :a AND platform = :p
ORDER BY id ASC
"""),
{"a": artist_id, "p": platform},
).fetchall()
# Canonical: first row whose URL doesn't look like a per-post URL;
# else min(id).
canonical_id = None
for sid, url in rows:
if not _matches_post_url(url):
canonical_id = sid
break
if canonical_id is None:
canonical_id = rows[0][0]
other_ids = [sid for sid, _ in rows if sid != canonical_id]
if not other_ids:
continue
# STEP 2: PRE-merge ALL Posts with duplicate external_post_id
# across the entire (canonical + others) group, BEFORE the bulk
# reparent. Two cases must both be handled:
# (A) canonical has Post X with epid=N; an "other" source has
# Post Y with epid=N → after bulk UPDATE, (canonical, N)
# collides with itself.
# (B) two different "other" sources each have a Post with
# epid=N; canonical has none → after bulk UPDATE, both
# are repointed to (canonical, N) and the second collides.
# The earlier version of this migration only handled (A); the
# operator's deploy 2026-05-26 tripped (B) at line 139.
# Fix: group ALL Posts in the (artist, platform) by epid; for
# any group with count>1, pick the keep (prefer one already
# under canonical; else lowest id) and merge the rest into it.
all_posts = conn.execute(
text("""
SELECT external_post_id, id, source_id
FROM post
WHERE source_id = :canonical OR source_id = ANY(:others)
ORDER BY external_post_id, id
"""),
{"canonical": canonical_id, "others": other_ids},
).fetchall()
by_epid: dict = {}
for epid, post_id, src_id in all_posts:
by_epid.setdefault(epid, []).append((post_id, src_id))
for _epid, posts in by_epid.items():
if len(posts) <= 1:
continue
# Prefer a Post already under canonical as the keep.
canonical_posts = [p for p in posts if p[1] == canonical_id]
if canonical_posts:
keep_id = canonical_posts[0][0]
else:
keep_id = posts[0][0] # already sorted by id ASC
drop_ids = [p[0] for p in posts if p[0] != keep_id]
for drop_id in drop_ids:
# Pre-delete image_provenance rows under drop_ whose
# image_record_id ALREADY has a provenance under keep —
# the UPDATE below would otherwise repoint them and
# trip uq_image_provenance_image_post (alembic 0021)
# row-by-row before any after-the-fact dedupe could
# run. Operator's v26.05.26.3 deploy 2026-05-26 tripped
# this at line 123.
conn.execute(
text("""
DELETE FROM image_provenance
WHERE post_id = :drop_
AND image_record_id IN (
SELECT image_record_id FROM image_provenance
WHERE post_id = :keep
)
"""),
{"keep": keep_id, "drop_": drop_id},
)
# Now safe to repoint the survivors.
conn.execute(
text("""
UPDATE image_provenance SET post_id = :keep
WHERE post_id = :drop_
"""),
{"keep": keep_id, "drop_": drop_id},
)
conn.execute(
text("""
UPDATE image_record SET primary_post_id = :keep
WHERE primary_post_id = :drop_
"""),
{"keep": keep_id, "drop_": drop_id},
)
conn.execute(
text("DELETE FROM post WHERE id = :drop_"),
{"drop_": drop_id},
)
# STEP 3: Bulk reparent the remaining Posts off the other
# Sources. After step 2, no collisions on
# (canonical, external_post_id) are possible.
conn.execute(
text("""
UPDATE post SET source_id = :canonical
WHERE source_id = ANY(:others)
"""),
{"canonical": canonical_id, "others": other_ids},
)
# STEP 4: Reparent ImageProvenance.source_id (denormalized FK).
# No UNIQUE on source_id; safe bulk update.
conn.execute(
text("""
UPDATE image_provenance SET source_id = :canonical
WHERE source_id = ANY(:others)
"""),
{"canonical": canonical_id, "others": other_ids},
)
# STEP 5: Drop the orphan Sources.
conn.execute(
text("DELETE FROM source WHERE id = ANY(:others)"),
{"others": other_ids},
)
# If the canonical's URL still looks per-post (no campaign URL
# existed among the candidates), rewrite to a synthetic anchor so
# the artist detail page renders something readable.
canonical_url = conn.execute(
text("SELECT url FROM source WHERE id = :id"),
{"id": canonical_id},
).scalar_one()
if _matches_post_url(canonical_url):
slug = conn.execute(
text("SELECT slug FROM artist WHERE id = :id"),
{"id": artist_id},
).scalar_one()
conn.execute(
text("""
UPDATE source
SET url = :new_url, enabled = false
WHERE id = :id
"""),
{
"id": canonical_id,
"new_url": f"sidecar:{platform}:{slug}",
},
)
def downgrade() -> None:
# Lossy migration — orphan Sources deleted, Posts reparented, Posts
# merged. No safe downgrade. If you need to roll back the schema
# invariant, fork from 0021 and re-run filesystem imports.
pass
def _matches_post_url(url: str) -> bool:
"""True if url ends with /posts/<token> (gallery-dl-style per-post URL)."""
import re
return bool(re.search(_POST_URL_RE, url or ""))
@@ -1,99 +0,0 @@
"""drop meta + rating tag kinds — operator-retired 2026-05-26
Revision ID: 0023
Revises: 0022
Create Date: 2026-05-26
Operator decided meta + rating aren't valid tag kinds for FC. Per-row
behavior: DELETE existing rows (operator chose "clean break" over
"convert to general"). All cascading FKs (image_tag, tag_alias,
tag_allowlist, tag_reference_embedding, tag_suggestion_rejection,
series_page) use ondelete="CASCADE" so a single DELETE on tag cleans
the related rows in one go.
After the data cleanup, recreate the tag_kind ENUM without 'meta' /
'rating' (Postgres has no `ALTER TYPE ... DROP VALUE`; standard
rename-create-cast-drop dance). The server default 'general' is
dropped before the type swap and restored after.
"""
from typing import Sequence, Union
from alembic import op
revision: str = "0023"
down_revision: Union[str, None] = "0022"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
# 1. Delete tags of the retired kinds. CASCADE handles related tables.
op.execute("DELETE FROM tag WHERE kind IN ('meta', 'rating')")
# 2. Drop the CHECK constraint that references the enum's literal
# values. Postgres can't resolve `kind = 'character'` across the
# type swap below — the literal would bind to the new tag_kind
# but the column is on tag_kind_old, producing
# "operator does not exist: tag_kind = tag_kind_old".
# (Operator-hit during the v26.05.26.5 deploy attempt; ck was
# originally added by alembic 0002.) Recreated post-swap.
op.drop_constraint(
"ck_tag_fandom_requires_character", "tag", type_="check"
)
# 3. Drop the server default — ALTER COLUMN TYPE can't carry it
# across the type swap below.
op.execute("ALTER TABLE tag ALTER COLUMN kind DROP DEFAULT")
# 4. Recreate the tag_kind enum without meta/rating.
op.execute("ALTER TYPE tag_kind RENAME TO tag_kind_old")
op.execute(
"CREATE TYPE tag_kind AS ENUM ("
"'artist', 'character', 'fandom', 'general', "
"'series', 'archive', 'post'"
")"
)
op.execute(
"ALTER TABLE tag "
"ALTER COLUMN kind TYPE tag_kind "
"USING kind::text::tag_kind"
)
op.execute("DROP TYPE tag_kind_old")
# 5. Restore the server default.
op.execute("ALTER TABLE tag ALTER COLUMN kind SET DEFAULT 'general'")
# 6. Restore the CHECK constraint (now bound to the new tag_kind).
op.create_check_constraint(
"ck_tag_fandom_requires_character",
"tag",
"(fandom_id IS NULL) OR (kind = 'character')",
)
def downgrade() -> None:
# Add the values back to the enum so old code can boot. The deleted
# tag rows are gone permanently — no safe restore.
op.drop_constraint(
"ck_tag_fandom_requires_character", "tag", type_="check"
)
op.execute("ALTER TABLE tag ALTER COLUMN kind DROP DEFAULT")
op.execute("ALTER TYPE tag_kind RENAME TO tag_kind_old")
op.execute(
"CREATE TYPE tag_kind AS ENUM ("
"'artist', 'character', 'fandom', 'general', "
"'series', 'archive', 'post', 'meta', 'rating'"
")"
)
op.execute(
"ALTER TABLE tag "
"ALTER COLUMN kind TYPE tag_kind "
"USING kind::text::tag_kind"
)
op.execute("DROP TYPE tag_kind_old")
op.execute("ALTER TABLE tag ALTER COLUMN kind SET DEFAULT 'general'")
op.create_check_constraint(
"ck_tag_fandom_requires_character",
"tag",
"(fandom_id IS NULL) OR (kind = 'character')",
)
@@ -1,80 +0,0 @@
"""backfill post.post_title from description first-line — 2026-05-27
Revision ID: 0024
Revises: 0023
Create Date: 2026-05-27
SubscribeStar gallery-dl always writes `title: ""` and embeds the leading
sentence inside `content` HTML. FC's sidecar parser was leaving
post_title NULL for every SubscribeStar post since FC-3 shipped. The
parser fix (sidecar._first_line_text fallback) now synthesizes a title
at parse time; this migration applies the same logic retroactively to
existing rows.
Operator-flagged 2026-05-27 after inspecting
/mnt/Data/Patreon/Cheunart/subscribestar/ sidecars.
Idempotent: only touches rows where post_title IS NULL or empty AND
description IS NOT NULL. Re-running the migration is a no-op.
"""
from __future__ import annotations
import re
from typing import Sequence, Union
from alembic import op
from sqlalchemy import text
revision: str = "0024"
down_revision: Union[str, None] = "0023"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
_TAG_RE = re.compile(r"<[^>]+>")
_WS_RE = re.compile(r"\s+")
def _first_line_text(body: str, limit: int = 120) -> str | None:
"""Mirror of sidecar._first_line_text. Kept inline so the migration
doesn't carry a runtime import dependency from app code that may
have moved by the time the migration is replayed years from now."""
if not body:
return None
text_ = _TAG_RE.sub(" ", body)
text_ = text_.replace("\xa0", " ")
for line in text_.splitlines():
line = _WS_RE.sub(" ", line).strip()
if line:
if len(line) > limit:
return line[: limit - 1].rstrip() + ""
return line
return None
def upgrade() -> None:
bind = op.get_bind()
rows = bind.execute(
text(
"SELECT id, description FROM post "
"WHERE (post_title IS NULL OR post_title = '') "
"AND description IS NOT NULL AND description <> ''"
)
).fetchall()
updated = 0
for row in rows:
derived = _first_line_text(row.description)
if not derived:
continue
bind.execute(
text("UPDATE post SET post_title = :t WHERE id = :id"),
{"t": derived, "id": row.id},
)
updated += 1
print(f"0024: backfilled post_title on {updated} row(s)")
def downgrade() -> None:
# No safe restore — we can't tell which post_titles were derived vs
# genuinely present. Leave the column alone on rollback.
pass
@@ -1,288 +0,0 @@
"""sidecar-audit followup: correct external_post_id + post_url across all platforms
Revision ID: 0025
Revises: 0024
Create Date: 2026-05-27
Closes the operator-flagged 2026-05-27 sidecar audit findings. Three
data-correctness bugs across non-Patreon platforms had been silently
corrupting Posts since FC-3 shipped; the parser fix (sidecar.py, same
commit) addresses new imports. This migration cleans up existing rows.
Per-platform actions:
subscribestar — gallery-dl wrote the per-attachment id in `id` and
the actual post id in `post_id`. FC's parser picked `id`, so every
multi-image SubscribeStar post was fragmented into N Post rows.
1. For each SubscribeStar Post, read its sidecar (via the related
ImageRecord's on-disk path), pull `post_id`, overwrite
external_post_id and post_url.
2. Merge groups of Posts under one source that now share an
external_post_id (fragments of the same actual post). Same
ImageProvenance pre-delete + repoint dance as alembic 0022.
hentaifoundry — sidecars have NO `url` field; `src` is the image
URL. FC's parser stored post_url=NULL. Read each HF Post's sidecar
for `user` + `index`, derive the canonical /pictures/user/<u>/<i>
permalink. external_post_id (= `index`) was already correct.
discord — gallery-dl wrote the CDN attachment URL in `url`. FC's
parser stored that as post_url. Read each Discord Post's sidecar
for the server/channel/message triple, derive the proper
discord.com/channels/.../<message> permalink. external_post_id (=
`message_id`) was already correct.
pixiv — pure-SQL backfill: replace any `i.pximg.net`-style URL on
Post.post_url with the derived `/artworks/<id>` permalink. Pixiv
external_post_id (= `id`) was already correct; no sidecar IO
needed.
Idempotent: re-running on already-corrected data is a no-op (skips
rows whose derived value matches what's already stored).
Posts whose related ImageRecord paths don't resolve on disk (orphaned
filesystem state) are skipped with a count in the migration output —
those will be picked up by a future deep-scan.
"""
from __future__ import annotations
import json
import re
from pathlib import Path
from typing import Sequence, Union
from alembic import op
from sqlalchemy import text
revision: str = "0025"
down_revision: Union[str, None] = "0024"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
# Mirror of sidecar._NUMBERING_PREFIX. Kept inline so the migration is
# self-contained (the operator's banked rule:
# reference_postgres_enum_swap_drop_checks.md says migrations shouldn't
# import from runtime app code).
_NUMBERING_PREFIX = re.compile(r"^\d+_(.+)$")
def _find_sidecar(media_path: Path) -> Path | None:
"""gallery-dl writes the sidecar under the unprefixed stem
(`HOLLOW-ICHIGO.json`) while the media file gets a NN_ ordering
prefix (`01_HOLLOW-ICHIGO.png`). Try in order:
1. <stem>.json next to the media
2. <media>.json next to the media (full-name variant)
3. strip the NN_ prefix from the stem, then <stripped>.json
"""
if not media_path:
return None
cand = media_path.with_suffix(".json")
if cand.is_file():
return cand
cand = media_path.parent / f"{media_path.name}.json"
if cand.is_file():
return cand
m = _NUMBERING_PREFIX.match(media_path.stem)
if m:
cand = media_path.parent / f"{m.group(1)}.json"
if cand.is_file():
return cand
return None
def _str_id(v) -> str | None:
"""str() a JSON scalar id; reject bool (JSON booleans are ints in
Python's eyes but they aren't valid sidecar ids)."""
if isinstance(v, bool):
return None
if isinstance(v, (str, int)) and str(v).strip():
return str(v).strip()
return None
def _str_field(v) -> str | None:
if isinstance(v, str) and v.strip():
return v.strip()
return None
def upgrade() -> None:
conn = op.get_bind()
# ── PART 1: Per-platform corrections requiring filesystem IO ─────
# SubscribeStar, HentaiFoundry, Discord all need fields from the
# sidecar to construct the right post_url. We walk each Post's
# related ImageRecord.path to find the sidecar, read it, derive,
# and update.
targets = conn.execute(text("""
SELECT p.id, p.external_post_id, p.post_url, s.platform
FROM post p
JOIN source s ON s.id = p.source_id
WHERE s.platform IN ('subscribestar', 'hentaifoundry', 'discord')
""")).fetchall()
stats: dict[str, dict[str, int]] = {
plat: {"read": 0, "updated": 0, "no_sidecar": 0}
for plat in ("subscribestar", "hentaifoundry", "discord")
}
for post_row in targets:
plat = post_row.platform
path = _first_attachment_path(conn, post_row.id)
if not path:
stats[plat]["no_sidecar"] += 1
continue
sidecar = _find_sidecar(Path(path))
if sidecar is None:
stats[plat]["no_sidecar"] += 1
continue
try:
data = json.loads(sidecar.read_text(encoding="utf-8"))
except (OSError, json.JSONDecodeError):
stats[plat]["no_sidecar"] += 1
continue
stats[plat]["read"] += 1
new_epid = post_row.external_post_id
new_url = None
if plat == "subscribestar":
pid = _str_id(data.get("post_id"))
if pid:
new_epid = pid
new_url = f"https://www.subscribestar.com/posts/{pid}"
elif plat == "hentaifoundry":
user = _str_field(data.get("user")) or _str_field(data.get("artist"))
idx = _str_id(data.get("index"))
if user and idx:
new_url = f"https://www.hentai-foundry.com/pictures/user/{user}/{idx}"
elif plat == "discord":
sid = _str_id(data.get("server_id"))
cid = _str_id(data.get("channel_id"))
mid = _str_id(data.get("message_id"))
if sid and cid and mid:
new_url = f"https://discord.com/channels/{sid}/{cid}/{mid}"
# Idempotent: skip if nothing changed.
if new_epid == post_row.external_post_id and new_url == post_row.post_url:
continue
conn.execute(
text("""
UPDATE post
SET external_post_id = :epid, post_url = :url
WHERE id = :id
"""),
{"epid": new_epid, "url": new_url, "id": post_row.id},
)
stats[plat]["updated"] += 1
for plat, s in stats.items():
print(
f"0025: {plat} — read {s['read']} sidecars, "
f"updated {s['updated']} Posts, "
f"{s['no_sidecar']} Posts had no resolvable sidecar"
)
# ── PART 2: Merge SubscribeStar fragments now sharing epid ───────
# After Part 1, each group of Posts under one source with the SAME
# new external_post_id is a fragment-set of the same actual post.
# Merge to one canonical row. Pre-handle the same ImageProvenance
# collision pattern as alembic 0022 (uq_image_provenance_image_post).
fragment_groups = conn.execute(text("""
SELECT p.source_id, p.external_post_id,
ARRAY_AGG(p.id ORDER BY p.id ASC) AS post_ids
FROM post p
JOIN source s ON s.id = p.source_id
WHERE s.platform = 'subscribestar'
AND p.external_post_id IS NOT NULL
GROUP BY p.source_id, p.external_post_id
HAVING COUNT(*) > 1
""")).fetchall()
merged = 0
for grp in fragment_groups:
post_ids = list(grp.post_ids)
keep_id, *drop_ids = post_ids
for drop_id in drop_ids:
# Pre-DELETE colliding ImageProvenance under drop_ that
# already exist under keep (alembic 0022 banked the pattern).
conn.execute(
text("""
DELETE FROM image_provenance
WHERE post_id = :drop_
AND image_record_id IN (
SELECT image_record_id FROM image_provenance
WHERE post_id = :keep
)
"""),
{"keep": keep_id, "drop_": drop_id},
)
conn.execute(
text("""
UPDATE image_provenance SET post_id = :keep
WHERE post_id = :drop_
"""),
{"keep": keep_id, "drop_": drop_id},
)
conn.execute(
text("""
UPDATE image_record SET primary_post_id = :keep
WHERE primary_post_id = :drop_
"""),
{"keep": keep_id, "drop_": drop_id},
)
conn.execute(
text("""
UPDATE post_attachment SET post_id = :keep
WHERE post_id = :drop_
"""),
{"keep": keep_id, "drop_": drop_id},
)
conn.execute(
text("DELETE FROM post WHERE id = :drop_"),
{"drop_": drop_id},
)
merged += 1
print(f"0025: subscribestar — merged {merged} duplicate Post fragments")
# ── PART 3: Pixiv post_url backfill (pure SQL) ───────────────────
# Pixiv's external_post_id is already correct (gallery-dl's `id` is
# the post id). Only post_url needs derivation: replace anything
# under i.pximg.net (the file URL) with the /artworks/<id> permalink.
pixiv_updated = conn.execute(text("""
UPDATE post p
SET post_url = 'https://www.pixiv.net/artworks/' || p.external_post_id
FROM source s
WHERE p.source_id = s.id
AND s.platform = 'pixiv'
AND p.external_post_id IS NOT NULL
AND (p.post_url IS NULL
OR p.post_url LIKE 'https://i.pximg.net/%'
OR p.post_url LIKE 'http://i.pximg.net/%')
""")).rowcount
print(f"0025: pixiv — backfilled post_url on {pixiv_updated} Posts")
def _first_attachment_path(conn, post_id: int) -> str | None:
"""Return any ImageRecord.path attached to this post (via
ImageProvenance). Lowest-id row keeps the migration deterministic
so re-running on the same DB picks the same sidecar."""
row = conn.execute(
text("""
SELECT ir.path
FROM image_provenance ip
JOIN image_record ir ON ir.id = ip.image_record_id
WHERE ip.post_id = :pid
ORDER BY ip.id ASC
LIMIT 1
"""),
{"pid": post_id},
).first()
return row[0] if row else None
def downgrade() -> None:
# Lossy: external_post_id values were overwritten with the correct
# post_id; original per-attachment ids weren't preserved. Post-merge
# also deleted drop rows. No safe restore. To roll back the schema
# invariant, fork from 0024 and re-run sidecar imports.
pass
@@ -1,53 +0,0 @@
"""import_task.recovery_count + refetched — poison-pill circuit breaker
Revision ID: 0026
Revises: 0025
Create Date: 2026-05-28
Backs the import-task resilience work (operator-flagged 2026-05-28):
- recovery_count: how many times recover_interrupted_tasks has
re-queued this row from a stuck 'processing' state. A row that
hard-crashes the worker (OOM / segfault on a corrupt or oversized
input) leaves no terminal flip, so the sweep re-queues it — and
without a cap it would loop forever, re-crashing the worker each
time. After MAX_RECOVERY_ATTEMPTS the sweep marks it 'failed' with a
diagnostic instead.
- refetched: whether a one-shot re-download has already been attempted
for this task's file. Bounds the Layer-2 re-fetch remediation to a
single attempt so source-side corruption doesn't loop.
Both default to 0 / false; additive, no backfill needed.
"""
from typing import Sequence, Union
import sqlalchemy as sa
from alembic import op
revision: str = "0026"
down_revision: Union[str, None] = "0025"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
op.add_column(
"import_task",
sa.Column(
"recovery_count", sa.Integer(), nullable=False,
server_default="0",
),
)
op.add_column(
"import_task",
sa.Column(
"refetched", sa.Boolean(), nullable=False,
server_default=sa.false(),
),
)
def downgrade() -> None:
op.drop_column("import_task", "refetched")
op.drop_column("import_task", "recovery_count")
@@ -1,50 +0,0 @@
"""drop migration_run — one-and-done GS/IR migration tooling removed
Revision ID: 0027
Revises: 0026
Create Date: 2026-05-29
The GS/IR migration tooling (services/migrators, /api/migrate, the
run_migration task, LegacyMigrationCard, and the MigrationRun model) was
removed after the migration cutover completed. This drops its now-orphaned
run-log table. Downgrade recreates the table (mirrors the old model) so the
migration is reversible.
"""
from typing import Sequence, Union
import sqlalchemy as sa
from alembic import op
from sqlalchemy.dialects.postgresql import JSONB
revision: str = "0027"
down_revision: Union[str, None] = "0026"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
op.drop_table("migration_run")
def downgrade() -> None:
op.create_table(
"migration_run",
sa.Column("id", sa.Integer(), primary_key=True),
sa.Column("kind", sa.String(length=32), nullable=False),
sa.Column("status", sa.String(length=32), nullable=False),
sa.Column("dry_run", sa.Boolean(), nullable=False, server_default=sa.false()),
sa.Column(
"started_at", sa.DateTime(timezone=True), nullable=False,
server_default=sa.func.now(),
),
sa.Column("finished_at", sa.DateTime(timezone=True), nullable=True),
sa.Column(
"counts", JSONB(), nullable=False, server_default=sa.text("'{}'::jsonb"),
),
sa.Column("error", sa.Text(), nullable=True),
sa.Column(
"metadata", JSONB(), nullable=False, server_default=sa.text("'{}'::jsonb"),
),
)
op.create_index("ix_migration_run_kind", "migration_run", ["kind"])
op.create_index("ix_migration_run_status", "migration_run", ["status"])
@@ -1,190 +0,0 @@
"""collapse-sidecar-synthetic: repoint Posts/ImageProvenance/DownloadEvents
from `sidecar:<platform>:<slug>` synthetic Source anchors onto the real
Source for the same (artist, platform) when one exists, then delete the
synthetic.
Revision ID: 0028
Revises: 0027
Create Date: 2026-05-31
Background: alembic 0022 (2026-05-26) consolidated the old per-post-URL
Source rows into one canonical Source per (artist, platform). When NO
real campaign URL was salvageable among the candidates, it rewrote the
canonical row to url='sidecar:<platform>:<slug>' enabled=false as a
disabled anchor for any Posts already attached.
That was fine while it was the only Source for that artist+platform.
But: the unique constraint on Source is (artist_id, platform, url), not
(artist_id, platform). When the operator later added the real
subscription via the UI / extension / etc., a SECOND row landed —
the real one — with id > the synthetic. Both coexisted.
Two follow-on problems surfaced 2026-05-31:
1. The Subscriptions UI listed both rows. The synthetic was disabled
so the scheduler never polled it, but it looked like a phantom
subscription. (Fixed in same commit by SourceService.list filter.)
2. importer._source_for_sidecar picked Source by `ORDER BY id ASC
LIMIT 1`, so EVERY gallery-dl download since the real Source was
added attached its Post to the SYNTHETIC anchor, not the real
Source. (Fixed in same commit by preferring non-sidecar URLs.)
This migration is the data half of the cleanup: for every (artist,
platform) with both a synthetic AND a real Source, repoint the
synthetic's children (Posts, ImageProvenance, DownloadEvents) onto the
real Source and delete the synthetic. Reuses the same epid/provenance
collision dance from alembic 0022 because the same uniqueness
constraints fire row-by-row during bulk UPDATEs.
Lone synthetic anchors — those where no real Source for the same
(artist, platform) exists (e.g., filesystem-imported artist with no
subscription added) — are LEFT INTACT. They anchor real imported
content; deleting them would CASCADE-delete the Posts the operator
imported. The SourceService.list filter hides them from the UI; the
operator can delete them by hand if they want the underlying imports
gone.
"""
from typing import Sequence, Union
from alembic import op
from sqlalchemy import text
revision: str = "0028"
down_revision: Union[str, None] = "0027"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
conn = op.get_bind()
# Find (artist_id, platform) groups where BOTH a sidecar synthetic
# and at least one real Source exist.
groups = conn.execute(text("""
SELECT artist_id, platform
FROM source
GROUP BY artist_id, platform
HAVING bool_or(url LIKE 'sidecar:%')
AND bool_or(url NOT LIKE 'sidecar:%')
""")).fetchall()
for artist_id, platform in groups:
rows = conn.execute(
text("""
SELECT id, url FROM source
WHERE artist_id = :a AND platform = :p
ORDER BY id ASC
"""),
{"a": artist_id, "p": platform},
).fetchall()
synthetic_ids = [sid for sid, url in rows if url.startswith("sidecar:")]
real_rows = [(sid, url) for sid, url in rows if not url.startswith("sidecar:")]
if not synthetic_ids or not real_rows:
continue # belt+suspenders; the GROUP BY already filtered
# Canonical real: lowest-id non-sidecar Source.
canonical_id = real_rows[0][0]
# STEP A: PRE-merge Post collisions on (canonical, external_post_id).
# Mirror alembic 0022's pre-merge logic — when synth has Post X
# epid=N and real has Post Y epid=N, the bulk UPDATE below would
# trip uq_post_source_external_id row-by-row. Group all Posts
# under (canonical + synthetics) by epid; for any group >1,
# pick a keep (prefer one already under canonical, else lowest
# id) and merge the rest into it.
all_posts = conn.execute(
text("""
SELECT external_post_id, id, source_id
FROM post
WHERE source_id = :canonical OR source_id = ANY(:synths)
ORDER BY external_post_id, id
"""),
{"canonical": canonical_id, "synths": synthetic_ids},
).fetchall()
by_epid: dict = {}
for epid, post_id, src_id in all_posts:
by_epid.setdefault(epid, []).append((post_id, src_id))
for _epid, posts in by_epid.items():
if len(posts) <= 1:
continue
canonical_side = [p for p in posts if p[1] == canonical_id]
keep_id = canonical_side[0][0] if canonical_side else posts[0][0]
drop_ids = [p[0] for p in posts if p[0] != keep_id]
for drop_id in drop_ids:
# Pre-delete image_provenance rows under drop_ whose
# image_record_id already has provenance under keep —
# avoids tripping uq_image_provenance_image_post (0021)
# row-by-row during the repoint UPDATE.
conn.execute(
text("""
DELETE FROM image_provenance
WHERE post_id = :drop_
AND image_record_id IN (
SELECT image_record_id FROM image_provenance
WHERE post_id = :keep
)
"""),
{"keep": keep_id, "drop_": drop_id},
)
conn.execute(
text("""
UPDATE image_provenance SET post_id = :keep
WHERE post_id = :drop_
"""),
{"keep": keep_id, "drop_": drop_id},
)
conn.execute(
text("""
UPDATE image_record SET primary_post_id = :keep
WHERE primary_post_id = :drop_
"""),
{"keep": keep_id, "drop_": drop_id},
)
conn.execute(
text("DELETE FROM post WHERE id = :drop_"),
{"drop_": drop_id},
)
# STEP B: Bulk reparent the remaining Posts off the synthetics.
conn.execute(
text("""
UPDATE post SET source_id = :canonical
WHERE source_id = ANY(:synths)
"""),
{"canonical": canonical_id, "synths": synthetic_ids},
)
# STEP C: Reparent ImageProvenance.source_id (denormalized FK;
# no UNIQUE on source_id, safe bulk).
conn.execute(
text("""
UPDATE image_provenance SET source_id = :canonical
WHERE source_id = ANY(:synths)
"""),
{"canonical": canonical_id, "synths": synthetic_ids},
)
# STEP D: Reparent any DownloadEvent.source_id. Synthetics are
# enabled=false so the scheduler never created events for them;
# this is belt+suspenders for any rows planted by manual force
# or older code paths.
conn.execute(
text("""
UPDATE download_event SET source_id = :canonical
WHERE source_id = ANY(:synths)
"""),
{"canonical": canonical_id, "synths": synthetic_ids},
)
# STEP E: Drop the now-empty synthetics.
conn.execute(
text("DELETE FROM source WHERE id = ANY(:synths)"),
{"synths": synthetic_ids},
)
def downgrade() -> None:
# Lossy migration — synthetic Sources deleted, Posts repointed and
# potentially merged. No safe downgrade.
pass
@@ -1,71 +0,0 @@
"""drop artist + copyright ml thresholds; lower general default to 0.50
Revision ID: 0029
Revises: 0028
Create Date: 2026-06-01
Operator-flagged 2026-06-01: the view modal's Suggestions panel hides
most general-category predictions because the default threshold is
0.95. Lowering the default to 0.50 (matches character) so general
suggestions surface more aggressively; the value remains tunable in
Settings → ML.
Same change retires two ML suggestion categories whose Tag.kind
surfaces are unused:
- `artist`: retired in FC-2d-vii-c — artist identity is acquisition-
derived (image_record.artist_id), never ML-inferred. The threshold
column was a leftover from before that retirement.
- `copyright`: retired 2026-06-01 — the app uses `fandom` for the
franchise/copyright concept (per TagsView.vue's doc comment); no
Tag rows of kind=copyright exist, and the threshold column never
fed anything user-visible.
Both columns are dropped from ml_settings; the existing row's
suggestion_threshold_general value is bumped from 0.95 to 0.50 iff
it's still at the old default, so deployed installs pick up the new
UX without overriding any operator tuning.
"""
from typing import Sequence, Union
from alembic import op
from sqlalchemy import text
revision: str = "0029"
down_revision: Union[str, None] = "0028"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
# Bump the general threshold for installs still at the old default.
op.execute(text(
"UPDATE ml_settings "
"SET suggestion_threshold_general = 0.50 "
"WHERE id = 1 AND suggestion_threshold_general = 0.95"
))
op.drop_column("ml_settings", "suggestion_threshold_artist")
op.drop_column("ml_settings", "suggestion_threshold_copyright")
def downgrade() -> None:
# Restore the columns with their prior defaults. The bump from
# 0.95 → 0.50 isn't reversible without remembering whether the
# operator had explicitly set 0.95 (unlikely — that was just the
# default) so we leave the current general value as-is.
from sqlalchemy import Column, Float
op.add_column(
"ml_settings",
Column(
"suggestion_threshold_artist",
Float, nullable=False, server_default="0.30",
),
)
op.add_column(
"ml_settings",
Column(
"suggestion_threshold_copyright",
Float, nullable=False, server_default="0.50",
),
)
@@ -1,145 +0,0 @@
"""nullable post.source_id + denormalized post.artist_id; retire sidecar synthetics
Revision ID: 0030
Revises: 0029
Create Date: 2026-06-01
Operator-asked 2026-06-01 after the Dymkens orphan investigation: the
sidecar synthetic Source pattern (`sidecar:<platform>:<slug>` rows
with enabled=false) was technically correct but misled the operator
into thinking they had phantom subscriptions. The synthetics existed
solely to satisfy `Post.source_id NOT NULL` for filesystem-imported
content with no real subscription.
This migration makes the data model honest:
1. **Post gets a denormalized `artist_id` column** so artist filters
work without traversing `Post → Source.artist_id`. Backfilled from
the existing Source linkage, then NOT NULL'd.
2. **`Post.source_id` becomes nullable**, FK ondelete `CASCADE` → `SET
NULL`. Deleting a Source detaches its Posts instead of destroying
imported content (semantically: subscription ends, archive stays).
3. **`ImageProvenance.source_id` becomes nullable** with the same FK
semantic change.
4. **Sidecar synthetic Sources are deleted** — first NULL out the
FKs from Post + ImageProvenance pointing at them (so the implicit
CASCADE doesn't fire), then delete. DownloadEvent FK is unchanged
(still CASCADE'd, NOT NULL'd) — synthetics have `enabled=false`
so no events exist for them.
Uniqueness handling: the existing `uq_post_source_external_id`
(source_id, external_post_id) keeps working for source-bound Posts
(Postgres treats NULL != NULL so NULL-source rows aren't deduped by
it). A second partial unique index covers the NULL-source case on
(artist_id, external_post_id) so filesystem-imported posts still
dedupe within an artist.
"""
from typing import Sequence, Union
import sqlalchemy as sa
from alembic import op
from sqlalchemy import text
revision: str = "0030"
down_revision: Union[str, None] = "0029"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
conn = op.get_bind()
# Step 1: add Post.artist_id, initially nullable for backfill.
# FK naming follows the Base.metadata naming_convention
# (fk_<table>_<column>_<referred_table>) — alembic 0001 set this up.
op.add_column(
"post",
sa.Column("artist_id", sa.Integer, nullable=True),
)
op.create_foreign_key(
"fk_post_artist_id_artist", "post", "artist",
["artist_id"], ["id"], ondelete="CASCADE",
)
# Step 2: backfill from Source.artist_id (every existing Post has a
# Source today, so every row gets populated).
conn.execute(text("""
UPDATE post p
SET artist_id = s.artist_id
FROM source s
WHERE p.source_id = s.id AND p.artist_id IS NULL
"""))
# Sanity: count any remaining NULLs. Should be zero pre-this-migration.
remaining = conn.execute(text(
"SELECT COUNT(*) FROM post WHERE artist_id IS NULL"
)).scalar_one()
if remaining:
raise RuntimeError(
f"alembic 0030: {remaining} post rows have no resolvable "
f"artist_id after backfill. Investigate before continuing."
)
# Step 3: enforce NOT NULL + add index for artist-filter queries.
op.alter_column("post", "artist_id", nullable=False)
op.create_index("ix_post_artist_id", "post", ["artist_id"])
# Step 4: relax post.source_id + flip FK to SET NULL. The original FK
# name from alembic 0001 is `fk_post_source_id_source` per the
# NAMING_CONVENTION in models/base.py.
op.alter_column("post", "source_id", nullable=True)
op.drop_constraint("fk_post_source_id_source", "post", type_="foreignkey")
op.create_foreign_key(
"fk_post_source_id_source", "post", "source",
["source_id"], ["id"], ondelete="SET NULL",
)
# Step 5: relax image_provenance.source_id + flip FK to SET NULL.
op.alter_column("image_provenance", "source_id", nullable=True)
op.drop_constraint(
"fk_image_provenance_source_id_source", "image_provenance",
type_="foreignkey",
)
op.create_foreign_key(
"fk_image_provenance_source_id_source", "image_provenance", "source",
["source_id"], ["id"], ondelete="SET NULL",
)
# Step 6: partial unique index on (artist_id, external_post_id) for
# NULL-source Posts. The existing uq_post_source_external_id keeps
# guarding source-bound rows; NULL-source rows now dedupe within
# an artist.
op.execute(
"CREATE UNIQUE INDEX uq_post_artist_external_id_null_source "
"ON post (artist_id, external_post_id) "
"WHERE source_id IS NULL"
)
# Step 7: retire sidecar synthetic Sources. NULL out the references
# FIRST (the new FK is SET NULL so CASCADE wouldn't fire anyway, but
# being explicit makes the intent clear). Then delete the synthetic
# source rows. Any DownloadEvent rows under synthetics CASCADE-die
# with the source — synthetics have enabled=false so there shouldn't
# be any in practice.
conn.execute(text("""
UPDATE post
SET source_id = NULL
WHERE source_id IN (SELECT id FROM source WHERE url LIKE 'sidecar:%')
"""))
conn.execute(text("""
UPDATE image_provenance
SET source_id = NULL
WHERE source_id IN (SELECT id FROM source WHERE url LIKE 'sidecar:%')
"""))
deleted = conn.execute(text(
"DELETE FROM source WHERE url LIKE 'sidecar:%' RETURNING id"
)).rowcount
print(f"alembic 0030: deleted {deleted} sidecar synthetic source rows")
def downgrade() -> None:
# Lossy migration — the deleted sidecar synthetics can't be
# restored from the orphan post.source_id / image_provenance.source_id
# values, and the partial unique index encodes a constraint that
# NULL-source Posts may now exist. No safe downgrade.
pass
@@ -1,45 +0,0 @@
"""source.backfill_runs_remaining: sticky deep-scan mode
Revision ID: 0031
Revises: 0030
Create Date: 2026-06-01
Tick vs backfill mode for subscription downloads. When
`backfill_runs_remaining > 0`, the next N download runs use
`skip: True` + 30-min timeout (walk full history). When 0, runs use
`skip: "exit:20"` + 14.5-min timeout (catch-up mode, exits early once
20 contiguous archived items are seen).
Operator-flagged 2026-06-01 (Knuxy run #38887): a creator with ~550
archived posts saturates the 870s catch-up timeout even when there is
no new content, because gallery-dl's default `skip: True` keeps walking.
Tick mode short-circuits that; backfill mode is the explicit opt-in for
deep history scans.
Default 0 (all existing subscriptions start in tick mode).
"""
from typing import Sequence, Union
import sqlalchemy as sa
from alembic import op
revision: str = "0031"
down_revision: Union[str, None] = "0030"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
op.add_column(
"source",
sa.Column(
"backfill_runs_remaining",
sa.Integer,
nullable=False,
server_default="0",
),
)
def downgrade() -> None:
op.drop_column("source", "backfill_runs_remaining")
@@ -1,41 +0,0 @@
"""source.error_type: surface ErrorType taxonomy in FailingSourcesCard
Revision ID: 0032
Revises: 0031
Create Date: 2026-06-02
Audit 2026-06-02: the backend computes 13 ErrorType categories (auth_error,
rate_limited, not_found, access_denied, validation_failed, etc.) and
stamps each one on DownloadEvent.metadata, but the Source row only carried
the free-text last_error. Operators couldn't bulk-triage failing sources
("all auth_error → rotate cookies, all rate_limited → just wait") without
opening Logs per row.
This column receives the last error_type from _update_source_health
and gets cleared on a successful run. Nullable + indexed so the failing-
sources rollup can filter/group cheaply.
"""
from typing import Sequence, Union
import sqlalchemy as sa
from alembic import op
revision: str = "0032"
down_revision: Union[str, None] = "0031"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
op.add_column(
"source",
sa.Column("error_type", sa.String(length=32), nullable=True),
)
op.create_index(
"ix_source_error_type", "source", ["error_type"],
)
def downgrade() -> None:
op.drop_index("ix_source_error_type", table_name="source")
op.drop_column("source", "error_type")
@@ -1,48 +0,0 @@
"""suggestion_threshold default 0.50 → 0.70
Revision ID: 0033
Revises: 0032
Create Date: 2026-06-02
Operator-flagged 2026-06-02 — the 0.50 default (set on 2026-06-01) is
too noisy in practice; raise to 0.70 for both suggestion categories.
Only conditionally updates singletons whose current value is still the
2026-06-01 default (0.50). Operators who deliberately tuned their row
to some other value (0.55, 0.65, 0.80, etc. via the Settings UI) keep
their pick — the migration only catches the unchanged-default case.
"""
from typing import Sequence, Union
from alembic import op
revision: str = "0033"
down_revision: Union[str, None] = "0032"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
op.execute(
"UPDATE ml_settings "
"SET suggestion_threshold_character = 0.70 "
"WHERE id = 1 AND suggestion_threshold_character = 0.50"
)
op.execute(
"UPDATE ml_settings "
"SET suggestion_threshold_general = 0.70 "
"WHERE id = 1 AND suggestion_threshold_general = 0.50"
)
def downgrade() -> None:
op.execute(
"UPDATE ml_settings "
"SET suggestion_threshold_character = 0.50 "
"WHERE id = 1 AND suggestion_threshold_character = 0.70"
)
op.execute(
"UPDATE ml_settings "
"SET suggestion_threshold_general = 0.50 "
"WHERE id = 1 AND suggestion_threshold_general = 0.70"
)
-53
View File
@@ -1,53 +0,0 @@
"""artist_visit: per-artist last-viewed timestamp for the "+N new" badge
Revision ID: 0034
Revises: 0033
Create Date: 2026-06-03
Powers the artists-directory "+N new since last visit" badge + ArtistView
banner. Single row per artist (no user_id yet — rule #47 multi-user ACL
is aspirational; widens to (user_id, artist_id) PK when User lands).
Seed every existing artist with `last_viewed_at = NOW()` so the badge
starts at 0 across the board — no noisy "you have 5000 unseen images"
on first deploy. New artists auto-get a row via
`ArtistService.find_or_create`.
"""
from typing import Sequence, Union
import sqlalchemy as sa
from alembic import op
revision: str = "0034"
down_revision: Union[str, None] = "0033"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
op.create_table(
"artist_visit",
sa.Column(
"artist_id",
sa.Integer,
sa.ForeignKey("artist.id", ondelete="CASCADE"),
primary_key=True,
),
sa.Column(
"last_viewed_at",
sa.DateTime(timezone=True),
nullable=False,
server_default=sa.text("NOW()"),
),
)
# Seed: every existing artist starts "fully caught up". Without this,
# every operator with N artists would see N badges (worth of every
# image ever imported) on first deploy.
op.execute(
"INSERT INTO artist_visit (artist_id, last_viewed_at) "
"SELECT id, NOW() FROM artist"
)
def downgrade() -> None:
op.drop_table("artist_visit")
@@ -1,70 +0,0 @@
"""image_record.effective_date: materialized gallery sort key + index
Revision ID: 0035
Revises: 0034
Create Date: 2026-06-04
The gallery ordered/cursored on COALESCE(post.post_date,
image_record.created_at) across the Post outer join. That expression spans
two tables, so no index can serve it — every /scroll sorted a large slice
of the library, and the frontend fired ten of them serially per initial
load. Materialize the value into image_record.effective_date and index
(effective_date DESC, id DESC) so the cursor scroll is an index range scan.
Backfill = COALESCE(primary post's post_date, created_at) so existing rows
keep their exact ordering. New rows get the created_at-equivalent server
default; services/importer.py overrides it with the post's date when a
primary post with a date is linked.
"""
from typing import Sequence, Union
import sqlalchemy as sa
from alembic import op
revision: str = "0035"
down_revision: Union[str, None] = "0034"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
# Add nullable first so the backfill can populate before NOT NULL.
op.add_column(
"image_record",
sa.Column("effective_date", sa.DateTime(timezone=True), nullable=True),
)
# Pure set-based UPDATEs (no per-row params) — immune to the 65535
# bind-parameter ceiling regardless of library size.
op.execute(
"""
UPDATE image_record AS ir
SET effective_date = COALESCE(p.post_date, ir.created_at)
FROM post AS p
WHERE ir.primary_post_id = p.id
"""
)
op.execute(
"""
UPDATE image_record
SET effective_date = created_at
WHERE effective_date IS NULL
"""
)
op.alter_column(
"image_record",
"effective_date",
nullable=False,
server_default=sa.text("now()"),
)
# DESC/DESC matches the gallery's ORDER BY effective_date DESC, id DESC
# so the scroll is a forward index scan; raw SQL because alembic's
# column list doesn't express per-column DESC cleanly.
op.execute(
"CREATE INDEX ix_image_record_effective_date "
"ON image_record (effective_date DESC, id DESC)"
)
def downgrade() -> None:
op.drop_index("ix_image_record_effective_date", table_name="image_record")
op.drop_column("image_record", "effective_date")
@@ -1,41 +0,0 @@
"""image_record.siglip_embedding: HNSW cosine index for "more like this"
Revision ID: 0036
Revises: 0035
Create Date: 2026-06-04
Gallery Phase 3 (visual similarity search) ranks images by
`siglip_embedding.cosine_distance(source_embedding)`. Without an index that's
a sequential scan computing a 1152-dim distance for every row — fine at small
scale, but it grows linearly with the library. Add an HNSW index with
`vector_cosine_ops` so the top-N nearest search is sub-50ms ANN.
1152 dims is under pgvector's 2000-dim HNSW limit, so HNSW (no training,
better recall than IVFFlat) is the right choice. ONE-TIME COST: building the
index over the existing embeddings (~57k vectors on the operator's library)
locks image_record for ~30-60s during this migration on deploy — acceptable
for a single-operator homelab. NULL embeddings (videos / not-yet-embedded
rows) are simply not indexed.
"""
from typing import Sequence, Union
from alembic import op
revision: str = "0036"
down_revision: Union[str, None] = "0035"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
# Raw SQL: alembic's create_index doesn't express the `USING hnsw (...
# vector_cosine_ops)` access-method + opclass cleanly. Must match the
# query's cosine_distance operator class to be usable by the planner.
op.execute(
"CREATE INDEX ix_image_record_siglip_hnsw "
"ON image_record USING hnsw (siglip_embedding vector_cosine_ops)"
)
def downgrade() -> None:
op.drop_index("ix_image_record_siglip_hnsw", table_name="image_record")
@@ -1,53 +0,0 @@
"""patreon_seen_media: per-source ledger of already-ingested Patreon media
Revision ID: 0037
Revises: 0036
Create Date: 2026-06-05
Native Patreon ingester (build step 2a). Replaces gallery-dl's
archive.sqlite3 with our own queryable table. The downloader upserts one
row per (source, media) so routine walks skip media we've already
processed; a future "recovery" mode bypasses the ledger to re-walk.
`filehash` is a 32-hex Patreon CDN MD5, OR a video sentinel of the form
``video:<post_id>:<media_id>`` — hence String(128). The unique
constraint on (source_id, filehash) is the dedup upsert key.
"""
from typing import Sequence, Union
import sqlalchemy as sa
from alembic import op
revision: str = "0037"
down_revision: Union[str, None] = "0036"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
op.create_table(
"patreon_seen_media",
sa.Column("id", sa.Integer, primary_key=True),
sa.Column(
"source_id",
sa.Integer,
sa.ForeignKey("source.id", ondelete="CASCADE"),
nullable=False,
index=True,
),
sa.Column("filehash", sa.String(128), nullable=False),
sa.Column("post_id", sa.String(64), nullable=True),
sa.Column(
"seen_at",
sa.DateTime(timezone=True),
nullable=False,
server_default=sa.text("NOW()"),
),
sa.UniqueConstraint(
"source_id", "filehash", name="uq_patreon_seen_media_source_id"
),
)
def downgrade() -> None:
op.drop_table("patreon_seen_media")
@@ -1,58 +0,0 @@
"""patreon_failed_media: per-source dead-letter ledger for failing Patreon media
Revision ID: 0038
Revises: 0037
Create Date: 2026-06-06
Plan #705 (#7). Media that keeps failing to download/validate (404'd CDN,
deleted post, geo-blocked Mux, persistently-corrupt bytes) gets recorded here
with an attempt counter; once it crosses the dead-letter threshold the ingester
skips it on routine walks (recovery still re-attempts). A clean download clears
the row. UNIQUE (source_id, filehash) is the upsert key (same media key the
seen-ledger uses).
"""
from typing import Sequence, Union
import sqlalchemy as sa
from alembic import op
revision: str = "0038"
down_revision: Union[str, None] = "0037"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
op.create_table(
"patreon_failed_media",
sa.Column("id", sa.Integer, primary_key=True),
sa.Column(
"source_id",
sa.Integer,
sa.ForeignKey("source.id", ondelete="CASCADE"),
nullable=False,
index=True,
),
sa.Column("filehash", sa.String(128), nullable=False),
sa.Column("attempts", sa.Integer, nullable=False, server_default="1"),
sa.Column("last_error", sa.Text, nullable=True),
sa.Column(
"first_failed_at",
sa.DateTime(timezone=True),
nullable=False,
server_default=sa.text("NOW()"),
),
sa.Column(
"last_failed_at",
sa.DateTime(timezone=True),
nullable=False,
server_default=sa.text("NOW()"),
),
sa.UniqueConstraint(
"source_id", "filehash", name="uq_patreon_failed_media_source_id"
),
)
def downgrade() -> None:
op.drop_table("patreon_failed_media")
@@ -1,40 +0,0 @@
"""library_audit_run: resume cursor + progress timestamp for chunked scans
Revision ID: 0039
Revises: 0038
Create Date: 2026-06-07
scan_library_for_rule used to run one 2h pass that timed out on large libraries
and monopolized the concurrency-1 maintenance queue (operator-flagged). It now
runs short time-boxed chunks that re-enqueue: `resume_after_id` persists the
keyset cursor so the next chunk continues where it left off, and
`last_progress_at` lets the recovery sweep tell a progressing multi-chunk audit
from a genuinely stuck one.
"""
from typing import Sequence, Union
import sqlalchemy as sa
from alembic import op
revision: str = "0039"
down_revision: Union[str, None] = "0038"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
op.add_column(
"library_audit_run",
sa.Column(
"resume_after_id", sa.Integer, nullable=False, server_default="0"
),
)
op.add_column(
"library_audit_run",
sa.Column("last_progress_at", sa.DateTime(timezone=True), nullable=True),
)
def downgrade() -> None:
op.drop_column("library_audit_run", "last_progress_at")
op.drop_column("library_audit_run", "resume_after_id")
-108
View File
@@ -1,108 +0,0 @@
"""series chapters: chapter layer over series_page (FC-6.1)
Revision ID: 0040
Revises: 0039
Create Date: 2026-06-07
A series (Tag kind='series') gains an ordered chapter layer. Reading order
becomes (series_chapter.chapter_number, series_page.page_number). Every existing
series is backfilled into a single auto-chapter (chapter_number=1) holding its
current flat pages, so no data is lost and the old flat ordering is preserved.
"""
from typing import Sequence, Union
import sqlalchemy as sa
from alembic import op
revision: str = "0040"
down_revision: Union[str, None] = "0039"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
op.create_table(
"series_chapter",
sa.Column("id", sa.Integer, primary_key=True),
sa.Column(
"series_tag_id",
sa.Integer,
sa.ForeignKey("tag.id", ondelete="CASCADE"),
nullable=False,
),
sa.Column("chapter_number", sa.Integer, nullable=False),
sa.Column("title", sa.Text, nullable=True),
sa.Column(
"is_placeholder", sa.Boolean, nullable=False, server_default="false"
),
sa.Column("stated_page_start", sa.Integer, nullable=True),
sa.Column("stated_page_end", sa.Integer, nullable=True),
sa.Column(
"created_at",
sa.DateTime(timezone=True),
nullable=False,
server_default=sa.text("now()"),
),
sa.Column(
"updated_at",
sa.DateTime(timezone=True),
nullable=False,
server_default=sa.text("now()"),
),
)
op.create_index(
"ix_series_chapter_series_tag_id", "series_chapter", ["series_tag_id"]
)
# New columns on series_page; chapter_id starts nullable so we can backfill.
op.add_column(
"series_page", sa.Column("chapter_id", sa.Integer, nullable=True)
)
op.add_column(
"series_page", sa.Column("stated_page", sa.Integer, nullable=True)
)
conn = op.get_bind()
# One auto-chapter per existing series (any series_tag_id present in pages).
conn.execute(
sa.text(
"INSERT INTO series_chapter "
"(series_tag_id, chapter_number, is_placeholder, created_at, updated_at) "
"SELECT DISTINCT series_tag_id, 1, false, now(), now() "
"FROM series_page"
)
)
# Point every existing page at its series' auto-chapter.
conn.execute(
sa.text(
"UPDATE series_page sp "
"SET chapter_id = sc.id "
"FROM series_chapter sc "
"WHERE sc.series_tag_id = sp.series_tag_id"
)
)
# Now lock chapter_id down: NOT NULL + FK (cascade) + index.
op.alter_column("series_page", "chapter_id", nullable=False)
op.create_foreign_key(
"fk_series_page_chapter_id",
"series_page",
"series_chapter",
["chapter_id"],
["id"],
ondelete="CASCADE",
)
op.create_index(
"ix_series_page_chapter_id", "series_page", ["chapter_id"]
)
def downgrade() -> None:
op.drop_index("ix_series_page_chapter_id", table_name="series_page")
op.drop_constraint(
"fk_series_page_chapter_id", "series_page", type_="foreignkey"
)
op.drop_column("series_page", "stated_page")
op.drop_column("series_page", "chapter_id")
op.drop_index("ix_series_chapter_series_tag_id", table_name="series_chapter")
op.drop_table("series_chapter")
@@ -1,98 +0,0 @@
"""series suggestions: assisted-continuation matcher (FC-6.3)
Revision ID: 0041
Revises: 0040
Create Date: 2026-06-07
A confirm-only queue of "this post may continue this series" hints, plus two
import_settings knobs (enable + score threshold) for the matcher.
"""
from typing import Sequence, Union
import sqlalchemy as sa
from alembic import op
revision: str = "0041"
down_revision: Union[str, None] = "0040"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
op.create_table(
"series_suggestion",
sa.Column("id", sa.Integer, primary_key=True),
sa.Column(
"post_id",
sa.Integer,
sa.ForeignKey("post.id", ondelete="CASCADE"),
nullable=False,
),
sa.Column(
"series_tag_id",
sa.Integer,
sa.ForeignKey("tag.id", ondelete="CASCADE"),
nullable=False,
),
sa.Column("score", sa.Float, nullable=False),
sa.Column("signals", sa.JSON, nullable=True),
sa.Column(
"status", sa.String(16), nullable=False, server_default="pending"
),
sa.Column(
"created_at",
sa.DateTime(timezone=True),
nullable=False,
server_default=sa.text("now()"),
),
sa.Column(
"updated_at",
sa.DateTime(timezone=True),
nullable=False,
server_default=sa.text("now()"),
),
sa.UniqueConstraint(
"post_id", "series_tag_id", name="uq_series_suggestion_post_series"
),
)
op.create_index(
"ix_series_suggestion_post_id", "series_suggestion", ["post_id"]
)
op.create_index(
"ix_series_suggestion_series_tag_id",
"series_suggestion",
["series_tag_id"],
)
op.create_index(
"ix_series_suggestion_status", "series_suggestion", ["status"]
)
op.add_column(
"import_settings",
sa.Column(
"series_suggest_enabled",
sa.Boolean,
nullable=False,
server_default=sa.true(),
),
)
op.add_column(
"import_settings",
sa.Column(
"series_suggest_threshold",
sa.Float,
nullable=False,
server_default="0.5",
),
)
def downgrade() -> None:
op.drop_column("import_settings", "series_suggest_threshold")
op.drop_column("import_settings", "series_suggest_enabled")
op.drop_index("ix_series_suggestion_status", table_name="series_suggestion")
op.drop_index(
"ix_series_suggestion_series_tag_id", table_name="series_suggestion"
)
op.drop_index("ix_series_suggestion_post_id", table_name="series_suggestion")
op.drop_table("series_suggestion")
@@ -1,32 +0,0 @@
"""series chapter stated_part: operator-facing Part N label (FC-6.4)
Revision ID: 0042
Revises: 0041
Create Date: 2026-06-07
A chapter's positional chapter_number is auto-managed (rewritten 1..N on
reorder/delete), so it can't double as the installment number the operator wants
to type (e.g. a series authored from a post that is Part 2). Add a nullable
stated_part alongside it — the same split as series_page.page_number (order) vs
series_page.stated_page (printed number). Nullable; the UI falls back to
chapter_number when unset.
"""
from typing import Sequence, Union
import sqlalchemy as sa
from alembic import op
revision: str = "0042"
down_revision: Union[str, None] = "0041"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
op.add_column(
"series_chapter", sa.Column("stated_part", sa.Integer, nullable=True)
)
def downgrade() -> None:
op.drop_column("series_chapter", "stated_part")
@@ -1,62 +0,0 @@
"""post_attachment: per-post sha uniqueness (empty-post flood fix)
Revision ID: 0043
Revises: 0042
Create Date: 2026-06-08
PostAttachment.sha256 was GLOBALLY unique, so a non-art file the creator attaches
to many posts (a standard pdf/zip/link-card) only ever got ONE row — on the first
post — leaving every later post a bare shell (no image, no attachment). The native
Patreon backfill of Anduo surfaced 1589 such shells (operator-flagged 2026-06-08).
Switch to PER-POST uniqueness: the on-disk blob stays sha-deduped, but each post
gets its own row. Replace the unique sha256 index with a plain lookup index plus
two partial uniques — (post_id, sha256) for real posts and (sha256) for the
NULL-post filesystem case (still one row per file there).
Existing data has ≤1 row per sha (the old global unique), so the new partial
uniques can't be violated on upgrade — no data backfill needed here. The bare-post
shells themselves are removed by the separate prune-empty-posts cleanup tool.
"""
from typing import Sequence, Union
import sqlalchemy as sa
from alembic import op
revision: str = "0043"
down_revision: Union[str, None] = "0042"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
# Drop the global unique index; recreate it as a plain (non-unique) lookup
# index so sha-based reads keep their index (matches the model's index=True).
op.drop_index("ix_post_attachment_sha256", table_name="post_attachment")
op.create_index(
"ix_post_attachment_sha256", "post_attachment", ["sha256"],
)
op.create_index(
"uq_post_attachment_post_sha", "post_attachment",
["post_id", "sha256"], unique=True,
postgresql_where=sa.text("post_id IS NOT NULL"),
)
op.create_index(
"uq_post_attachment_null_post_sha", "post_attachment",
["sha256"], unique=True,
postgresql_where=sa.text("post_id IS NULL"),
)
def downgrade() -> None:
op.drop_index(
"uq_post_attachment_null_post_sha", table_name="post_attachment"
)
op.drop_index(
"uq_post_attachment_post_sha", table_name="post_attachment"
)
op.drop_index("ix_post_attachment_sha256", table_name="post_attachment")
op.create_index(
"ix_post_attachment_sha256", "post_attachment", ["sha256"],
unique=True,
)
@@ -1,37 +0,0 @@
"""ml_settings.tagger_store_floor
The ingest confidence floor below which tagger predictions are not stored,
promoted from the TAGGER_STORE_FLOOR env var to a DB-backed, UI-tunable
setting. Default 0.70 (was an env default of 0.05): the suggestion path
already filters at 0.70 and the centroid/learned path covers low-confidence
preferred tags, so the sub-0.70 tail was redundant weight — it had grown
image_record's TOAST to ~100 GB. See plan-task #764.
Revision ID: 0044
Revises: 0043
Create Date: 2026-06-10
"""
from typing import Sequence, Union
import sqlalchemy as sa
from alembic import op
revision: str = "0044"
down_revision: Union[str, None] = "0043"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
op.add_column(
"ml_settings",
sa.Column(
"tagger_store_floor", sa.Float(),
nullable=False, server_default="0.7",
),
)
def downgrade() -> None:
op.drop_column("ml_settings", "tagger_store_floor")
@@ -1,69 +0,0 @@
"""image_prediction table (DDL only — backfill runs as a background task)
Normalizes the per-image tagger predictions out of the JSON blob into a
queryable table (#768). This migration creates ONLY the table + indexes — it
is pure DDL and commits instantly, so web boots immediately.
The data backfill from the existing image_record.tagger_predictions JSON is
deliberately NOT done here. Doing it inline made the whole migration one
transaction over the ~100 GB TOAST: nothing committed until the very end, it
was invisible/unmonitorable mid-run, and an early MATERIALIZED-CTE form spilled
the full 100 GB to temp. Instead the backfill is the
backend.app.tasks.admin.backfill_image_predictions_task — batched by id window,
committed per chunk (visible progress + resumable), idempotent
(ON CONFLICT DO NOTHING). Trigger it from Settings → Maintenance once web is up.
The old image_record.tagger_predictions column is left in place (vestigial) and
dropped in a follow-up once the backfill + code cutover are verified — dropping
it needs an ACCESS EXCLUSIVE lock on the hot image_record table (the 0044 lock
class), so it's deferred to a quiesced-worker window.
Revision ID: 0045
Revises: 0044
Create Date: 2026-06-10
"""
from typing import Sequence, Union
import sqlalchemy as sa
from alembic import op
revision: str = "0045"
down_revision: Union[str, None] = "0044"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
op.create_table(
"image_prediction",
sa.Column("id", sa.Integer(), primary_key=True),
sa.Column(
"image_record_id", sa.Integer(),
sa.ForeignKey("image_record.id", ondelete="CASCADE"),
nullable=False,
),
sa.Column("raw_name", sa.String(length=255), nullable=False),
sa.Column("category", sa.String(length=64), nullable=False),
sa.Column("score", sa.Float(), nullable=False),
sa.UniqueConstraint(
"image_record_id", "raw_name", name="image_raw_name",
),
)
op.create_index(
"ix_image_prediction_image", "image_prediction", ["image_record_id"],
)
op.create_index(
"ix_image_prediction_name_score", "image_prediction",
["raw_name", "score"],
)
# No data backfill here — see the module docstring. The one-time copy from
# image_record.tagger_predictions runs as backfill_image_predictions_task
# (batched, resumable, idempotent), kept out of this transaction so web boots
# without waiting on a ~100 GB pass.
def downgrade() -> None:
op.drop_index("ix_image_prediction_name_score", "image_prediction")
op.drop_index("ix_image_prediction_image", "image_prediction")
op.drop_table("image_prediction")
@@ -1,43 +0,0 @@
"""drop image_record.tagger_predictions (predictions normalized to image_prediction)
Final step of #768. The per-tag predictions now live in the image_prediction
table (backfilled from the JSON, read by suggestions + allowlist, written by
tag_and_embed). The old JSON column is dead weight — and it's the ~100 GB of
sub-0.70 score tail that bloated image_record's TOAST and broke DB backups
(#739). Dropping it is a fast catalog change; it does NOT reclaim the disk on
its own — run `VACUUM FULL image_record` (or pg_repack) afterward, off-hours,
to return the space to the OS so backups go small.
DROP COLUMN needs a brief ACCESS EXCLUSIVE lock on image_record; env.py's
lock_timeout guards it, so quiesce the ml-worker if a tagging run is in flight
(see the migration-lock reference). tagger_model_version is kept — it's the
"has this been tagged / is it current?" signal the backfill sweep reads.
Revision ID: 0046
Revises: 0045
Create Date: 2026-06-11
"""
from typing import Sequence, Union
import sqlalchemy as sa
from alembic import op
revision: str = "0046"
down_revision: Union[str, None] = "0045"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
op.drop_column("image_record", "tagger_predictions")
def downgrade() -> None:
# Re-add the column empty. The JSON data is not restored (it lived only in
# this column); a downgrade would re-tag or backfill from image_prediction
# separately if ever needed.
op.add_column(
"image_record",
sa.Column("tagger_predictions", sa.JSON(), nullable=True),
)
@@ -1,175 +0,0 @@
"""series chapters become cosmetic dividers; pages become one series-global run
FC-6.x reframe (#789). A series is now ONE flat, series-global ordered run of
pages; chapters stop owning pages and become labeled dividers anchored to the
page that begins them.
Migration (order matters — series_page.chapter_id cascades, so it must be
dropped BEFORE any chapter row is deleted, or pages would cascade away):
a. Renumber series_page.page_number to a series-global 1..N (ordered by the
OLD (chapter_number, page_number)).
b. Add series_chapter.anchor_page_id and populate it with each chapter's first
page (lowest new page_number).
c. Drop series_page.chapter_id (severs the cascade link).
d. Prune chapters that shouldn't become dividers: empty/placeholder ones (no
anchor) and the redundant unlabeled chapter that would sit at page 1.
e. Reshape series_chapter into the divider: drop chapter_number,
is_placeholder, stated_page_start/end; make anchor_page_id NOT NULL +
UNIQUE + FK→series_page ON DELETE CASCADE.
Revision ID: 0047
Revises: 0046
Create Date: 2026-06-11
"""
from typing import Sequence, Union
import sqlalchemy as sa
from alembic import op
revision: str = "0047"
down_revision: Union[str, None] = "0046"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
# a. series-global page numbering, preserving the old reading order.
op.execute(
"""
WITH ordered AS (
SELECT sp.id,
ROW_NUMBER() OVER (
PARTITION BY sp.series_tag_id
ORDER BY sc.chapter_number, sp.page_number, sp.id
) AS rn
FROM series_page sp
JOIN series_chapter sc ON sc.id = sp.chapter_id
)
UPDATE series_page sp
SET page_number = ordered.rn
FROM ordered
WHERE sp.id = ordered.id
"""
)
# b. anchor each existing chapter at its first page (lowest new page_number).
op.add_column(
"series_chapter",
sa.Column("anchor_page_id", sa.Integer(), nullable=True),
)
op.execute(
"""
WITH firsts AS (
SELECT DISTINCT ON (sp.chapter_id)
sp.chapter_id, sp.id AS page_id
FROM series_page sp
ORDER BY sp.chapter_id, sp.page_number, sp.id
)
UPDATE series_chapter sc
SET anchor_page_id = firsts.page_id
FROM firsts
WHERE firsts.chapter_id = sc.id
"""
)
# c. sever the ownership link (drops the FK + index with the column) BEFORE
# pruning chapters, so deleting a chapter can't cascade-delete its pages.
op.drop_column("series_page", "chapter_id")
# d. prune chapters that don't become dividers: placeholders / empty ones
# (no anchor), and the unlabeled chapter that would land redundantly at
# page 1 (the series just starts — no divider needed there).
op.execute(
"""
DELETE FROM series_chapter sc
USING (
SELECT sc2.id
FROM series_chapter sc2
LEFT JOIN series_page sp ON sp.id = sc2.anchor_page_id
WHERE sc2.anchor_page_id IS NULL
OR (sp.page_number = 1
AND sc2.title IS NULL
AND sc2.stated_part IS NULL)
) gone
WHERE sc.id = gone.id
"""
)
# e. reshape into the divider model.
op.drop_column("series_chapter", "chapter_number")
op.drop_column("series_chapter", "is_placeholder")
op.drop_column("series_chapter", "stated_page_start")
op.drop_column("series_chapter", "stated_page_end")
op.alter_column("series_chapter", "anchor_page_id", nullable=False)
op.create_unique_constraint(
"uq_series_chapter_anchor_page", "series_chapter", ["anchor_page_id"]
)
op.create_foreign_key(
"fk_series_chapter_anchor_page",
"series_chapter",
"series_page",
["anchor_page_id"],
["id"],
ondelete="CASCADE",
)
def downgrade() -> None:
# Lossy: dividers can't be reconstructed as owning chapters. Collapse back to
# exactly one chapter per series that owns all its pages in order.
op.add_column(
"series_page", sa.Column("chapter_id", sa.Integer(), nullable=True)
)
op.drop_constraint(
"fk_series_chapter_anchor_page", "series_chapter", type_="foreignkey"
)
op.drop_constraint(
"uq_series_chapter_anchor_page", "series_chapter", type_="unique"
)
op.drop_column("series_chapter", "anchor_page_id")
op.add_column(
"series_chapter",
sa.Column(
"chapter_number", sa.Integer(), nullable=False, server_default="1"
),
)
op.add_column(
"series_chapter",
sa.Column(
"is_placeholder", sa.Boolean(), nullable=False,
server_default="false",
),
)
op.add_column(
"series_chapter",
sa.Column("stated_page_start", sa.Integer(), nullable=True),
)
op.add_column(
"series_chapter",
sa.Column("stated_page_end", sa.Integer(), nullable=True),
)
op.execute("DELETE FROM series_chapter")
op.execute(
"""
INSERT INTO series_chapter (series_tag_id, chapter_number)
SELECT DISTINCT series_tag_id, 1 FROM series_page
"""
)
op.execute(
"""
UPDATE series_page sp
SET chapter_id = sc.id
FROM series_chapter sc
WHERE sc.series_tag_id = sp.series_tag_id
"""
)
op.alter_column("series_page", "chapter_id", nullable=False)
op.create_foreign_key(
"fk_series_page_chapter",
"series_page",
"series_chapter",
["chapter_id"],
["id"],
ondelete="CASCADE",
)
@@ -1,45 +0,0 @@
"""series_page pending staging: status + nullable page_number (#789 Phase 2)
Pages added from a post no longer append straight into the run — they land
'pending' with a NULL page_number, staged grouped by their source post so the
operator can drop junk (text-free alts, bumpers) and place the keepers into the
sequence. A page only gets a series-global page_number once it's 'placed'.
Revision ID: 0048
Revises: 0047
Create Date: 2026-06-11
"""
from typing import Sequence, Union
import sqlalchemy as sa
from alembic import op
revision: str = "0048"
down_revision: Union[str, None] = "0047"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
op.add_column(
"series_page",
sa.Column(
"status", sa.String(length=16), nullable=False,
server_default="placed",
),
)
op.alter_column(
"series_page", "page_number",
existing_type=sa.Integer(), nullable=True,
)
def downgrade() -> None:
# Lossy: pending pages are unsorted staging rows with no order — drop them.
op.execute("DELETE FROM series_page WHERE status = 'pending'")
op.alter_column(
"series_page", "page_number",
existing_type=sa.Integer(), nullable=False,
)
op.drop_column("series_page", "status")
@@ -1,90 +0,0 @@
"""external_link table — off-platform file-host links found in post bodies
Creators host the real files on mega.nz / Google Drive / MediaFire / Dropbox /
Pixeldrain and link them in the post text. This table records each such link
(so nothing is silently dropped), and doubles as the dedup + dead-letter ledger
the download worker (a later slice) walks. `url` keeps the FULL link including
the `#fragment` — mega.nz's decryption key lives there; truncating it makes the
file undownloadable.
CHECK whitelists for host + status include the full enum up front (incl. the
download-worker statuses) so the worker slice needs no constraint migration.
Revision ID: 0049
Revises: 0048
Create Date: 2026-06-14
"""
from typing import Sequence, Union
import sqlalchemy as sa
from alembic import op
revision: str = "0049"
down_revision: Union[str, None] = "0048"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
op.create_table(
"external_link",
sa.Column("id", sa.Integer(), primary_key=True),
sa.Column(
"post_id", sa.Integer(),
sa.ForeignKey("post.id", ondelete="CASCADE"), nullable=False,
),
sa.Column(
"artist_id", sa.Integer(),
sa.ForeignKey("artist.id", ondelete="SET NULL"), nullable=True,
),
sa.Column("host", sa.String(length=16), nullable=False),
sa.Column("url", sa.Text(), nullable=False),
sa.Column("label", sa.Text(), nullable=True),
sa.Column(
"status", sa.String(length=16), nullable=False,
server_default="pending",
),
sa.Column("attempts", sa.Integer(), nullable=False, server_default="0"),
sa.Column("last_error", sa.Text(), nullable=True),
sa.Column(
"attachment_id", sa.Integer(),
sa.ForeignKey("post_attachment.id", ondelete="SET NULL"),
nullable=True,
),
sa.Column(
"created_at", sa.DateTime(timezone=True), nullable=False,
server_default=sa.func.now(),
),
sa.Column("completed_at", sa.DateTime(timezone=True), nullable=True),
sa.Column("duration_seconds", sa.Float(), nullable=True),
sa.CheckConstraint(
"host IN ('mega','gdrive','mediafire','dropbox','pixeldrain')",
name="ck_external_link_host",
),
sa.CheckConstraint(
"status IN ('pending','downloading','downloaded','failed',"
"'skipped','dead')",
name="ck_external_link_status",
),
)
op.create_index(
"ix_external_link_post_id", "external_link", ["post_id"],
)
op.create_index(
"ix_external_link_artist_id", "external_link", ["artist_id"],
)
op.create_index(
"ix_external_link_status", "external_link", ["status"],
)
op.create_index(
"uq_external_link_post_url", "external_link", ["post_id", "url"],
unique=True,
)
def downgrade() -> None:
op.drop_index("uq_external_link_post_url", table_name="external_link")
op.drop_index("ix_external_link_status", table_name="external_link")
op.drop_index("ix_external_link_artist_id", table_name="external_link")
op.drop_index("ix_external_link_post_id", table_name="external_link")
op.drop_table("external_link")
@@ -1,38 +0,0 @@
"""import_settings: per-host enable toggles for external file-host downloads
Operator levers (#830): disable a single host (e.g. mega.nz when it's
rate-limiting/banning) without touching the others. The worker reads these via
getattr and defaults to enabled, so the toggles default TRUE (works out of the
box, rule #26).
Revision ID: 0050
Revises: 0049
Create Date: 2026-06-14
"""
from typing import Sequence, Union
import sqlalchemy as sa
from alembic import op
revision: str = "0050"
down_revision: Union[str, None] = "0049"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
_HOSTS = ("mega", "gdrive", "mediafire", "dropbox", "pixeldrain")
def upgrade() -> None:
for host in _HOSTS:
op.add_column(
"import_settings",
sa.Column(
f"extdl_{host}_enabled", sa.Boolean(), nullable=False,
server_default=sa.true(),
),
)
def downgrade() -> None:
for host in _HOSTS:
op.drop_column("import_settings", f"extdl_{host}_enabled")
@@ -1,38 +0,0 @@
"""image_record: source_url + source_filehash (inline-image localization)
#830 Phase 2. To render a post body faithfully we serve LOCAL copies of inline
images instead of hotlinking the public CDN. The join key between a body
`<img src=CDN>` and the local file is the CDN's 32-hex filehash (the same
identity extract_media dedups by). Persist it (indexed) plus the full source
URL for provenance/debugging. Both NULL for filesystem-imported / pre-existing
rows — those fall back to hotlinking until re-downloaded.
Revision ID: 0051
Revises: 0050
Create Date: 2026-06-14
"""
from typing import Sequence, Union
import sqlalchemy as sa
from alembic import op
revision: str = "0051"
down_revision: Union[str, None] = "0050"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
op.add_column("image_record", sa.Column("source_url", sa.Text(), nullable=True))
op.add_column(
"image_record", sa.Column("source_filehash", sa.String(length=32), nullable=True)
)
op.create_index(
"ix_image_record_source_filehash", "image_record", ["source_filehash"]
)
def downgrade() -> None:
op.drop_index("ix_image_record_source_filehash", table_name="image_record")
op.drop_column("image_record", "source_filehash")
op.drop_column("image_record", "source_url")
@@ -1,32 +0,0 @@
"""image_record: duration_seconds (Tier-1 video near-dup key)
#871. Videos previously deduped on sha256 only (pHash is images-only), so a
different encode/remux of the same video imported as a distinct record. Persist
the container duration so the importer can treat same-artist videos with matching
duration (+ aspect ratio) as the same content and dedup/supersede like images.
NULL for images and for video rows imported before this column existed (a
backfill re-probes those so they participate in dedup).
Revision ID: 0052
Revises: 0051
Create Date: 2026-06-16
"""
from typing import Sequence, Union
import sqlalchemy as sa
from alembic import op
revision: str = "0052"
down_revision: Union[str, None] = "0051"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
op.add_column(
"image_record", sa.Column("duration_seconds", sa.Float(), nullable=True)
)
def downgrade() -> None:
op.drop_column("image_record", "duration_seconds")
@@ -1,49 +0,0 @@
"""ml_settings: video tagging knobs (cadence sampling + noise floor)
#747. Video tag quality/perf: sample frames at a fixed cadence (interval) so a
tag's frame-presence reflects real screen time, cap total frames so long videos
stay bounded, and keep a tag only if it appears in >= min_tag_frames sampled
frames. Operator-tunable via Settings → ML (replaces the VIDEO_ML_FRAMES env var).
Revision ID: 0053
Revises: 0052
Create Date: 2026-06-16
"""
from typing import Sequence, Union
import sqlalchemy as sa
from alembic import op
revision: str = "0053"
down_revision: Union[str, None] = "0052"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
op.add_column(
"ml_settings",
sa.Column(
"video_frame_interval_seconds", sa.Float(), nullable=False,
server_default="4.0",
),
)
op.add_column(
"ml_settings",
sa.Column(
"video_max_frames", sa.Integer(), nullable=False, server_default="64",
),
)
op.add_column(
"ml_settings",
sa.Column(
"video_min_tag_frames", sa.Integer(), nullable=False,
server_default="3",
),
)
def downgrade() -> None:
op.drop_column("ml_settings", "video_min_tag_frames")
op.drop_column("ml_settings", "video_max_frames")
op.drop_column("ml_settings", "video_frame_interval_seconds")
@@ -1,82 +0,0 @@
"""subscribestar_seen_media + subscribestar_failed_media: per-source ledgers
Revision ID: 0054
Revises: 0053
Create Date: 2026-06-17
SubscribeStar native ingester (phase 1 of the gallery-dl → native-core
migration). Mirrors the Patreon ledger tables (0037/0038): a seen-ledger so
routine walks skip already-ingested media (recovery bypasses it) and a
dead-letter ledger so persistently-failing media stops re-burning backfill
chunks. `filehash` is a CDN content hash when present, else a synthesized
``<post_id>:<filename>`` key — hence String(128). UNIQUE (source_id, filehash)
is the upsert key on each.
"""
from typing import Sequence, Union
import sqlalchemy as sa
from alembic import op
revision: str = "0054"
down_revision: Union[str, None] = "0053"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
op.create_table(
"subscribestar_seen_media",
sa.Column("id", sa.Integer, primary_key=True),
sa.Column(
"source_id",
sa.Integer,
sa.ForeignKey("source.id", ondelete="CASCADE"),
nullable=False,
index=True,
),
sa.Column("filehash", sa.String(128), nullable=False),
sa.Column("post_id", sa.String(64), nullable=True),
sa.Column(
"seen_at",
sa.DateTime(timezone=True),
nullable=False,
server_default=sa.text("NOW()"),
),
sa.UniqueConstraint(
"source_id", "filehash", name="uq_subscribestar_seen_media_source_id"
),
)
op.create_table(
"subscribestar_failed_media",
sa.Column("id", sa.Integer, primary_key=True),
sa.Column(
"source_id",
sa.Integer,
sa.ForeignKey("source.id", ondelete="CASCADE"),
nullable=False,
index=True,
),
sa.Column("filehash", sa.String(128), nullable=False),
sa.Column("attempts", sa.Integer, nullable=False, server_default="1"),
sa.Column("last_error", sa.Text, nullable=True),
sa.Column(
"first_failed_at",
sa.DateTime(timezone=True),
nullable=False,
server_default=sa.text("NOW()"),
),
sa.Column(
"last_failed_at",
sa.DateTime(timezone=True),
nullable=False,
server_default=sa.text("NOW()"),
),
sa.UniqueConstraint(
"source_id", "filehash", name="uq_subscribestar_failed_media_source_id"
),
)
def downgrade() -> None:
op.drop_table("subscribestar_failed_media")
op.drop_table("subscribestar_seen_media")
@@ -1,55 +0,0 @@
"""image_provenance: from_attachment_id (which archive an image was extracted from)
Milestone #87. When an image is pulled out of a .zip/.rar, record WHICH archive
PostAttachment it came from, so the provenance UI can show the single archive a
file lives inside instead of every attachment on the post. Nullable FK with
ON DELETE SET NULL — a loose (non-archive) download leaves it NULL, and deleting
the archive attachment forgets the linkage without destroying the (image, post)
provenance edge. Existing rows are NULL until the reextract backfill stamps them.
Revision ID: 0055
Revises: 0054
Create Date: 2026-06-22
"""
from typing import Sequence, Union
import sqlalchemy as sa
from alembic import op
revision: str = "0055"
down_revision: Union[str, None] = "0054"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
op.add_column(
"image_provenance",
sa.Column("from_attachment_id", sa.Integer(), nullable=True),
)
op.create_index(
"ix_image_provenance_from_attachment_id",
"image_provenance",
["from_attachment_id"],
)
op.create_foreign_key(
"fk_image_provenance_from_attachment",
"image_provenance",
"post_attachment",
["from_attachment_id"],
["id"],
ondelete="SET NULL",
)
def downgrade() -> None:
op.drop_constraint(
"fk_image_provenance_from_attachment",
"image_provenance",
type_="foreignkey",
)
op.drop_index(
"ix_image_provenance_from_attachment_id",
table_name="image_provenance",
)
op.drop_column("image_provenance", "from_attachment_id")
-43
View File
@@ -1,43 +0,0 @@
"""tag_eval_run: persisted head-vs-centroid tagging eval runs (#1130)
Milestone #114 slice 1. A long ml-queue eval whose full report must SURVIVE
navigation, so the run + report live in a row the admin card rehydrates from
(mirrors library_audit_run). running -> ready / error.
Revision ID: 0056
Revises: 0055
Create Date: 2026-06-28
"""
from typing import Sequence, Union
import sqlalchemy as sa
from alembic import op
from sqlalchemy.dialects.postgresql import JSONB
revision: str = "0056"
down_revision: Union[str, None] = "0055"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
op.create_table(
"tag_eval_run",
sa.Column("id", sa.Integer(), primary_key=True),
sa.Column("params", JSONB(), nullable=False),
sa.Column("status", sa.String(length=16), nullable=False, server_default="running"),
sa.Column(
"started_at", sa.DateTime(timezone=True), nullable=False,
server_default=sa.func.now(),
),
sa.Column("finished_at", sa.DateTime(timezone=True), nullable=True),
sa.Column("report", JSONB(), nullable=True),
sa.Column("error", sa.Text(), nullable=True),
sa.Column("last_progress_at", sa.DateTime(timezone=True), nullable=True),
)
op.create_index("ix_tag_eval_run_status", "tag_eval_run", ["status"])
def downgrade() -> None:
op.drop_index("ix_tag_eval_run_status", table_name="tag_eval_run")
op.drop_table("tag_eval_run")
@@ -1,40 +0,0 @@
"""tag_positive_confirmation: operator-affirmed correct positives (#1130)
Mirror of tag_suggestion_rejection. "Keep" on a doubted positive records here so
the eval's doubts list stops resurfacing confirmed-correct images every run.
Revision ID: 0057
Revises: 0056
Create Date: 2026-06-28
"""
from typing import Sequence, Union
import sqlalchemy as sa
from alembic import op
revision: str = "0057"
down_revision: Union[str, None] = "0056"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
op.create_table(
"tag_positive_confirmation",
sa.Column(
"image_record_id", sa.Integer(),
sa.ForeignKey("image_record.id", ondelete="CASCADE"), primary_key=True,
),
sa.Column(
"tag_id", sa.Integer(),
sa.ForeignKey("tag.id", ondelete="CASCADE"), primary_key=True, index=True,
),
sa.Column(
"confirmed_at", sa.DateTime(timezone=True), nullable=False,
server_default=sa.func.now(),
),
)
def downgrade() -> None:
op.drop_table("tag_positive_confirmation")
-95
View File
@@ -1,95 +0,0 @@
"""tag_head + head_training_run: production heads that learn from tags (#114)
The eval (#1130) proved the frozen-embedding + trained-head spine; this lands its
production form. tag_head stores one logistic-regression head per concept (the
new suggestion source, replacing Camie + centroid); head_training_run tracks the
batch that (re)trains them. Adds two head-training tunables to ml_settings.
Revision ID: 0058
Revises: 0057
Create Date: 2026-06-28
"""
from typing import Sequence, Union
import sqlalchemy as sa
from alembic import op
from pgvector.sqlalchemy import Vector
from sqlalchemy.dialects.postgresql import JSONB
revision: str = "0058"
down_revision: Union[str, None] = "0057"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
_HEAD_DIM = 1152
def upgrade() -> None:
op.create_table(
"tag_head",
sa.Column(
"tag_id", sa.Integer(),
sa.ForeignKey("tag.id", ondelete="CASCADE"), primary_key=True,
),
sa.Column("embedding_version", sa.String(length=128), nullable=False),
sa.Column("weights", Vector(_HEAD_DIM), nullable=False),
sa.Column("bias", sa.Float(), nullable=False),
sa.Column("suggest_threshold", sa.Float(), nullable=False),
sa.Column("auto_apply_threshold", sa.Float(), nullable=True),
sa.Column("n_pos", sa.Integer(), nullable=False),
sa.Column("n_neg", sa.Integer(), nullable=False),
sa.Column("ap", sa.Float(), nullable=False),
sa.Column("precision_cv", sa.Float(), nullable=False),
sa.Column("recall", sa.Float(), nullable=False),
sa.Column(
"trained_at", sa.DateTime(timezone=True), nullable=False,
server_default=sa.func.now(),
),
sa.Column("metrics", JSONB(), nullable=True),
)
op.create_table(
"head_training_run",
sa.Column("id", sa.Integer(), primary_key=True),
sa.Column("params", JSONB(), nullable=False),
sa.Column(
"status", sa.String(length=16), nullable=False,
server_default="running",
),
sa.Column(
"started_at", sa.DateTime(timezone=True), nullable=False,
server_default=sa.func.now(),
),
sa.Column("finished_at", sa.DateTime(timezone=True), nullable=True),
sa.Column("n_trained", sa.Integer(), nullable=True),
sa.Column("n_skipped", sa.Integer(), nullable=True),
sa.Column("error", sa.Text(), nullable=True),
sa.Column("last_progress_at", sa.DateTime(timezone=True), nullable=True),
)
op.create_index(
"ix_head_training_run_status", "head_training_run", ["status"],
)
# Head-training tunables on the ml_settings singleton.
op.add_column(
"ml_settings",
sa.Column(
"head_min_positives", sa.Integer(), nullable=False,
server_default="8",
),
)
op.add_column(
"ml_settings",
sa.Column(
"head_auto_apply_precision", sa.Float(), nullable=False,
server_default="0.97",
),
)
def downgrade() -> None:
op.drop_column("ml_settings", "head_auto_apply_precision")
op.drop_column("ml_settings", "head_min_positives")
op.drop_index("ix_head_training_run_status", table_name="head_training_run")
op.drop_table("head_training_run")
op.drop_table("tag_head")
-70
View File
@@ -1,70 +0,0 @@
"""head_auto_apply_run + earned-auto-apply settings (#114)
A graduated head can apply its tag without a human, gated by a master switch +
a support floor. head_auto_apply_run tracks each sweep / dry-run preview.
Revision ID: 0059
Revises: 0058
Create Date: 2026-06-29
"""
from typing import Sequence, Union
import sqlalchemy as sa
from alembic import op
from sqlalchemy.dialects.postgresql import JSONB
revision: str = "0059"
down_revision: Union[str, None] = "0058"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
op.create_table(
"head_auto_apply_run",
sa.Column("id", sa.Integer(), primary_key=True),
sa.Column(
"dry_run", sa.Boolean(), nullable=False, server_default=sa.false()
),
sa.Column("params", JSONB(), nullable=False),
sa.Column(
"status", sa.String(length=16), nullable=False,
server_default="running",
),
sa.Column(
"started_at", sa.DateTime(timezone=True), nullable=False,
server_default=sa.func.now(),
),
sa.Column("finished_at", sa.DateTime(timezone=True), nullable=True),
sa.Column("n_applied", sa.Integer(), nullable=True),
sa.Column("report", JSONB(), nullable=True),
sa.Column("error", sa.Text(), nullable=True),
sa.Column("last_progress_at", sa.DateTime(timezone=True), nullable=True),
)
op.create_index(
"ix_head_auto_apply_run_status", "head_auto_apply_run", ["status"],
)
op.add_column(
"ml_settings",
sa.Column(
"head_auto_apply_enabled", sa.Boolean(), nullable=False,
server_default=sa.true(), # opt-out: on by default (operator-asked)
),
)
op.add_column(
"ml_settings",
sa.Column(
"head_auto_apply_min_positives", sa.Integer(), nullable=False,
server_default="30",
),
)
def downgrade() -> None:
op.drop_column("ml_settings", "head_auto_apply_min_positives")
op.drop_column("ml_settings", "head_auto_apply_enabled")
op.drop_index(
"ix_head_auto_apply_run_status", table_name="head_auto_apply_run"
)
op.drop_table("head_auto_apply_run")
-74
View File
@@ -1,74 +0,0 @@
"""head_metric + head_metrics_snapshot: auto-apply observability (#114)
Running misfire/under-fire counters per concept (captured at correction time,
since image_tag.source is lost on delete) + a daily per-concept time-series so
the operator can tune the precision target + support floor from real data.
Revision ID: 0060
Revises: 0059
Create Date: 2026-06-29
"""
from typing import Sequence, Union
import sqlalchemy as sa
from alembic import op
revision: str = "0060"
down_revision: Union[str, None] = "0059"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
op.create_table(
"head_metric",
sa.Column(
"tag_id", sa.Integer(),
sa.ForeignKey("tag.id", ondelete="CASCADE"), primary_key=True,
),
sa.Column("n_misfires", sa.Integer(), nullable=False, server_default="0"),
sa.Column("n_underfires", sa.Integer(), nullable=False, server_default="0"),
sa.Column(
"updated_at", sa.DateTime(timezone=True), nullable=False,
server_default=sa.func.now(),
),
)
op.create_table(
"head_metrics_snapshot",
sa.Column("id", sa.Integer(), primary_key=True),
sa.Column(
"tag_id", sa.Integer(),
sa.ForeignKey("tag.id", ondelete="CASCADE"),
),
sa.Column("name", sa.String(length=255), nullable=False),
sa.Column(
"snapshot_at", sa.DateTime(timezone=True), nullable=False,
server_default=sa.func.now(),
),
sa.Column("n_auto_applied", sa.Integer(), nullable=False, server_default="0"),
sa.Column("n_misfires", sa.Integer(), nullable=False, server_default="0"),
sa.Column("n_underfires", sa.Integer(), nullable=False, server_default="0"),
sa.Column("ap", sa.Float(), nullable=True),
sa.Column("precision_cv", sa.Float(), nullable=True),
sa.Column("recall", sa.Float(), nullable=True),
sa.Column("n_pos", sa.Integer(), nullable=True),
)
op.create_index(
"ix_head_metrics_snapshot_tag_id", "head_metrics_snapshot", ["tag_id"],
)
op.create_index(
"ix_head_metrics_snapshot_snapshot_at", "head_metrics_snapshot",
["snapshot_at"],
)
def downgrade() -> None:
op.drop_index(
"ix_head_metrics_snapshot_snapshot_at", table_name="head_metrics_snapshot"
)
op.drop_index(
"ix_head_metrics_snapshot_tag_id", table_name="head_metrics_snapshot"
)
op.drop_table("head_metrics_snapshot")
op.drop_table("head_metric")
-59
View File
@@ -1,59 +0,0 @@
"""image_region: detected/proposed regions + their crop embeddings (#114)
Storage backbone of the crop pipeline. A region = normalized bbox + the crop's
embedding (CCIP for face/figure → character id; SigLIP for concept regions →
head bag-of-embeddings). Also serves as grounded-tag bbox provenance.
Revision ID: 0061
Revises: 0060
Create Date: 2026-06-29
"""
from typing import Sequence, Union
import sqlalchemy as sa
from alembic import op
from pgvector.sqlalchemy import Vector
revision: str = "0061"
down_revision: Union[str, None] = "0060"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
_CCIP_DIM = 768
_SIGLIP_DIM = 1152
def upgrade() -> None:
op.create_table(
"image_region",
sa.Column("id", sa.Integer(), primary_key=True),
sa.Column(
"image_record_id", sa.Integer(),
sa.ForeignKey("image_record.id", ondelete="CASCADE"), nullable=False,
),
sa.Column("kind", sa.String(length=16), nullable=False),
# Video/animated: source frame timestamp (seconds); NULL for stills.
sa.Column("frame_time", sa.Float(), nullable=True),
sa.Column("rx", sa.Float(), nullable=False),
sa.Column("ry", sa.Float(), nullable=False),
sa.Column("rw", sa.Float(), nullable=False),
sa.Column("rh", sa.Float(), nullable=False),
sa.Column("score", sa.Float(), nullable=True),
sa.Column("detector_version", sa.String(length=64), nullable=True),
sa.Column("crop_version", sa.String(length=64), nullable=True),
sa.Column("embedding_version", sa.String(length=128), nullable=True),
sa.Column("ccip_embedding", Vector(_CCIP_DIM), nullable=True),
sa.Column("siglip_embedding", Vector(_SIGLIP_DIM), nullable=True),
sa.Column(
"created_at", sa.DateTime(timezone=True), nullable=False,
server_default=sa.func.now(),
),
)
op.create_index(
"ix_image_region_image_record_id", "image_region", ["image_record_id"],
)
def downgrade() -> None:
op.drop_index("ix_image_region_image_record_id", table_name="image_region")
op.drop_table("image_region")
-55
View File
@@ -1,55 +0,0 @@
"""gpu_job: the HTTP-leased GPU work queue for the desktop agent (#114)
The agent stays HTTP-only — the server enqueues per-(image, task) jobs here and
the agent leases/submits over the web API; Redis/Postgres stay private.
Revision ID: 0062
Revises: 0061
Create Date: 2026-06-29
"""
from typing import Sequence, Union
import sqlalchemy as sa
from alembic import op
revision: str = "0062"
down_revision: Union[str, None] = "0061"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
op.create_table(
"gpu_job",
sa.Column("id", sa.Integer(), primary_key=True),
sa.Column(
"image_record_id", sa.Integer(),
sa.ForeignKey("image_record.id", ondelete="CASCADE"), nullable=False,
),
sa.Column("task", sa.String(length=32), nullable=False),
sa.Column(
"status", sa.String(length=16), nullable=False,
server_default="pending",
),
sa.Column("lease_token", sa.String(length=64), nullable=True),
sa.Column("leased_at", sa.DateTime(timezone=True), nullable=True),
sa.Column("lease_expires_at", sa.DateTime(timezone=True), nullable=True),
sa.Column("attempts", sa.Integer(), nullable=False, server_default="0"),
sa.Column("error", sa.Text(), nullable=True),
sa.Column(
"created_at", sa.DateTime(timezone=True), nullable=False,
server_default=sa.func.now(),
),
sa.Column(
"updated_at", sa.DateTime(timezone=True), nullable=False,
server_default=sa.func.now(),
),
)
op.create_index("ix_gpu_job_image_record_id", "gpu_job", ["image_record_id"])
op.create_index("ix_gpu_job_status", "gpu_job", ["status"])
def downgrade() -> None:
op.drop_index("ix_gpu_job_status", table_name="gpu_job")
op.drop_index("ix_gpu_job_image_record_id", table_name="gpu_job")
op.drop_table("gpu_job")
@@ -1,33 +0,0 @@
"""ml_settings.ccip_match_threshold — tunable CCIP character-match cut (#114)
The v1 matcher used a flat 0.75 cosine; live data showed that over-fires (a
high-reference character matched a scatter of images). 0.85 keeps the confident
single-character matches and drops the noise. Tunable from the GPU agent card.
Revision ID: 0063
Revises: 0062
Create Date: 2026-06-29
"""
from typing import Sequence, Union
import sqlalchemy as sa
from alembic import op
revision: str = "0063"
down_revision: Union[str, None] = "0062"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
op.add_column(
"ml_settings",
sa.Column(
"ccip_match_threshold", sa.Float(), nullable=False,
server_default="0.85",
),
)
def downgrade() -> None:
op.drop_column("ml_settings", "ccip_match_threshold")
-42
View File
@@ -1,42 +0,0 @@
"""ml_settings: CCIP auto-apply switch + threshold (#114)
Confident CCIP character matches auto-tag (source='ccip_auto') on a daily sweep,
so identity tags keep flowing without pressing a button. ON by default (opt-out,
like head auto-apply); the high threshold (0.92, above the 0.85 suggest cut) +
single-character references keep it safe, and every auto-tag is reversible.
Revision ID: 0064
Revises: 0063
Create Date: 2026-06-30
"""
from typing import Sequence, Union
import sqlalchemy as sa
from alembic import op
revision: str = "0064"
down_revision: Union[str, None] = "0063"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
op.add_column(
"ml_settings",
sa.Column(
"ccip_auto_apply_enabled", sa.Boolean(), nullable=False,
server_default=sa.true(),
),
)
op.add_column(
"ml_settings",
sa.Column(
"ccip_auto_apply_threshold", sa.Float(), nullable=False,
server_default="0.92",
),
)
def downgrade() -> None:
op.drop_column("ml_settings", "ccip_auto_apply_threshold")
op.drop_column("ml_settings", "ccip_auto_apply_enabled")
@@ -1,35 +0,0 @@
"""ml_settings: embedder_model_name (#1190 operator model swap)
The embedder MODEL VERSION was already a setting (and stamps image_record.
siglip_model_version); the HF model NAME was env-only, so an operator couldn't
actually point the pipeline at a different embedder. Storing the name as a
setting makes the model an operator choice: set name + version → re-embed (the
GPU agent) → retrain heads. Default = the current SigLIP so400m.
Revision ID: 0065
Revises: 0064
Create Date: 2026-06-30
"""
from typing import Sequence, Union
import sqlalchemy as sa
from alembic import op
revision: str = "0065"
down_revision: Union[str, None] = "0064"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
op.add_column(
"ml_settings",
sa.Column(
"embedder_model_name", sa.String(length=128), nullable=False,
server_default="google/siglip-so400m-patch14-384",
),
)
def downgrade() -> None:
op.drop_column("ml_settings", "embedder_model_name")
-57
View File
@@ -1,57 +0,0 @@
"""drop the dead per-tag centroid subsystem (#1189 cleanup)
The v2 pivot replaced per-tag SigLIP centroids with learned heads + CCIP.
Nothing read the centroids anymore — they were recomputed (on merge + a daily
beat) but never consumed for suggestions or auto-apply. Remove the storage +
its two now-unused settings columns. (The recompute tasks, beat, endpoint,
service, and UI card are removed in the same change.)
Revision ID: 0066
Revises: 0065
Create Date: 2026-06-30
"""
from typing import Sequence, Union
import sqlalchemy as sa
from alembic import op
revision: str = "0066"
down_revision: Union[str, None] = "0065"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
op.drop_table("tag_reference_embedding")
op.drop_column("ml_settings", "centroid_similarity_threshold")
op.drop_column("ml_settings", "min_reference_images")
def downgrade() -> None:
op.add_column(
"ml_settings",
sa.Column(
"min_reference_images", sa.Integer(), nullable=False,
server_default="5",
),
)
op.add_column(
"ml_settings",
sa.Column(
"centroid_similarity_threshold", sa.Float(), nullable=False,
server_default="0.55",
),
)
op.create_table(
"tag_reference_embedding",
sa.Column("tag_id", sa.Integer(), nullable=False),
sa.Column("embedding", sa.LargeBinary(), nullable=False),
sa.Column("reference_count", sa.Integer(), nullable=False),
sa.Column("model_version", sa.String(length=128), nullable=False),
sa.Column(
"updated_at", sa.DateTime(timezone=True),
server_default=sa.func.now(), nullable=False,
),
sa.ForeignKeyConstraint(["tag_id"], ["tag.id"], ondelete="CASCADE"),
sa.PrimaryKeyConstraint("tag_id"),
)
@@ -1,66 +0,0 @@
"""retire the Camie tagger + allowlist bulk-apply (#1189)
The v2 pivot made heads + CCIP the tag source and head auto-apply the earned
propagation. The Camie tagger ran only to feed the allowlist bulk-apply (its
predictions had no other consumer), and the allowlist was a second, un-earned
auto-apply path parallel to heads. Both are retired — drop their storage.
(image_prediction = Camie's per-image predictions; tag_allowlist = the bulk-
apply allowlist. Nothing references INTO these tables, so the drop is clean.)
Revision ID: 0067
Revises: 0066
Create Date: 2026-06-30
"""
from typing import Sequence, Union
import sqlalchemy as sa
from alembic import op
revision: str = "0067"
down_revision: Union[str, None] = "0066"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
op.drop_table("image_prediction")
op.drop_table("tag_allowlist")
def downgrade() -> None:
op.create_table(
"tag_allowlist",
sa.Column("tag_id", sa.Integer(), nullable=False),
sa.Column(
"min_confidence", sa.Float(), nullable=False, server_default="0.9"
),
sa.Column(
"created_at", sa.DateTime(timezone=True),
server_default=sa.func.now(), nullable=False,
),
sa.ForeignKeyConstraint(["tag_id"], ["tag.id"], ondelete="CASCADE"),
sa.PrimaryKeyConstraint("tag_id"),
sa.CheckConstraint(
"min_confidence >= 0 AND min_confidence <= 1",
name="ck_tag_allowlist_confidence_range",
),
)
op.create_table(
"image_prediction",
sa.Column("id", sa.Integer(), primary_key=True),
sa.Column("image_record_id", sa.Integer(), nullable=False),
sa.Column("raw_name", sa.String(length=255), nullable=False),
sa.Column("category", sa.String(length=32), nullable=False),
sa.Column("score", sa.Float(), nullable=False),
sa.ForeignKeyConstraint(
["image_record_id"], ["image_record.id"], ondelete="CASCADE"
),
)
op.create_index(
"ix_image_prediction_image", "image_prediction", ["image_record_id"]
)
op.create_index(
"ix_image_prediction_name_score", "image_prediction",
["raw_name", "score"],
)
@@ -1,80 +0,0 @@
"""drop dead tagger/suggestion settings + columns left after Camie retirement (#1199)
Hygiene follow-up to #1189. These were left inert to bound that change; nothing
reads them now:
- ml_settings: tagger_store_floor + tagger_model_version (only the deleted Camie
tagger used them), suggestion_threshold_character/general (already dead pre-
retirement — scoring uses per-head thresholds), video_min_tag_frames (only the
deleted video-prediction aggregator used it).
- image_record: tagger_model_version (no writer now), centroid_scores (long-dead
JSON cache, no reader).
Revision ID: 0068
Revises: 0067
Create Date: 2026-06-30
"""
from typing import Sequence, Union
import sqlalchemy as sa
from alembic import op
revision: str = "0068"
down_revision: Union[str, None] = "0067"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
op.drop_column("ml_settings", "suggestion_threshold_character")
op.drop_column("ml_settings", "suggestion_threshold_general")
op.drop_column("ml_settings", "tagger_store_floor")
op.drop_column("ml_settings", "video_min_tag_frames")
op.drop_column("ml_settings", "tagger_model_version")
op.drop_column("image_record", "tagger_model_version")
op.drop_column("image_record", "centroid_scores")
def downgrade() -> None:
op.add_column(
"image_record",
sa.Column("centroid_scores", sa.JSON(), nullable=True),
)
op.add_column(
"image_record",
sa.Column("tagger_model_version", sa.String(length=128), nullable=True),
)
op.add_column(
"ml_settings",
sa.Column(
"tagger_model_version", sa.String(length=128), nullable=False,
server_default="camie-tagger-v2",
),
)
op.add_column(
"ml_settings",
sa.Column(
"video_min_tag_frames", sa.Integer(), nullable=False,
server_default="3",
),
)
op.add_column(
"ml_settings",
sa.Column(
"tagger_store_floor", sa.Float(), nullable=False,
server_default="0.7",
),
)
op.add_column(
"ml_settings",
sa.Column(
"suggestion_threshold_general", sa.Float(), nullable=False,
server_default="0.7",
),
)
op.add_column(
"ml_settings",
sa.Column(
"suggestion_threshold_character", sa.Float(), nullable=False,
server_default="0.7",
),
)
-51
View File
@@ -1,51 +0,0 @@
"""default the embedder to SigLIP 2 — for FRESH installs only (#1203)
Make SigLIP 2 (so400m, 512px; a 1152-d drop-in) the default embedder. New
installs start on it. An EXISTING library is NOT touched: flipping its stored
embedder version would mark every embedding stale (the scorer is version-gated)
and kill suggestions until a full re-embed+retrain — so an existing instance
switches deliberately via Settings → GPU agent → Embedding model → Re-embed →
Retrain. We detect "fresh" by the absence of any embedded image.
Revision ID: 0069
Revises: 0068
Create Date: 2026-06-30
"""
from typing import Sequence, Union
from alembic import op
revision: str = "0069"
down_revision: Union[str, None] = "0068"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
_NEW_NAME = "google/siglip2-so400m-patch16-512"
_NEW_VERSION = "siglip2-so400m-patch16-512"
_OLD_NAME = "google/siglip-so400m-patch14-384"
_OLD_VERSION = "siglip-so400m-patch14-384"
def upgrade() -> None:
# Fresh install (nothing embedded yet) → adopt SigLIP 2.
op.execute(
f"""
UPDATE ml_settings SET
embedder_model_name = '{_NEW_NAME}',
embedder_model_version = '{_NEW_VERSION}'
WHERE NOT EXISTS (
SELECT 1 FROM image_record WHERE siglip_embedding IS NOT NULL
)
"""
)
op.alter_column("ml_settings", "embedder_model_name", server_default=_NEW_NAME)
op.alter_column(
"ml_settings", "embedder_model_version", server_default=_NEW_VERSION
)
def downgrade() -> None:
op.alter_column("ml_settings", "embedder_model_name", server_default=_OLD_NAME)
op.alter_column(
"ml_settings", "embedder_model_version", server_default=_OLD_VERSION
)
@@ -1,44 +0,0 @@
"""partial indexes so GPU-job leasing stays O(batch), not O(completed)
The lease claims the lowest-id pending (or expired-leased) jobs. With only a
plain `status` index, `... ORDER BY id LIMIT n` walked the primary-key index from
the start, skipping the entire prefix of already-done/error rows before reaching
pending ones — so leasing slowed to a crawl as `done` piled up (the whole reason
throughput fell off a cliff mid-run and /status stalled). Two partial indexes fix
it: the pending one is id-ordered so the hot path reads just the first n entries,
and the leased-expiry one keeps the crash-recovery reclaim + the orphan sweep
cheap. They cover only the small live slice of the table, so they stay tiny even
as the done/error history grows to millions.
Revision ID: 0070
Revises: 0069
Create Date: 2026-06-30
"""
from typing import Sequence, Union
import sqlalchemy as sa
from alembic import op
revision: str = "0070"
down_revision: Union[str, None] = "0069"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
# Hot path: lowest-id pending jobs. Index on id, restricted to pending, so
# `WHERE status='pending' ORDER BY id LIMIT n` is a short index-order scan.
op.create_index(
"ix_gpu_job_pending", "gpu_job", ["id"],
postgresql_where=sa.text("status = 'pending'"),
)
# Crash-recovery: expired leases, for the lease backstop + recover_orphaned.
op.create_index(
"ix_gpu_job_leased_expires", "gpu_job", ["lease_expires_at"],
postgresql_where=sa.text("status = 'leased'"),
)
def downgrade() -> None:
op.drop_index("ix_gpu_job_leased_expires", table_name="gpu_job")
op.drop_index("ix_gpu_job_pending", table_name="gpu_job")
@@ -1,80 +0,0 @@
"""image_record.earliest_post_date: original-publish gallery sort key + index
Revision ID: 0071
Revises: 0070
Create Date: 2026-07-01
effective_date (0035) keys off the PRIMARY post — which is often the repost /
download the file actually came from — and falls back to created_at, so the
gallery's default order surfaces download dates rather than when content was
first posted (operator-flagged 2026-07-01). Materialize a second sort key,
earliest_post_date = MIN(post_date) across ALL of an image's provenance posts
(every post it appears in), falling back to created_at only when no linked post
carries a date. Indexed (DESC, id DESC) so the "post date" gallery sort is an
index range scan just like effective_date.
Backfill mirrors 0035: created_at baseline, then override with the MIN over
image_provenance ⋈ post. New rows get the created_at-equivalent server default;
services/importer.py recomputes it whenever a dated post is linked.
"""
from typing import Sequence, Union
import sqlalchemy as sa
from alembic import op
revision: str = "0071"
down_revision: Union[str, None] = "0070"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
# Add nullable first so the backfill can populate before NOT NULL.
op.add_column(
"image_record",
sa.Column("earliest_post_date", sa.DateTime(timezone=True), nullable=True),
)
# Baseline: download date. Set-based (no per-row binds) → immune to the
# 65535 bind-parameter ceiling regardless of library size.
op.execute(
"""
UPDATE image_record
SET earliest_post_date = created_at
"""
)
# Override with the earliest post_date across EVERY post the image appears
# in (image_provenance is the many-to-many edge; ignore posts with no date).
op.execute(
"""
UPDATE image_record AS ir
SET earliest_post_date = sub.min_date
FROM (
SELECT ip.image_record_id AS iid, MIN(p.post_date) AS min_date
FROM image_provenance AS ip
JOIN post AS p ON p.id = ip.post_id
WHERE p.post_date IS NOT NULL
GROUP BY ip.image_record_id
) AS sub
WHERE ir.id = sub.iid
"""
)
op.alter_column(
"image_record",
"earliest_post_date",
nullable=False,
server_default=sa.text("now()"),
)
# DESC/DESC matches the gallery's ORDER BY earliest_post_date DESC, id DESC
# so the "post date" scroll is a forward index scan; raw SQL because
# alembic's column list doesn't express per-column DESC cleanly.
op.execute(
"CREATE INDEX ix_image_record_earliest_post_date "
"ON image_record (earliest_post_date DESC, id DESC)"
)
def downgrade() -> None:
op.drop_index(
"ix_image_record_earliest_post_date", table_name="image_record"
)
op.drop_column("image_record", "earliest_post_date")
@@ -1,32 +0,0 @@
"""gpu_job.triage_status — the probe's verdict on an errored job's FILE
Failure triage (#125): a periodic sweep probes each errored image's file
(sha256 + decode, verify_integrity's machinery) exactly once and stores the
verdict here — 'defect' (the file is bad: recovery material, excluded from
/retry_errors) or 'file_ok' (failure was operational, safe to retry). NULL
means not yet probed; selecting on NULL is what makes the sweep resumable.
No index: the errored slice the sweep scans is tiny by design (tombstones).
Revision ID: 0072
Revises: 0071
Create Date: 2026-07-02
"""
from typing import Sequence, Union
import sqlalchemy as sa
from alembic import op
revision: str = "0072"
down_revision: Union[str, None] = "0071"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
op.add_column(
"gpu_job", sa.Column("triage_status", sa.String(16), nullable=True)
)
def downgrade() -> None:
op.drop_column("gpu_job", "triage_status")
@@ -1,46 +0,0 @@
"""drop tag_eval_run — the head-vs-centroid eval harness is retired
The eval (#1130) existed to prove the heads tagging spine on the operator's own
data. It did; the operator accepted the system and retired the harness
(2026-07-02) — card, API, task, model and this table all go. The eval's data
loaders + metric helpers live on in services/ml/training_data.py, where the
production heads trainer uses them nightly.
Revision ID: 0073
Revises: 0072
Create Date: 2026-07-02
"""
from typing import Sequence, Union
import sqlalchemy as sa
from alembic import op
from sqlalchemy.dialects import postgresql
revision: str = "0073"
down_revision: Union[str, None] = "0072"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
op.drop_index("ix_tag_eval_run_status", table_name="tag_eval_run")
op.drop_table("tag_eval_run")
def downgrade() -> None:
# Recreates the shape from 0056 (data is not restorable).
op.create_table(
"tag_eval_run",
sa.Column("id", sa.Integer(), primary_key=True),
sa.Column("params", postgresql.JSONB(), nullable=False),
sa.Column("status", sa.String(length=16), nullable=False,
server_default="running"),
sa.Column("started_at", sa.DateTime(timezone=True), nullable=False,
server_default=sa.func.now()),
sa.Column("finished_at", sa.DateTime(timezone=True), nullable=True),
sa.Column("report", postgresql.JSONB(), nullable=True),
sa.Column("error", sa.Text(), nullable=True),
sa.Column("last_progress_at", sa.DateTime(timezone=True),
nullable=True),
)
op.create_index("ix_tag_eval_run_status", "tag_eval_run", ["status"])
@@ -1,35 +0,0 @@
"""ml_settings.cpu_embed_enabled — the CPU embed fallback becomes a switch
B3 (operator 2026-07-02): the ml-worker's only processing role is the CPU
whole-image embed for stacks without a GPU agent. ON by default (a fresh
install works agent-less); agent-equipped stacks that drop the ml-worker
container turn it off so import hooks stop queueing embed work into a queue
nothing consumes — the daily GPU 'embed' backfill covers those images.
Revision ID: 0074
Revises: 0073
Create Date: 2026-07-02
"""
from typing import Sequence, Union
import sqlalchemy as sa
from alembic import op
revision: str = "0074"
down_revision: Union[str, None] = "0073"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
op.add_column(
"ml_settings",
sa.Column(
"cpu_embed_enabled", sa.Boolean(), nullable=False,
server_default=sa.true(),
),
)
def downgrade() -> None:
op.drop_column("ml_settings", "cpu_embed_enabled")
-60
View File
@@ -1,60 +0,0 @@
"""tag.is_system + seed the three hygiene system tags
Training hygiene (operator 2026-07-03, milestone #128): rough WIPs tagged as a
character poison that character's head and CCIP references; banners/editor
screenshots pollute whole-image similarity. The fix keys on SYSTEM tags the
product ships — not operator configuration — so the seed lives here.
Seeding ADOPTS an existing same-(name, kind=general) tag (case-insensitive,
matching TagService.rename's collision stance) instead of inserting a
duplicate, so an operator who already tagged `wip` keeps their applications.
Revision ID: 0075
Revises: 0074
Create Date: 2026-07-03
"""
from typing import Sequence, Union
import sqlalchemy as sa
from alembic import op
revision: str = "0075"
down_revision: Union[str, None] = "0074"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
SYSTEM_TAG_NAMES = ("wip", "banner", "editor screenshot")
def upgrade() -> None:
op.add_column(
"tag",
sa.Column(
"is_system", sa.Boolean(), nullable=False,
server_default=sa.false(),
),
)
conn = op.get_bind()
for name in SYSTEM_TAG_NAMES:
adopted = conn.execute(
sa.text(
"UPDATE tag SET is_system = true "
"WHERE lower(name) = lower(:name) AND kind = 'general'"
),
{"name": name},
)
if adopted.rowcount == 0:
conn.execute(
sa.text(
"INSERT INTO tag (name, kind, is_system) "
"VALUES (:name, 'general', true)"
),
{"name": name},
)
def downgrade() -> None:
# The seeded rows survive as ordinary general tags — dropping the flag is
# enough to disarm the mechanism, and deleting rows would orphan any
# operator applications made while the flag existed.
op.drop_column("tag", "is_system")
-82
View File
@@ -1,82 +0,0 @@
"""pixiv_seen_media + pixiv_failed_media: per-source ledgers
Revision ID: 0076
Revises: 0075
Create Date: 2026-07-03
Pixiv native ingester (milestone #129, gallery-dl → native-core migration).
Mirrors the Patreon (0037/0038) and SubscribeStar (0054) ledger tables: a
seen-ledger so routine walks skip already-ingested media (recovery bypasses
it) and a dead-letter ledger so persistently-failing media stops re-burning
backfill chunks. Pixiv URLs carry no content hash, so `filehash` is always the
synthesized ``<illust_id>:p<num>`` / ``<illust_id>:ugoira`` key — String(128)
matches the siblings. UNIQUE (source_id, filehash) is the upsert key on each.
"""
from typing import Sequence, Union
import sqlalchemy as sa
from alembic import op
revision: str = "0076"
down_revision: Union[str, None] = "0075"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
op.create_table(
"pixiv_seen_media",
sa.Column("id", sa.Integer, primary_key=True),
sa.Column(
"source_id",
sa.Integer,
sa.ForeignKey("source.id", ondelete="CASCADE"),
nullable=False,
index=True,
),
sa.Column("filehash", sa.String(128), nullable=False),
sa.Column("post_id", sa.String(64), nullable=True),
sa.Column(
"seen_at",
sa.DateTime(timezone=True),
nullable=False,
server_default=sa.text("NOW()"),
),
sa.UniqueConstraint(
"source_id", "filehash", name="uq_pixiv_seen_media_source_id"
),
)
op.create_table(
"pixiv_failed_media",
sa.Column("id", sa.Integer, primary_key=True),
sa.Column(
"source_id",
sa.Integer,
sa.ForeignKey("source.id", ondelete="CASCADE"),
nullable=False,
index=True,
),
sa.Column("filehash", sa.String(128), nullable=False),
sa.Column("attempts", sa.Integer, nullable=False, server_default="1"),
sa.Column("last_error", sa.Text, nullable=True),
sa.Column(
"first_failed_at",
sa.DateTime(timezone=True),
nullable=False,
server_default=sa.text("NOW()"),
),
sa.Column(
"last_failed_at",
sa.DateTime(timezone=True),
nullable=False,
server_default=sa.text("NOW()"),
),
sa.UniqueConstraint(
"source_id", "filehash", name="uq_pixiv_failed_media_source_id"
),
)
def downgrade() -> None:
op.drop_table("pixiv_failed_media")
op.drop_table("pixiv_seen_media")
@@ -1,32 +0,0 @@
"""drop uq_artist_name — decouple display name from identity/storage
Revision ID: 0077
Revises: 0076
Create Date: 2026-07-04
Artist model fragility fix (milestone #130). One `slug` column was doing
identity + storage-path + display, and BOTH `name` and `slug` were UNIQUE, so
the display name couldn't be edited freely and two genuinely different creators
collided. Decouple: `slug` stays the immutable, unique storage/identity key (the
on-disk path component — untouched here); `name` becomes freely editable, NON-
unique display text. This migration only drops the `uq_artist_name` constraint;
no data moves and no path changes.
"""
from typing import Sequence, Union
from alembic import op
revision: str = "0077"
down_revision: Union[str, None] = "0076"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
op.drop_constraint("uq_artist_name", "artist", type_="unique")
def downgrade() -> None:
# Re-adding the UNIQUE would fail if duplicate names now exist; callers that
# need to reverse this must dedupe names first.
op.create_unique_constraint("uq_artist_name", "artist", ["name"])
@@ -1,83 +0,0 @@
"""ml_settings crop-proposer / detector config (#134)
Move the WHERE-to-crop detector config (per-proposer enable + weights + conf,
plus caps + dedupe IoU) into the DB so it's UI-tunable and announced to the GPU
agent in the lease (like the embedder model) — no restart, agent env is now
bootstrap-only. All server_defaults are the working values so existing rows +
fresh installs crop out-of-the-box with all three proposers ON.
Revision ID: 0078
Revises: 0077
Create Date: 2026-07-05
"""
from typing import Sequence, Union
import sqlalchemy as sa
from alembic import op
revision: str = "0078"
down_revision: Union[str, None] = "0077"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
_ANATOMY_DEFAULT = (
"https://github.com/aperveyev/booru_yolo/raw/main/models/yolov11m_aa22.pt"
)
_PANEL_DEFAULT = "mosesb/best-comic-panel-detection::best.pt"
def upgrade() -> None:
op.add_column("ml_settings", sa.Column(
"detector_person_enabled", sa.Boolean(), nullable=False,
server_default=sa.true()))
op.add_column("ml_settings", sa.Column(
"detector_person_weights", sa.String(512), nullable=False,
server_default="yolo11n.pt"))
op.add_column("ml_settings", sa.Column(
"detector_person_conf", sa.Float(), nullable=False,
server_default=sa.text("0.35")))
op.add_column("ml_settings", sa.Column(
"detector_anatomy_enabled", sa.Boolean(), nullable=False,
server_default=sa.true()))
op.add_column("ml_settings", sa.Column(
"detector_anatomy_weights", sa.String(512), nullable=False,
server_default=_ANATOMY_DEFAULT))
op.add_column("ml_settings", sa.Column(
"detector_anatomy_conf", sa.Float(), nullable=False,
server_default=sa.text("0.30")))
op.add_column("ml_settings", sa.Column(
"detector_panel_enabled", sa.Boolean(), nullable=False,
server_default=sa.true()))
op.add_column("ml_settings", sa.Column(
"detector_panel_weights", sa.String(512), nullable=False,
server_default=_PANEL_DEFAULT))
op.add_column("ml_settings", sa.Column(
"detector_panel_conf", sa.Float(), nullable=False,
server_default=sa.text("0.30")))
op.add_column("ml_settings", sa.Column(
"detector_max_figures", sa.Integer(), nullable=False,
server_default=sa.text("8")))
op.add_column("ml_settings", sa.Column(
"detector_max_components", sa.Integer(), nullable=False,
server_default=sa.text("8")))
op.add_column("ml_settings", sa.Column(
"detector_max_panels", sa.Integer(), nullable=False,
server_default=sa.text("8")))
op.add_column("ml_settings", sa.Column(
"detector_max_regions", sa.Integer(), nullable=False,
server_default=sa.text("128")))
op.add_column("ml_settings", sa.Column(
"detector_dedupe_iou", sa.Float(), nullable=False,
server_default=sa.text("0.85")))
def downgrade() -> None:
for col in (
"detector_person_enabled", "detector_person_weights", "detector_person_conf",
"detector_anatomy_enabled", "detector_anatomy_weights", "detector_anatomy_conf",
"detector_panel_enabled", "detector_panel_weights", "detector_panel_conf",
"detector_max_figures", "detector_max_components", "detector_max_panels",
"detector_max_regions", "detector_dedupe_iou",
):
op.drop_column("ml_settings", col)
@@ -1,77 +0,0 @@
"""character prototype store (#1317) — precomputed, incremental CCIP references
New tables character_prototype + ccip_prototype_state, plus MLSettings columns
ccip_ref_signature (cheap global change gate) + ccip_prototype_cap (per-character
reference cap). The reference set the CCIP matcher uses becomes a precomputed
artifact refreshed incrementally off the request path. See milestone 138 /
backend.app.services.ml.character_prototypes.
Revision ID: 0079
Revises: 0078
Create Date: 2026-07-06
"""
from typing import Sequence, Union
import sqlalchemy as sa
from alembic import op
from pgvector.sqlalchemy import Vector
revision: str = "0079"
down_revision: Union[str, None] = "0078"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
# Matches models.image_region.CCIP_DIM (the CCIP figure-embedding width).
_CCIP_DIM = 768
def upgrade() -> None:
op.create_table(
"character_prototype",
sa.Column("id", sa.Integer(), primary_key=True),
sa.Column(
"tag_id", sa.Integer(),
sa.ForeignKey("tag.id", ondelete="CASCADE"), nullable=False,
),
sa.Column("ccip_embedding", Vector(_CCIP_DIM), nullable=False),
sa.Column(
"region_id", sa.Integer(),
sa.ForeignKey("image_region.id", ondelete="SET NULL"), nullable=True,
),
)
op.create_index(
"ix_character_prototype_tag_id", "character_prototype", ["tag_id"]
)
op.create_table(
"ccip_prototype_state",
sa.Column(
"tag_id", sa.Integer(),
sa.ForeignKey("tag.id", ondelete="CASCADE"), primary_key=True,
),
sa.Column("fingerprint", sa.String(64), nullable=False),
sa.Column(
"updated_at", sa.DateTime(timezone=True), nullable=False,
server_default=sa.func.now(),
),
)
op.add_column(
"ml_settings",
sa.Column("ccip_ref_signature", sa.String(128), nullable=True),
)
op.add_column(
"ml_settings",
sa.Column(
"ccip_prototype_cap", sa.Integer(), nullable=False,
server_default=sa.text("64"),
),
)
def downgrade() -> None:
op.drop_column("ml_settings", "ccip_prototype_cap")
op.drop_column("ml_settings", "ccip_ref_signature")
op.drop_table("ccip_prototype_state")
op.drop_index(
"ix_character_prototype_tag_id", table_name="character_prototype"
)
op.drop_table("character_prototype")
@@ -1,31 +0,0 @@
"""tag_head.train_fingerprint (#1317 phase 2) — incremental head retraining
A per-head training-data fingerprint (positive + rejection count/latest-timestamp)
so a manual Retrain refits only the tags whose data changed; the nightly run
ignores it (full reconcile). Nullable — a NULL fingerprint (existing heads) forces
a refit on the first incremental run, then it's stamped.
Revision ID: 0080
Revises: 0079
Create Date: 2026-07-06
"""
from typing import Sequence, Union
import sqlalchemy as sa
from alembic import op
revision: str = "0080"
down_revision: Union[str, None] = "0079"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
op.add_column(
"tag_head",
sa.Column("train_fingerprint", sa.String(128), nullable=True),
)
def downgrade() -> None:
op.drop_column("tag_head", "train_fingerprint")
@@ -1,43 +0,0 @@
"""stricter auto-apply defaults (milestone 139) — cut auto-apply misfires
head_auto_apply_min_positives 30→50 and ccip_auto_apply_threshold 0.92→0.95
(operator-asked 2026-07-06). The head graduation precision bar stays 0.97 — the
operator confirmed the general-tag confidence was already well tuned; only the
support floor + the CCIP match confidence are raised. The model defaults change
for fresh installs; here we bump the existing singleton row IFF it is still at
the previous default, so a deliberate operator change is NOT clobbered.
Revision ID: 0081
Revises: 0080
Create Date: 2026-07-06
"""
from typing import Sequence, Union
from alembic import op
revision: str = "0081"
down_revision: Union[str, None] = "0080"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
op.execute(
"UPDATE ml_settings SET head_auto_apply_min_positives = 50 "
"WHERE head_auto_apply_min_positives = 30"
)
op.execute(
"UPDATE ml_settings SET ccip_auto_apply_threshold = 0.95 "
"WHERE ccip_auto_apply_threshold = 0.92"
)
def downgrade() -> None:
op.execute(
"UPDATE ml_settings SET head_auto_apply_min_positives = 30 "
"WHERE head_auto_apply_min_positives = 50"
)
op.execute(
"UPDATE ml_settings SET ccip_auto_apply_threshold = 0.92 "
"WHERE ccip_auto_apply_threshold = 0.95"
)
@@ -1,85 +0,0 @@
"""presentation-chrome auto-hide (#141) — settings knobs + review table
MLSettings gains presentation_auto_apply_enabled / _threshold and
presentation_conflict_threshold: banner + editor-screenshot auto-hide on the
sweep with a FLAT threshold (decoupled from content-head graduation), and a
conflict threshold that flags an auto-hide that "also looks like content".
New table presentation_review records an auto-hidden chrome image that also
scored high on a content head, surfaced in the Hidden view for a keep-hidden /
un-hide decision. Resolved rows are pruned by retention.
Revision ID: 0082
Revises: 0081
Create Date: 2026-07-07
"""
from typing import Sequence, Union
import sqlalchemy as sa
from alembic import op
revision: str = "0082"
down_revision: Union[str, None] = "0081"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
op.add_column(
"ml_settings",
sa.Column(
"presentation_auto_apply_enabled", sa.Boolean(), nullable=False,
server_default=sa.text("true"),
),
)
op.add_column(
"ml_settings",
sa.Column(
"presentation_auto_apply_threshold", sa.Float(), nullable=False,
server_default=sa.text("0.90"),
),
)
op.add_column(
"ml_settings",
sa.Column(
"presentation_conflict_threshold", sa.Float(), nullable=False,
server_default=sa.text("0.50"),
),
)
op.create_table(
"presentation_review",
sa.Column(
"image_record_id", sa.Integer(),
sa.ForeignKey("image_record.id", ondelete="CASCADE"),
primary_key=True,
),
sa.Column(
"tag_id", sa.Integer(),
sa.ForeignKey("tag.id", ondelete="CASCADE"), primary_key=True,
),
sa.Column(
"conflict_tag_id", sa.Integer(),
sa.ForeignKey("tag.id", ondelete="SET NULL"), nullable=True,
),
sa.Column("conflict_score", sa.Float(), nullable=False),
sa.Column(
"created_at", sa.DateTime(timezone=True), nullable=False,
server_default=sa.func.now(),
),
sa.Column("resolved_at", sa.DateTime(timezone=True), nullable=True),
)
# The review list queries the unresolved flags (resolved_at IS NULL).
op.create_index(
"ix_presentation_review_resolved_at", "presentation_review",
["resolved_at"],
)
def downgrade() -> None:
op.drop_index(
"ix_presentation_review_resolved_at", table_name="presentation_review"
)
op.drop_table("presentation_review")
op.drop_column("ml_settings", "presentation_conflict_threshold")
op.drop_column("ml_settings", "presentation_auto_apply_threshold")
op.drop_column("ml_settings", "presentation_auto_apply_enabled")
-73
View File
@@ -1,73 +0,0 @@
"""post-text translation via Interpreter (milestone 143) — Post columns + settings
Post gains the translated title/description + the detected source language,
Interpreter engine_version (cache key), and translated_at — filled by the
translate sweep. ImportSettings gains translation_enabled (OFF by default),
interpreter_base_url (EMPTY — the operator sets their own, behind a reverse
proxy), and translation_target_lang (en).
Revision ID: 0083
Revises: 0082
Create Date: 2026-07-07
"""
from typing import Sequence, Union
import sqlalchemy as sa
from alembic import op
revision: str = "0083"
down_revision: Union[str, None] = "0082"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
op.add_column(
"post", sa.Column("post_title_translated", sa.Text(), nullable=True)
)
op.add_column(
"post", sa.Column("description_translated", sa.Text(), nullable=True)
)
op.add_column(
"post",
sa.Column("translated_source_lang", sa.String(8), nullable=True),
)
op.add_column(
"post",
sa.Column("translation_engine_version", sa.String(128), nullable=True),
)
op.add_column(
"post",
sa.Column("translated_at", sa.DateTime(timezone=True), nullable=True),
)
op.add_column(
"import_settings",
sa.Column(
"translation_enabled", sa.Boolean(), nullable=False,
server_default=sa.text("false"),
),
)
op.add_column(
"import_settings",
sa.Column(
"interpreter_base_url", sa.Text(), nullable=False, server_default="",
),
)
op.add_column(
"import_settings",
sa.Column(
"translation_target_lang", sa.Text(), nullable=False,
server_default="en",
),
)
def downgrade() -> None:
op.drop_column("import_settings", "translation_target_lang")
op.drop_column("import_settings", "interpreter_base_url")
op.drop_column("import_settings", "translation_enabled")
op.drop_column("post", "translated_at")
op.drop_column("post", "translation_engine_version")
op.drop_column("post", "translated_source_lang")
op.drop_column("post", "description_translated")
op.drop_column("post", "post_title_translated")
@@ -1,51 +0,0 @@
"""translation strictness setting + per-post translation override (milestone 155)
ImportSettings gains ``translation_min_confidence`` (the latin-script acceptance
floor, now operator-tunable in the UI; default 0.9 — stricter than the old
hardcoded 0.8, since Interpreter confidently mis-detects short ASCII English at
~0.86). Post gains ``translation_override`` — a sticky per-post choice of
auto / force / original so the operator can force a skipped translation on, or
knock a wrongly-translated one back to the original, and have it survive a
Re-translate-all.
Revision ID: 0084
Revises: 0083
Create Date: 2026-07-10
"""
from typing import Sequence, Union
import sqlalchemy as sa
from alembic import op
revision: str = "0084"
down_revision: Union[str, None] = "0083"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
op.add_column(
"import_settings",
sa.Column(
"translation_min_confidence", sa.Float(), nullable=False,
server_default=sa.text("0.9"),
),
)
op.add_column(
"post",
sa.Column(
"translation_override", sa.String(16), nullable=False,
server_default="auto",
),
)
op.create_check_constraint(
"ck_post_translation_override",
"post",
"translation_override IN ('auto', 'force', 'original')",
)
def downgrade() -> None:
op.drop_constraint("ck_post_translation_override", "post", type_="check")
op.drop_column("post", "translation_override")
op.drop_column("import_settings", "translation_min_confidence")
@@ -1,35 +0,0 @@
"""title-based WIP auto-tagging (task #1458) — ImportSettings toggle
ImportSettings gains wip_title_tagging_enabled (ON by default): when a freshly
imported post's title explicitly declares work-in-progress ("WIP" / "work in
progress"), the importer applies the `wip` system tag to its images. No new
table — the tag itself is the seeded `wip` system tag (migration 0075) and the
application reuses image_tag with source='wip_title'.
Revision ID: 0085
Revises: 0084
Create Date: 2026-07-12
"""
from typing import Sequence, Union
import sqlalchemy as sa
from alembic import op
revision: str = "0085"
down_revision: Union[str, None] = "0084"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
op.add_column(
"import_settings",
sa.Column(
"wip_title_tagging_enabled", sa.Boolean(), nullable=False,
server_default=sa.text("true"),
),
)
def downgrade() -> None:
op.drop_column("import_settings", "wip_title_tagging_enabled")
@@ -1,61 +0,0 @@
"""process auto-apply settings + review mode (#1464) — system-tag refactor
The system-tag behavior refactor gives `wip` / `editor screenshot` (the PROCESS
group) their own provisional auto-apply, parallel to the presentation (chrome)
sweep. MLSettings gains three knobs: enabled (OFF by default — a new whole-library
auto-tagger is opt-in), the flat apply threshold, and the ring-loud conflict
threshold. presentation_review gains a `mode` column so one review surface serves
both chrome and process flags (existing rows backfill 'chrome'). server_defaults
so the existing rows fill cleanly.
Revision ID: 0086
Revises: 0085
Create Date: 2026-07-13
"""
from typing import Sequence, Union
import sqlalchemy as sa
from alembic import op
revision: str = "0086"
down_revision: Union[str, None] = "0085"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
op.add_column(
"ml_settings",
sa.Column(
"process_auto_apply_enabled", sa.Boolean(), nullable=False,
server_default=sa.text("false"),
),
)
op.add_column(
"ml_settings",
sa.Column(
"process_auto_apply_threshold", sa.Float(), nullable=False,
server_default="0.90",
),
)
op.add_column(
"ml_settings",
sa.Column(
"process_conflict_threshold", sa.Float(), nullable=False,
server_default="0.50",
),
)
op.add_column(
"presentation_review",
sa.Column(
"mode", sa.String(16), nullable=False,
server_default="chrome",
),
)
def downgrade() -> None:
op.drop_column("presentation_review", "mode")
op.drop_column("ml_settings", "process_conflict_threshold")
op.drop_column("ml_settings", "process_auto_apply_threshold")
op.drop_column("ml_settings", "process_auto_apply_enabled")
+872
View File
@@ -0,0 +1,872 @@
"""Collapsed baseline — the whole schema in one revision.
Replaces revisions 0001..0087, which narrated the build-out of this project
and were deleted in milestone 328 step 1. A new install creates the schema in
one step instead of replaying that history.
WHY THE REVISION ID IS "0087" AND NOT "0001"
--------------------------------------------
It is deliberately the id of the LAST revision this baseline collapses, so an
existing database needs no intervention at all:
* a fresh install finds current=none, head=0087, runs this file once, and
ends stamped at 0087.
* an existing install is ALREADY at 0087, so `alembic upgrade head` finds
current == head and does nothing.
The alternative — numbering this 0001 and stamping every existing database —
means running `alembic stamp` against live data, and stamp VALIDATES NOTHING.
It writes a version string whether or not the schema actually matches, so a
wrong baseline would be discovered later, by the next real migration, with no
clean way back. Keeping the id removes that operation instead of making it
safe. Future revisions continue at 0088.
The one case this makes worse, and it fails LOUDLY rather than silently: a
database still sitting between 0001 and 0086 (i.e. never upgraded to head)
cannot be located in this chain and errors out. Upgrade to 0087 on a
pre-squash build first, then take this one.
WHAT IS HAND-WRITTEN HERE
-------------------------
Most of this file is `alembic revision --autogenerate` output, but four
things are NOT in SQLAlchemy metadata and the generator cannot produce them.
Each fails differently, and none of them fail at generation time:
1. CREATE EXTENSION vector (was 0001) — without it the VECTOR
columns below cannot be created at all.
2. CREATE EXTENSION tsm_system_rows (was 0004) — used by the random-sample
query path; its absence surfaces only when that query runs.
3. The HNSW index on image_record.siglip_embedding (was 0036). Raw SQL
because alembic's create_index cannot express `USING hnsw (...
vector_cosine_ops)`. Its absence is the quietest failure of the four:
everything works, similarity search just stops using an index.
4. `import pgvector.sqlalchemy.vector`. Autogenerate EMITS references to
pgvector.sqlalchemy.vector.VECTOR but does not add the import, so the
generated file dies with NameError on first run.
The acceptance test for this file is not that it reads correctly — it is
`.forgejo/workflows/baseline.yml`, which builds a database from the old
0001..0087 chain (read out of git) and one from this file, and diffs
pg_dump --schema-only output. That is what proves nothing was missed.
Revision ID: 0087
Revises:
Create Date: 2026-08-30
"""
from typing import Sequence, Union
from alembic import op
import sqlalchemy as sa
from sqlalchemy.dialects import postgresql
# Autogenerate references pgvector.sqlalchemy.vector.VECTOR without importing
# it. Item 4 above.
import pgvector.sqlalchemy.vector
revision: str = "0087"
down_revision: Union[str, None] = None
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
# Extensions FIRST: the VECTOR columns below cannot be created without
# `vector`, so ordering here is load-bearing, not tidiness.
op.execute("CREATE EXTENSION IF NOT EXISTS vector")
op.execute("CREATE EXTENSION IF NOT EXISTS tsm_system_rows")
op.create_table('app_setting',
sa.Column('key', sa.String(length=64), nullable=False),
sa.Column('value', sa.Text(), nullable=False),
sa.Column('updated_at', sa.DateTime(timezone=True), server_default=sa.text('now()'), nullable=False),
sa.PrimaryKeyConstraint('key', name=op.f('pk_app_setting'))
)
op.create_table('artist',
sa.Column('id', sa.Integer(), nullable=False),
sa.Column('name', sa.String(length=255), nullable=False),
sa.Column('slug', sa.String(length=255), nullable=False),
sa.Column('notes', sa.Text(), nullable=True),
sa.Column('is_subscription', sa.Boolean(), nullable=False),
sa.Column('auto_check', sa.Boolean(), nullable=False),
sa.Column('check_interval_seconds', sa.Integer(), nullable=True),
sa.Column('created_at', sa.DateTime(timezone=True), server_default=sa.text('now()'), nullable=False),
sa.PrimaryKeyConstraint('id', name=op.f('pk_artist')),
sa.UniqueConstraint('slug', name=op.f('uq_artist_slug'))
)
op.create_table('backup_run',
sa.Column('id', sa.Integer(), nullable=False),
sa.Column('kind', sa.String(length=16), nullable=False),
sa.Column('status', sa.String(length=16), nullable=False),
sa.Column('tag', sa.String(length=64), nullable=True),
sa.Column('triggered_by', sa.String(length=32), nullable=False),
sa.Column('started_at', sa.DateTime(timezone=True), nullable=False),
sa.Column('finished_at', sa.DateTime(timezone=True), nullable=True),
sa.Column('sql_path', sa.Text(), nullable=True),
sa.Column('tar_path', sa.Text(), nullable=True),
sa.Column('size_bytes', sa.BigInteger(), nullable=True),
sa.Column('error', sa.Text(), nullable=True),
sa.Column('manifest', sa.JSON(), server_default='{}', nullable=False),
sa.Column('restored_from_id', sa.Integer(), nullable=True),
sa.ForeignKeyConstraint(['restored_from_id'], ['backup_run.id'], name=op.f('fk_backup_run_restored_from_id_backup_run'), ondelete='SET NULL'),
sa.PrimaryKeyConstraint('id', name=op.f('pk_backup_run'))
)
op.create_index(op.f('ix_backup_run_finished_at'), 'backup_run', ['finished_at'], unique=False)
op.create_index(op.f('ix_backup_run_kind'), 'backup_run', ['kind'], unique=False)
op.create_index(op.f('ix_backup_run_started_at'), 'backup_run', ['started_at'], unique=False)
op.create_index(op.f('ix_backup_run_status'), 'backup_run', ['status'], unique=False)
op.create_index(op.f('ix_backup_run_tag'), 'backup_run', ['tag'], unique=False)
op.create_table('credential',
sa.Column('id', sa.Integer(), nullable=False),
sa.Column('platform', sa.String(length=64), nullable=False),
sa.Column('credential_type', sa.String(length=32), nullable=False),
sa.Column('encrypted_blob', sa.LargeBinary(), nullable=False),
sa.Column('captured_at', sa.DateTime(timezone=True), server_default=sa.text('now()'), nullable=False),
sa.Column('expires_at', sa.DateTime(timezone=True), nullable=True),
sa.Column('last_verified', sa.DateTime(timezone=True), nullable=True),
sa.PrimaryKeyConstraint('id', name=op.f('pk_credential')),
sa.UniqueConstraint('platform', name=op.f('uq_credential_platform'))
)
op.create_table('head_auto_apply_run',
sa.Column('id', sa.Integer(), nullable=False),
sa.Column('dry_run', sa.Boolean(), nullable=False),
sa.Column('params', postgresql.JSONB(astext_type=sa.Text()), nullable=False),
sa.Column('status', sa.String(length=16), nullable=False),
sa.Column('started_at', sa.DateTime(timezone=True), server_default=sa.text('now()'), nullable=False),
sa.Column('finished_at', sa.DateTime(timezone=True), nullable=True),
sa.Column('n_applied', sa.Integer(), nullable=True),
sa.Column('report', postgresql.JSONB(astext_type=sa.Text()), nullable=True),
sa.Column('error', sa.Text(), nullable=True),
sa.Column('last_progress_at', sa.DateTime(timezone=True), nullable=True),
sa.PrimaryKeyConstraint('id', name=op.f('pk_head_auto_apply_run'))
)
op.create_index(op.f('ix_head_auto_apply_run_status'), 'head_auto_apply_run', ['status'], unique=False)
op.create_table('head_training_run',
sa.Column('id', sa.Integer(), nullable=False),
sa.Column('params', postgresql.JSONB(astext_type=sa.Text()), nullable=False),
sa.Column('status', sa.String(length=16), nullable=False),
sa.Column('started_at', sa.DateTime(timezone=True), server_default=sa.text('now()'), nullable=False),
sa.Column('finished_at', sa.DateTime(timezone=True), nullable=True),
sa.Column('n_trained', sa.Integer(), nullable=True),
sa.Column('n_skipped', sa.Integer(), nullable=True),
sa.Column('error', sa.Text(), nullable=True),
sa.Column('last_progress_at', sa.DateTime(timezone=True), nullable=True),
sa.PrimaryKeyConstraint('id', name=op.f('pk_head_training_run'))
)
op.create_index(op.f('ix_head_training_run_status'), 'head_training_run', ['status'], unique=False)
op.create_table('import_batch',
sa.Column('id', sa.Integer(), nullable=False),
sa.Column('triggered_by', sa.String(length=32), nullable=False),
sa.Column('source_path', sa.Text(), nullable=False),
sa.Column('scan_mode', sa.String(length=16), nullable=False),
sa.Column('started_at', sa.DateTime(timezone=True), server_default=sa.text('now()'), nullable=False),
sa.Column('finished_at', sa.DateTime(timezone=True), nullable=True),
sa.Column('total_files', sa.Integer(), nullable=False),
sa.Column('imported', sa.Integer(), nullable=False),
sa.Column('skipped', sa.Integer(), nullable=False),
sa.Column('failed', sa.Integer(), nullable=False),
sa.Column('attachments', sa.Integer(), nullable=False),
sa.Column('refreshed', sa.Integer(), nullable=False),
sa.Column('status', sa.String(length=16), nullable=False),
sa.PrimaryKeyConstraint('id', name=op.f('pk_import_batch'))
)
op.create_index(op.f('ix_import_batch_status'), 'import_batch', ['status'], unique=False)
op.create_table('import_settings',
sa.Column('id', sa.Integer(), nullable=False),
sa.Column('import_scan_path', sa.Text(), nullable=False),
sa.Column('min_width', sa.Integer(), nullable=False),
sa.Column('min_height', sa.Integer(), nullable=False),
sa.Column('skip_transparent', sa.Boolean(), nullable=False),
sa.Column('transparency_threshold', sa.Float(), nullable=False),
sa.Column('skip_single_color', sa.Boolean(), nullable=False),
sa.Column('single_color_threshold', sa.Float(), nullable=False),
sa.Column('single_color_tolerance', sa.Integer(), nullable=False),
sa.Column('phash_threshold', sa.Integer(), nullable=False),
sa.Column('download_rate_limit_seconds', sa.Float(), nullable=False),
sa.Column('download_validate_files', sa.Boolean(), nullable=False),
sa.Column('download_schedule_default_seconds', sa.Integer(), nullable=False),
sa.Column('download_event_retention_days', sa.Integer(), nullable=False),
sa.Column('download_failure_warning_threshold', sa.Integer(), nullable=False),
sa.Column('backup_db_nightly_enabled', sa.Boolean(), nullable=False),
sa.Column('backup_db_nightly_hour_utc', sa.Integer(), nullable=False),
sa.Column('backup_db_keep_last_n', sa.Integer(), nullable=False),
sa.Column('backup_images_keep_last_n', sa.Integer(), nullable=False),
sa.Column('series_suggest_enabled', sa.Boolean(), nullable=False),
sa.Column('series_suggest_threshold', sa.Float(), nullable=False),
sa.Column('extdl_mega_enabled', sa.Boolean(), server_default='true', nullable=False),
sa.Column('extdl_gdrive_enabled', sa.Boolean(), server_default='true', nullable=False),
sa.Column('extdl_mediafire_enabled', sa.Boolean(), server_default='true', nullable=False),
sa.Column('extdl_dropbox_enabled', sa.Boolean(), server_default='true', nullable=False),
sa.Column('extdl_pixeldrain_enabled', sa.Boolean(), server_default='true', nullable=False),
sa.Column('translation_enabled', sa.Boolean(), server_default='false', nullable=False),
sa.Column('interpreter_base_url', sa.Text(), server_default='', nullable=False),
sa.Column('translation_target_lang', sa.Text(), server_default='en', nullable=False),
sa.Column('translation_min_confidence', sa.Float(), server_default='0.9', nullable=False),
sa.Column('wip_title_tagging_enabled', sa.Boolean(), server_default='true', nullable=False),
sa.Column('wip_soft_title_tagging_enabled', sa.Boolean(), server_default='false', nullable=False),
sa.CheckConstraint('id = 1', name=op.f('ck_import_settings_singleton')),
sa.PrimaryKeyConstraint('id', name=op.f('pk_import_settings'))
)
op.create_table('library_audit_run',
sa.Column('id', sa.Integer(), nullable=False),
sa.Column('rule', sa.String(length=32), nullable=False),
sa.Column('params', postgresql.JSONB(astext_type=sa.Text()), nullable=False),
sa.Column('status', sa.String(length=16), nullable=False),
sa.Column('started_at', sa.DateTime(timezone=True), server_default=sa.text('now()'), nullable=False),
sa.Column('finished_at', sa.DateTime(timezone=True), nullable=True),
sa.Column('scanned_count', sa.Integer(), nullable=False),
sa.Column('matched_count', sa.Integer(), nullable=False),
sa.Column('matched_ids', postgresql.JSONB(astext_type=sa.Text()), nullable=False),
sa.Column('error', sa.Text(), nullable=True),
sa.Column('resume_after_id', sa.Integer(), nullable=False),
sa.Column('last_progress_at', sa.DateTime(timezone=True), nullable=True),
sa.PrimaryKeyConstraint('id', name=op.f('pk_library_audit_run'))
)
op.create_index(op.f('ix_library_audit_run_rule'), 'library_audit_run', ['rule'], unique=False)
op.create_index(op.f('ix_library_audit_run_status'), 'library_audit_run', ['status'], unique=False)
op.create_table('ml_settings',
sa.Column('id', sa.Integer(), nullable=False),
sa.Column('cpu_embed_enabled', sa.Boolean(), nullable=False),
sa.Column('video_frame_interval_seconds', sa.Float(), nullable=False),
sa.Column('video_max_frames', sa.Integer(), nullable=False),
sa.Column('head_min_positives', sa.Integer(), nullable=False),
sa.Column('head_auto_apply_precision', sa.Float(), nullable=False),
sa.Column('head_auto_apply_enabled', sa.Boolean(), nullable=False),
sa.Column('head_auto_apply_min_positives', sa.Integer(), nullable=False),
sa.Column('ccip_match_threshold', sa.Float(), nullable=False),
sa.Column('ccip_auto_apply_enabled', sa.Boolean(), nullable=False),
sa.Column('ccip_auto_apply_threshold', sa.Float(), nullable=False),
sa.Column('presentation_auto_apply_enabled', sa.Boolean(), nullable=False),
sa.Column('presentation_auto_apply_threshold', sa.Float(), nullable=False),
sa.Column('presentation_conflict_threshold', sa.Float(), nullable=False),
sa.Column('process_auto_apply_enabled', sa.Boolean(), nullable=False),
sa.Column('process_auto_apply_threshold', sa.Float(), nullable=False),
sa.Column('process_conflict_threshold', sa.Float(), nullable=False),
sa.Column('embedder_model_version', sa.String(length=128), nullable=False),
sa.Column('embedder_model_name', sa.String(length=128), nullable=False),
sa.Column('detector_person_enabled', sa.Boolean(), nullable=False),
sa.Column('detector_person_weights', sa.String(length=512), nullable=False),
sa.Column('detector_person_conf', sa.Float(), nullable=False),
sa.Column('detector_anatomy_enabled', sa.Boolean(), nullable=False),
sa.Column('detector_anatomy_weights', sa.String(length=512), nullable=False),
sa.Column('detector_anatomy_conf', sa.Float(), nullable=False),
sa.Column('detector_panel_enabled', sa.Boolean(), nullable=False),
sa.Column('detector_panel_weights', sa.String(length=512), nullable=False),
sa.Column('detector_panel_conf', sa.Float(), nullable=False),
sa.Column('detector_max_figures', sa.Integer(), nullable=False),
sa.Column('detector_max_components', sa.Integer(), nullable=False),
sa.Column('detector_max_panels', sa.Integer(), nullable=False),
sa.Column('detector_max_regions', sa.Integer(), nullable=False),
sa.Column('detector_dedupe_iou', sa.Float(), nullable=False),
sa.Column('ccip_ref_signature', sa.String(length=128), nullable=True),
sa.Column('ccip_prototype_cap', sa.Integer(), nullable=False),
sa.Column('updated_at', sa.DateTime(timezone=True), server_default=sa.text('now()'), nullable=False),
sa.CheckConstraint('id = 1', name=op.f('ck_ml_settings_singleton')),
sa.PrimaryKeyConstraint('id', name=op.f('pk_ml_settings'))
)
op.create_table('tag',
sa.Column('id', sa.Integer(), nullable=False),
sa.Column('name', sa.String(length=255), nullable=False),
sa.Column('kind', sa.Enum('artist', 'character', 'fandom', 'general', 'series', 'archive', 'post', name='tag_kind'), nullable=False),
sa.Column('fandom_id', sa.Integer(), nullable=True),
sa.Column('is_system', sa.Boolean(), server_default=sa.text('false'), nullable=False),
sa.Column('created_at', sa.DateTime(timezone=True), server_default=sa.text('now()'), nullable=False),
sa.CheckConstraint("(fandom_id IS NULL) OR (kind = 'character')", name=op.f('ck_tag_ck_tag_fandom_requires_character')),
sa.ForeignKeyConstraint(['fandom_id'], ['tag.id'], name=op.f('fk_tag_fandom_id_tag'), ondelete='SET NULL'),
sa.PrimaryKeyConstraint('id', name=op.f('pk_tag'))
)
op.create_index(op.f('ix_tag_fandom_id'), 'tag', ['fandom_id'], unique=False)
op.create_table('task_run',
sa.Column('id', sa.Integer(), nullable=False),
sa.Column('celery_task_id', sa.String(length=64), nullable=False),
sa.Column('queue', sa.String(length=32), nullable=False),
sa.Column('task_name', sa.String(length=128), nullable=False),
sa.Column('target_id', sa.Integer(), nullable=True),
sa.Column('started_at', sa.DateTime(timezone=True), nullable=False),
sa.Column('finished_at', sa.DateTime(timezone=True), nullable=True),
sa.Column('duration_ms', sa.Integer(), nullable=True),
sa.Column('status', sa.String(length=16), nullable=False),
sa.Column('error_type', sa.String(length=128), nullable=True),
sa.Column('error_message', sa.Text(), nullable=True),
sa.Column('retry_count', sa.Integer(), nullable=True),
sa.Column('worker_hostname', sa.String(length=128), nullable=True),
sa.Column('args_summary', sa.String(length=255), nullable=True),
sa.PrimaryKeyConstraint('id', name=op.f('pk_task_run'))
)
op.create_index(op.f('ix_task_run_celery_task_id'), 'task_run', ['celery_task_id'], unique=False)
op.create_index(op.f('ix_task_run_finished_at'), 'task_run', ['finished_at'], unique=False)
op.create_index(op.f('ix_task_run_queue'), 'task_run', ['queue'], unique=False)
op.create_index(op.f('ix_task_run_started_at'), 'task_run', ['started_at'], unique=False)
op.create_index(op.f('ix_task_run_status'), 'task_run', ['status'], unique=False)
op.create_index(op.f('ix_task_run_task_name'), 'task_run', ['task_name'], unique=False)
op.create_table('artist_visit',
sa.Column('artist_id', sa.Integer(), nullable=False),
sa.Column('last_viewed_at', sa.DateTime(timezone=True), server_default=sa.text('now()'), nullable=False),
sa.ForeignKeyConstraint(['artist_id'], ['artist.id'], name=op.f('fk_artist_visit_artist_id_artist'), ondelete='CASCADE'),
sa.PrimaryKeyConstraint('artist_id', name=op.f('pk_artist_visit'))
)
op.create_table('ccip_prototype_state',
sa.Column('tag_id', sa.Integer(), nullable=False),
sa.Column('fingerprint', sa.String(length=64), nullable=False),
sa.Column('updated_at', sa.DateTime(timezone=True), server_default=sa.text('now()'), nullable=False),
sa.ForeignKeyConstraint(['tag_id'], ['tag.id'], name=op.f('fk_ccip_prototype_state_tag_id_tag'), ondelete='CASCADE'),
sa.PrimaryKeyConstraint('tag_id', name=op.f('pk_ccip_prototype_state'))
)
op.create_table('head_metric',
sa.Column('tag_id', sa.Integer(), nullable=False),
sa.Column('n_misfires', sa.Integer(), nullable=False),
sa.Column('n_underfires', sa.Integer(), nullable=False),
sa.Column('updated_at', sa.DateTime(timezone=True), server_default=sa.text('now()'), nullable=False),
sa.ForeignKeyConstraint(['tag_id'], ['tag.id'], name=op.f('fk_head_metric_tag_id_tag'), ondelete='CASCADE'),
sa.PrimaryKeyConstraint('tag_id', name=op.f('pk_head_metric'))
)
op.create_table('head_metrics_snapshot',
sa.Column('id', sa.Integer(), nullable=False),
sa.Column('tag_id', sa.Integer(), nullable=False),
sa.Column('name', sa.String(length=255), nullable=False),
sa.Column('snapshot_at', sa.DateTime(timezone=True), server_default=sa.text('now()'), nullable=False),
sa.Column('n_auto_applied', sa.Integer(), nullable=False),
sa.Column('n_misfires', sa.Integer(), nullable=False),
sa.Column('n_underfires', sa.Integer(), nullable=False),
sa.Column('ap', sa.Float(), nullable=True),
sa.Column('precision_cv', sa.Float(), nullable=True),
sa.Column('recall', sa.Float(), nullable=True),
sa.Column('n_pos', sa.Integer(), nullable=True),
sa.ForeignKeyConstraint(['tag_id'], ['tag.id'], name=op.f('fk_head_metrics_snapshot_tag_id_tag'), ondelete='CASCADE'),
sa.PrimaryKeyConstraint('id', name=op.f('pk_head_metrics_snapshot'))
)
op.create_index(op.f('ix_head_metrics_snapshot_snapshot_at'), 'head_metrics_snapshot', ['snapshot_at'], unique=False)
op.create_index(op.f('ix_head_metrics_snapshot_tag_id'), 'head_metrics_snapshot', ['tag_id'], unique=False)
op.create_table('source',
sa.Column('id', sa.Integer(), nullable=False),
sa.Column('artist_id', sa.Integer(), nullable=False),
sa.Column('platform', sa.String(length=64), nullable=False),
sa.Column('url', sa.Text(), nullable=False),
sa.Column('enabled', sa.Boolean(), nullable=False),
sa.Column('config_overrides', sa.JSON(), nullable=True),
sa.Column('last_checked_at', sa.DateTime(timezone=True), nullable=True),
sa.Column('last_error', sa.Text(), nullable=True),
sa.Column('error_type', sa.String(length=32), nullable=True),
sa.Column('check_interval_override', sa.Integer(), nullable=True),
sa.Column('consecutive_failures', sa.Integer(), nullable=False),
sa.Column('backfill_runs_remaining', sa.Integer(), server_default='0', nullable=False),
sa.ForeignKeyConstraint(['artist_id'], ['artist.id'], name=op.f('fk_source_artist_id_artist'), ondelete='CASCADE'),
sa.PrimaryKeyConstraint('id', name=op.f('pk_source'))
)
op.create_index(op.f('ix_source_artist_id'), 'source', ['artist_id'], unique=False)
op.create_index(op.f('ix_source_error_type'), 'source', ['error_type'], unique=False)
op.create_table('tag_alias',
sa.Column('alias_string', sa.String(length=255), nullable=False),
sa.Column('alias_category', sa.String(length=32), nullable=False),
sa.Column('canonical_tag_id', sa.Integer(), nullable=False),
sa.Column('created_at', sa.DateTime(timezone=True), server_default=sa.text('now()'), nullable=False),
sa.ForeignKeyConstraint(['canonical_tag_id'], ['tag.id'], name=op.f('fk_tag_alias_canonical_tag_id_tag'), ondelete='CASCADE'),
sa.PrimaryKeyConstraint('alias_string', 'alias_category', name=op.f('pk_tag_alias'))
)
op.create_index(op.f('ix_tag_alias_canonical_tag_id'), 'tag_alias', ['canonical_tag_id'], unique=False)
op.create_table('tag_head',
sa.Column('tag_id', sa.Integer(), nullable=False),
sa.Column('embedding_version', sa.String(length=128), nullable=False),
sa.Column('weights', pgvector.sqlalchemy.vector.VECTOR(dim=1152), nullable=False),
sa.Column('bias', sa.Float(), nullable=False),
sa.Column('suggest_threshold', sa.Float(), nullable=False),
sa.Column('auto_apply_threshold', sa.Float(), nullable=True),
sa.Column('n_pos', sa.Integer(), nullable=False),
sa.Column('n_neg', sa.Integer(), nullable=False),
sa.Column('ap', sa.Float(), nullable=False),
sa.Column('precision_cv', sa.Float(), nullable=False),
sa.Column('recall', sa.Float(), nullable=False),
sa.Column('trained_at', sa.DateTime(timezone=True), server_default=sa.text('now()'), nullable=False),
sa.Column('train_fingerprint', sa.String(length=128), nullable=True),
sa.Column('metrics', postgresql.JSONB(astext_type=sa.Text()), nullable=True),
sa.ForeignKeyConstraint(['tag_id'], ['tag.id'], name=op.f('fk_tag_head_tag_id_tag'), ondelete='CASCADE'),
sa.PrimaryKeyConstraint('tag_id', name=op.f('pk_tag_head'))
)
op.create_table('patreon_failed_media',
sa.Column('id', sa.Integer(), nullable=False),
sa.Column('source_id', sa.Integer(), nullable=False),
sa.Column('filehash', sa.String(length=128), nullable=False),
sa.Column('attempts', sa.Integer(), nullable=False),
sa.Column('last_error', sa.Text(), nullable=True),
sa.Column('first_failed_at', sa.DateTime(timezone=True), server_default=sa.text('now()'), nullable=False),
sa.Column('last_failed_at', sa.DateTime(timezone=True), server_default=sa.text('now()'), nullable=False),
sa.ForeignKeyConstraint(['source_id'], ['source.id'], name=op.f('fk_patreon_failed_media_source_id_source'), ondelete='CASCADE'),
sa.PrimaryKeyConstraint('id', name=op.f('pk_patreon_failed_media')),
sa.UniqueConstraint('source_id', 'filehash', name='uq_patreon_failed_media_source_id')
)
op.create_index(op.f('ix_patreon_failed_media_source_id'), 'patreon_failed_media', ['source_id'], unique=False)
op.create_table('patreon_seen_media',
sa.Column('id', sa.Integer(), nullable=False),
sa.Column('source_id', sa.Integer(), nullable=False),
sa.Column('filehash', sa.String(length=128), nullable=False),
sa.Column('post_id', sa.String(length=64), nullable=True),
sa.Column('seen_at', sa.DateTime(timezone=True), server_default=sa.text('now()'), nullable=False),
sa.ForeignKeyConstraint(['source_id'], ['source.id'], name=op.f('fk_patreon_seen_media_source_id_source'), ondelete='CASCADE'),
sa.PrimaryKeyConstraint('id', name=op.f('pk_patreon_seen_media')),
sa.UniqueConstraint('source_id', 'filehash', name='uq_patreon_seen_media_source_id')
)
op.create_index(op.f('ix_patreon_seen_media_source_id'), 'patreon_seen_media', ['source_id'], unique=False)
op.create_table('pixiv_failed_media',
sa.Column('id', sa.Integer(), nullable=False),
sa.Column('source_id', sa.Integer(), nullable=False),
sa.Column('filehash', sa.String(length=128), nullable=False),
sa.Column('attempts', sa.Integer(), nullable=False),
sa.Column('last_error', sa.Text(), nullable=True),
sa.Column('first_failed_at', sa.DateTime(timezone=True), server_default=sa.text('now()'), nullable=False),
sa.Column('last_failed_at', sa.DateTime(timezone=True), server_default=sa.text('now()'), nullable=False),
sa.ForeignKeyConstraint(['source_id'], ['source.id'], name=op.f('fk_pixiv_failed_media_source_id_source'), ondelete='CASCADE'),
sa.PrimaryKeyConstraint('id', name=op.f('pk_pixiv_failed_media')),
sa.UniqueConstraint('source_id', 'filehash', name='uq_pixiv_failed_media_source_id')
)
op.create_index(op.f('ix_pixiv_failed_media_source_id'), 'pixiv_failed_media', ['source_id'], unique=False)
op.create_table('pixiv_seen_media',
sa.Column('id', sa.Integer(), nullable=False),
sa.Column('source_id', sa.Integer(), nullable=False),
sa.Column('filehash', sa.String(length=128), nullable=False),
sa.Column('post_id', sa.String(length=64), nullable=True),
sa.Column('seen_at', sa.DateTime(timezone=True), server_default=sa.text('now()'), nullable=False),
sa.ForeignKeyConstraint(['source_id'], ['source.id'], name=op.f('fk_pixiv_seen_media_source_id_source'), ondelete='CASCADE'),
sa.PrimaryKeyConstraint('id', name=op.f('pk_pixiv_seen_media')),
sa.UniqueConstraint('source_id', 'filehash', name='uq_pixiv_seen_media_source_id')
)
op.create_index(op.f('ix_pixiv_seen_media_source_id'), 'pixiv_seen_media', ['source_id'], unique=False)
op.create_table('post',
sa.Column('id', sa.Integer(), nullable=False),
sa.Column('source_id', sa.Integer(), nullable=True),
sa.Column('artist_id', sa.Integer(), nullable=False),
sa.Column('external_post_id', sa.String(length=128), nullable=False),
sa.Column('post_url', sa.Text(), nullable=True),
sa.Column('post_title', sa.Text(), nullable=True),
sa.Column('post_date', sa.DateTime(timezone=True), nullable=True),
sa.Column('raw_metadata', sa.JSON(), nullable=True),
sa.Column('description', sa.Text(), nullable=True),
sa.Column('attachment_count', sa.Integer(), nullable=True),
sa.Column('post_title_translated', sa.Text(), nullable=True),
sa.Column('description_translated', sa.Text(), nullable=True),
sa.Column('translated_source_lang', sa.String(length=8), nullable=True),
sa.Column('translation_engine_version', sa.String(length=128), nullable=True),
sa.Column('translated_at', sa.DateTime(timezone=True), nullable=True),
sa.Column('translation_override', sa.String(length=16), server_default='auto', nullable=False),
sa.Column('downloaded_at', sa.DateTime(timezone=True), server_default=sa.text('now()'), nullable=False),
sa.CheckConstraint("translation_override IN ('auto', 'force', 'original')", name=op.f('ck_post_ck_post_translation_override')),
sa.ForeignKeyConstraint(['artist_id'], ['artist.id'], name=op.f('fk_post_artist_id_artist'), ondelete='CASCADE'),
sa.ForeignKeyConstraint(['source_id'], ['source.id'], name=op.f('fk_post_source_id_source'), ondelete='SET NULL'),
sa.PrimaryKeyConstraint('id', name=op.f('pk_post')),
sa.UniqueConstraint('source_id', 'external_post_id', name='uq_post_source_external_id')
)
op.create_index(op.f('ix_post_artist_id'), 'post', ['artist_id'], unique=False)
op.create_index(op.f('ix_post_source_id'), 'post', ['source_id'], unique=False)
op.create_table('subscribestar_failed_media',
sa.Column('id', sa.Integer(), nullable=False),
sa.Column('source_id', sa.Integer(), nullable=False),
sa.Column('filehash', sa.String(length=128), nullable=False),
sa.Column('attempts', sa.Integer(), nullable=False),
sa.Column('last_error', sa.Text(), nullable=True),
sa.Column('first_failed_at', sa.DateTime(timezone=True), server_default=sa.text('now()'), nullable=False),
sa.Column('last_failed_at', sa.DateTime(timezone=True), server_default=sa.text('now()'), nullable=False),
sa.ForeignKeyConstraint(['source_id'], ['source.id'], name=op.f('fk_subscribestar_failed_media_source_id_source'), ondelete='CASCADE'),
sa.PrimaryKeyConstraint('id', name=op.f('pk_subscribestar_failed_media')),
sa.UniqueConstraint('source_id', 'filehash', name='uq_subscribestar_failed_media_source_id')
)
op.create_index(op.f('ix_subscribestar_failed_media_source_id'), 'subscribestar_failed_media', ['source_id'], unique=False)
op.create_table('subscribestar_seen_media',
sa.Column('id', sa.Integer(), nullable=False),
sa.Column('source_id', sa.Integer(), nullable=False),
sa.Column('filehash', sa.String(length=128), nullable=False),
sa.Column('post_id', sa.String(length=64), nullable=True),
sa.Column('seen_at', sa.DateTime(timezone=True), server_default=sa.text('now()'), nullable=False),
sa.ForeignKeyConstraint(['source_id'], ['source.id'], name=op.f('fk_subscribestar_seen_media_source_id_source'), ondelete='CASCADE'),
sa.PrimaryKeyConstraint('id', name=op.f('pk_subscribestar_seen_media')),
sa.UniqueConstraint('source_id', 'filehash', name='uq_subscribestar_seen_media_source_id')
)
op.create_index(op.f('ix_subscribestar_seen_media_source_id'), 'subscribestar_seen_media', ['source_id'], unique=False)
op.create_table('download_event',
sa.Column('id', sa.Integer(), nullable=False),
sa.Column('source_id', sa.Integer(), nullable=False),
sa.Column('post_id', sa.Integer(), nullable=True),
sa.Column('status', sa.String(length=32), nullable=False),
sa.Column('started_at', sa.DateTime(timezone=True), server_default=sa.text('now()'), nullable=False),
sa.Column('finished_at', sa.DateTime(timezone=True), nullable=True),
sa.Column('bytes_downloaded', sa.BigInteger(), nullable=False),
sa.Column('files_count', sa.Integer(), nullable=False),
sa.Column('error', sa.Text(), nullable=True),
sa.Column('metadata', postgresql.JSONB(astext_type=sa.Text()), server_default=sa.text("'{}'::jsonb"), nullable=False),
sa.ForeignKeyConstraint(['post_id'], ['post.id'], name=op.f('fk_download_event_post_id_post'), ondelete='SET NULL'),
sa.ForeignKeyConstraint(['source_id'], ['source.id'], name=op.f('fk_download_event_source_id_source'), ondelete='CASCADE'),
sa.PrimaryKeyConstraint('id', name=op.f('pk_download_event'))
)
op.create_index(op.f('ix_download_event_post_id'), 'download_event', ['post_id'], unique=False)
op.create_index(op.f('ix_download_event_source_id'), 'download_event', ['source_id'], unique=False)
op.create_table('image_record',
sa.Column('id', sa.Integer(), nullable=False),
sa.Column('path', sa.Text(), nullable=False),
sa.Column('sha256', sa.String(length=64), nullable=False),
sa.Column('phash', sa.String(length=32), nullable=True),
sa.Column('size_bytes', sa.BigInteger(), nullable=False),
sa.Column('mime', sa.String(length=64), nullable=False),
sa.Column('width', sa.Integer(), nullable=True),
sa.Column('height', sa.Integer(), nullable=True),
sa.Column('duration_seconds', sa.Float(), nullable=True),
sa.Column('integrity_status', sa.String(length=24), nullable=False),
sa.Column('thumbnail_path', sa.Text(), nullable=True),
sa.Column('source_url', sa.Text(), nullable=True),
sa.Column('source_filehash', sa.String(length=32), nullable=True),
sa.Column('origin', sa.Enum('downloaded', 'imported_filesystem', 'uploaded', name='origin_enum'), nullable=False),
sa.Column('primary_post_id', sa.Integer(), nullable=True),
sa.Column('artist_id', sa.Integer(), nullable=True),
sa.Column('siglip_embedding', pgvector.sqlalchemy.vector.VECTOR(dim=1152), nullable=True),
sa.Column('siglip_model_version', sa.String(length=128), nullable=True),
sa.Column('created_at', sa.DateTime(timezone=True), server_default=sa.text('now()'), nullable=False),
sa.Column('effective_date', sa.DateTime(timezone=True), server_default=sa.text('now()'), nullable=False),
sa.Column('earliest_post_date', sa.DateTime(timezone=True), server_default=sa.text('now()'), nullable=False),
sa.Column('updated_at', sa.DateTime(timezone=True), server_default=sa.text('now()'), nullable=False),
sa.ForeignKeyConstraint(['artist_id'], ['artist.id'], name=op.f('fk_image_record_artist_id_artist'), ondelete='SET NULL'),
sa.ForeignKeyConstraint(['primary_post_id'], ['post.id'], name=op.f('fk_image_record_primary_post_id_post'), ondelete='SET NULL'),
sa.PrimaryKeyConstraint('id', name=op.f('pk_image_record')),
sa.UniqueConstraint('path', name=op.f('uq_image_record_path'))
)
op.create_index(op.f('ix_image_record_artist_id'), 'image_record', ['artist_id'], unique=False)
op.create_index(op.f('ix_image_record_integrity_status'), 'image_record', ['integrity_status'], unique=False)
op.create_index(op.f('ix_image_record_phash'), 'image_record', ['phash'], unique=False)
op.create_index(op.f('ix_image_record_primary_post_id'), 'image_record', ['primary_post_id'], unique=False)
op.create_index(op.f('ix_image_record_sha256'), 'image_record', ['sha256'], unique=True)
op.create_index(op.f('ix_image_record_source_filehash'), 'image_record', ['source_filehash'], unique=False)
op.create_table('post_attachment',
sa.Column('id', sa.Integer(), nullable=False),
sa.Column('post_id', sa.Integer(), nullable=True),
sa.Column('artist_id', sa.Integer(), nullable=True),
sa.Column('sha256', sa.String(length=64), nullable=False),
sa.Column('path', sa.Text(), nullable=False),
sa.Column('original_filename', sa.Text(), nullable=False),
sa.Column('ext', sa.String(length=32), nullable=False),
sa.Column('mime', sa.String(length=128), nullable=True),
sa.Column('size_bytes', sa.BigInteger(), nullable=False),
sa.Column('captured_at', sa.DateTime(timezone=True), server_default=sa.text('now()'), nullable=False),
sa.ForeignKeyConstraint(['artist_id'], ['artist.id'], name=op.f('fk_post_attachment_artist_id_artist'), ondelete='SET NULL'),
sa.ForeignKeyConstraint(['post_id'], ['post.id'], name=op.f('fk_post_attachment_post_id_post'), ondelete='SET NULL'),
sa.PrimaryKeyConstraint('id', name=op.f('pk_post_attachment'))
)
op.create_index(op.f('ix_post_attachment_artist_id'), 'post_attachment', ['artist_id'], unique=False)
op.create_index(op.f('ix_post_attachment_post_id'), 'post_attachment', ['post_id'], unique=False)
op.create_index(op.f('ix_post_attachment_sha256'), 'post_attachment', ['sha256'], unique=False)
op.create_index('uq_post_attachment_null_post_sha', 'post_attachment', ['sha256'], unique=True, postgresql_where=sa.text('post_id IS NULL'))
op.create_index('uq_post_attachment_post_sha', 'post_attachment', ['post_id', 'sha256'], unique=True, postgresql_where=sa.text('post_id IS NOT NULL'))
op.create_table('series_suggestion',
sa.Column('id', sa.Integer(), nullable=False),
sa.Column('post_id', sa.Integer(), nullable=False),
sa.Column('series_tag_id', sa.Integer(), nullable=False),
sa.Column('score', sa.Float(), nullable=False),
sa.Column('signals', sa.JSON(), nullable=True),
sa.Column('status', sa.String(length=16), server_default='pending', nullable=False),
sa.Column('created_at', sa.DateTime(timezone=True), server_default=sa.text('now()'), nullable=False),
sa.Column('updated_at', sa.DateTime(timezone=True), server_default=sa.text('now()'), nullable=False),
sa.ForeignKeyConstraint(['post_id'], ['post.id'], name=op.f('fk_series_suggestion_post_id_post'), ondelete='CASCADE'),
sa.ForeignKeyConstraint(['series_tag_id'], ['tag.id'], name=op.f('fk_series_suggestion_series_tag_id_tag'), ondelete='CASCADE'),
sa.PrimaryKeyConstraint('id', name=op.f('pk_series_suggestion')),
sa.UniqueConstraint('post_id', 'series_tag_id', name='uq_series_suggestion_post_series')
)
op.create_index(op.f('ix_series_suggestion_post_id'), 'series_suggestion', ['post_id'], unique=False)
op.create_index(op.f('ix_series_suggestion_series_tag_id'), 'series_suggestion', ['series_tag_id'], unique=False)
op.create_index(op.f('ix_series_suggestion_status'), 'series_suggestion', ['status'], unique=False)
op.create_table('external_link',
sa.Column('id', sa.Integer(), nullable=False),
sa.Column('post_id', sa.Integer(), nullable=False),
sa.Column('artist_id', sa.Integer(), nullable=True),
sa.Column('host', sa.String(length=16), nullable=False),
sa.Column('url', sa.Text(), nullable=False),
sa.Column('label', sa.Text(), nullable=True),
sa.Column('status', sa.String(length=16), server_default='pending', nullable=False),
sa.Column('attempts', sa.Integer(), server_default=sa.text('0'), nullable=False),
sa.Column('last_error', sa.Text(), nullable=True),
sa.Column('attachment_id', sa.Integer(), nullable=True),
sa.Column('created_at', sa.DateTime(timezone=True), server_default=sa.text('now()'), nullable=False),
sa.Column('completed_at', sa.DateTime(timezone=True), nullable=True),
sa.Column('duration_seconds', sa.Float(), nullable=True),
sa.ForeignKeyConstraint(['artist_id'], ['artist.id'], name=op.f('fk_external_link_artist_id_artist'), ondelete='SET NULL'),
sa.ForeignKeyConstraint(['attachment_id'], ['post_attachment.id'], name=op.f('fk_external_link_attachment_id_post_attachment'), ondelete='SET NULL'),
sa.ForeignKeyConstraint(['post_id'], ['post.id'], name=op.f('fk_external_link_post_id_post'), ondelete='CASCADE'),
sa.PrimaryKeyConstraint('id', name=op.f('pk_external_link'))
)
op.create_index(op.f('ix_external_link_artist_id'), 'external_link', ['artist_id'], unique=False)
op.create_index(op.f('ix_external_link_post_id'), 'external_link', ['post_id'], unique=False)
op.create_index('ix_external_link_status', 'external_link', ['status'], unique=False)
op.create_index('uq_external_link_post_url', 'external_link', ['post_id', 'url'], unique=True)
op.create_table('gpu_job',
sa.Column('id', sa.Integer(), nullable=False),
sa.Column('image_record_id', sa.Integer(), nullable=False),
sa.Column('task', sa.String(length=32), nullable=False),
sa.Column('status', sa.String(length=16), nullable=False),
sa.Column('lease_token', sa.String(length=64), nullable=True),
sa.Column('leased_at', sa.DateTime(timezone=True), nullable=True),
sa.Column('lease_expires_at', sa.DateTime(timezone=True), nullable=True),
sa.Column('attempts', sa.Integer(), nullable=False),
sa.Column('error', sa.Text(), nullable=True),
sa.Column('triage_status', sa.String(length=16), nullable=True),
sa.Column('created_at', sa.DateTime(timezone=True), server_default=sa.text('now()'), nullable=False),
sa.Column('updated_at', sa.DateTime(timezone=True), server_default=sa.text('now()'), nullable=False),
sa.ForeignKeyConstraint(['image_record_id'], ['image_record.id'], name=op.f('fk_gpu_job_image_record_id_image_record'), ondelete='CASCADE'),
sa.PrimaryKeyConstraint('id', name=op.f('pk_gpu_job'))
)
op.create_index(op.f('ix_gpu_job_image_record_id'), 'gpu_job', ['image_record_id'], unique=False)
op.create_index('ix_gpu_job_leased_expires', 'gpu_job', ['lease_expires_at'], unique=False, postgresql_where=sa.text("status = 'leased'"))
op.create_index('ix_gpu_job_pending', 'gpu_job', ['id'], unique=False, postgresql_where=sa.text("status = 'pending'"))
op.create_index(op.f('ix_gpu_job_status'), 'gpu_job', ['status'], unique=False)
op.create_table('image_provenance',
sa.Column('id', sa.Integer(), nullable=False),
sa.Column('image_record_id', sa.Integer(), nullable=False),
sa.Column('post_id', sa.Integer(), nullable=False),
sa.Column('source_id', sa.Integer(), nullable=True),
sa.Column('from_attachment_id', sa.Integer(), nullable=True),
sa.Column('captured_metadata', sa.JSON(), nullable=True),
sa.Column('captured_at', sa.DateTime(timezone=True), server_default=sa.text('now()'), nullable=False),
sa.ForeignKeyConstraint(['from_attachment_id'], ['post_attachment.id'], name=op.f('fk_image_provenance_from_attachment_id_post_attachment'), ondelete='SET NULL'),
sa.ForeignKeyConstraint(['image_record_id'], ['image_record.id'], name=op.f('fk_image_provenance_image_record_id_image_record'), ondelete='CASCADE'),
sa.ForeignKeyConstraint(['post_id'], ['post.id'], name=op.f('fk_image_provenance_post_id_post'), ondelete='CASCADE'),
sa.ForeignKeyConstraint(['source_id'], ['source.id'], name=op.f('fk_image_provenance_source_id_source'), ondelete='SET NULL'),
sa.PrimaryKeyConstraint('id', name=op.f('pk_image_provenance')),
sa.UniqueConstraint('image_record_id', 'post_id', name='uq_image_provenance_image_post')
)
op.create_index(op.f('ix_image_provenance_from_attachment_id'), 'image_provenance', ['from_attachment_id'], unique=False)
op.create_index(op.f('ix_image_provenance_image_record_id'), 'image_provenance', ['image_record_id'], unique=False)
op.create_index(op.f('ix_image_provenance_post_id'), 'image_provenance', ['post_id'], unique=False)
op.create_index(op.f('ix_image_provenance_source_id'), 'image_provenance', ['source_id'], unique=False)
op.create_table('image_region',
sa.Column('id', sa.Integer(), nullable=False),
sa.Column('image_record_id', sa.Integer(), nullable=False),
sa.Column('kind', sa.String(length=16), nullable=False),
sa.Column('frame_time', sa.Float(), nullable=True),
sa.Column('rx', sa.Float(), nullable=False),
sa.Column('ry', sa.Float(), nullable=False),
sa.Column('rw', sa.Float(), nullable=False),
sa.Column('rh', sa.Float(), nullable=False),
sa.Column('score', sa.Float(), nullable=True),
sa.Column('detector_version', sa.String(length=64), nullable=True),
sa.Column('crop_version', sa.String(length=64), nullable=True),
sa.Column('embedding_version', sa.String(length=128), nullable=True),
sa.Column('ccip_embedding', pgvector.sqlalchemy.vector.VECTOR(dim=768), nullable=True),
sa.Column('siglip_embedding', pgvector.sqlalchemy.vector.VECTOR(dim=1152), nullable=True),
sa.Column('created_at', sa.DateTime(timezone=True), server_default=sa.text('now()'), nullable=False),
sa.ForeignKeyConstraint(['image_record_id'], ['image_record.id'], name=op.f('fk_image_region_image_record_id_image_record'), ondelete='CASCADE'),
sa.PrimaryKeyConstraint('id', name=op.f('pk_image_region'))
)
op.create_index(op.f('ix_image_region_image_record_id'), 'image_region', ['image_record_id'], unique=False)
op.create_table('image_tag',
sa.Column('image_record_id', sa.Integer(), nullable=False),
sa.Column('tag_id', sa.Integer(), nullable=False),
sa.Column('source', sa.String(length=32), nullable=False),
sa.Column('created_at', sa.DateTime(timezone=True), server_default=sa.text('now()'), nullable=False),
sa.ForeignKeyConstraint(['image_record_id'], ['image_record.id'], name=op.f('fk_image_tag_image_record_id_image_record'), ondelete='CASCADE'),
sa.ForeignKeyConstraint(['tag_id'], ['tag.id'], name=op.f('fk_image_tag_tag_id_tag'), ondelete='CASCADE'),
sa.PrimaryKeyConstraint('image_record_id', 'tag_id', name=op.f('pk_image_tag'))
)
op.create_table('import_task',
sa.Column('id', sa.Integer(), nullable=False),
sa.Column('batch_id', sa.Integer(), nullable=False),
sa.Column('source_path', sa.Text(), nullable=False),
sa.Column('task_type', sa.String(length=16), nullable=False),
sa.Column('status', sa.String(length=16), nullable=False),
sa.Column('recovery_count', sa.Integer(), nullable=False),
sa.Column('refetched', sa.Boolean(), nullable=False),
sa.Column('result_image_id', sa.Integer(), nullable=True),
sa.Column('error', sa.Text(), nullable=True),
sa.Column('size_bytes', sa.BigInteger(), nullable=True),
sa.Column('created_at', sa.DateTime(timezone=True), server_default=sa.text('now()'), nullable=False),
sa.Column('started_at', sa.DateTime(timezone=True), nullable=True),
sa.Column('finished_at', sa.DateTime(timezone=True), nullable=True),
sa.ForeignKeyConstraint(['batch_id'], ['import_batch.id'], name=op.f('fk_import_task_batch_id_import_batch'), ondelete='CASCADE'),
sa.ForeignKeyConstraint(['result_image_id'], ['image_record.id'], name=op.f('fk_import_task_result_image_id_image_record'), ondelete='SET NULL'),
sa.PrimaryKeyConstraint('id', name=op.f('pk_import_task'))
)
op.create_index(op.f('ix_import_task_batch_id'), 'import_task', ['batch_id'], unique=False)
op.create_index(op.f('ix_import_task_status'), 'import_task', ['status'], unique=False)
op.create_table('presentation_review',
sa.Column('image_record_id', sa.Integer(), nullable=False),
sa.Column('tag_id', sa.Integer(), nullable=False),
sa.Column('conflict_tag_id', sa.Integer(), nullable=True),
sa.Column('conflict_score', sa.Float(), nullable=False),
sa.Column('mode', sa.String(length=16), server_default='chrome', nullable=False),
sa.Column('created_at', sa.DateTime(timezone=True), server_default=sa.text('now()'), nullable=False),
sa.Column('resolved_at', sa.DateTime(timezone=True), nullable=True),
sa.ForeignKeyConstraint(['conflict_tag_id'], ['tag.id'], name=op.f('fk_presentation_review_conflict_tag_id_tag'), ondelete='SET NULL'),
sa.ForeignKeyConstraint(['image_record_id'], ['image_record.id'], name=op.f('fk_presentation_review_image_record_id_image_record'), ondelete='CASCADE'),
sa.ForeignKeyConstraint(['tag_id'], ['tag.id'], name=op.f('fk_presentation_review_tag_id_tag'), ondelete='CASCADE'),
sa.PrimaryKeyConstraint('image_record_id', 'tag_id', name=op.f('pk_presentation_review'))
)
op.create_table('series_page',
sa.Column('id', sa.Integer(), nullable=False),
sa.Column('series_tag_id', sa.Integer(), nullable=False),
sa.Column('image_id', sa.Integer(), nullable=False),
sa.Column('status', sa.String(length=16), server_default='placed', nullable=False),
sa.Column('page_number', sa.Integer(), nullable=True),
sa.Column('stated_page', sa.Integer(), nullable=True),
sa.Column('created_at', sa.DateTime(timezone=True), server_default=sa.text('now()'), nullable=False),
sa.Column('updated_at', sa.DateTime(timezone=True), server_default=sa.text('now()'), nullable=False),
sa.ForeignKeyConstraint(['image_id'], ['image_record.id'], name=op.f('fk_series_page_image_id_image_record'), ondelete='CASCADE'),
sa.ForeignKeyConstraint(['series_tag_id'], ['tag.id'], name=op.f('fk_series_page_series_tag_id_tag'), ondelete='CASCADE'),
sa.PrimaryKeyConstraint('id', name=op.f('pk_series_page')),
sa.UniqueConstraint('image_id', name=op.f('uq_series_page_image_id'))
)
op.create_index(op.f('ix_series_page_series_tag_id'), 'series_page', ['series_tag_id'], unique=False)
op.create_table('tag_positive_confirmation',
sa.Column('image_record_id', sa.Integer(), nullable=False),
sa.Column('tag_id', sa.Integer(), nullable=False),
sa.Column('confirmed_at', sa.DateTime(timezone=True), server_default=sa.text('now()'), nullable=False),
sa.ForeignKeyConstraint(['image_record_id'], ['image_record.id'], name=op.f('fk_tag_positive_confirmation_image_record_id_image_record'), ondelete='CASCADE'),
sa.ForeignKeyConstraint(['tag_id'], ['tag.id'], name=op.f('fk_tag_positive_confirmation_tag_id_tag'), ondelete='CASCADE'),
sa.PrimaryKeyConstraint('image_record_id', 'tag_id', name=op.f('pk_tag_positive_confirmation'))
)
op.create_index(op.f('ix_tag_positive_confirmation_tag_id'), 'tag_positive_confirmation', ['tag_id'], unique=False)
op.create_table('tag_suggestion_rejection',
sa.Column('image_record_id', sa.Integer(), nullable=False),
sa.Column('tag_id', sa.Integer(), nullable=False),
sa.Column('rejected_at', sa.DateTime(timezone=True), server_default=sa.text('now()'), nullable=False),
sa.ForeignKeyConstraint(['image_record_id'], ['image_record.id'], name=op.f('fk_tag_suggestion_rejection_image_record_id_image_record'), ondelete='CASCADE'),
sa.ForeignKeyConstraint(['tag_id'], ['tag.id'], name=op.f('fk_tag_suggestion_rejection_tag_id_tag'), ondelete='CASCADE'),
sa.PrimaryKeyConstraint('image_record_id', 'tag_id', name=op.f('pk_tag_suggestion_rejection'))
)
op.create_index(op.f('ix_tag_suggestion_rejection_tag_id'), 'tag_suggestion_rejection', ['tag_id'], unique=False)
op.create_table('character_prototype',
sa.Column('id', sa.Integer(), nullable=False),
sa.Column('tag_id', sa.Integer(), nullable=False),
sa.Column('ccip_embedding', pgvector.sqlalchemy.vector.VECTOR(dim=768), nullable=False),
sa.Column('region_id', sa.Integer(), nullable=True),
sa.ForeignKeyConstraint(['region_id'], ['image_region.id'], name=op.f('fk_character_prototype_region_id_image_region'), ondelete='SET NULL'),
sa.ForeignKeyConstraint(['tag_id'], ['tag.id'], name=op.f('fk_character_prototype_tag_id_tag'), ondelete='CASCADE'),
sa.PrimaryKeyConstraint('id', name=op.f('pk_character_prototype'))
)
op.create_index(op.f('ix_character_prototype_tag_id'), 'character_prototype', ['tag_id'], unique=False)
op.create_table('series_chapter',
sa.Column('id', sa.Integer(), nullable=False),
sa.Column('series_tag_id', sa.Integer(), nullable=False),
sa.Column('anchor_page_id', sa.Integer(), nullable=False),
sa.Column('title', sa.Text(), nullable=True),
sa.Column('stated_part', sa.Integer(), nullable=True),
sa.Column('created_at', sa.DateTime(timezone=True), server_default=sa.text('now()'), nullable=False),
sa.Column('updated_at', sa.DateTime(timezone=True), server_default=sa.text('now()'), nullable=False),
sa.ForeignKeyConstraint(['anchor_page_id'], ['series_page.id'], name=op.f('fk_series_chapter_anchor_page_id_series_page'), ondelete='CASCADE'),
sa.ForeignKeyConstraint(['series_tag_id'], ['tag.id'], name=op.f('fk_series_chapter_series_tag_id_tag'), ondelete='CASCADE'),
sa.PrimaryKeyConstraint('id', name=op.f('pk_series_chapter')),
sa.UniqueConstraint('anchor_page_id', name=op.f('uq_series_chapter_anchor_page_id'))
)
op.create_index(op.f('ix_series_chapter_series_tag_id'), 'series_chapter', ['series_tag_id'], unique=False)
# The HNSW index, item 3 above. Must match the query's cosine-distance
# operator class or the planner will not use it.
op.execute(
"CREATE INDEX ix_image_record_siglip_hnsw "
"ON image_record USING hnsw (siglip_embedding vector_cosine_ops)"
)
def downgrade() -> None:
# Dropping image_record takes its indexes with it, so the HNSW index needs
# no separate drop. The extensions are deliberately left in place: they are
# database-scoped and something else may be using them.
op.drop_index(op.f('ix_series_chapter_series_tag_id'), table_name='series_chapter')
op.drop_table('series_chapter')
op.drop_index(op.f('ix_character_prototype_tag_id'), table_name='character_prototype')
op.drop_table('character_prototype')
op.drop_index(op.f('ix_tag_suggestion_rejection_tag_id'), table_name='tag_suggestion_rejection')
op.drop_table('tag_suggestion_rejection')
op.drop_index(op.f('ix_tag_positive_confirmation_tag_id'), table_name='tag_positive_confirmation')
op.drop_table('tag_positive_confirmation')
op.drop_index(op.f('ix_series_page_series_tag_id'), table_name='series_page')
op.drop_table('series_page')
op.drop_table('presentation_review')
op.drop_index(op.f('ix_import_task_status'), table_name='import_task')
op.drop_index(op.f('ix_import_task_batch_id'), table_name='import_task')
op.drop_table('import_task')
op.drop_table('image_tag')
op.drop_index(op.f('ix_image_region_image_record_id'), table_name='image_region')
op.drop_table('image_region')
op.drop_index(op.f('ix_image_provenance_source_id'), table_name='image_provenance')
op.drop_index(op.f('ix_image_provenance_post_id'), table_name='image_provenance')
op.drop_index(op.f('ix_image_provenance_image_record_id'), table_name='image_provenance')
op.drop_index(op.f('ix_image_provenance_from_attachment_id'), table_name='image_provenance')
op.drop_table('image_provenance')
op.drop_index(op.f('ix_gpu_job_status'), table_name='gpu_job')
op.drop_index('ix_gpu_job_pending', table_name='gpu_job', postgresql_where=sa.text("status = 'pending'"))
op.drop_index('ix_gpu_job_leased_expires', table_name='gpu_job', postgresql_where=sa.text("status = 'leased'"))
op.drop_index(op.f('ix_gpu_job_image_record_id'), table_name='gpu_job')
op.drop_table('gpu_job')
op.drop_index('uq_external_link_post_url', table_name='external_link')
op.drop_index('ix_external_link_status', table_name='external_link')
op.drop_index(op.f('ix_external_link_post_id'), table_name='external_link')
op.drop_index(op.f('ix_external_link_artist_id'), table_name='external_link')
op.drop_table('external_link')
op.drop_index(op.f('ix_series_suggestion_status'), table_name='series_suggestion')
op.drop_index(op.f('ix_series_suggestion_series_tag_id'), table_name='series_suggestion')
op.drop_index(op.f('ix_series_suggestion_post_id'), table_name='series_suggestion')
op.drop_table('series_suggestion')
op.drop_index('uq_post_attachment_post_sha', table_name='post_attachment', postgresql_where=sa.text('post_id IS NOT NULL'))
op.drop_index('uq_post_attachment_null_post_sha', table_name='post_attachment', postgresql_where=sa.text('post_id IS NULL'))
op.drop_index(op.f('ix_post_attachment_sha256'), table_name='post_attachment')
op.drop_index(op.f('ix_post_attachment_post_id'), table_name='post_attachment')
op.drop_index(op.f('ix_post_attachment_artist_id'), table_name='post_attachment')
op.drop_table('post_attachment')
op.drop_index(op.f('ix_image_record_source_filehash'), table_name='image_record')
op.drop_index(op.f('ix_image_record_sha256'), table_name='image_record')
op.drop_index(op.f('ix_image_record_primary_post_id'), table_name='image_record')
op.drop_index(op.f('ix_image_record_phash'), table_name='image_record')
op.drop_index(op.f('ix_image_record_integrity_status'), table_name='image_record')
op.drop_index(op.f('ix_image_record_artist_id'), table_name='image_record')
op.drop_table('image_record')
op.drop_index(op.f('ix_download_event_source_id'), table_name='download_event')
op.drop_index(op.f('ix_download_event_post_id'), table_name='download_event')
op.drop_table('download_event')
op.drop_index(op.f('ix_subscribestar_seen_media_source_id'), table_name='subscribestar_seen_media')
op.drop_table('subscribestar_seen_media')
op.drop_index(op.f('ix_subscribestar_failed_media_source_id'), table_name='subscribestar_failed_media')
op.drop_table('subscribestar_failed_media')
op.drop_index(op.f('ix_post_source_id'), table_name='post')
op.drop_index(op.f('ix_post_artist_id'), table_name='post')
op.drop_table('post')
op.drop_index(op.f('ix_pixiv_seen_media_source_id'), table_name='pixiv_seen_media')
op.drop_table('pixiv_seen_media')
op.drop_index(op.f('ix_pixiv_failed_media_source_id'), table_name='pixiv_failed_media')
op.drop_table('pixiv_failed_media')
op.drop_index(op.f('ix_patreon_seen_media_source_id'), table_name='patreon_seen_media')
op.drop_table('patreon_seen_media')
op.drop_index(op.f('ix_patreon_failed_media_source_id'), table_name='patreon_failed_media')
op.drop_table('patreon_failed_media')
op.drop_table('tag_head')
op.drop_index(op.f('ix_tag_alias_canonical_tag_id'), table_name='tag_alias')
op.drop_table('tag_alias')
op.drop_index(op.f('ix_source_error_type'), table_name='source')
op.drop_index(op.f('ix_source_artist_id'), table_name='source')
op.drop_table('source')
op.drop_index(op.f('ix_head_metrics_snapshot_tag_id'), table_name='head_metrics_snapshot')
op.drop_index(op.f('ix_head_metrics_snapshot_snapshot_at'), table_name='head_metrics_snapshot')
op.drop_table('head_metrics_snapshot')
op.drop_table('head_metric')
op.drop_table('ccip_prototype_state')
op.drop_table('artist_visit')
op.drop_index(op.f('ix_task_run_task_name'), table_name='task_run')
op.drop_index(op.f('ix_task_run_status'), table_name='task_run')
op.drop_index(op.f('ix_task_run_started_at'), table_name='task_run')
op.drop_index(op.f('ix_task_run_queue'), table_name='task_run')
op.drop_index(op.f('ix_task_run_finished_at'), table_name='task_run')
op.drop_index(op.f('ix_task_run_celery_task_id'), table_name='task_run')
op.drop_table('task_run')
op.drop_index(op.f('ix_tag_fandom_id'), table_name='tag')
op.drop_table('tag')
op.drop_table('ml_settings')
op.drop_index(op.f('ix_library_audit_run_status'), table_name='library_audit_run')
op.drop_index(op.f('ix_library_audit_run_rule'), table_name='library_audit_run')
op.drop_table('library_audit_run')
op.drop_table('import_settings')
op.drop_index(op.f('ix_import_batch_status'), table_name='import_batch')
op.drop_table('import_batch')
op.drop_index(op.f('ix_head_training_run_status'), table_name='head_training_run')
op.drop_table('head_training_run')
op.drop_index(op.f('ix_head_auto_apply_run_status'), table_name='head_auto_apply_run')
op.drop_table('head_auto_apply_run')
op.drop_table('credential')
op.drop_index(op.f('ix_backup_run_tag'), table_name='backup_run')
op.drop_index(op.f('ix_backup_run_status'), table_name='backup_run')
op.drop_index(op.f('ix_backup_run_started_at'), table_name='backup_run')
op.drop_index(op.f('ix_backup_run_kind'), table_name='backup_run')
op.drop_index(op.f('ix_backup_run_finished_at'), table_name='backup_run')
op.drop_table('backup_run')
op.drop_table('artist')
op.drop_table('app_setting')
@@ -1,33 +0,0 @@
"""soft WIP title tier toggle (#1474) — ImportSettings.wip_soft_title_tagging_enabled
The soft tier also tags sketch/doodle/scribble titles, but with a provisional source
that never trains the head. OFF by default (a lower-precision tier is opt-in).
server_default so the existing singleton row (id=1) fills cleanly.
Revision ID: 0087
Revises: 0086
Create Date: 2026-07-13
"""
from typing import Sequence, Union
import sqlalchemy as sa
from alembic import op
revision: str = "0087"
down_revision: Union[str, None] = "0086"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
op.add_column(
"import_settings",
sa.Column(
"wip_soft_title_tagging_enabled", sa.Boolean(), nullable=False,
server_default=sa.text("false"),
),
)
def downgrade() -> None:
op.drop_column("import_settings", "wip_soft_title_tagging_enabled")
+28 -1
View File
@@ -13,6 +13,7 @@ from pathlib import Path
from quart import Blueprint, jsonify, request
from sqlalchemy import select
from ..build_info import FC_CHANNEL as _FC_CHANNEL
from ..extensions import get_session
from ..models import AppSetting
from ..services.extension_service import (
@@ -31,6 +32,15 @@ XPI_DIR = Path("/app/frontend/dist/extension")
_XPI_VERSION_RE = re.compile(r"fabledcurator-(?P<version>[\w.-]+)\.xpi$")
# Which channel this image belongs to — "dev" or "main" — baked in at build
# time (milestone 271 step 7). Read from build_info rather than the environment
# a second time: /api/health reports the same value, and two independent
# `os.environ.get` calls are two things that can drift.
#
# Still bound as a module-level name here, so tests monkeypatch
# `extension.FC_CHANNEL` exactly as they did before, same as XPI_DIR above.
FC_CHANNEL = _FC_CHANNEL
async def _ext_key_required(session) -> bool:
"""Unlike /api/credentials (which accepts the browser path with no
@@ -133,13 +143,30 @@ def _read_manifest_sync() -> dict | None:
return None
versioned.sort(key=lambda p: p.stat().st_mtime)
latest = versioned[-1]
return {
info = {
"installed": True,
"version": _extract_version(latest.name),
"xpi_url": f"/extension/{latest.name}",
"latest_url": "/extension/fabledcurator-latest.xpi",
"sha256": _sha256(latest),
}
# The channel goes BESIDE the version, never inside it. A `-dev` suffix is
# what silently disabled the dev channel in the sibling project this design
# comes from: the comparator returned nothing for a non-integer segment, so
# every dev version compared equal and "no update available" became
# indistinguishable from "I cannot read this version".
#
# Omitted rather than defaulted when unset. Absence already has a meaning
# every reader must handle — an image built before this field existed says
# exactly the same thing by not having the key — so a blank channel reuses
# that path instead of inventing a second "unknown" spelling.
#
# Reported verbatim, not validated against {"dev", "main"}: if an image
# declares something else, showing what it actually claims is more useful
# to whoever is debugging it than dropping the value on the floor.
if FC_CHANNEL:
info["channel"] = FC_CHANNEL
return info
@extension_bp.route("/manifest", methods=["GET"])
+17 -2
View File
@@ -1,5 +1,20 @@
"""Health endpoint — no DB or Redis touch; just liveness."""
"""Health endpoint — no DB or Redis touch; liveness, plus the build's identity.
The identity rides here rather than on a route of its own because it answers
at the same cost: two module constants, no I/O, nothing that can be slow or
fail. It is also already fetched app-wide — TopNav calls `refreshHealth` on
mount — so a separate endpoint would mean a second request for two strings.
Both fields are OMITTED when unset rather than sent empty. See build_info.
"""
from ..build_info import FC_CHANNEL, FC_VERSION
async def get_health():
return {"status": "ok"}, 200
body = {"status": "ok"}
if FC_VERSION:
body["version"] = FC_VERSION
if FC_CHANNEL:
body["channel"] = FC_CHANNEL
return body, 200
+30
View File
@@ -0,0 +1,30 @@
"""What this build IS — stamped at image build time, not configurable.
Deliberately separate from `config.py`. Those are operator settings, read from
the environment and meant to be changed. These describe the artifact itself and
are baked in by CI (the `FC_VERSION` / `FC_CHANNEL` build args); an operator
setting them by hand is not a supported thing to do, it is just how a value
gets from the build into the running process.
**Absent rather than empty when unknown.** A locally-built image has no version,
and neither did any image predating the field — one spelling of "cannot say",
which every reader already has to handle, instead of a second one to
special-case (note #3127 §7).
**Why this matters more than it used to.** Milestone 318 stopped publishing
version image tags, so a running instance's self-report is now the *only*
answer to "which build is this?" — there is no registry name left to check it
against. A wrong value here has nothing to contradict it. That is why the UI
renders `unknown` rather than a blank or a plausible default: an empty footer
reads as "no version", which is a different and false claim.
The channel lives BESIDE the version and is never folded into it (rule 149).
A `-dev` suffix would be parsed by the extension's comparator as a segment
worth 0, making every dev build compare equal to every other — issue #2993's
exact failure.
"""
import os
FC_VERSION = os.environ.get("FC_VERSION", "").strip()
FC_CHANNEL = os.environ.get("FC_CHANNEL", "").strip()
-44
View File
@@ -1,44 +0,0 @@
"""Literal SQL for the FC-2d-vii-c artist backfill / artist-tag delete.
Intentionally pure string constants — NO model/slug imports, NO logic —
so migration 0008 and its test share one drift-proof source of truth.
Backfill steps are ordered primary -> provenance -> artist-tag and each
only touches rows still NULL (idempotent, first match wins). The
artist-tag step matches Artist.name = Tag.name: the importer always
created both from the same artist_name string.
"""
BACKFILL_PRIMARY_SQL = """
UPDATE image_record AS ir
SET artist_id = s.artist_id
FROM post p
JOIN source s ON s.id = p.source_id
WHERE ir.primary_post_id = p.id
AND ir.artist_id IS NULL
"""
BACKFILL_PROVENANCE_SQL = """
UPDATE image_record AS ir
SET artist_id = s.artist_id
FROM (
SELECT DISTINCT ON (ip.image_record_id)
ip.image_record_id, src.artist_id
FROM image_provenance ip
JOIN source src ON src.id = ip.source_id
ORDER BY ip.image_record_id, ip.id
) AS s
WHERE ir.id = s.image_record_id
AND ir.artist_id IS NULL
"""
BACKFILL_TAG_SQL = """
UPDATE image_record AS ir
SET artist_id = a.id
FROM image_tag it
JOIN tag t ON t.id = it.tag_id AND t.kind = 'artist'
JOIN artist a ON a.name = t.name
WHERE it.image_record_id = ir.id
AND ir.artist_id IS NULL
"""
DELETE_ARTIST_TAGS_SQL = "DELETE FROM tag WHERE kind = 'artist'"
+174 -12
View File
@@ -9,14 +9,17 @@ git.fabledsword.com/bvandeusen/ci-python:3.14
## Image deps used
- python 3.14
- ruff (analyzer for `backend/`, `tests/`, `alembic/`)
- ruff (analyzer for `backend/`, `tests/`, `alembic/`, `agent/`, `scripts/`)
- node (frontend job: `npm install` + vitest + vite build)
- docker CLI + buildx (`.forgejo/workflows/build.yml`: build-web, build-ml — Fabled-Git registry push)
- docker CLI + buildx (`.forgejo/workflows/build.yml`: build-web, build-ml, build-agent — Fabled-Git registry push, and `imagetools inspect`/`create` for the reuse path)
## Secondary runtime image
node:24-bookworm-slim — `.forgejo/workflows/extension.yml` only.
`.forgejo/workflows/release.yml` runs on `ci-python:3.14` like everything else
and installs nothing: it needs git and stdlib python, and builds no image.
The extension lane is the one job that does NOT run on `ci-python:3.14`: it
needs a current Node for `web-ext` and vitest and nothing Python at all. Kept
on the upstream slim image rather than adding a Node toolchain to `ci-python`,
@@ -55,16 +58,175 @@ per `docs/process.md`'s "add deps to the image when used by >1 project".
modules, so the specs exercise exactly the bytes packaged into the XPI.
- **`extension/scripts/packaging.sh` is the single definition of what ships
inside the XPI.** Three consumers read from it rather than keeping their own
copy: web-ext's `--ignore-files` (`extension/package.json`), the `:(exclude)`
pathspec in `ci.yml`'s `extension-version` guard, and the `git log` pathspec
that derives the extension version. Three hand-kept copies of that one fact
is what allowed issue #2397.
- Jobs that derive the extension version check out with `fetch-depth: 0`. The
version is the commit TIME of the newest packaged-extension change (minutes
since 2020-01-01, per family rule 149 — never a commit count, which orders
by branch rather than by recency). A depth-1 clone sees one commit and
derives a wrong, too-low value rather than failing, so the full-history
checkout is load-bearing wherever `packaging.sh version` is called.
copy: web-ext's `--ignore-files` (`extension/package.json`), the `git log`
pathspec inside the script's own version derivation, and `scripts/artifacts.sh`,
which appends the extension's set to web's because the web image bundles the
signed XPI. Hand-kept copies of that one fact is what allowed issue #2397, so
`extension/test/version.spec.js` asserts no workflow has reintroduced a
literal `:(exclude)extension/…`.
- **Packaged and version-relevant are two different sets** (#3156). `scripts/`
is excluded from the XPI and is NOT excluded from the version derivation,
because `packaging.sh` decides the version string stamped into the packaged
`manifest.json`. The membership test is *"can changing this file change the
published bytes?"*, not *"is this file copied in?"* — which is why the script
keeps two lists rather than one.
- **The shipped extension version is derived, not committed.** It is the commit
TIME of the newest packaged-extension change, rendered `YYYY.M.D.HHMM` UTC
(family rules 148/149 — never a commit count, which orders by branch rather
than by recency). `build.yml`'s `sign-extension` computes it and stamps it
into `extension/manifest.json` + `package.json` in the working tree before
signing; the stamp is never committed. The version in the repo is **wholly
inert** — since milestone 318 step 8 there is no hand-set MAJOR.MINOR either.
- **The extension is the one artifact that does not zero-pad, and that is not a
drift** (#3138). Mozilla's grammar for AMO is
`^(0|[1-9][0-9]{0,8})([.](0|[1-9][0-9]{0,8})){0,3}$` — a segment is the single
digit `0` or starts 1-9, and there are at most four. `2026.08.29.0201` is
rejected; `2026.8.29.201` is the same value one character narrower per
segment, and rule 148 defines comparison as numeric per segment, so nothing is
reordered. `ci.yml`'s `extension-version` lane asserts the derived string
against that exact regex, plus a `YYYY.M.D.HHMM` shape check that would catch
a regression to the pre-318 `1.0.<minutes>` — which AMO would accept and which
orders below everything already signed. Checking here is the whole point: AMO
409s on re-signing, so a version it rejects is burned and cannot be reused.
`scripts/artifacts.sh version extension` **delegates** to `packaging.sh` so
the two cannot answer differently.
- Every job that derives anything checks out with `fetch-depth: 0` — all four
`build.yml` jobs, `ci.yml`'s `extension-version` and `backend-lint-and-test`
(for `tests/test_artifact_paths.py` and `test_artifact_identity.py`), and
`release.yml`, which additionally walks the tag graph. A depth-1 clone sees
one commit and derives a wrong, too-low value **rather than failing**, so the
full-history checkout is load-bearing rather than incidental.
- **`scripts/artifacts.sh` is the same shape one level up: one definition per
artifact of what it is built from, and the two values derived from it.**
`revision` (12 hex of the newest commit touching that set) and `version`
(`YYYY.MM.DD.HHMM` UTC, rule 148). Four artifacts, four independent answers,
so a push touching only `agent/` leaves web and ml alone.
`tests/test_artifact_paths.py` reads each Dockerfile and asserts every COPY
source is covered, so adding a COPY without updating the script fails CI.
- **A file that DECIDES an artifact's identity belongs in its set even though it
is copied into nothing** — `packaging.sh` for the extension and web (#3156),
and `artifacts.sh` itself for web (#3202), which decides the `FC_VERSION`
baked into that image. Only web needs the second entry: every artifact stamps
a revision, but a revision has a backstop (a changed derivation stops matching
the published label and forces a rebuild) and a version has none, because
nothing compares it to anything. `tests/test_artifact_paths.py`'s `DERIVERS`
table is the guard.
- **Builds are skipped when the content is already published.** Each image
carries its revision as an `fc.revision` LABEL, and `build.yml` reads that
label back off the moving channel tag (`imagetools inspect --format`). Equal
to the derived revision means the bytes are already published, so the job
repoints the remaining tags at the existing manifest instead of rebuilding.
Two things this depends on: an inspect that errors for ANY reason reads as a
MISS so no needed build is ever skipped, and the repoint must EXCLUDE the
source tag — `imagetools create` wraps its source in a manifest index, and
config labels do not resolve through an index, so writing the channel tag
from itself destroys the label the next run reads (#3183).
- **The build pushes exactly ONE tag — the channel's — and every other tag is
written registry-side afterwards** (#3190). buildx on this runner pushes the
first tag to the registry and then re-pushes the rest through the docker
driver, out of a local image store that a registry-direct build never fills;
it fails intermittently with `tag does not exist`. On `dev` that only reddens
a job, but on `main` it silently skips `:c-<sha>` while `:latest` publishes
fine — a missing rollback tag has no consumer that fails, so nothing but the
red job would notice until somebody needs to roll back. `imagetools create`
has no local store to be absent from, and it is the code the reuse path
already ran, so both paths now share one proven route. The cost: `:c-<sha>`
is an index rather than a plain image, so `fc.revision` does not resolve
through it — nothing reads it there, and the index names the same manifest.
- **The image builds run on a `docker-container` buildx builder, and
`provenance`/`sbom` are explicitly OFF** (milestone 326 step 1). The builder
is what makes a registry layer cache possible at all — the default `docker`
driver cannot export one (#3114) — and it is #3190's leading suspect, since
it is the driver that resolves image metadata against a local store a
registry-direct push never fills. **The attestation flags are load-bearing,
not tidiness:** on the container driver `build-push-action@v5` defaults
`provenance` to true when pushing, an attestation manifest makes the pushed
tag a manifest INDEX, and config labels do not resolve through an index — so
leaving them on would make every push read `fc.revision=<none>`, miss, and
rebuild forever with every lane green. Same failure as #3183, different door.
These jobs run inside a container against a mounted docker socket, so the
buildkit container is a sibling rather than a child.
- **All three images import and export a registry layer cache**
(`<image>:buildcache`, `mode=max`). This is not an optimisation bolted onto
the driver change — it is the other half of it. The `docker-container`
driver gets a fresh buildkit instance per job and therefore has **no local
layer store at all**, where the old `docker` driver at least reused whatever
the runner's dockerd happened to hold. Measured on run 4896, the first builds
after the driver moved: web 3m44s (was 2m23s), ml 3m49s (was 3m20s), agent
11m12s (was 9m26s) — every one slower. A `:buildcache` tag is read by every
build that runs, is one moving ref per image, holds cache blobs rather than a
shippable artifact, and is overwritten in place, so it is not a return of the
per-version tags milestone 318 withdrew (#3114).
- **`build.yml` accepts a `workflow_dispatch` with `force_build`**, which
bypasses the reuse check for all three images. It exists because
skip-if-exists makes its own build path untestable: `agent/` has not changed
since 2026-07-17, so the agent build has not run in six weeks and cannot be
exercised on demand — and #3190 lives on exactly that path. Editing
`build.yml` does not force a build either, deliberately: the workflow is not
shipped bytes and is in no artifact's path set. The flag is read through
`github.event.inputs` into an env var rather than interpolated into a run
block, and it is checked inside the reuse step so that one decision drives
both the build and the repoint.
- **A weekly `schedule` rebuilds all three images against fresh base layers**
(Sunday 06:00 UTC, milestone 326 step 4, #3154). Skip-if-exists is keyed on
OUR source, so an artifact whose source stops moving stops picking up base
updates — `agent/` has not changed since 2026-07-17 and would otherwise serve
that day's `nvidia/cuda` layers forever. Four things make it work:
- It **builds `main`, not the branch that triggered it.** Forgejo registers a
cron from the DEFAULT branch (`dev` here), so a scheduled run arrives with
`github.ref` on dev. The ref is decided once in a top-level `env:
BUILD_REF` that every checkout in the file takes, rather than per job —
otherwise `sign-extension` would derive dev's extension version while
`build-web` bundled main's, and the release download would 404 on a version
that exists perfectly well. Every job then ASSERTS its checkout is `main`
before doing anything, because `env` inside `with:` is not a context this
runner is known to evaluate — if it silently resolved to empty, checkout
would fall back to the triggering ref and the refresh would publish dev's
source to `:latest` with every lane green.
- It **publishes only `:latest`.** `:c-<sha>` for main's HEAD already names
the bytes that commit built; re-pushing it over refreshed layers would
break the one tag rule 145 makes immutable, and it is the rollback unit.
The repoint step needs no schedule case for this — the tag list is the
channel tag alone, so SOURCE is the only entry, it is excluded as always,
and the step correctly does nothing.
- **`:latest` and `:c-<sha>` therefore diverge between a refresh and the next
`main` push, by design.** They re-converge on that push: it hits reuse (a
refresh does not move `fc.revision`, because it does not touch the source),
and the repoint writes the NEW `:c-<sha>` from the refreshed `:latest`. The
push path needed no change for this, because the repoint already excluded
the source tag — the same rule that keeps the label readable also keeps a
refresh from being undone.
- **`pull: true` on the scheduled path only** is the mechanism: a moved base
tag changes the `FROM` layer's cache key and everything above it rebuilds.
**It does not currently make the unmoved case free.** Measured on the first
real fire (run 4934, 2026-08-30): every content step reported `CACHED` and
the bases resolved to unchanged digests, yet all three `:latest` tags got a
NEW manifest digest, because buildkit mints a fresh image config per run and
republishes identical layers under it. So `:latest` is rewritten weekly
whether or not anything changed, and `:c-<sha>` is handed a new manifest to
diverge from on the same cadence — a digest change stops meaning anything.
Tracked as #3265; the likely fix is a deterministic `SOURCE_DATE_EPOCH`.
Separately not caught: a Debian package update inside the `apt-get install`
layer while the base tag stands still — a lag rather than a hole, since the
official python/cuda images rebuild with those updates baked in.
- **`FC_CHANNEL` and `FC_VERSION` are build args, not runtime settings.**
`build.yml` passes them to the web image only — the ml and agent images have
nothing to report them to. `/api/health` returns both, the foot of Settings
renders them, and `/api/extension/manifest` reports the channel beside the
extension version so an install can be traced to a channel. With no version image tags, that
self-report is the ONLY answer to "which build is this?" — which is why a
missing version renders `unknown` rather than a blank: an empty footer reads
as "no version", a different and false claim.
Both are declared LAST in the Dockerfile on purpose: an ARG invalidates every
layer below it, and these are the values that differ between the dev and main
builds of identical source, so placing them earlier would stop the two
channels ever sharing a cached `pip install`. Empty by default — a local build
then reports nothing rather than claiming a channel it is not on.
- **The channel is never folded into the version.** A `-dev` suffix makes the
extension's per-segment `parseInt` comparator read that segment as 0, so every
dev build compares equal to every other — issue #2993 exactly (rule 149).
`frontend/test/systemBuild.spec.js` pins the rendered version to the bare
number.
- Callers MUST `set -f` before substituting the script's output. Without it the
shell expands `test/**` against the working tree and silently narrows the
pattern to whatever files exist at that moment — a failure that looks like

Some files were not shown because too many files have changed in this diff Show More