diff --git a/.forgejo/workflows/build.yml b/.forgejo/workflows/build.yml index 23ddd18..72660ca 100644 --- a/.forgejo/workflows/build.yml +++ b/.forgejo/workflows/build.yml @@ -116,8 +116,9 @@ jobs: # STILL. If web moves, its path set is too wide. # * a push touching only docs moves nothing. # * a push touching the extension moves the extension AND web, since - # web bakes in the XPI. If web does not move, its set is too narrow - # — the direction that serves stale bytes on a pin. + # web bakes in the XPI. If web does not move, its set is too narrow: + # the reuse check hits, and the channel serves a web image bundling + # the PREVIOUS XPI while the freshly signed one is orphaned (#3156). # * dev and main derive the same values for the same source. - name: Shadow — derived artifact version (informational) run: | @@ -345,12 +346,11 @@ jobs: fetch-depth: 0 # --- derived values, one line (milestone 313) ------------------------ - # These stopped being shadow output at step 3: `tag` is published on - # main and `revision` decides whether the build below runs at all. This - # step prints all three anyway, because the load-bearing steps each - # print only the one they use, and on dev the date tag is computed - # nowhere else. When a build is skipped or a pin looks wrong, this is - # the line that says what the commit derived. + # These stopped being shadow output at step 3. `revision` decides + # whether the build below runs at all and `version` is what the image + # reports about itself; the load-bearing steps each print only the one + # they use, so this is the only place the pair appears together. When a + # build is skipped, this is the line that says what the commit derived. # # Still diagnostic, so it still must not fail the build — no `set -e`, # and every derivation falls back to UNAVAILABLE. A broken echo must @@ -361,8 +361,9 @@ jobs: # STILL. If web moves, its path set is too wide. # * a push touching only docs moves nothing. # * a push touching the extension moves the extension AND web, since - # web bakes in the XPI. If web does not move, its set is too narrow - # — the direction that serves stale bytes on a pin. + # web bakes in the XPI. If web does not move, its set is too narrow: + # the reuse check hits, and the channel serves a web image bundling + # the PREVIOUS XPI while the freshly signed one is orphaned (#3156). # * dev and main derive the same values for the same source. - name: Report the derived artifact version run: | @@ -463,15 +464,13 @@ jobs: # that skips a build that was actually needed; the worst case is paying # for a build we could have avoided. # - # BASE-IMAGE FRESHNESS, decided rather than left implicit: an artifact - # whose source stops moving stops picking up base-image updates under - # its pinned tag. That is what a pin MEANS — a date tag has to keep - # serving the bytes it served (fabledcurator:2026.7.17 still - # resolves to July's image), or it is not a pin — and family rule - # 145 already says where the refresh goes instead: a rebuild with - # different contents publishes only the MOVING tag, never the immutable - # one. A scheduled channel-only refresh is tracked separately (#3154); - # it does not belong in the push path. + # BASE-IMAGE FRESHNESS: an artifact whose source stops moving stops + # picking up base-image updates. Milestone 318 removed the argument this + # used to need rather than answering it — with no version tags there is + # no immutable name a refresh could contradict, and rule 145 already + # allows a rebuild with different contents to republish a MOVING tag. + # So a refresh is just a build. A scheduled channel-only one is tracked + # separately (#3154); it does not belong in the push path. - name: Is this content already published? id: reuse env: @@ -624,8 +623,8 @@ jobs: # Registry-side manifest copy: no layer transfer, no local daemon, no # rebuild. Each -t becomes another reference to the SAME manifest the - # channel tag already holds, so :c- and the date pin are - # byte-identical to what is published rather than a lookalike rebuild. + # channel tag already holds, so :c- is byte-identical to what is + # published rather than a lookalike rebuild. # # Runs on EVERY reuse, which is what keeps family rule 146 true: a # rolling channel refreshes itself, so skipping a build must never mean @@ -691,12 +690,11 @@ jobs: fetch-depth: 0 # --- derived values, one line (milestone 313) ------------------------ - # These stopped being shadow output at step 3: `tag` is published on - # main and `revision` decides whether the build below runs at all. This - # step prints all three anyway, because the load-bearing steps each - # print only the one they use, and on dev the date tag is computed - # nowhere else. When a build is skipped or a pin looks wrong, this is - # the line that says what the commit derived. + # These stopped being shadow output at step 3. `revision` decides + # whether the build below runs at all and `version` is what the image + # reports about itself; the load-bearing steps each print only the one + # they use, so this is the only place the pair appears together. When a + # build is skipped, this is the line that says what the commit derived. # # Still diagnostic, so it still must not fail the build — no `set -e`, # and every derivation falls back to UNAVAILABLE. A broken echo must @@ -707,8 +705,9 @@ jobs: # STILL. If web moves, its path set is too wide. # * a push touching only docs moves nothing. # * a push touching the extension moves the extension AND web, since - # web bakes in the XPI. If web does not move, its set is too narrow - # — the direction that serves stale bytes on a pin. + # web bakes in the XPI. If web does not move, its set is too narrow: + # the reuse check hits, and the channel serves a web image bundling + # the PREVIOUS XPI while the freshly signed one is orphaned (#3156). # * dev and main derive the same values for the same source. - name: Report the derived artifact version run: | @@ -770,15 +769,13 @@ jobs: # that skips a build that was actually needed; the worst case is paying # for a build we could have avoided. # - # BASE-IMAGE FRESHNESS, decided rather than left implicit: an artifact - # whose source stops moving stops picking up base-image updates under - # its pinned tag. That is what a pin MEANS — a date tag has to keep - # serving the bytes it served (fabledcurator-ml:2026.7.17 still - # resolves to July's image), or it is not a pin — and family rule - # 145 already says where the refresh goes instead: a rebuild with - # different contents publishes only the MOVING tag, never the immutable - # one. A scheduled channel-only refresh is tracked separately (#3154); - # it does not belong in the push path. + # BASE-IMAGE FRESHNESS: an artifact whose source stops moving stops + # picking up base-image updates. Milestone 318 removed the argument this + # used to need rather than answering it — with no version tags there is + # no immutable name a refresh could contradict, and rule 145 already + # allows a rebuild with different contents to republish a MOVING tag. + # So a refresh is just a build. A scheduled channel-only one is tracked + # separately (#3154); it does not belong in the push path. - name: Is this content already published? id: reuse env: @@ -845,8 +842,8 @@ jobs: # Registry-side manifest copy: no layer transfer, no local daemon, no # rebuild. Each -t becomes another reference to the SAME manifest the - # channel tag already holds, so :c- and the date pin are - # byte-identical to what is published rather than a lookalike rebuild. + # channel tag already holds, so :c- is byte-identical to what is + # published rather than a lookalike rebuild. # # Runs on EVERY reuse, which is what keeps family rule 146 true: a # rolling channel refreshes itself, so skipping a build must never mean @@ -915,12 +912,11 @@ jobs: fetch-depth: 0 # --- derived values, one line (milestone 313) ------------------------ - # These stopped being shadow output at step 3: `tag` is published on - # main and `revision` decides whether the build below runs at all. This - # step prints all three anyway, because the load-bearing steps each - # print only the one they use, and on dev the date tag is computed - # nowhere else. When a build is skipped or a pin looks wrong, this is - # the line that says what the commit derived. + # These stopped being shadow output at step 3. `revision` decides + # whether the build below runs at all and `version` is what the image + # reports about itself; the load-bearing steps each print only the one + # they use, so this is the only place the pair appears together. When a + # build is skipped, this is the line that says what the commit derived. # # Still diagnostic, so it still must not fail the build — no `set -e`, # and every derivation falls back to UNAVAILABLE. A broken echo must @@ -931,8 +927,9 @@ jobs: # STILL. If web moves, its path set is too wide. # * a push touching only docs moves nothing. # * a push touching the extension moves the extension AND web, since - # web bakes in the XPI. If web does not move, its set is too narrow - # — the direction that serves stale bytes on a pin. + # web bakes in the XPI. If web does not move, its set is too narrow: + # the reuse check hits, and the channel serves a web image bundling + # the PREVIOUS XPI while the freshly signed one is orphaned (#3156). # * dev and main derive the same values for the same source. - name: Report the derived artifact version run: | @@ -989,15 +986,13 @@ jobs: # that skips a build that was actually needed; the worst case is paying # for a build we could have avoided. # - # BASE-IMAGE FRESHNESS, decided rather than left implicit: an artifact - # whose source stops moving stops picking up base-image updates under - # its pinned tag. That is what a pin MEANS — a date tag has to keep - # serving the bytes it served (fabledcurator-agent:2026.7.17 still - # resolves to July's image), or it is not a pin — and family rule - # 145 already says where the refresh goes instead: a rebuild with - # different contents publishes only the MOVING tag, never the immutable - # one. A scheduled channel-only refresh is tracked separately (#3154); - # it does not belong in the push path. + # BASE-IMAGE FRESHNESS: an artifact whose source stops moving stops + # picking up base-image updates. Milestone 318 removed the argument this + # used to need rather than answering it — with no version tags there is + # no immutable name a refresh could contradict, and rule 145 already + # allows a rebuild with different contents to republish a MOVING tag. + # So a refresh is just a build. A scheduled channel-only one is tracked + # separately (#3154); it does not belong in the push path. - name: Is this content already published? id: reuse env: @@ -1064,8 +1059,8 @@ jobs: # Registry-side manifest copy: no layer transfer, no local daemon, no # rebuild. Each -t becomes another reference to the SAME manifest the - # channel tag already holds, so :c- and the date pin are - # byte-identical to what is published rather than a lookalike rebuild. + # channel tag already holds, so :c- is byte-identical to what is + # published rather than a lookalike rebuild. # # Runs on EVERY reuse, which is what keeps family rule 146 true: a # rolling channel refreshes itself, so skipping a build must never mean diff --git a/README.md b/README.md index 915ffa5..5531d81 100644 --- a/README.md +++ b/README.md @@ -10,6 +10,35 @@ In production. `main` is continuously deployed — every merge to `main` builds and publishes `:latest` images, so whatever is on `main` is what is running. Day-to-day work happens on `dev`, which publishes `:dev` images. +## Versions and tags + +Three image tags exist, and no others: + +| Tag | Branch | Meaning | +| --- | --- | --- | +| `:latest` | `main` | Production. Moves on every merge. | +| `:c-` | `main` | Immutable — the rollback unit, all three images together. | +| `:dev` | `dev` | The rolling test channel. Moves on every push. | + +There are deliberately **no version tags**. Nothing pins one, and a per-build +name nobody reads is upkeep for a model FC does not run (family rule 145; the +reasoning is note #3127 §5). Rolling back is `docker pull …:c-`. + +Each artifact still has a version, derived rather than chosen: the commit time +of the newest change to that artifact's *own* shipped files, as +`YYYY.MM.DD.HHMM` UTC (rule 148). Four artifacts, four independent versions — +a push touching only `agent/` re-versions the agent and leaves web and ml +alone, and CI skips the builds whose content did not move. + +Because no registry name carries it, the running instance's own report is the +only answer to "which build is this?". The foot of Settings shows +`FabledCurator 2026.08.29.0201 · dev`, and `/api/health` returns the same two +fields. + +Release tags are optional bookmarks — FC went twelve weeks without one and +nothing was wrong. Pushing `v` publishes a Forgejo release listing the +commits since the previous tag; it builds no image. + ## What's in here Five deployable pieces, built by `.forgejo/workflows/build.yml`: @@ -52,9 +81,10 @@ FabledCurator is designed to run inside a self-hosted homelab environment over p ## CI / Forgejo setup -Three workflows: `ci.yml` (lint, extension-version check, backend unit tests, +Four workflows: `ci.yml` (lint, extension-version check, backend unit tests, frontend build, integration), `extension.yml` (extension lint, vitest, XPI -content verification), and `build.yml` (sign + publish). +content verification), `build.yml` (sign + publish), and `release.yml`, which +runs only on a `v*` tag and publishes a changelog without building anything. **The toolchain each job runs in is its `container.image`, not its `runs-on` label.** `runs-on: python-ci` only schedules the job onto a runner; every job @@ -71,8 +101,12 @@ The repo expects one secret: Generate at https://git.fabledsword.com/user/settings/applications. The injected `GITHUB_TOKEN` cannot be used because it lacks `write:package`. -AMO signing additionally needs `MOZILLA_AMO_JWT_KEY` / `MOZILLA_AMO_JWT_SECRET`; it runs on -`main` only and is cached per version, since AMO rejects a re-signed version. +AMO signing additionally needs `MOZILLA_AMO_JWT_KEY` / `MOZILLA_AMO_JWT_SECRET`. +It runs on **both** channels and is cached per version: because the version is +derived from commit time, `dev` and `main` derive the same number for the same +source, so `main` finds `dev`'s signature already cached and makes no second AMO +call. That cache is why signing must be one-shot — AMO rejects a re-signed +version. ## License diff --git a/ci-requirements.md b/ci-requirements.md index 0be64a8..f57e46d 100644 --- a/ci-requirements.md +++ b/ci-requirements.md @@ -9,14 +9,17 @@ git.fabledsword.com/bvandeusen/ci-python:3.14 ## Image deps used - python 3.14 -- ruff (analyzer for `backend/`, `tests/`, `alembic/`) +- ruff (analyzer for `backend/`, `tests/`, `alembic/`, `agent/`, `scripts/`) - node (frontend job: `npm install` + vitest + vite build) -- docker CLI + buildx (`.forgejo/workflows/build.yml`: build-web, build-ml — Fabled-Git registry push) +- docker CLI + buildx (`.forgejo/workflows/build.yml`: build-web, build-ml, build-agent — Fabled-Git registry push, and `imagetools inspect`/`create` for the reuse path) ## Secondary runtime image node:24-bookworm-slim — `.forgejo/workflows/extension.yml` only. +`.forgejo/workflows/release.yml` runs on `ci-python:3.14` like everything else +and installs nothing: it needs git and stdlib python, and builds no image. + The extension lane is the one job that does NOT run on `ci-python:3.14`: it needs a current Node for `web-ext` and vitest and nothing Python at all. Kept on the upstream slim image rather than adding a Node toolchain to `ci-python`, @@ -54,13 +57,19 @@ per `docs/process.md`'s "add deps to the image when used by >1 project". shims to production code — the libs ship as `background.scripts`, not ES modules, so the specs exercise exactly the bytes packaged into the XPI. - **`extension/scripts/packaging.sh` is the single definition of what ships - inside the XPI.** Two consumers read from it rather than keeping their own - copy: web-ext's `--ignore-files` (`extension/package.json`), and the `git log` - pathspec inside the script's own version derivation. It was three until - 2026-08-27 — `ci.yml`'s `extension-version` guard held the third and went when - the manual bump it guarded did (milestone 271 step 5). Hand-kept copies of - that one fact is what allowed issue #2397, so `extension/test/version.spec.js` - asserts no workflow has reintroduced a literal `:(exclude)extension/…`. + inside the XPI.** Three consumers read from it rather than keeping their own + copy: web-ext's `--ignore-files` (`extension/package.json`), the `git log` + pathspec inside the script's own version derivation, and `scripts/artifacts.sh`, + which appends the extension's set to web's because the web image bundles the + signed XPI. Hand-kept copies of that one fact is what allowed issue #2397, so + `extension/test/version.spec.js` asserts no workflow has reintroduced a + literal `:(exclude)extension/…`. +- **Packaged and version-relevant are two different sets** (#3156). `scripts/` + is excluded from the XPI and is NOT excluded from the version derivation, + because `packaging.sh` decides the version string stamped into the packaged + `manifest.json`. The membership test is *"can changing this file change the + published bytes?"*, not *"is this file copied in?"* — which is why the script + keeps two lists rather than one. - **The shipped extension version is derived, not committed.** It is the commit TIME of the newest packaged-extension change (minutes since 2020-01-01, per family rule 149 — never a commit count, which orders by branch rather than by @@ -68,20 +77,55 @@ per `docs/process.md`'s "add deps to the image when used by >1 project". `extension/manifest.json` + `package.json` in the working tree before signing; the stamp is never committed. Treat the version in the repo as a base: only its MAJOR.MINOR is read, and its patch component is inert. -- Every job that calls `packaging.sh version` checks out with `fetch-depth: 0` — - `build.yml`'s `sign-extension` and `build-web`, and `ci.yml`'s - `extension-version`. A depth-1 clone sees one commit and derives a wrong, - too-low value **rather than failing**, so the full-history checkout is - load-bearing rather than incidental. -- **`FC_CHANNEL` is a build arg, not a runtime setting.** `build.yml` passes - `dev` / `main` to the web image only (the ml and agent images have nothing to - report it to), and `/api/extension/manifest` reports it beside the version so - an install can be traced to a channel. It is declared LAST in the Dockerfile - on purpose: an ARG invalidates every layer below it, and this is the one value - that differs between the dev and main builds of identical source, so placing - it earlier would stop the two channels ever sharing a cached `pip install`. - Empty by default — a local build then reports no channel at all rather than - claiming one. +- Every job that derives anything checks out with `fetch-depth: 0` — all four + `build.yml` jobs, `ci.yml`'s `extension-version` and `backend-lint-and-test` + (for `tests/test_artifact_paths.py` and `test_artifact_identity.py`), and + `release.yml`, which additionally walks the tag graph. A depth-1 clone sees + one commit and derives a wrong, too-low value **rather than failing**, so the + full-history checkout is load-bearing rather than incidental. +- **`scripts/artifacts.sh` is the same shape one level up: one definition per + artifact of what it is built from, and the two values derived from it.** + `revision` (12 hex of the newest commit touching that set) and `version` + (`YYYY.MM.DD.HHMM` UTC, rule 148). Four artifacts, four independent answers, + so a push touching only `agent/` leaves web and ml alone. + `tests/test_artifact_paths.py` reads each Dockerfile and asserts every COPY + source is covered, so adding a COPY without updating the script fails CI. +- **A file that DECIDES an artifact's identity belongs in its set even though it + is copied into nothing** — `packaging.sh` for the extension and web (#3156), + and `artifacts.sh` itself for web (#3202), which decides the `FC_VERSION` + baked into that image. Only web needs the second entry: every artifact stamps + a revision, but a revision has a backstop (a changed derivation stops matching + the published label and forces a rebuild) and a version has none, because + nothing compares it to anything. `tests/test_artifact_paths.py`'s `DERIVERS` + table is the guard. +- **Builds are skipped when the content is already published.** Each image + carries its revision as an `fc.revision` LABEL, and `build.yml` reads that + label back off the moving channel tag (`imagetools inspect --format`). Equal + to the derived revision means the bytes are already published, so the job + repoints the remaining tags at the existing manifest instead of rebuilding. + Two things this depends on: an inspect that errors for ANY reason reads as a + MISS so no needed build is ever skipped, and the repoint must EXCLUDE the + source tag — `imagetools create` wraps its source in a manifest index, and + config labels do not resolve through an index, so writing the channel tag + from itself destroys the label the next run reads (#3183). +- **`FC_CHANNEL` and `FC_VERSION` are build args, not runtime settings.** + `build.yml` passes them to the web image only — the ml and agent images have + nothing to report them to. `/api/health` returns both, the foot of Settings + renders them, and `/api/extension/manifest` reports the channel beside the + extension version so an install can be traced to a channel. With no version image tags, that + self-report is the ONLY answer to "which build is this?" — which is why a + missing version renders `unknown` rather than a blank: an empty footer reads + as "no version", a different and false claim. + Both are declared LAST in the Dockerfile on purpose: an ARG invalidates every + layer below it, and these are the values that differ between the dev and main + builds of identical source, so placing them earlier would stop the two + channels ever sharing a cached `pip install`. Empty by default — a local build + then reports nothing rather than claiming a channel it is not on. +- **The channel is never folded into the version.** A `-dev` suffix makes the + extension's per-segment `parseInt` comparator read that segment as 0, so every + dev build compares equal to every other — issue #2993 exactly (rule 149). + `frontend/test/systemBuild.spec.js` pins the rendered version to the bare + number. - Callers MUST `set -f` before substituting the script's output. Without it the shell expands `test/**` against the working tree and silently narrows the pattern to whatever files exist at that moment — a failure that looks like