From d38585ed94db2192e40a6233ae802de9ed07a54d Mon Sep 17 00:00:00 2001 From: Bryan Van Deusen Date: Sat, 29 Aug 2026 00:09:52 -0400 Subject: [PATCH] docs: true up the tag scheme against what the pipeline publishes (318 step 9) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A doc describing a tag scheme the pipeline stopped using is worse than no doc — it is a confident wrong answer, and #3159 named it as the drift that had just bitten rule 148. build.yml (each edit ×3, once per build job): * The derived-values block claimed three values including a `tag` subcommand that no longer exists, and said "on dev the date tag is computed nowhere else". There are two values and no date tag. * The BASE-IMAGE FRESHNESS paragraph argued from a pin that no longer exists — "a date tag has to keep serving the bytes it served (fabledcurator:2026.7.17 still resolves to July's image)". Milestone 318 removed the argument's premise rather than answering it: with no version tags there is no immutable name a refresh could contradict, and rule 145 already allows a moving tag to be republished. Deleted the argument, kept a line saying why there is nothing to argue about (#3159's phrasing: the workflow becomes obviously correct instead of carefully correct). * The repoint comment said ":c- and the date pin". There is no date pin. * "its set is too narrow — the direction that serves stale bytes on a pin" now names the failure that actually exists: the reuse check hits and the channel serves a web image bundling the previous XPI (#3156). README gains a "Versions and tags" section — the three tags, why there is no fourth, and the fact that with no registry name carrying it, the instance's own report is the only answer to "which build is this?". Also fixes two stale claims: three workflows (four since release.yml) and "AMO signing runs on main only", which has been false since milestone 271 step 6 signed on both channels. ci-requirements.md was missing milestones 313 and 318 entirely: no artifacts.sh, no label-keyed reuse, no FC_VERSION. Added those, plus the #3156/#3202 membership test (can changing this file change the published bytes — not is it copied in), and corrected packaging.sh's "two consumers" to three now that artifacts.sh reads it. docker-compose.yml needed nothing — it names `:dev` directly and never offered a pin target. Note #3127 and rule 145's relation notes are updated in Scribe. --- .forgejo/workflows/build.yml | 113 +++++++++++++++++------------------ README.md | 42 +++++++++++-- ci-requirements.md | 90 +++++++++++++++++++++------- 3 files changed, 159 insertions(+), 86 deletions(-) diff --git a/.forgejo/workflows/build.yml b/.forgejo/workflows/build.yml index 23ddd18..72660ca 100644 --- a/.forgejo/workflows/build.yml +++ b/.forgejo/workflows/build.yml @@ -116,8 +116,9 @@ jobs: # STILL. If web moves, its path set is too wide. # * a push touching only docs moves nothing. # * a push touching the extension moves the extension AND web, since - # web bakes in the XPI. If web does not move, its set is too narrow - # — the direction that serves stale bytes on a pin. + # web bakes in the XPI. If web does not move, its set is too narrow: + # the reuse check hits, and the channel serves a web image bundling + # the PREVIOUS XPI while the freshly signed one is orphaned (#3156). # * dev and main derive the same values for the same source. - name: Shadow — derived artifact version (informational) run: | @@ -345,12 +346,11 @@ jobs: fetch-depth: 0 # --- derived values, one line (milestone 313) ------------------------ - # These stopped being shadow output at step 3: `tag` is published on - # main and `revision` decides whether the build below runs at all. This - # step prints all three anyway, because the load-bearing steps each - # print only the one they use, and on dev the date tag is computed - # nowhere else. When a build is skipped or a pin looks wrong, this is - # the line that says what the commit derived. + # These stopped being shadow output at step 3. `revision` decides + # whether the build below runs at all and `version` is what the image + # reports about itself; the load-bearing steps each print only the one + # they use, so this is the only place the pair appears together. When a + # build is skipped, this is the line that says what the commit derived. # # Still diagnostic, so it still must not fail the build — no `set -e`, # and every derivation falls back to UNAVAILABLE. A broken echo must @@ -361,8 +361,9 @@ jobs: # STILL. If web moves, its path set is too wide. # * a push touching only docs moves nothing. # * a push touching the extension moves the extension AND web, since - # web bakes in the XPI. If web does not move, its set is too narrow - # — the direction that serves stale bytes on a pin. + # web bakes in the XPI. If web does not move, its set is too narrow: + # the reuse check hits, and the channel serves a web image bundling + # the PREVIOUS XPI while the freshly signed one is orphaned (#3156). # * dev and main derive the same values for the same source. - name: Report the derived artifact version run: | @@ -463,15 +464,13 @@ jobs: # that skips a build that was actually needed; the worst case is paying # for a build we could have avoided. # - # BASE-IMAGE FRESHNESS, decided rather than left implicit: an artifact - # whose source stops moving stops picking up base-image updates under - # its pinned tag. That is what a pin MEANS — a date tag has to keep - # serving the bytes it served (fabledcurator:2026.7.17 still - # resolves to July's image), or it is not a pin — and family rule - # 145 already says where the refresh goes instead: a rebuild with - # different contents publishes only the MOVING tag, never the immutable - # one. A scheduled channel-only refresh is tracked separately (#3154); - # it does not belong in the push path. + # BASE-IMAGE FRESHNESS: an artifact whose source stops moving stops + # picking up base-image updates. Milestone 318 removed the argument this + # used to need rather than answering it — with no version tags there is + # no immutable name a refresh could contradict, and rule 145 already + # allows a rebuild with different contents to republish a MOVING tag. + # So a refresh is just a build. A scheduled channel-only one is tracked + # separately (#3154); it does not belong in the push path. - name: Is this content already published? id: reuse env: @@ -624,8 +623,8 @@ jobs: # Registry-side manifest copy: no layer transfer, no local daemon, no # rebuild. Each -t becomes another reference to the SAME manifest the - # channel tag already holds, so :c- and the date pin are - # byte-identical to what is published rather than a lookalike rebuild. + # channel tag already holds, so :c- is byte-identical to what is + # published rather than a lookalike rebuild. # # Runs on EVERY reuse, which is what keeps family rule 146 true: a # rolling channel refreshes itself, so skipping a build must never mean @@ -691,12 +690,11 @@ jobs: fetch-depth: 0 # --- derived values, one line (milestone 313) ------------------------ - # These stopped being shadow output at step 3: `tag` is published on - # main and `revision` decides whether the build below runs at all. This - # step prints all three anyway, because the load-bearing steps each - # print only the one they use, and on dev the date tag is computed - # nowhere else. When a build is skipped or a pin looks wrong, this is - # the line that says what the commit derived. + # These stopped being shadow output at step 3. `revision` decides + # whether the build below runs at all and `version` is what the image + # reports about itself; the load-bearing steps each print only the one + # they use, so this is the only place the pair appears together. When a + # build is skipped, this is the line that says what the commit derived. # # Still diagnostic, so it still must not fail the build — no `set -e`, # and every derivation falls back to UNAVAILABLE. A broken echo must @@ -707,8 +705,9 @@ jobs: # STILL. If web moves, its path set is too wide. # * a push touching only docs moves nothing. # * a push touching the extension moves the extension AND web, since - # web bakes in the XPI. If web does not move, its set is too narrow - # — the direction that serves stale bytes on a pin. + # web bakes in the XPI. If web does not move, its set is too narrow: + # the reuse check hits, and the channel serves a web image bundling + # the PREVIOUS XPI while the freshly signed one is orphaned (#3156). # * dev and main derive the same values for the same source. - name: Report the derived artifact version run: | @@ -770,15 +769,13 @@ jobs: # that skips a build that was actually needed; the worst case is paying # for a build we could have avoided. # - # BASE-IMAGE FRESHNESS, decided rather than left implicit: an artifact - # whose source stops moving stops picking up base-image updates under - # its pinned tag. That is what a pin MEANS — a date tag has to keep - # serving the bytes it served (fabledcurator-ml:2026.7.17 still - # resolves to July's image), or it is not a pin — and family rule - # 145 already says where the refresh goes instead: a rebuild with - # different contents publishes only the MOVING tag, never the immutable - # one. A scheduled channel-only refresh is tracked separately (#3154); - # it does not belong in the push path. + # BASE-IMAGE FRESHNESS: an artifact whose source stops moving stops + # picking up base-image updates. Milestone 318 removed the argument this + # used to need rather than answering it — with no version tags there is + # no immutable name a refresh could contradict, and rule 145 already + # allows a rebuild with different contents to republish a MOVING tag. + # So a refresh is just a build. A scheduled channel-only one is tracked + # separately (#3154); it does not belong in the push path. - name: Is this content already published? id: reuse env: @@ -845,8 +842,8 @@ jobs: # Registry-side manifest copy: no layer transfer, no local daemon, no # rebuild. Each -t becomes another reference to the SAME manifest the - # channel tag already holds, so :c- and the date pin are - # byte-identical to what is published rather than a lookalike rebuild. + # channel tag already holds, so :c- is byte-identical to what is + # published rather than a lookalike rebuild. # # Runs on EVERY reuse, which is what keeps family rule 146 true: a # rolling channel refreshes itself, so skipping a build must never mean @@ -915,12 +912,11 @@ jobs: fetch-depth: 0 # --- derived values, one line (milestone 313) ------------------------ - # These stopped being shadow output at step 3: `tag` is published on - # main and `revision` decides whether the build below runs at all. This - # step prints all three anyway, because the load-bearing steps each - # print only the one they use, and on dev the date tag is computed - # nowhere else. When a build is skipped or a pin looks wrong, this is - # the line that says what the commit derived. + # These stopped being shadow output at step 3. `revision` decides + # whether the build below runs at all and `version` is what the image + # reports about itself; the load-bearing steps each print only the one + # they use, so this is the only place the pair appears together. When a + # build is skipped, this is the line that says what the commit derived. # # Still diagnostic, so it still must not fail the build — no `set -e`, # and every derivation falls back to UNAVAILABLE. A broken echo must @@ -931,8 +927,9 @@ jobs: # STILL. If web moves, its path set is too wide. # * a push touching only docs moves nothing. # * a push touching the extension moves the extension AND web, since - # web bakes in the XPI. If web does not move, its set is too narrow - # — the direction that serves stale bytes on a pin. + # web bakes in the XPI. If web does not move, its set is too narrow: + # the reuse check hits, and the channel serves a web image bundling + # the PREVIOUS XPI while the freshly signed one is orphaned (#3156). # * dev and main derive the same values for the same source. - name: Report the derived artifact version run: | @@ -989,15 +986,13 @@ jobs: # that skips a build that was actually needed; the worst case is paying # for a build we could have avoided. # - # BASE-IMAGE FRESHNESS, decided rather than left implicit: an artifact - # whose source stops moving stops picking up base-image updates under - # its pinned tag. That is what a pin MEANS — a date tag has to keep - # serving the bytes it served (fabledcurator-agent:2026.7.17 still - # resolves to July's image), or it is not a pin — and family rule - # 145 already says where the refresh goes instead: a rebuild with - # different contents publishes only the MOVING tag, never the immutable - # one. A scheduled channel-only refresh is tracked separately (#3154); - # it does not belong in the push path. + # BASE-IMAGE FRESHNESS: an artifact whose source stops moving stops + # picking up base-image updates. Milestone 318 removed the argument this + # used to need rather than answering it — with no version tags there is + # no immutable name a refresh could contradict, and rule 145 already + # allows a rebuild with different contents to republish a MOVING tag. + # So a refresh is just a build. A scheduled channel-only one is tracked + # separately (#3154); it does not belong in the push path. - name: Is this content already published? id: reuse env: @@ -1064,8 +1059,8 @@ jobs: # Registry-side manifest copy: no layer transfer, no local daemon, no # rebuild. Each -t becomes another reference to the SAME manifest the - # channel tag already holds, so :c- and the date pin are - # byte-identical to what is published rather than a lookalike rebuild. + # channel tag already holds, so :c- is byte-identical to what is + # published rather than a lookalike rebuild. # # Runs on EVERY reuse, which is what keeps family rule 146 true: a # rolling channel refreshes itself, so skipping a build must never mean diff --git a/README.md b/README.md index 915ffa5..5531d81 100644 --- a/README.md +++ b/README.md @@ -10,6 +10,35 @@ In production. `main` is continuously deployed — every merge to `main` builds and publishes `:latest` images, so whatever is on `main` is what is running. Day-to-day work happens on `dev`, which publishes `:dev` images. +## Versions and tags + +Three image tags exist, and no others: + +| Tag | Branch | Meaning | +| --- | --- | --- | +| `:latest` | `main` | Production. Moves on every merge. | +| `:c-` | `main` | Immutable — the rollback unit, all three images together. | +| `:dev` | `dev` | The rolling test channel. Moves on every push. | + +There are deliberately **no version tags**. Nothing pins one, and a per-build +name nobody reads is upkeep for a model FC does not run (family rule 145; the +reasoning is note #3127 §5). Rolling back is `docker pull …:c-`. + +Each artifact still has a version, derived rather than chosen: the commit time +of the newest change to that artifact's *own* shipped files, as +`YYYY.MM.DD.HHMM` UTC (rule 148). Four artifacts, four independent versions — +a push touching only `agent/` re-versions the agent and leaves web and ml +alone, and CI skips the builds whose content did not move. + +Because no registry name carries it, the running instance's own report is the +only answer to "which build is this?". The foot of Settings shows +`FabledCurator 2026.08.29.0201 · dev`, and `/api/health` returns the same two +fields. + +Release tags are optional bookmarks — FC went twelve weeks without one and +nothing was wrong. Pushing `v` publishes a Forgejo release listing the +commits since the previous tag; it builds no image. + ## What's in here Five deployable pieces, built by `.forgejo/workflows/build.yml`: @@ -52,9 +81,10 @@ FabledCurator is designed to run inside a self-hosted homelab environment over p ## CI / Forgejo setup -Three workflows: `ci.yml` (lint, extension-version check, backend unit tests, +Four workflows: `ci.yml` (lint, extension-version check, backend unit tests, frontend build, integration), `extension.yml` (extension lint, vitest, XPI -content verification), and `build.yml` (sign + publish). +content verification), `build.yml` (sign + publish), and `release.yml`, which +runs only on a `v*` tag and publishes a changelog without building anything. **The toolchain each job runs in is its `container.image`, not its `runs-on` label.** `runs-on: python-ci` only schedules the job onto a runner; every job @@ -71,8 +101,12 @@ The repo expects one secret: Generate at https://git.fabledsword.com/user/settings/applications. The injected `GITHUB_TOKEN` cannot be used because it lacks `write:package`. -AMO signing additionally needs `MOZILLA_AMO_JWT_KEY` / `MOZILLA_AMO_JWT_SECRET`; it runs on -`main` only and is cached per version, since AMO rejects a re-signed version. +AMO signing additionally needs `MOZILLA_AMO_JWT_KEY` / `MOZILLA_AMO_JWT_SECRET`. +It runs on **both** channels and is cached per version: because the version is +derived from commit time, `dev` and `main` derive the same number for the same +source, so `main` finds `dev`'s signature already cached and makes no second AMO +call. That cache is why signing must be one-shot — AMO rejects a re-signed +version. ## License diff --git a/ci-requirements.md b/ci-requirements.md index 0be64a8..f57e46d 100644 --- a/ci-requirements.md +++ b/ci-requirements.md @@ -9,14 +9,17 @@ git.fabledsword.com/bvandeusen/ci-python:3.14 ## Image deps used - python 3.14 -- ruff (analyzer for `backend/`, `tests/`, `alembic/`) +- ruff (analyzer for `backend/`, `tests/`, `alembic/`, `agent/`, `scripts/`) - node (frontend job: `npm install` + vitest + vite build) -- docker CLI + buildx (`.forgejo/workflows/build.yml`: build-web, build-ml — Fabled-Git registry push) +- docker CLI + buildx (`.forgejo/workflows/build.yml`: build-web, build-ml, build-agent — Fabled-Git registry push, and `imagetools inspect`/`create` for the reuse path) ## Secondary runtime image node:24-bookworm-slim — `.forgejo/workflows/extension.yml` only. +`.forgejo/workflows/release.yml` runs on `ci-python:3.14` like everything else +and installs nothing: it needs git and stdlib python, and builds no image. + The extension lane is the one job that does NOT run on `ci-python:3.14`: it needs a current Node for `web-ext` and vitest and nothing Python at all. Kept on the upstream slim image rather than adding a Node toolchain to `ci-python`, @@ -54,13 +57,19 @@ per `docs/process.md`'s "add deps to the image when used by >1 project". shims to production code — the libs ship as `background.scripts`, not ES modules, so the specs exercise exactly the bytes packaged into the XPI. - **`extension/scripts/packaging.sh` is the single definition of what ships - inside the XPI.** Two consumers read from it rather than keeping their own - copy: web-ext's `--ignore-files` (`extension/package.json`), and the `git log` - pathspec inside the script's own version derivation. It was three until - 2026-08-27 — `ci.yml`'s `extension-version` guard held the third and went when - the manual bump it guarded did (milestone 271 step 5). Hand-kept copies of - that one fact is what allowed issue #2397, so `extension/test/version.spec.js` - asserts no workflow has reintroduced a literal `:(exclude)extension/…`. + inside the XPI.** Three consumers read from it rather than keeping their own + copy: web-ext's `--ignore-files` (`extension/package.json`), the `git log` + pathspec inside the script's own version derivation, and `scripts/artifacts.sh`, + which appends the extension's set to web's because the web image bundles the + signed XPI. Hand-kept copies of that one fact is what allowed issue #2397, so + `extension/test/version.spec.js` asserts no workflow has reintroduced a + literal `:(exclude)extension/…`. +- **Packaged and version-relevant are two different sets** (#3156). `scripts/` + is excluded from the XPI and is NOT excluded from the version derivation, + because `packaging.sh` decides the version string stamped into the packaged + `manifest.json`. The membership test is *"can changing this file change the + published bytes?"*, not *"is this file copied in?"* — which is why the script + keeps two lists rather than one. - **The shipped extension version is derived, not committed.** It is the commit TIME of the newest packaged-extension change (minutes since 2020-01-01, per family rule 149 — never a commit count, which orders by branch rather than by @@ -68,20 +77,55 @@ per `docs/process.md`'s "add deps to the image when used by >1 project". `extension/manifest.json` + `package.json` in the working tree before signing; the stamp is never committed. Treat the version in the repo as a base: only its MAJOR.MINOR is read, and its patch component is inert. -- Every job that calls `packaging.sh version` checks out with `fetch-depth: 0` — - `build.yml`'s `sign-extension` and `build-web`, and `ci.yml`'s - `extension-version`. A depth-1 clone sees one commit and derives a wrong, - too-low value **rather than failing**, so the full-history checkout is - load-bearing rather than incidental. -- **`FC_CHANNEL` is a build arg, not a runtime setting.** `build.yml` passes - `dev` / `main` to the web image only (the ml and agent images have nothing to - report it to), and `/api/extension/manifest` reports it beside the version so - an install can be traced to a channel. It is declared LAST in the Dockerfile - on purpose: an ARG invalidates every layer below it, and this is the one value - that differs between the dev and main builds of identical source, so placing - it earlier would stop the two channels ever sharing a cached `pip install`. - Empty by default — a local build then reports no channel at all rather than - claiming one. +- Every job that derives anything checks out with `fetch-depth: 0` — all four + `build.yml` jobs, `ci.yml`'s `extension-version` and `backend-lint-and-test` + (for `tests/test_artifact_paths.py` and `test_artifact_identity.py`), and + `release.yml`, which additionally walks the tag graph. A depth-1 clone sees + one commit and derives a wrong, too-low value **rather than failing**, so the + full-history checkout is load-bearing rather than incidental. +- **`scripts/artifacts.sh` is the same shape one level up: one definition per + artifact of what it is built from, and the two values derived from it.** + `revision` (12 hex of the newest commit touching that set) and `version` + (`YYYY.MM.DD.HHMM` UTC, rule 148). Four artifacts, four independent answers, + so a push touching only `agent/` leaves web and ml alone. + `tests/test_artifact_paths.py` reads each Dockerfile and asserts every COPY + source is covered, so adding a COPY without updating the script fails CI. +- **A file that DECIDES an artifact's identity belongs in its set even though it + is copied into nothing** — `packaging.sh` for the extension and web (#3156), + and `artifacts.sh` itself for web (#3202), which decides the `FC_VERSION` + baked into that image. Only web needs the second entry: every artifact stamps + a revision, but a revision has a backstop (a changed derivation stops matching + the published label and forces a rebuild) and a version has none, because + nothing compares it to anything. `tests/test_artifact_paths.py`'s `DERIVERS` + table is the guard. +- **Builds are skipped when the content is already published.** Each image + carries its revision as an `fc.revision` LABEL, and `build.yml` reads that + label back off the moving channel tag (`imagetools inspect --format`). Equal + to the derived revision means the bytes are already published, so the job + repoints the remaining tags at the existing manifest instead of rebuilding. + Two things this depends on: an inspect that errors for ANY reason reads as a + MISS so no needed build is ever skipped, and the repoint must EXCLUDE the + source tag — `imagetools create` wraps its source in a manifest index, and + config labels do not resolve through an index, so writing the channel tag + from itself destroys the label the next run reads (#3183). +- **`FC_CHANNEL` and `FC_VERSION` are build args, not runtime settings.** + `build.yml` passes them to the web image only — the ml and agent images have + nothing to report them to. `/api/health` returns both, the foot of Settings + renders them, and `/api/extension/manifest` reports the channel beside the + extension version so an install can be traced to a channel. With no version image tags, that + self-report is the ONLY answer to "which build is this?" — which is why a + missing version renders `unknown` rather than a blank: an empty footer reads + as "no version", a different and false claim. + Both are declared LAST in the Dockerfile on purpose: an ARG invalidates every + layer below it, and these are the values that differ between the dev and main + builds of identical source, so placing them earlier would stop the two + channels ever sharing a cached `pip install`. Empty by default — a local build + then reports nothing rather than claiming a channel it is not on. +- **The channel is never folded into the version.** A `-dev` suffix makes the + extension's per-segment `parseInt` comparator read that segment as 0, so every + dev build compares equal to every other — issue #2993 exactly (rule 149). + `frontend/test/systemBuild.spec.js` pins the rendered version to the bare + number. - Callers MUST `set -f` before substituting the script's output. Without it the shell expands `test/**` against the working tree and silently narrows the pattern to whatever files exist at that moment — a failure that looks like