bvandeusen/FabledCurator
1
0
Fork 0
Code Issues 2 Pull Requests Actions Packages Projects Releases 12 Wiki Activity

dev→main: milestone 318 steps 5–7 and 9, plus #3202 #240

Merged
bvandeusen merged 5 commits from dev into main 2026-08-29 00:14:34 -04:00
Conversation 0 Commits 5 Files Changed 19
+159 -86
3 changed files with 159 additions and 86 deletions
Download Patch File Download Diff File Expand all files Collapse all files
Showing only changes of commit d38585ed94 - Show all commits
.forgejo/workflows/build.yml
+54 -59
View File
@@ -116,8 +116,9 @@ jobs:
# STILL. If web moves, its path set is too wide.
# * a push touching only docs moves nothing.
# * a push touching the extension moves the extension AND web, since
# web bakes in the XPI. If web does not move, its set is too narrow
# — the direction that serves stale bytes on a pin.
# web bakes in the XPI. If web does not move, its set is too narrow:
# the reuse check hits, and the channel serves a web image bundling
# the PREVIOUS XPI while the freshly signed one is orphaned (#3156).
# * dev and main derive the same values for the same source.
- name: Shadow — derived artifact version (informational)
run: |
@@ -345,12 +346,11 @@ jobs:
fetch-depth: 0
# --- derived values, one line (milestone 313) ------------------------
# These stopped being shadow output at step 3: `tag` is published on
# main and `revision` decides whether the build below runs at all. This
# step prints all three anyway, because the load-bearing steps each
# print only the one they use, and on dev the date tag is computed
# nowhere else. When a build is skipped or a pin looks wrong, this is
# the line that says what the commit derived.
# These stopped being shadow output at step 3. `revision` decides
# whether the build below runs at all and `version` is what the image
# reports about itself; the load-bearing steps each print only the one
# they use, so this is the only place the pair appears together. When a
# build is skipped, this is the line that says what the commit derived.
#
# Still diagnostic, so it still must not fail the build — no `set -e`,
# and every derivation falls back to UNAVAILABLE. A broken echo must
@@ -361,8 +361,9 @@ jobs:
# STILL. If web moves, its path set is too wide.
# * a push touching only docs moves nothing.
# * a push touching the extension moves the extension AND web, since
# web bakes in the XPI. If web does not move, its set is too narrow
# — the direction that serves stale bytes on a pin.
# web bakes in the XPI. If web does not move, its set is too narrow:
# the reuse check hits, and the channel serves a web image bundling
# the PREVIOUS XPI while the freshly signed one is orphaned (#3156).
# * dev and main derive the same values for the same source.
- name: Report the derived artifact version
run: |
@@ -463,15 +464,13 @@ jobs:
# that skips a build that was actually needed; the worst case is paying
# for a build we could have avoided.
#
# BASE-IMAGE FRESHNESS, decided rather than left implicit: an artifact
# whose source stops moving stops picking up base-image updates under
# its pinned tag. That is what a pin MEANS — a date tag has to keep
# serving the bytes it served (fabledcurator:2026.7.17 still
# resolves to July's image), or it is not a pin — and family rule
# 145 already says where the refresh goes instead: a rebuild with
# different contents publishes only the MOVING tag, never the immutable
# one. A scheduled channel-only refresh is tracked separately (#3154);
# it does not belong in the push path.
# BASE-IMAGE FRESHNESS: an artifact whose source stops moving stops
# picking up base-image updates. Milestone 318 removed the argument this
# used to need rather than answering it — with no version tags there is
# no immutable name a refresh could contradict, and rule 145 already
# allows a rebuild with different contents to republish a MOVING tag.
# So a refresh is just a build. A scheduled channel-only one is tracked
# separately (#3154); it does not belong in the push path.
- name: Is this content already published?
id: reuse
env:
@@ -624,8 +623,8 @@ jobs:
# Registry-side manifest copy: no layer transfer, no local daemon, no
# rebuild. Each -t becomes another reference to the SAME manifest the
# channel tag already holds, so :c-<sha> and the date pin are
# byte-identical to what is published rather than a lookalike rebuild.
# channel tag already holds, so :c-<sha> is byte-identical to what is
# published rather than a lookalike rebuild.
#
# Runs on EVERY reuse, which is what keeps family rule 146 true: a
# rolling channel refreshes itself, so skipping a build must never mean
@@ -691,12 +690,11 @@ jobs:
fetch-depth: 0
# --- derived values, one line (milestone 313) ------------------------
# These stopped being shadow output at step 3: `tag` is published on
# main and `revision` decides whether the build below runs at all. This
# step prints all three anyway, because the load-bearing steps each
# print only the one they use, and on dev the date tag is computed
# nowhere else. When a build is skipped or a pin looks wrong, this is
# the line that says what the commit derived.
# These stopped being shadow output at step 3. `revision` decides
# whether the build below runs at all and `version` is what the image
# reports about itself; the load-bearing steps each print only the one
# they use, so this is the only place the pair appears together. When a
# build is skipped, this is the line that says what the commit derived.
#
# Still diagnostic, so it still must not fail the build — no `set -e`,
# and every derivation falls back to UNAVAILABLE. A broken echo must
@@ -707,8 +705,9 @@ jobs:
# STILL. If web moves, its path set is too wide.
# * a push touching only docs moves nothing.
# * a push touching the extension moves the extension AND web, since
# web bakes in the XPI. If web does not move, its set is too narrow
# — the direction that serves stale bytes on a pin.
# web bakes in the XPI. If web does not move, its set is too narrow:
# the reuse check hits, and the channel serves a web image bundling
# the PREVIOUS XPI while the freshly signed one is orphaned (#3156).
# * dev and main derive the same values for the same source.
- name: Report the derived artifact version
run: |
@@ -770,15 +769,13 @@ jobs:
# that skips a build that was actually needed; the worst case is paying
# for a build we could have avoided.
#
# BASE-IMAGE FRESHNESS, decided rather than left implicit: an artifact
# whose source stops moving stops picking up base-image updates under
# its pinned tag. That is what a pin MEANS — a date tag has to keep
# serving the bytes it served (fabledcurator-ml:2026.7.17 still
# resolves to July's image), or it is not a pin — and family rule
# 145 already says where the refresh goes instead: a rebuild with
# different contents publishes only the MOVING tag, never the immutable
# one. A scheduled channel-only refresh is tracked separately (#3154);
# it does not belong in the push path.
# BASE-IMAGE FRESHNESS: an artifact whose source stops moving stops
# picking up base-image updates. Milestone 318 removed the argument this
# used to need rather than answering it — with no version tags there is
# no immutable name a refresh could contradict, and rule 145 already
# allows a rebuild with different contents to republish a MOVING tag.
# So a refresh is just a build. A scheduled channel-only one is tracked
# separately (#3154); it does not belong in the push path.
- name: Is this content already published?
id: reuse
env:
@@ -845,8 +842,8 @@ jobs:
# Registry-side manifest copy: no layer transfer, no local daemon, no
# rebuild. Each -t becomes another reference to the SAME manifest the
# channel tag already holds, so :c-<sha> and the date pin are
# byte-identical to what is published rather than a lookalike rebuild.
# channel tag already holds, so :c-<sha> is byte-identical to what is
# published rather than a lookalike rebuild.
#
# Runs on EVERY reuse, which is what keeps family rule 146 true: a
# rolling channel refreshes itself, so skipping a build must never mean
@@ -915,12 +912,11 @@ jobs:
fetch-depth: 0
# --- derived values, one line (milestone 313) ------------------------
# These stopped being shadow output at step 3: `tag` is published on
# main and `revision` decides whether the build below runs at all. This
# step prints all three anyway, because the load-bearing steps each
# print only the one they use, and on dev the date tag is computed
# nowhere else. When a build is skipped or a pin looks wrong, this is
# the line that says what the commit derived.
# These stopped being shadow output at step 3. `revision` decides
# whether the build below runs at all and `version` is what the image
# reports about itself; the load-bearing steps each print only the one
# they use, so this is the only place the pair appears together. When a
# build is skipped, this is the line that says what the commit derived.
#
# Still diagnostic, so it still must not fail the build — no `set -e`,
# and every derivation falls back to UNAVAILABLE. A broken echo must
@@ -931,8 +927,9 @@ jobs:
# STILL. If web moves, its path set is too wide.
# * a push touching only docs moves nothing.
# * a push touching the extension moves the extension AND web, since
# web bakes in the XPI. If web does not move, its set is too narrow
# — the direction that serves stale bytes on a pin.
# web bakes in the XPI. If web does not move, its set is too narrow:
# the reuse check hits, and the channel serves a web image bundling
# the PREVIOUS XPI while the freshly signed one is orphaned (#3156).
# * dev and main derive the same values for the same source.
- name: Report the derived artifact version
run: |
@@ -989,15 +986,13 @@ jobs:
# that skips a build that was actually needed; the worst case is paying
# for a build we could have avoided.
#
# BASE-IMAGE FRESHNESS, decided rather than left implicit: an artifact
# whose source stops moving stops picking up base-image updates under
# its pinned tag. That is what a pin MEANS — a date tag has to keep
# serving the bytes it served (fabledcurator-agent:2026.7.17 still
# resolves to July's image), or it is not a pin — and family rule
# 145 already says where the refresh goes instead: a rebuild with
# different contents publishes only the MOVING tag, never the immutable
# one. A scheduled channel-only refresh is tracked separately (#3154);
# it does not belong in the push path.
# BASE-IMAGE FRESHNESS: an artifact whose source stops moving stops
# picking up base-image updates. Milestone 318 removed the argument this
# used to need rather than answering it — with no version tags there is
# no immutable name a refresh could contradict, and rule 145 already
# allows a rebuild with different contents to republish a MOVING tag.
# So a refresh is just a build. A scheduled channel-only one is tracked
# separately (#3154); it does not belong in the push path.
- name: Is this content already published?
id: reuse
env:
@@ -1064,8 +1059,8 @@ jobs:
# Registry-side manifest copy: no layer transfer, no local daemon, no
# rebuild. Each -t becomes another reference to the SAME manifest the
# channel tag already holds, so :c-<sha> and the date pin are
# byte-identical to what is published rather than a lookalike rebuild.
# channel tag already holds, so :c-<sha> is byte-identical to what is
# published rather than a lookalike rebuild.
#
# Runs on EVERY reuse, which is what keeps family rule 146 true: a
# rolling channel refreshes itself, so skipping a build must never mean
README.md
+38 -4
View File
@@ -10,6 +10,35 @@ In production. `main` is continuously deployed — every merge to `main` builds
and publishes `:latest` images, so whatever is on `main` is what is running.
Day-to-day work happens on `dev`, which publishes `:dev` images.
## Versions and tags
Three image tags exist, and no others:
| Tag | Branch | Meaning |
| --- | --- | --- |
| `:latest` | `main` | Production. Moves on every merge. |
| `:c-<sha>` | `main` | Immutable — the rollback unit, all three images together. |
| `:dev` | `dev` | The rolling test channel. Moves on every push. |
There are deliberately **no version tags**. Nothing pins one, and a per-build
name nobody reads is upkeep for a model FC does not run (family rule 145; the
reasoning is note #3127 §5). Rolling back is `docker pull …:c-<sha>`.
Each artifact still has a version, derived rather than chosen: the commit time
of the newest change to that artifact's *own* shipped files, as
`YYYY.MM.DD.HHMM` UTC (rule 148). Four artifacts, four independent versions —
a push touching only `agent/` re-versions the agent and leaves web and ml
alone, and CI skips the builds whose content did not move.
Because no registry name carries it, the running instance's own report is the
only answer to "which build is this?". The foot of Settings shows
`FabledCurator 2026.08.29.0201 · dev`, and `/api/health` returns the same two
fields.
Release tags are optional bookmarks — FC went twelve weeks without one and
nothing was wrong. Pushing `v<version>` publishes a Forgejo release listing the
commits since the previous tag; it builds no image.
## What's in here
Five deployable pieces, built by `.forgejo/workflows/build.yml`:
@@ -52,9 +81,10 @@ FabledCurator is designed to run inside a self-hosted homelab environment over p
## CI / Forgejo setup
Three workflows: `ci.yml` (lint, extension-version check, backend unit tests,
Four workflows: `ci.yml` (lint, extension-version check, backend unit tests,
frontend build, integration), `extension.yml` (extension lint, vitest, XPI
content verification), and `build.yml` (sign + publish).
content verification), `build.yml` (sign + publish), and `release.yml`, which
runs only on a `v*` tag and publishes a changelog without building anything.
**The toolchain each job runs in is its `container.image`, not its `runs-on`
label.** `runs-on: python-ci` only schedules the job onto a runner; every job
@@ -71,8 +101,12 @@ The repo expects one secret:
Generate at https://git.fabledsword.com/user/settings/applications. The injected `GITHUB_TOKEN` cannot be used because it lacks `write:package`.
AMO signing additionally needs `MOZILLA_AMO_JWT_KEY` / `MOZILLA_AMO_JWT_SECRET`; it runs on
`main` only and is cached per version, since AMO rejects a re-signed version.
AMO signing additionally needs `MOZILLA_AMO_JWT_KEY` / `MOZILLA_AMO_JWT_SECRET`.
It runs on **both** channels and is cached per version: because the version is
derived from commit time, `dev` and `main` derive the same number for the same
source, so `main` finds `dev`'s signature already cached and makes no second AMO
call. That cache is why signing must be one-shot — AMO rejects a re-signed
version.
## License
ci-requirements.md
+67 -23
View File
@@ -9,14 +9,17 @@ git.fabledsword.com/bvandeusen/ci-python:3.14
## Image deps used
- python 3.14
- ruff (analyzer for `backend/`, `tests/`, `alembic/`)
- ruff (analyzer for `backend/`, `tests/`, `alembic/`, `agent/`, `scripts/`)
- node (frontend job: `npm install` + vitest + vite build)
- docker CLI + buildx (`.forgejo/workflows/build.yml`: build-web, build-ml — Fabled-Git registry push)
- docker CLI + buildx (`.forgejo/workflows/build.yml`: build-web, build-ml, build-agent — Fabled-Git registry push, and `imagetools inspect`/`create` for the reuse path)
## Secondary runtime image
node:24-bookworm-slim — `.forgejo/workflows/extension.yml` only.
`.forgejo/workflows/release.yml` runs on `ci-python:3.14` like everything else
and installs nothing: it needs git and stdlib python, and builds no image.
The extension lane is the one job that does NOT run on `ci-python:3.14`: it
needs a current Node for `web-ext` and vitest and nothing Python at all. Kept
on the upstream slim image rather than adding a Node toolchain to `ci-python`,
@@ -54,13 +57,19 @@ per `docs/process.md`'s "add deps to the image when used by >1 project".
shims to production code — the libs ship as `background.scripts`, not ES
modules, so the specs exercise exactly the bytes packaged into the XPI.
- **`extension/scripts/packaging.sh` is the single definition of what ships
inside the XPI.** Two consumers read from it rather than keeping their own
copy: web-ext's `--ignore-files` (`extension/package.json`), and the `git log`
pathspec inside the script's own version derivation. It was three until
2026-08-27 — `ci.yml`'s `extension-version` guard held the third and went when
the manual bump it guarded did (milestone 271 step 5). Hand-kept copies of
that one fact is what allowed issue #2397, so `extension/test/version.spec.js`
asserts no workflow has reintroduced a literal `:(exclude)extension/…`.
inside the XPI.** Three consumers read from it rather than keeping their own
copy: web-ext's `--ignore-files` (`extension/package.json`), the `git log`
pathspec inside the script's own version derivation, and `scripts/artifacts.sh`,
which appends the extension's set to web's because the web image bundles the
signed XPI. Hand-kept copies of that one fact is what allowed issue #2397, so
`extension/test/version.spec.js` asserts no workflow has reintroduced a
literal `:(exclude)extension/…`.
- **Packaged and version-relevant are two different sets** (#3156). `scripts/`
is excluded from the XPI and is NOT excluded from the version derivation,
because `packaging.sh` decides the version string stamped into the packaged
`manifest.json`. The membership test is *"can changing this file change the
published bytes?"*, not *"is this file copied in?"* — which is why the script
keeps two lists rather than one.
- **The shipped extension version is derived, not committed.** It is the commit
TIME of the newest packaged-extension change (minutes since 2020-01-01, per
family rule 149 — never a commit count, which orders by branch rather than by
@@ -68,20 +77,55 @@ per `docs/process.md`'s "add deps to the image when used by >1 project".
`extension/manifest.json` + `package.json` in the working tree before signing;
the stamp is never committed. Treat the version in the repo as a base: only
its MAJOR.MINOR is read, and its patch component is inert.
- Every job that calls `packaging.sh version` checks out with `fetch-depth: 0` —
`build.yml`'s `sign-extension` and `build-web`, and `ci.yml`'s
`extension-version`. A depth-1 clone sees one commit and derives a wrong,
too-low value **rather than failing**, so the full-history checkout is
load-bearing rather than incidental.
- **`FC_CHANNEL` is a build arg, not a runtime setting.** `build.yml` passes
`dev` / `main` to the web image only (the ml and agent images have nothing to
report it to), and `/api/extension/manifest` reports it beside the version so
an install can be traced to a channel. It is declared LAST in the Dockerfile
on purpose: an ARG invalidates every layer below it, and this is the one value
that differs between the dev and main builds of identical source, so placing
it earlier would stop the two channels ever sharing a cached `pip install`.
Empty by default — a local build then reports no channel at all rather than
claiming one.
- Every job that derives anything checks out with `fetch-depth: 0` — all four
`build.yml` jobs, `ci.yml`'s `extension-version` and `backend-lint-and-test`
(for `tests/test_artifact_paths.py` and `test_artifact_identity.py`), and
`release.yml`, which additionally walks the tag graph. A depth-1 clone sees
one commit and derives a wrong, too-low value **rather than failing**, so the
full-history checkout is load-bearing rather than incidental.
- **`scripts/artifacts.sh` is the same shape one level up: one definition per
artifact of what it is built from, and the two values derived from it.**
`revision` (12 hex of the newest commit touching that set) and `version`
(`YYYY.MM.DD.HHMM` UTC, rule 148). Four artifacts, four independent answers,
so a push touching only `agent/` leaves web and ml alone.
`tests/test_artifact_paths.py` reads each Dockerfile and asserts every COPY
source is covered, so adding a COPY without updating the script fails CI.
- **A file that DECIDES an artifact's identity belongs in its set even though it
is copied into nothing** — `packaging.sh` for the extension and web (#3156),
and `artifacts.sh` itself for web (#3202), which decides the `FC_VERSION`
baked into that image. Only web needs the second entry: every artifact stamps
a revision, but a revision has a backstop (a changed derivation stops matching
the published label and forces a rebuild) and a version has none, because
nothing compares it to anything. `tests/test_artifact_paths.py`'s `DERIVERS`
table is the guard.
- **Builds are skipped when the content is already published.** Each image
carries its revision as an `fc.revision` LABEL, and `build.yml` reads that
label back off the moving channel tag (`imagetools inspect --format`). Equal
to the derived revision means the bytes are already published, so the job
repoints the remaining tags at the existing manifest instead of rebuilding.
Two things this depends on: an inspect that errors for ANY reason reads as a
MISS so no needed build is ever skipped, and the repoint must EXCLUDE the
source tag — `imagetools create` wraps its source in a manifest index, and
config labels do not resolve through an index, so writing the channel tag
from itself destroys the label the next run reads (#3183).
- **`FC_CHANNEL` and `FC_VERSION` are build args, not runtime settings.**
`build.yml` passes them to the web image only — the ml and agent images have
nothing to report them to. `/api/health` returns both, the foot of Settings
renders them, and `/api/extension/manifest` reports the channel beside the
extension version so an install can be traced to a channel. With no version image tags, that
self-report is the ONLY answer to "which build is this?" — which is why a
missing version renders `unknown` rather than a blank: an empty footer reads
as "no version", a different and false claim.
Both are declared LAST in the Dockerfile on purpose: an ARG invalidates every
layer below it, and these are the values that differ between the dev and main
builds of identical source, so placing them earlier would stop the two
channels ever sharing a cached `pip install`. Empty by default — a local build
then reports nothing rather than claiming a channel it is not on.
- **The channel is never folded into the version.** A `-dev` suffix makes the
extension's per-segment `parseInt` comparator read that segment as 0, so every
dev build compares equal to every other — issue #2993 exactly (rule 149).
`frontend/test/systemBuild.spec.js` pins the rendered version to the bare
number.
- Callers MUST `set -f` before substituting the script's output. Without it the
shell expands `test/**` against the working tree and silently narrows the
pattern to whatever files exist at that moment — a failure that looks like