From c0370069e000d44b2ac161163778b6cfaecd51aa Mon Sep 17 00:00:00 2001 From: Bryan Van Deusen Date: Wed, 2 Sep 2026 12:28:48 -0400 Subject: [PATCH 1/4] release: the first release describes the product; it has nothing to diff against MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Step 7 needs a release that reads as "what is FabledCurator and how do I run it". What the script would actually have published is "changes since v26.06.04.0" over 533 commits, truncated to 200 — a release page whose first screen is the internal build-out that milestone 328 exists to stop shipping, addressed to a reader who has never seen this project. Two causes, fixed separately. **A pre-convention tag is history, not a predecessor.** The 28 `v26.*` tags were kept when their releases were deleted, so `--match v*` walks ancestry straight back to one of them. Reachable is not comparable: nobody has run v26.06.04.0 and its release page no longer exists to compare against. The match is now `v[0-9][0-9][0-9][0-9].*` — rule 148's shape, which is exactly the set of tags naming a release a reader could have been running. **With that narrowed, the first rule-148 tag reaches no predecessor**, and the old fallback — diff against the whole history — is worse than the problem it replaced. A release with no predecessor now renders the product overview and no commit list at all. The overview is READ OUT OF README.md between `` and ``, not written into the script, for the same reason the changelog is derived: two hand-maintained descriptions of one product drift and nothing ever catches it. The release page and the repo front page are one source. Missing markers are reported as a note and publish anyway, on cross_checks()'s reasoning — the release is still the useful object. Every later release goes back to being a changelog, which is what note #3127 §5 says a release is for. MAX_COMMITS still guards the case it now guards: two real releases far enough apart that the list stops being readable. Also corrected while marking up the README: "Importing — ingests an existing library from disk" was still advertising the folder-import feature that 3590c47 documented as deliberately retired. Replaced with what FC actually does with what arrives — content-hash dedup, sidecar metadata, provenance. Tests: the two that asserted the old no-predecessor behaviour are rewritten rather than left; synthetic repos now carry their own copy of the script, since the overview resolves relative to `__file__` (correct in production, where release.yml checks out the tag) and would otherwise have every fixture silently quoting FabledCurator's real README. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01TTjbZZ6JirCMSaJzQV1RhA --- README.md | 7 +- scripts/release_notes.py | 132 ++++++++++++++++++++++++++++++++---- tests/test_release_notes.py | 118 +++++++++++++++++++++++++++----- 3 files changed, 224 insertions(+), 33 deletions(-) diff --git a/README.md b/README.md index 22c89d5..365032d 100644 --- a/README.md +++ b/README.md @@ -1,5 +1,6 @@ # FabledCurator + Self-hosted media curation — a gallery, ML auto-tagging, and subscription-driven downloading in one application. Part of the FabledSword family. @@ -19,13 +20,15 @@ through afterwards. characters, find near-duplicates and power similarity search. Suggestions are reviewable — it proposes, you confirm, and it learns which proposals you keep rejecting. -- **Importing.** Ingests an existing library from disk, deduplicates it by - content hash, and reads metadata sidecars. +- **Deduplication and provenance.** Everything that arrives is hashed and + deduplicated by content, metadata sidecars are read wherever the source + writes them, and every file keeps a record of where it came from. - **Maintenance.** Backups, library audits, thumbnail and embedding backfills, orphan cleanup — all from the UI, all as background jobs you can watch. Everything is configured from the Settings UI and stored in the database. There is no config file to edit beyond a handful of bootstrap environment variables. + ## Before you expose it diff --git a/scripts/release_notes.py b/scripts/release_notes.py index 0511d99..70487ad 100644 --- a/scripts/release_notes.py +++ b/scripts/release_notes.py @@ -33,6 +33,32 @@ history. Ancestry is immune to the shape change, and it is also the more honest question: "what is in this that was not in the last one" IS a reachability question. +Ancestry alone is not enough, though, and milestone 328 is where that showed. +The 28 `v26.*` tags are still in the repo — the operator kept them as history +when their releases were deleted — so `--match v*` walks straight back to +`v26.06.04.0` and reports 533 commits. That span is not a changelog: nobody has +run `v26.06.04.0`, its release page no longer exists to compare against, and +the 200 lines that survive truncation are precisely the internal build-out that +milestone 328 exists to stop shipping. So the match is `v[0-9][0-9][0-9][0-9].*` +— rule 148's four-digit-year shape — which is exactly the set of tags that name +a release a reader could have been running. A pre-convention tag is history, +not a predecessor. + +## The first release has no changelog, and should not pretend to + +Once the match is narrowed, the first rule-148 tag reaches no predecessor at +all, and the old fallback — diff against the whole history — is worse than the +problem it replaced. The honest content for a release nobody has a previous +version of is what the thing IS. + +So a release with no reachable predecessor renders the product overview instead +of a commit list. It is read out of README.md between `` +and `` rather than written here, for the same reason the +changelog is derived: two hand-maintained descriptions of one product drift, +and nothing ever catches it. The release page and the repo front page are one +source. Every later release goes back to being a changelog, which is what §5 of +note #3127 says a release is for. + ## Re-runs update, they do not fall through Note #3127 §6.7: a publisher that POSTs and recovers the id from a `409` never @@ -91,18 +117,45 @@ def git_ok(*args: str) -> str | None: def previous_tag(ref: str, tag: str | None) -> str | None: - """The most recent `v*` tag reachable from `ref`, excluding `tag` itself. + """The most recent rule-148 tag reachable from `ref`, excluding `tag` itself. `--exclude` rather than `^` so this is the same call whether or not `ref` is the tag being released — and so it does not blow up on a root commit that has no parent to walk to. + + The glob deliberately does NOT match the old `v26.*` tags. They are kept as + history and their releases are gone, so naming one as the predecessor emits + a span nobody can look up. See the module docstring. """ - args = ["describe", "--tags", "--abbrev=0", "--match", "v*"] + args = ["describe", "--tags", "--abbrev=0", "--match", "v[0-9][0-9][0-9][0-9].*"] if tag: args += ["--exclude", tag] return git_ok(*args, ref) +def product_overview() -> str | None: + """The product description, lifted verbatim from README.md. + + Returns None if the markers are absent or empty — a missing overview is + reported as a note and the release still publishes, on the same reasoning + as cross_checks(): the release is the useful object even when one part of + the derivation could not run. + """ + root = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) + try: + with open(os.path.join(root, "README.md"), encoding="utf-8") as fh: + readme = fh.read() + except OSError: + return None + match = re.search( + r"(.*?)", + readme, re.S, + ) + if not match: + return None + return match.group(1).strip() or None + + def commits(previous: str | None, ref: str) -> list[str]: """The subjects between the previous release and this one. @@ -125,7 +178,10 @@ def truncate(log: list[str]) -> tuple[list[str], str | None]: ) -def render(tag: str, sha: str, previous: str | None, log: list[str], notes: list[str]) -> str: +def render( + tag: str, sha: str, previous: str | None, log: list[str], notes: list[str], + overview: str | None, +) -> str: short = sha[:7] parts = [] @@ -135,6 +191,25 @@ def render(tag: str, sha: str, previous: str | None, log: list[str], notes: list # is the failure this whole milestone is about. parts.append("\n".join(f"> **Note:** {n}" for n in notes)) + # No predecessor means nobody reading this has run an earlier one, so the + # release describes the product rather than a diff. The overview is + # README.md's own words — see the module docstring on why it is not + # written here. + if previous is None and overview: + parts.append(overview) + parts.append( + "## Installing\n\n" + "```\ncurl -O https://git.fabledsword.com/bvandeusen/FabledCurator/raw/" + f"tag/{tag}/docker-compose.yml\ncurl -O https://git.fabledsword.com/" + f"bvandeusen/FabledCurator/raw/tag/{tag}/.env.example\n" + "mv .env.example .env # then set SECRET_KEY, DB_PASSWORD\n" + "docker compose -f docker-compose.yml up -d\n```\n\n" + "**Read \"Before you expose it\" in the README first.** FabledCurator " + "has no login, and it stores live platform session cookies for " + "accounts that usually have a payment method attached. Bind it to a " + "network you trust." + ) + parts.append( f"Built from `{short}`. The rollback unit is the immutable `:c-` tag " f"(rule 145) — these three move together:\n\n```\n" @@ -142,7 +217,19 @@ def render(tag: str, sha: str, previous: str | None, log: list[str], notes: list + "\n```" ) - heading = f"## Changes since {previous}" if previous else "## Changes" + if previous is None: + # Deliberately NOT a commit list. The alternative is the whole history + # truncated to MAX_COMMITS, which is 200 lines of internal build-out + # presented to someone who has never seen this project. + parts.append( + "---\n\n_First release under rule 148's `vYYYY.MM.DD.HHMM` shape, so " + "there is no predecessor to diff against and no changelog to derive. " + "The description above is README.md's, quoted at publish time. Later " + "releases carry the commits since the previous one._" + ) + return "\n\n".join(parts) + + heading = f"## Changes since {previous}" if log: parts.append(heading + "\n\n" + "\n".join(f"- {line}" for line in log)) else: @@ -152,10 +239,9 @@ def render(tag: str, sha: str, previous: str | None, log: list[str], notes: list "names the same source under a new name._" ) - span = f"{previous}..{tag}" if previous else tag parts.append( - f"---\n\n_Derived at publish time from `git log --no-merges {span}`. " - f"Nothing here is hand-maintained._" + f"---\n\n_Derived at publish time from " + f"`git log --no-merges {previous}..{tag}`. Nothing here is hand-maintained._" ) return "\n\n".join(parts) @@ -289,13 +375,31 @@ def main() -> None: for note in notes: print(f"release: NOTE {note}") - log = commits(previous, ref) - print(f"release: {len(log)} non-merge commits in the span") - log, overflow = truncate(log) - if overflow: - print(f"release: NOTE {overflow}") - notes.append(overflow) - body = render(tag or ref, sha, previous, log, notes) + # A first release renders the overview instead of a changelog, so the + # commit walk is skipped entirely rather than computed and discarded — + # `commits(None, ref)` is the whole history and there is no reason to ask + # for it. + overview = None + log: list[str] = [] + if previous is None: + overview = product_overview() + if overview is None: + note = ( + "No `` block found in README.md, so this " + "first release has no product description. Published anyway; add " + "the markers and re-run the workflow to fill it in." + ) + print(f"release: NOTE {note}") + notes.append(note) + print("release: no rule-148 predecessor — rendering the product overview") + else: + log = commits(previous, ref) + print(f"release: {len(log)} non-merge commits in the span") + log, overflow = truncate(log) + if overflow: + print(f"release: NOTE {overflow}") + notes.append(overflow) + body = render(tag or ref, sha, previous, log, notes, overview) if args.dry_run or not tag: print("--- body ---") diff --git a/tests/test_release_notes.py b/tests/test_release_notes.py index 129be14..34b354c 100644 --- a/tests/test_release_notes.py +++ b/tests/test_release_notes.py @@ -24,12 +24,36 @@ SCRIPT = ROOT / "scripts" / "release_notes.py" def notes(*args: str, cwd: Path | None = None) -> str: + """Run the script the way release.yml does. + + A synthetic repo runs its OWN copy of the script, because the overview is + read relative to `__file__` rather than to the cwd — which is right in + production (release.yml checks out the tag, so the script IS the tagged + tree's copy) and would otherwise make every synthetic repo silently quote + FabledCurator's real README. + """ + root = cwd or ROOT + script = root / "scripts" / "release_notes.py" return subprocess.run( - ["python3", str(SCRIPT), "--dry-run", *args], - capture_output=True, text=True, check=True, cwd=cwd or ROOT, + ["python3", str(script if script.exists() else SCRIPT), "--dry-run", *args], + capture_output=True, text=True, check=True, cwd=root, ).stdout +OVERVIEW_TEXT = "A synthetic product, described once." + + +def install_script(repo: Path, *, overview: bool = True) -> None: + """Give a synthetic repo the script and a README to quote.""" + (repo / "scripts").mkdir(exist_ok=True) + (repo / "scripts" / "release_notes.py").write_text(SCRIPT.read_text()) + (repo / "scripts" / "artifacts.sh").write_text("#!/bin/sh\nexit 1\n") + readme = "# Synthetic\n\n" + if overview: + readme += f"\n{OVERVIEW_TEXT}\n\n" + (repo / "README.md").write_text(readme) + + def body_of(out: str) -> str: assert "--- body ---" in out, f"no body was rendered:\n{out}" return out.split("--- body ---", 1)[1] @@ -56,9 +80,10 @@ def shaped_history(tmp_path: Path) -> Path: repo = tmp_path / "shaped" repo.mkdir() git(repo, "init", "-q", "-b", "main") + install_script(repo) for i, tag in enumerate(("v26.06.04.0", "v2026.08.28.2208", "v2026.08.29.1000")): (repo / "f.txt").write_text(f"{i}\n") - git(repo, "add", "f.txt") + git(repo, "add", "-A") git(repo, "commit", "-q", "-m", f"work landing in {tag}") git(repo, "tag", tag) # One more commit and a merge, so the merge-exclusion test has something to @@ -109,12 +134,57 @@ def test_merges_are_excluded_so_the_list_is_the_work(shaped_history): assert "Merge pull request #999" not in body -def test_the_first_release_still_renders_with_nothing_behind_it(shaped_history): - """No previous tag is reachable from the oldest one. That is a real state, - not an error, and it must not take the release down with it.""" - out = notes("v26.06.04.0", cwd=shaped_history) +def test_a_pre_convention_tag_is_history_not_a_predecessor(shaped_history): + """The defect milestone 328 hit, and the reason the match glob narrowed. + + The 28 `v26.*` tags are kept as history while their releases were deleted. + Ancestry alone happily names `v26.06.04.0` as the predecessor of the first + rule-148 tag — and then the body offers "changes since" a release that no + longer exists, over a span (533 commits in the real repo) that is the + internal build-out this milestone exists to stop publishing. + + Reachable is not the same as comparable. Only a `vYYYY.` tag names a + release a reader could have been running. + """ + out = notes("v2026.08.28.2208", cwd=shaped_history) assert "previous=" in out - assert "## Changes" in body_of(out) + assert "v26.06.04.0" not in body_of(out) + + +def test_the_first_release_describes_the_product_instead_of_diffing(shaped_history): + """No predecessor means nobody reading has run an earlier version, so a + changelog has no referent. The alternative the script used to take — diff + against the whole history, truncated — puts 200 lines of internal build-out + in front of someone meeting the project for the first time.""" + body = body_of(notes("v2026.08.28.2208", cwd=shaped_history)) + assert OVERVIEW_TEXT in body + assert "## Changes" not in body + assert not [ln for ln in body.split("\n") if ln.startswith("- work landing")] + + +def test_the_overview_is_readmes_words_not_a_second_copy(shaped_history): + """Two hand-maintained descriptions of one product drift and nothing + catches it. The release page quotes README.md so there is one source.""" + readme = (shaped_history / "README.md").read_text() + assert OVERVIEW_TEXT in readme + assert OVERVIEW_TEXT in body_of(notes("v2026.08.28.2208", cwd=shaped_history)) + + +def test_a_missing_overview_block_is_reported_and_still_publishes(tmp_path): + """Same reasoning as cross_checks(): the release is the useful object even + when part of the derivation could not run. Say what is missing, publish + anyway — do not leave the operator with a tag and no release.""" + repo = tmp_path / "no-markers" + repo.mkdir() + git(repo, "init", "-q", "-b", "main") + install_script(repo, overview=False) + git(repo, "add", "-A") + git(repo, "commit", "-q", "-m", "first") + git(repo, "tag", "v2026.09.01.1200") + + out = notes("v2026.09.01.1200", cwd=repo) + assert "No `` block found" in out + assert "No `` block found" in body_of(out) def test_a_non_tag_ref_renders_but_refuses_to_claim_it_published(): @@ -135,15 +205,29 @@ def test_the_rollback_refs_name_all_three_images(): assert f"bvandeusen/{image}:c-" in body, f"{image} missing from the rollback refs" -def test_an_unbounded_span_is_truncated_and_says_so(): - """With no reachable previous tag the span is the whole history. Emitting - eleven hundred lines would bury the one line explaining why there are - eleven hundred of them, so the cap is part of the message, not a silent - slice.""" - out = notes("HEAD") - if "previous=" not in out: - pytest.skip("a previous tag is reachable from HEAD in this checkout") +def test_a_long_span_between_two_releases_is_truncated_and_says_so(tmp_path): + """The cap is still reachable, just not by the route it used to be. + + It no longer fires on "no predecessor" — that renders the overview now. + What it still guards is two real releases far enough apart that the list + stops being something anyone reads, which is the ordinary case for a + project that cuts a bookmark twice a year. The cap is part of the message, + not a silent slice. + """ + repo = tmp_path / "long" + repo.mkdir() + git(repo, "init", "-q", "-b", "main") + install_script(repo) + git(repo, "add", "-A") + git(repo, "commit", "-q", "-m", "scaffold") + git(repo, "tag", "v2026.01.01.0000") + for i in range(205): + git(repo, "commit", "-q", "--allow-empty", "-m", f"fix: change {i}") + git(repo, "tag", "v2026.07.01.0000") + + out = notes("v2026.07.01.0000", cwd=repo) + assert "previous=v2026.01.01.0000" in out body = body_of(out) listed = [ln for ln in body.split("\n") if ln.startswith("- ")] - assert len(listed) <= 200 + assert len(listed) == 200 assert "more than a changelog is for" in body -- 2.54.0 From 635138b0d17d498a11f6a7437f09d202f7a0ebc7 Mon Sep 17 00:00:00 2001 From: Bryan Van Deusen Date: Wed, 2 Sep 2026 13:46:16 -0400 Subject: [PATCH 2/4] ci: pin the build clock to the commit, so an unchanged refresh publishes nothing MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Milestone 362 step 1, closing #3265's root cause. The weekly base refresh rewrote all three `:latest` tags on 2026-08-30 with nothing changed in any of them. Not a cache miss — run 4934's log shows every content step CACHED and both bases resolved to unchanged pinned digests. buildkit stamps the image config with the wall clock of the build, so identical layers get republished under a new config blob and therefore a new manifest digest. The cost is not storage, it is meaning: `:latest` moved on a calendar, so a digest change stopped being evidence that anything was different. That is the one thing a digest is any use for, and it is load-bearing here — the reuse check, the `:c-` rollback story and any future redeploy signal all rest on it. SOURCE_DATE_EPOCH normalises `created` and the history timestamps, so the same source produces the same config bytes and the same digest, and pushing it is a registry no-op. The value is routed through artifacts.sh's existing `newest()` rather than taken from git separately. `revision`, `version` and now `epoch` are three fields of ONE lookup, so they cannot drift into naming different commits — a divergence that would stamp an image reproducibly against one commit while it reported being another, with both values looking perfectly well-formed. Note #3127 §2 is the record of what a second clock costs; this adds a view, not a clock. Also corrected: the build step comment and ci-requirements.md both described the churn as current behaviour with the fix as a "likely" future. They now describe what the file does. Tests pin the property the fix depends on, not the fix: epoch is the same commit version names, in both renderings including the extension's unpadded one, and it does not move between two calls on one checkout. A future refactor that gave epoch its own `git log` would pass every other test in that file. Not yet verified end to end — proving it needs two consecutive refreshes to land on the same digest, which is the next thing, and is the step #3265 exists because nobody did last time. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01TTjbZZ6JirCMSaJzQV1RhA --- .forgejo/workflows/build.yml | 111 +++++++++++++++++++++----------- ci-requirements.md | 23 ++++--- scripts/artifacts.sh | 23 ++++++- tests/test_artifact_identity.py | 53 +++++++++++++++ 4 files changed, 162 insertions(+), 48 deletions(-) diff --git a/.forgejo/workflows/build.yml b/.forgejo/workflows/build.yml index 7bd8066..c6bd756 100644 --- a/.forgejo/workflows/build.yml +++ b/.forgejo/workflows/build.yml @@ -638,6 +638,14 @@ jobs: # adds no variability the reuse check would have to account for. echo "version=$(sh scripts/artifacts.sh version web)" >> "$GITHUB_OUTPUT" + # The build clock, pinned to the same commit (#3265). Without it + # buildkit stamps the image config with the wall clock of the build, + # so identical layers republish under a new config blob and the + # channel tag gets a new manifest digest for no reason. Derived from + # `newest()` like revision and version, so all three name one commit + # and cannot drift apart. + echo "epoch=$(sh scripts/artifacts.sh epoch web)" >> "$GITHUB_OUTPUT" + # The moving tag for this channel. Which tag we ask IS the channel — # that is why the revision needs no -main/-dev qualifier any more. if [ "$CHANNEL" = "main" ]; then T=latest; else T=dev; fi @@ -764,6 +772,12 @@ jobs: - name: Build and push web image if: steps.reuse.outputs.hit != 'true' + # Read by buildx out of the ENVIRONMENT, not passed as a build-arg — + # it normalises the image config's `created` field and the history + # timestamps rather than being consumed by the Dockerfile. See #3265 + # and the reuse step's `epoch` output. + env: + SOURCE_DATE_EPOCH: ${{ steps.reuse.outputs.epoch }} uses: docker/build-push-action@v5 with: context: . @@ -776,20 +790,17 @@ jobs: # invalidates, and the image genuinely rebuilds. # # MEASURED on the first real fire, run 4934 (#3265): when the base - # did NOT move, the build is ~13s and every content step reports - # CACHED — but the channel tag STILL gets a new manifest digest. - # buildkit mints a fresh image config each run, so identical layers - # are republished under a new config blob. All three images moved - # that way on 2026-08-30 with nothing whatsoever changed in them. + # did NOT move, the build was ~13s with every content step CACHED — + # and the channel tag STILL got a new manifest digest, because + # buildkit stamps a fresh image config per run and republishes the + # identical layers under it. All three images moved that way on + # 2026-08-30 with nothing whatsoever changed in them. # - # So a refresh currently rewrites :latest every Sunday whether or - # not there is anything new in it, and :c- is handed a new - # manifest to diverge from on the same cadence. Layers are shared, - # so the storage cost is a config blob; the cost that matters is - # that a digest change no longer MEANS anything. Tracked in #3265 — - # the likely fix is a deterministic SOURCE_DATE_EPOCH, which would - # make "same source, same bytes" true and turn the no-op case into - # a genuine no-op. + # SOURCE_DATE_EPOCH (below) is the fix: pinned to the commit the + # content came from, the config is byte-identical across runs, so + # the manifest digest is too and the push is a registry no-op. A + # digest change means the content changed again, which is the only + # thing a digest is any use for. # # What `pull` does NOT catch either: a Debian package update inside # the `apt-get install` layer while the base tag itself stands @@ -1097,6 +1108,14 @@ jobs: DERIVED=$(sh scripts/artifacts.sh revision ml) echo "revision=$DERIVED" >> "$GITHUB_OUTPUT" + # The build clock, pinned to the same commit (#3265). Without it + # buildkit stamps the image config with the wall clock of the build, + # so identical layers republish under a new config blob and the + # channel tag gets a new manifest digest for no reason. Derived from + # `newest()` like revision and version, so all three name one commit + # and cannot drift apart. + echo "epoch=$(sh scripts/artifacts.sh epoch ml)" >> "$GITHUB_OUTPUT" + # The moving tag for this channel. Which tag we ask IS the channel — # that is why the revision needs no -main/-dev qualifier any more. if [ "$CHANNEL" = "main" ]; then T=latest; else T=dev; fi @@ -1147,6 +1166,12 @@ jobs: - name: Build and push ml image if: steps.reuse.outputs.hit != 'true' + # Read by buildx out of the ENVIRONMENT, not passed as a build-arg — + # it normalises the image config's `created` field and the history + # timestamps rather than being consumed by the Dockerfile. See #3265 + # and the reuse step's `epoch` output. + env: + SOURCE_DATE_EPOCH: ${{ steps.reuse.outputs.epoch }} uses: docker/build-push-action@v5 with: context: . @@ -1159,20 +1184,17 @@ jobs: # invalidates, and the image genuinely rebuilds. # # MEASURED on the first real fire, run 4934 (#3265): when the base - # did NOT move, the build is ~13s and every content step reports - # CACHED — but the channel tag STILL gets a new manifest digest. - # buildkit mints a fresh image config each run, so identical layers - # are republished under a new config blob. All three images moved - # that way on 2026-08-30 with nothing whatsoever changed in them. + # did NOT move, the build was ~13s with every content step CACHED — + # and the channel tag STILL got a new manifest digest, because + # buildkit stamps a fresh image config per run and republishes the + # identical layers under it. All three images moved that way on + # 2026-08-30 with nothing whatsoever changed in them. # - # So a refresh currently rewrites :latest every Sunday whether or - # not there is anything new in it, and :c- is handed a new - # manifest to diverge from on the same cadence. Layers are shared, - # so the storage cost is a config blob; the cost that matters is - # that a digest change no longer MEANS anything. Tracked in #3265 — - # the likely fix is a deterministic SOURCE_DATE_EPOCH, which would - # make "same source, same bytes" true and turn the no-op case into - # a genuine no-op. + # SOURCE_DATE_EPOCH (below) is the fix: pinned to the commit the + # content came from, the config is byte-identical across runs, so + # the manifest digest is too and the push is a registry no-op. A + # digest change means the content changed again, which is the only + # thing a digest is any use for. # # What `pull` does NOT catch either: a Debian package update inside # the `apt-get install` layer while the base tag itself stands @@ -1466,6 +1488,14 @@ jobs: DERIVED=$(sh scripts/artifacts.sh revision agent) echo "revision=$DERIVED" >> "$GITHUB_OUTPUT" + # The build clock, pinned to the same commit (#3265). Without it + # buildkit stamps the image config with the wall clock of the build, + # so identical layers republish under a new config blob and the + # channel tag gets a new manifest digest for no reason. Derived from + # `newest()` like revision and version, so all three name one commit + # and cannot drift apart. + echo "epoch=$(sh scripts/artifacts.sh epoch agent)" >> "$GITHUB_OUTPUT" + # The moving tag for this channel. Which tag we ask IS the channel — # that is why the revision needs no -main/-dev qualifier any more. if [ "$CHANNEL" = "main" ]; then T=latest; else T=dev; fi @@ -1516,6 +1546,12 @@ jobs: - name: Build and push agent image if: steps.reuse.outputs.hit != 'true' + # Read by buildx out of the ENVIRONMENT, not passed as a build-arg — + # it normalises the image config's `created` field and the history + # timestamps rather than being consumed by the Dockerfile. See #3265 + # and the reuse step's `epoch` output. + env: + SOURCE_DATE_EPOCH: ${{ steps.reuse.outputs.epoch }} uses: docker/build-push-action@v5 with: context: agent @@ -1528,20 +1564,17 @@ jobs: # invalidates, and the image genuinely rebuilds. # # MEASURED on the first real fire, run 4934 (#3265): when the base - # did NOT move, the build is ~13s and every content step reports - # CACHED — but the channel tag STILL gets a new manifest digest. - # buildkit mints a fresh image config each run, so identical layers - # are republished under a new config blob. All three images moved - # that way on 2026-08-30 with nothing whatsoever changed in them. + # did NOT move, the build was ~13s with every content step CACHED — + # and the channel tag STILL got a new manifest digest, because + # buildkit stamps a fresh image config per run and republishes the + # identical layers under it. All three images moved that way on + # 2026-08-30 with nothing whatsoever changed in them. # - # So a refresh currently rewrites :latest every Sunday whether or - # not there is anything new in it, and :c- is handed a new - # manifest to diverge from on the same cadence. Layers are shared, - # so the storage cost is a config blob; the cost that matters is - # that a digest change no longer MEANS anything. Tracked in #3265 — - # the likely fix is a deterministic SOURCE_DATE_EPOCH, which would - # make "same source, same bytes" true and turn the no-op case into - # a genuine no-op. + # SOURCE_DATE_EPOCH (below) is the fix: pinned to the commit the + # content came from, the config is byte-identical across runs, so + # the manifest digest is too and the push is a registry no-op. A + # digest change means the content changed again, which is the only + # thing a digest is any use for. # # What `pull` does NOT catch either: a Debian package update inside # the `apt-get install` layer while the base tag itself stands diff --git a/ci-requirements.md b/ci-requirements.md index ccd3081..8b05574 100644 --- a/ci-requirements.md +++ b/ci-requirements.md @@ -198,14 +198,21 @@ per `docs/process.md`'s "add deps to the image when used by >1 project". refresh from being undone. - **`pull: true` on the scheduled path only** is the mechanism: a moved base tag changes the `FROM` layer's cache key and everything above it rebuilds. - **It does not currently make the unmoved case free.** Measured on the first - real fire (run 4934, 2026-08-30): every content step reported `CACHED` and - the bases resolved to unchanged digests, yet all three `:latest` tags got a - NEW manifest digest, because buildkit mints a fresh image config per run and - republishes identical layers under it. So `:latest` is rewritten weekly - whether or not anything changed, and `:c-` is handed a new manifest to - diverge from on the same cadence — a digest change stops meaning anything. - Tracked as #3265; the likely fix is a deterministic `SOURCE_DATE_EPOCH`. + It did not always make the unmoved case free. Measured on the first real + fire (run 4934, 2026-08-30): every content step reported `CACHED` and the + bases resolved to unchanged digests, yet all three `:latest` tags got a NEW + manifest digest, because buildkit stamps a fresh image config per run and + republishes identical layers under it — so `:latest` was rewritten weekly + whether or not anything changed, and a digest change stopped meaning + anything (#3265). + - **`SOURCE_DATE_EPOCH` is what makes it free.** Set on each build step from + `artifacts.sh epoch ` — the unix timestamp of the same commit + `revision` and `version` name, so all three are views of one `newest()` + lookup and cannot drift into disagreeing. With the config's `created` field + and history timestamps pinned to the content rather than to the wall clock, + identical source produces an identical manifest digest and the push is a + registry no-op. That restores the property the whole scheme rests on: a + channel tag's digest changes when, and only when, its content does. Separately not caught: a Debian package update inside the `apt-get install` layer while the base tag stands still — a lag rather than a hole, since the official python/cuda images rebuild with those updates baked in. diff --git a/scripts/artifacts.sh b/scripts/artifacts.sh index 5f86a79..e93b9eb 100755 --- a/scripts/artifacts.sh +++ b/scripts/artifacts.sh @@ -90,7 +90,7 @@ DERIVER='scripts/artifacts.sh' usage() { - echo "usage: artifacts.sh {paths|revision|version} {web|ml|agent|extension}" >&2 + echo "usage: artifacts.sh {paths|revision|version|epoch} {web|ml|agent|extension}" >&2 exit 2 } @@ -149,6 +149,26 @@ cmd_revision() { echo "$(newest "$1")" | cut -d' ' -f2 | cut -c1-12 } +# The BUILD CLOCK: the same commit's unix timestamp, for SOURCE_DATE_EPOCH. +# +# buildkit stamps the image config's `created` field and every history entry +# with the wall clock of the build unless this is set, so two builds of +# identical source produce different config blobs and therefore different +# manifest digests. That is #3265: the weekly refresh republished all three +# `:latest` tags on 2026-08-30 with every content step CACHED and the bases +# resolved to unchanged digests — nothing was different, and the digest moved +# anyway. A digest that changes on a calendar cannot also mean "the content +# changed", which is the only thing anyone wants it for. +# +# It is the same commit `revision` and `version` name — deliberately, and this +# is the point of routing it through `newest()` rather than taking git's word +# separately. Three values derived from three lookups can disagree; three +# views of one lookup cannot. Note #3127 §2 is the record of what a second +# clock costs. +cmd_epoch() { + echo "$(newest "$1")" | cut -d' ' -f1 +} + # The VERSION: `YYYY.MM.DD.HHMM`, zero-padded, UTC. One shape across the whole # family (note #3127 §1, rule 148) — the number an instance reports about # itself, and, with a `v` in front, the release tag naming the same build. @@ -197,5 +217,6 @@ case "$1" in paths) cmd_paths "$2" ;; revision) cmd_revision "$2" ;; version) cmd_version "$2" ;; + epoch) cmd_epoch "$2" ;; *) usage ;; esac diff --git a/tests/test_artifact_identity.py b/tests/test_artifact_identity.py index fe614bf..a0beab3 100644 --- a/tests/test_artifact_identity.py +++ b/tests/test_artifact_identity.py @@ -234,3 +234,56 @@ def test_version_and_revision_describe_the_same_commit(artifact): f"in AMO_UNPADDED may differ here." ) assert sha.startswith(revision(artifact)) + + +@pytest.mark.parametrize("artifact", ARTIFACTS) +def test_epoch_is_the_same_commit_the_version_names(artifact): + """The build clock and the version must be one lookup, not two. + + `epoch` feeds SOURCE_DATE_EPOCH, which decides the image config's bytes and + therefore the manifest digest; `version` is what the instance reports about + itself. If they could name different commits, an image would be stamped + reproducibly against one commit while claiming to be another — and both + values would look perfectly well-formed, exactly like the divergence the + test above guards. + + They cannot, because `cmd_epoch` and `cmd_version` are two fields of one + `newest()` result. This pins that they stay that way: a future refactor + that gave epoch its own `git log` would pass every other test here. + """ + epoch = artifacts("epoch", artifact).strip() + assert epoch.isdigit(), f"{artifact} epoch is {epoch!r}, not a unix timestamp" + + sha = newest_by_commit_time(artifact) + committed = subprocess.run( + ["git", "show", "-s", "--format=%ct", sha], + capture_output=True, text=True, check=True, cwd=ROOT, + ).stdout.strip() + assert epoch == committed, ( + f"{artifact} derives epoch {epoch}, but its newest shipped commit " + f"{sha[:12]} was committed at {committed}. SOURCE_DATE_EPOCH would " + f"pin the image config to a commit the version does not name." + ) + + # And the two renderings must agree, which is the property that actually + # matters at build time: same commit in, same digest and same reported + # version out. + rendered = subprocess.run( + ["git", "show", "-s", "--format=%cd", "--date=format-local:%Y.%m.%d.%H%M", sha], + capture_output=True, text=True, check=True, cwd=ROOT, + env={"TZ": "UTC", "PATH": os.environ.get("PATH", "")}, + ).stdout.strip() + assert segments(artifacts("version", artifact).strip()) == segments(rendered) + + +def test_epoch_is_stable_across_calls(): + """SOURCE_DATE_EPOCH's entire job is to be the same on the next build. + + A value that moved between two invocations on one unchanged checkout would + reintroduce #3265 through the very mechanism meant to close it, and the + symptom would be indistinguishable: a digest that changes for no reason. + """ + for artifact in ARTIFACTS: + first = artifacts("epoch", artifact).strip() + second = artifacts("epoch", artifact).strip() + assert first == second, f"{artifact} epoch moved: {first} then {second}" -- 2.54.0 From b590d25f8f74f904d25e8c0d2af3de40d664cf36 Mon Sep 17 00:00:00 2001 From: Bryan Van Deusen Date: Wed, 2 Sep 2026 14:40:19 -0400 Subject: [PATCH 3/4] ci: the scheduled refresh builds a candidate, then names the channel MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Milestone 362 step 2. Structural: it creates a moment between "built" and "published" for step 3's gate to occupy. No behaviour change. A refresh rebuilds against freshly resolved base images, and the web image's runtime is a line of UNPINNED Debian packages — ffmpeg, libjpeg62-turbo, libpq5, megatools — re-resolved on every build. Nothing in ci.yml can see that: its lanes run on ci-python:3.14 and install requirements.txt, and a base bump changes neither. So refreshed bytes need proving before :latest names them, and proving needs somewhere to stand. On a push nothing changes: build_ref IS channel_ref, promote is false, and the build writes the channel tag directly the way it always has. On the schedule the build writes :refresh-candidate — one moving ref per image, overwritten in place, holding a build nobody is told to pull. That is the shape rule 145 already allows for :buildcache, not the per-build tag family 318 withdrew. Both values are decided in the reuse step beside `hit`, because that step already owns "what does this job do" (build.yml's own rule, at the force branch). A promote condition derived somewhere else could disagree with the tag the build actually wrote. **The promote is a manifest PUT, not `imagetools create`.** That distinction is the whole risk in this change. `imagetools create` wraps its source in an index, and an indexed channel tag is the one thing this pipeline cannot survive: `.Image.Config.Labels` does not resolve through an index, so the fc.revision the reuse check reads back would come up empty, every later push would miss and rebuild, and nothing would go red. That is #3183, observed on run 4751 — reuse worked exactly once and the only symptom was the bill. The repoint step already excludes its own source tag for this reason; a promote that re-introduced the wrap through another door would undo that care. A manifest PUT is what "make this tag name that image" means at the registry: same bytes, same media type, identical digest, no layer transfer. It reads the result back and fails if the tag does not name what was just written — a PUT that 2xx'd and landed something else is exactly the silent-and-plausible failure this pipeline keeps producing. Every call carries a deadline (rule 156); a registry that stops answering must fail the step, not hang the weekly refresh until the job times out. Promote is UNCONDITIONAL today, deliberately. Gating it before the gate exists would leave the refresh building something and publishing nothing for as long as this milestone takes. Step 4 wraps it in the smoke suite's verdict. Not yet verified on the refresh path — that needs a scheduled run, and the lever to trigger one on demand is the next commit. This one is verified by the push path being untouched. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01TTjbZZ6JirCMSaJzQV1RhA --- .forgejo/workflows/build.yml | 324 ++++++++++++++++++++++++++++++++++- 1 file changed, 321 insertions(+), 3 deletions(-) diff --git a/.forgejo/workflows/build.yml b/.forgejo/workflows/build.yml index c6bd756..cd895f2 100644 --- a/.forgejo/workflows/build.yml +++ b/.forgejo/workflows/build.yml @@ -651,6 +651,41 @@ jobs: if [ "$CHANNEL" = "main" ]; then T=latest; else T=dev; fi echo "channel_ref=$IMAGE:$T" >> "$GITHUB_OUTPUT" + # WHERE THE BUILD PUBLISHES, which is not always the channel — and + # whether the channel then has to be written separately. + # + # On a push the build writes the channel tag directly: the bytes came + # from a commit, and a commit is the thing CI tests. Nothing to hold + # it behind. + # + # On the scheduled refresh it writes a CANDIDATE tag instead. A + # refresh rebuilds against freshly resolved base images, and the web + # image's runtime is a line of UNPINNED Debian packages (ffmpeg, + # libjpeg62-turbo, libpq5, megatools…) re-resolved on every build. + # Nothing in ci.yml can see that: its lanes run on ci-python:3.14 and + # install requirements.txt, and a base bump changes neither. So + # refreshed bytes have to be proven before :latest names them, and + # proving needs a moment between "built" and "published" to occupy. + # This is that moment; :latest goes on naming the build that works + # until something says otherwise. + # + # `:refresh-candidate` is one moving ref per image, overwritten in + # place, holding a build nobody is told to pull — the shape rule 145 + # already allows for :buildcache, not the per-build tag family that + # milestone 318 withdrew. + # + # Both values are decided HERE, beside `hit`, for the reason the + # force/schedule branch below gives: one step decides what this job + # does. A promote condition derived independently could disagree with + # the tag the build actually wrote. + if [ "${EVENT:-}" = "schedule" ]; then + echo "build_ref=$IMAGE:refresh-candidate" >> "$GITHUB_OUTPUT" + echo "promote=true" >> "$GITHUB_OUTPUT" + else + echo "build_ref=$IMAGE:$T" >> "$GITHUB_OUTPUT" + echo "promote=false" >> "$GITHUB_OUTPUT" + fi + # Compare VALUES, never exit codes. Measured on buildx v0.36.1 # (run 4732): a missing key returns an empty string and exits 0, so # branching on the exit code would read "no label yet" as success. @@ -817,7 +852,7 @@ jobs: # out of a local image store a registry-direct build never filled — # #3190, which cost `main` its :c- on 2026-08-29 while :latest # published perfectly well. - tags: ${{ steps.reuse.outputs.channel_ref }} + tags: ${{ steps.reuse.outputs.build_ref }} # The reuse key. Read back off the channel tag on the next push to # decide whether that push needs to build at all, so this is not # decoration — an unstamped image is one that will always rebuild. @@ -864,6 +899,77 @@ jobs: FC_CHANNEL=${{ steps.tag.outputs.channel }} FC_VERSION=${{ steps.reuse.outputs.version }} + # Point the channel tag at the candidate the refresh just built. + # + # Unconditional TODAY, so this milestone never leaves the refresh in a + # state where it builds and publishes nothing. Step 4 wraps it in the + # smoke suite's verdict; until then the scheduled path behaves exactly + # as it did, just via two operations instead of one. + # + # NOT `imagetools create`. That wraps its source in an INDEX, and an + # indexed channel tag is the one thing this pipeline cannot survive: + # `.Image.Config.Labels` does not resolve through an index, so the + # fc.revision the reuse check reads off the channel tag would come back + # empty, every subsequent push would miss and rebuild, and nothing would + # go red. That is #3183, observed on run 4751 — reuse worked exactly once + # and the only symptom was the bill. The repoint step below excludes its + # own source tag for precisely this reason; a promote that re-introduced + # the wrap through a different door would undo that care. + # + # A manifest PUT is what "make this tag name that image" means at the + # registry level: the same bytes under the same media type, so the digest + # is identical, the media type is preserved, and no layer moves. + - name: Promote the refresh candidate to the channel + if: steps.reuse.outputs.promote == 'true' + env: + IMAGE: git.fabledsword.com/bvandeusen/fabledcurator + CHANNEL_REF: ${{ steps.reuse.outputs.channel_ref }} + TOKEN: ${{ secrets.RELEASE_TOKEN }} + ACTOR: ${{ github.actor }} + run: | + set -eu + REPO=${IMAGE#git.fabledsword.com/} + TAG=${CHANNEL_REF##*:} + + # Registry auth is its own token exchange — the `docker login` above + # authenticates the docker client, not curl. Deadline on every call + # (rule 156): a registry that stops answering must fail this step, + # not hang the weekly refresh until the job times out. + BEARER=$(curl -fsS --max-time 30 -u "$ACTOR:$TOKEN" \ + "https://git.fabledsword.com/v2/token?scope=repository:$REPO:pull,push&service=git.fabledsword.com" \ + | python3 -c 'import sys,json; print(json.load(sys.stdin)["token"])') + + # Ask for the image manifest media types ONLY. Offering the index + # types too would let the registry hand back an index if one ever + # existed at this tag, and we would faithfully copy the thing we are + # trying not to create. + ACCEPT='application/vnd.oci.image.manifest.v1+json, application/vnd.docker.distribution.manifest.v2+json' + CT=$(curl -fsS --max-time 60 -o manifest.json -D headers.txt \ + -H "Authorization: Bearer $BEARER" -H "Accept: $ACCEPT" \ + "https://git.fabledsword.com/v2/$REPO/manifests/refresh-candidate" \ + && tr -d '\r' < headers.txt | awk -F': ' '/^[Cc]ontent-[Tt]ype:/{print $2}') + test -n "$CT" + SRC_DIGEST=$(tr -d '\r' < headers.txt | awk -F': ' '/^[Dd]ocker-[Cc]ontent-[Dd]igest:/{print $2}') + echo "promote: candidate is $SRC_DIGEST ($CT)" + + curl -fsS --max-time 120 -X PUT \ + -H "Authorization: Bearer $BEARER" -H "Content-Type: $CT" \ + --data-binary @manifest.json \ + "https://git.fabledsword.com/v2/$REPO/manifests/$TAG" + + # Read it back. A PUT that returned 2xx but landed something else is + # exactly the silent-and-plausible failure this pipeline keeps + # producing, and the check costs one request. + NOW=$(curl -fsS --max-time 30 -o /dev/null -D - \ + -H "Authorization: Bearer $BEARER" -H "Accept: $ACCEPT" \ + "https://git.fabledsword.com/v2/$REPO/manifests/$TAG" \ + | tr -d '\r' | awk -F': ' '/^[Dd]ocker-[Cc]ontent-[Dd]igest:/{print $2}') + if [ "$NOW" != "$SRC_DIGEST" ]; then + echo "promote: $IMAGE:$TAG is $NOW, expected $SRC_DIGEST" >&2 + exit 1 + fi + echo "promote: $IMAGE:$TAG now names $NOW" + # Every tag but the channel's own is written HERE, registry-side, # whether or not a build ran. Each -t becomes another reference to the # SAME manifest the channel tag holds, so :c- is byte-identical to @@ -1121,6 +1227,41 @@ jobs: if [ "$CHANNEL" = "main" ]; then T=latest; else T=dev; fi echo "channel_ref=$IMAGE:$T" >> "$GITHUB_OUTPUT" + # WHERE THE BUILD PUBLISHES, which is not always the channel — and + # whether the channel then has to be written separately. + # + # On a push the build writes the channel tag directly: the bytes came + # from a commit, and a commit is the thing CI tests. Nothing to hold + # it behind. + # + # On the scheduled refresh it writes a CANDIDATE tag instead. A + # refresh rebuilds against freshly resolved base images, and the web + # image's runtime is a line of UNPINNED Debian packages (ffmpeg, + # libjpeg62-turbo, libpq5, megatools…) re-resolved on every build. + # Nothing in ci.yml can see that: its lanes run on ci-python:3.14 and + # install requirements.txt, and a base bump changes neither. So + # refreshed bytes have to be proven before :latest names them, and + # proving needs a moment between "built" and "published" to occupy. + # This is that moment; :latest goes on naming the build that works + # until something says otherwise. + # + # `:refresh-candidate` is one moving ref per image, overwritten in + # place, holding a build nobody is told to pull — the shape rule 145 + # already allows for :buildcache, not the per-build tag family that + # milestone 318 withdrew. + # + # Both values are decided HERE, beside `hit`, for the reason the + # force/schedule branch below gives: one step decides what this job + # does. A promote condition derived independently could disagree with + # the tag the build actually wrote. + if [ "${EVENT:-}" = "schedule" ]; then + echo "build_ref=$IMAGE:refresh-candidate" >> "$GITHUB_OUTPUT" + echo "promote=true" >> "$GITHUB_OUTPUT" + else + echo "build_ref=$IMAGE:$T" >> "$GITHUB_OUTPUT" + echo "promote=false" >> "$GITHUB_OUTPUT" + fi + # Compare VALUES, never exit codes. Measured on buildx v0.36.1 # (run 4732): a missing key returns an empty string and exits 0, so # branching on the exit code would read "no label yet" as success. @@ -1211,7 +1352,7 @@ jobs: # out of a local image store a registry-direct build never filled — # #3190, which cost `main` its :c- on 2026-08-29 while :latest # published perfectly well. - tags: ${{ steps.reuse.outputs.channel_ref }} + tags: ${{ steps.reuse.outputs.build_ref }} # The reuse key. Read back off the channel tag on the next push to # decide whether that push needs to build at all, so this is not # decoration — an unstamped image is one that will always rebuild. @@ -1252,6 +1393,77 @@ jobs: cache-from: type=registry,ref=git.fabledsword.com/bvandeusen/fabledcurator-ml:buildcache cache-to: type=registry,ref=git.fabledsword.com/bvandeusen/fabledcurator-ml:buildcache,mode=max + # Point the channel tag at the candidate the refresh just built. + # + # Unconditional TODAY, so this milestone never leaves the refresh in a + # state where it builds and publishes nothing. Step 4 wraps it in the + # smoke suite's verdict; until then the scheduled path behaves exactly + # as it did, just via two operations instead of one. + # + # NOT `imagetools create`. That wraps its source in an INDEX, and an + # indexed channel tag is the one thing this pipeline cannot survive: + # `.Image.Config.Labels` does not resolve through an index, so the + # fc.revision the reuse check reads off the channel tag would come back + # empty, every subsequent push would miss and rebuild, and nothing would + # go red. That is #3183, observed on run 4751 — reuse worked exactly once + # and the only symptom was the bill. The repoint step below excludes its + # own source tag for precisely this reason; a promote that re-introduced + # the wrap through a different door would undo that care. + # + # A manifest PUT is what "make this tag name that image" means at the + # registry level: the same bytes under the same media type, so the digest + # is identical, the media type is preserved, and no layer moves. + - name: Promote the refresh candidate to the channel + if: steps.reuse.outputs.promote == 'true' + env: + IMAGE: git.fabledsword.com/bvandeusen/fabledcurator-ml + CHANNEL_REF: ${{ steps.reuse.outputs.channel_ref }} + TOKEN: ${{ secrets.RELEASE_TOKEN }} + ACTOR: ${{ github.actor }} + run: | + set -eu + REPO=${IMAGE#git.fabledsword.com/} + TAG=${CHANNEL_REF##*:} + + # Registry auth is its own token exchange — the `docker login` above + # authenticates the docker client, not curl. Deadline on every call + # (rule 156): a registry that stops answering must fail this step, + # not hang the weekly refresh until the job times out. + BEARER=$(curl -fsS --max-time 30 -u "$ACTOR:$TOKEN" \ + "https://git.fabledsword.com/v2/token?scope=repository:$REPO:pull,push&service=git.fabledsword.com" \ + | python3 -c 'import sys,json; print(json.load(sys.stdin)["token"])') + + # Ask for the image manifest media types ONLY. Offering the index + # types too would let the registry hand back an index if one ever + # existed at this tag, and we would faithfully copy the thing we are + # trying not to create. + ACCEPT='application/vnd.oci.image.manifest.v1+json, application/vnd.docker.distribution.manifest.v2+json' + CT=$(curl -fsS --max-time 60 -o manifest.json -D headers.txt \ + -H "Authorization: Bearer $BEARER" -H "Accept: $ACCEPT" \ + "https://git.fabledsword.com/v2/$REPO/manifests/refresh-candidate" \ + && tr -d '\r' < headers.txt | awk -F': ' '/^[Cc]ontent-[Tt]ype:/{print $2}') + test -n "$CT" + SRC_DIGEST=$(tr -d '\r' < headers.txt | awk -F': ' '/^[Dd]ocker-[Cc]ontent-[Dd]igest:/{print $2}') + echo "promote: candidate is $SRC_DIGEST ($CT)" + + curl -fsS --max-time 120 -X PUT \ + -H "Authorization: Bearer $BEARER" -H "Content-Type: $CT" \ + --data-binary @manifest.json \ + "https://git.fabledsword.com/v2/$REPO/manifests/$TAG" + + # Read it back. A PUT that returned 2xx but landed something else is + # exactly the silent-and-plausible failure this pipeline keeps + # producing, and the check costs one request. + NOW=$(curl -fsS --max-time 30 -o /dev/null -D - \ + -H "Authorization: Bearer $BEARER" -H "Accept: $ACCEPT" \ + "https://git.fabledsword.com/v2/$REPO/manifests/$TAG" \ + | tr -d '\r' | awk -F': ' '/^[Dd]ocker-[Cc]ontent-[Dd]igest:/{print $2}') + if [ "$NOW" != "$SRC_DIGEST" ]; then + echo "promote: $IMAGE:$TAG is $NOW, expected $SRC_DIGEST" >&2 + exit 1 + fi + echo "promote: $IMAGE:$TAG now names $NOW" + # Every tag but the channel's own is written HERE, registry-side, # whether or not a build ran. Each -t becomes another reference to the # SAME manifest the channel tag holds, so :c- is byte-identical to @@ -1501,6 +1713,41 @@ jobs: if [ "$CHANNEL" = "main" ]; then T=latest; else T=dev; fi echo "channel_ref=$IMAGE:$T" >> "$GITHUB_OUTPUT" + # WHERE THE BUILD PUBLISHES, which is not always the channel — and + # whether the channel then has to be written separately. + # + # On a push the build writes the channel tag directly: the bytes came + # from a commit, and a commit is the thing CI tests. Nothing to hold + # it behind. + # + # On the scheduled refresh it writes a CANDIDATE tag instead. A + # refresh rebuilds against freshly resolved base images, and the web + # image's runtime is a line of UNPINNED Debian packages (ffmpeg, + # libjpeg62-turbo, libpq5, megatools…) re-resolved on every build. + # Nothing in ci.yml can see that: its lanes run on ci-python:3.14 and + # install requirements.txt, and a base bump changes neither. So + # refreshed bytes have to be proven before :latest names them, and + # proving needs a moment between "built" and "published" to occupy. + # This is that moment; :latest goes on naming the build that works + # until something says otherwise. + # + # `:refresh-candidate` is one moving ref per image, overwritten in + # place, holding a build nobody is told to pull — the shape rule 145 + # already allows for :buildcache, not the per-build tag family that + # milestone 318 withdrew. + # + # Both values are decided HERE, beside `hit`, for the reason the + # force/schedule branch below gives: one step decides what this job + # does. A promote condition derived independently could disagree with + # the tag the build actually wrote. + if [ "${EVENT:-}" = "schedule" ]; then + echo "build_ref=$IMAGE:refresh-candidate" >> "$GITHUB_OUTPUT" + echo "promote=true" >> "$GITHUB_OUTPUT" + else + echo "build_ref=$IMAGE:$T" >> "$GITHUB_OUTPUT" + echo "promote=false" >> "$GITHUB_OUTPUT" + fi + # Compare VALUES, never exit codes. Measured on buildx v0.36.1 # (run 4732): a missing key returns an empty string and exits 0, so # branching on the exit code would read "no label yet" as success. @@ -1591,7 +1838,7 @@ jobs: # out of a local image store a registry-direct build never filled — # #3190, which cost `main` its :c- on 2026-08-29 while :latest # published perfectly well. - tags: ${{ steps.reuse.outputs.channel_ref }} + tags: ${{ steps.reuse.outputs.build_ref }} # The reuse key. Read back off the channel tag on the next push to # decide whether that push needs to build at all, so this is not # decoration — an unstamped image is one that will always rebuild. @@ -1632,6 +1879,77 @@ jobs: cache-from: type=registry,ref=git.fabledsword.com/bvandeusen/fabledcurator-agent:buildcache cache-to: type=registry,ref=git.fabledsword.com/bvandeusen/fabledcurator-agent:buildcache,mode=max + # Point the channel tag at the candidate the refresh just built. + # + # Unconditional TODAY, so this milestone never leaves the refresh in a + # state where it builds and publishes nothing. Step 4 wraps it in the + # smoke suite's verdict; until then the scheduled path behaves exactly + # as it did, just via two operations instead of one. + # + # NOT `imagetools create`. That wraps its source in an INDEX, and an + # indexed channel tag is the one thing this pipeline cannot survive: + # `.Image.Config.Labels` does not resolve through an index, so the + # fc.revision the reuse check reads off the channel tag would come back + # empty, every subsequent push would miss and rebuild, and nothing would + # go red. That is #3183, observed on run 4751 — reuse worked exactly once + # and the only symptom was the bill. The repoint step below excludes its + # own source tag for precisely this reason; a promote that re-introduced + # the wrap through a different door would undo that care. + # + # A manifest PUT is what "make this tag name that image" means at the + # registry level: the same bytes under the same media type, so the digest + # is identical, the media type is preserved, and no layer moves. + - name: Promote the refresh candidate to the channel + if: steps.reuse.outputs.promote == 'true' + env: + IMAGE: git.fabledsword.com/bvandeusen/fabledcurator-agent + CHANNEL_REF: ${{ steps.reuse.outputs.channel_ref }} + TOKEN: ${{ secrets.RELEASE_TOKEN }} + ACTOR: ${{ github.actor }} + run: | + set -eu + REPO=${IMAGE#git.fabledsword.com/} + TAG=${CHANNEL_REF##*:} + + # Registry auth is its own token exchange — the `docker login` above + # authenticates the docker client, not curl. Deadline on every call + # (rule 156): a registry that stops answering must fail this step, + # not hang the weekly refresh until the job times out. + BEARER=$(curl -fsS --max-time 30 -u "$ACTOR:$TOKEN" \ + "https://git.fabledsword.com/v2/token?scope=repository:$REPO:pull,push&service=git.fabledsword.com" \ + | python3 -c 'import sys,json; print(json.load(sys.stdin)["token"])') + + # Ask for the image manifest media types ONLY. Offering the index + # types too would let the registry hand back an index if one ever + # existed at this tag, and we would faithfully copy the thing we are + # trying not to create. + ACCEPT='application/vnd.oci.image.manifest.v1+json, application/vnd.docker.distribution.manifest.v2+json' + CT=$(curl -fsS --max-time 60 -o manifest.json -D headers.txt \ + -H "Authorization: Bearer $BEARER" -H "Accept: $ACCEPT" \ + "https://git.fabledsword.com/v2/$REPO/manifests/refresh-candidate" \ + && tr -d '\r' < headers.txt | awk -F': ' '/^[Cc]ontent-[Tt]ype:/{print $2}') + test -n "$CT" + SRC_DIGEST=$(tr -d '\r' < headers.txt | awk -F': ' '/^[Dd]ocker-[Cc]ontent-[Dd]igest:/{print $2}') + echo "promote: candidate is $SRC_DIGEST ($CT)" + + curl -fsS --max-time 120 -X PUT \ + -H "Authorization: Bearer $BEARER" -H "Content-Type: $CT" \ + --data-binary @manifest.json \ + "https://git.fabledsword.com/v2/$REPO/manifests/$TAG" + + # Read it back. A PUT that returned 2xx but landed something else is + # exactly the silent-and-plausible failure this pipeline keeps + # producing, and the check costs one request. + NOW=$(curl -fsS --max-time 30 -o /dev/null -D - \ + -H "Authorization: Bearer $BEARER" -H "Accept: $ACCEPT" \ + "https://git.fabledsword.com/v2/$REPO/manifests/$TAG" \ + | tr -d '\r' | awk -F': ' '/^[Dd]ocker-[Cc]ontent-[Dd]igest:/{print $2}') + if [ "$NOW" != "$SRC_DIGEST" ]; then + echo "promote: $IMAGE:$TAG is $NOW, expected $SRC_DIGEST" >&2 + exit 1 + fi + echo "promote: $IMAGE:$TAG now names $NOW" + # Every tag but the channel's own is written HERE, registry-side, # whether or not a build ran. Each -t becomes another reference to the # SAME manifest the channel tag holds, so :c- is byte-identical to -- 2.54.0 From bfc4f9cec9023fb0b3cb1a9bf605ab2ff1a00127 Mon Sep 17 00:00:00 2001 From: Bryan Van Deusen Date: Wed, 2 Sep 2026 14:41:33 -0400 Subject: [PATCH 4/4] ci: one fact for "is this a base refresh", and a lever to trigger one MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Milestone 362, enabling step 2's verification and everything after it. The weekly refresh was testable once a week. That is not a cadence anything can be developed against, and milestone 362's whole point is a gate — which has to be watched rejecting something before anyone can believe it is wired up. So `refresh` joins `force_build` as a dispatch input, on the same reasoning that added that one (#3252: confirm #3190 was gone rather than wait for it to recur). Adding it meant confronting that "is this a refresh?" was asked in five places and spelled five ways: `github.event_name == 'schedule'` in an `if:`, `$GITHUB_EVENT_NAME` in one shell, an `EVENT:` env passed into another, and a bare expression on `pull:`. Five spellings of one fact is how half of them come to disagree once somebody adds a sixth trigger — which is precisely what this commit is. So it is derived once at the top, next to BUILD_REF, which already exists for exactly this reason on exactly this question. String comparison, not boolean: Forgejo delivers dispatch inputs as strings, so `inputs.refresh` is 'true'/'false' and `&&` on it would read the string 'false' as truthy. **A constraint this makes visible, which pre-dates it.** A refresh checks out `main` (BUILD_REF) while running the workflow definition from the branch that triggered it — the cron registers from the default branch. So dev's workflow builds main's source, and dev's workflow cannot depend on anything main's tree does not have yet. It does now: the reuse step calls `artifacts.sh epoch`, which lands on main with this batch. Until then a refresh dispatch fails loudly at that call, which is the right failure — the alternative is tolerating a missing epoch and silently rebuilding #3265 into every refresh. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01TTjbZZ6JirCMSaJzQV1RhA --- .forgejo/workflows/build.yml | 58 +++++++++++++++++++++++------------- 1 file changed, 38 insertions(+), 20 deletions(-) diff --git a/.forgejo/workflows/build.yml b/.forgejo/workflows/build.yml index cd895f2..aa49ad5 100644 --- a/.forgejo/workflows/build.yml +++ b/.forgejo/workflows/build.yml @@ -44,6 +44,10 @@ on: description: 'Rebuild every image even if the published revision matches' type: boolean default: false + refresh: + description: 'Behave as the weekly base refresh: build main against fresh bases, publish through the candidate tag' + type: boolean + default: false # The base-image refresh (milestone 326 step 4, #3154). # @@ -72,8 +76,25 @@ on: # Deriving it per job invites the two halves to disagree: sign-extension would # derive dev's extension version while build-web bundled main's, and the # release download would 404 on a version that exists perfectly well. +# IS THIS A BASE REFRESH? Asked in five places and previously spelled five +# ways — `github.event_name == 'schedule'` in an `if:`, `$GITHUB_EVENT_NAME` in +# one shell, an `EVENT:` env passed into another, and a bare expression on +# `pull:`. Five spellings of one fact is how half of them come to disagree +# after somebody adds a sixth trigger. +# +# The `refresh` dispatch input is here so this path can be EXERCISED. A weekly +# cron is otherwise testable once a week, which is not a cadence anything can +# be developed against — the same reason `force_build` exists (#3252, added to +# confirm #3190 was gone rather than wait for it to recur). It is also what +# makes the milestone-362 gate verifiable at all: a gate has to be watched +# rejecting something before anyone can believe it is wired up. +# +# Note this is a STRING comparison, not a boolean. Forgejo delivers +# workflow_dispatch inputs as strings, so `inputs.refresh` is 'true'/'false' +# and `&&` on it would treat the string 'false' as truthy. env: - BUILD_REF: ${{ github.event_name == 'schedule' && 'main' || github.ref }} + IS_REFRESH: ${{ (github.event_name == 'schedule' || github.event.inputs.refresh == 'true') && 'true' || 'false' }} + BUILD_REF: ${{ (github.event_name == 'schedule' || github.event.inputs.refresh == 'true') && 'main' || github.ref }} # Requires repo secret RELEASE_TOKEN — a Forgejo PAT with scopes: # - write:package, read:package (for docker push to git.fabledsword.com) @@ -143,7 +164,7 @@ jobs: # evaluate — this file already gates steps on it — so the guard cannot # be disabled by the same uncertainty it exists to cover. - name: Guard — a scheduled run must have checked out main - if: github.event_name == 'schedule' + if: env.IS_REFRESH == 'true' run: | set -eu BRANCH=$(git rev-parse --abbrev-ref HEAD) @@ -437,7 +458,7 @@ jobs: # See sign-extension's copy for why this guard exists. - name: Guard — a scheduled run must have checked out main - if: github.event_name == 'schedule' + if: env.IS_REFRESH == 'true' run: | set -eu BRANCH=$(git rev-parse --abbrev-ref HEAD) @@ -528,7 +549,7 @@ jobs: # Checked BEFORE the ref test, not after: a scheduled run's # GITHUB_REF is the default branch (dev), so the main test would # never fire on it. - if [ "${GITHUB_EVENT_NAME:-}" = "schedule" ]; then + if [ "${IS_REFRESH:-}" = "true" ]; then echo "tags=git.fabledsword.com/bvandeusen/fabledcurator:latest" >> "$GITHUB_OUTPUT" echo "channel=main" >> "$GITHUB_OUTPUT" elif [ "${GITHUB_REF##*/}" = "main" ]; then @@ -628,7 +649,6 @@ jobs: # A scheduled refresh has to bypass reuse by construction: it # rebuilds the SAME source, so fc.revision always matches and the # check would skip every refresh there has ever been. - EVENT: ${{ github.event_name }} run: | set -eu DERIVED=$(sh scripts/artifacts.sh revision web) @@ -678,7 +698,7 @@ jobs: # force/schedule branch below gives: one step decides what this job # does. A promote condition derived independently could disagree with # the tag the build actually wrote. - if [ "${EVENT:-}" = "schedule" ]; then + if [ "${IS_REFRESH:-}" = "true" ]; then echo "build_ref=$IMAGE:refresh-candidate" >> "$GITHUB_OUTPUT" echo "promote=true" >> "$GITHUB_OUTPUT" else @@ -718,7 +738,7 @@ jobs: if [ "${FORCE:-false}" = "true" ]; then echo "hit=false" >> "$GITHUB_OUTPUT" echo "reuse: force_build set — building regardless" - elif [ "${EVENT:-}" = "schedule" ]; then + elif [ "${IS_REFRESH:-}" = "true" ]; then echo "hit=false" >> "$GITHUB_OUTPUT" echo "reuse: scheduled base refresh — building regardless" elif [ -n "$PUBLISHED" ] && [ "$PUBLISHED" = "$DERIVED" ]; then @@ -845,7 +865,7 @@ jobs: # churn #3265 is about. # # Only on the schedule. An ordinary push wants the cached base. - pull: ${{ github.event_name == 'schedule' }} + pull: ${{ env.IS_REFRESH == 'true' }} # ONE tag, the channel's. Every other tag is written by the step # below, registry-side. buildx here pushes the first tag to the # registry and then re-pushes the rest through the DOCKER driver, @@ -1074,7 +1094,7 @@ jobs: # See sign-extension's copy for why this guard exists. - name: Guard — a scheduled run must have checked out main - if: github.event_name == 'schedule' + if: env.IS_REFRESH == 'true' run: | set -eu BRANCH=$(git rev-parse --abbrev-ref HEAD) @@ -1125,7 +1145,7 @@ jobs: SHORT_SHA=$(printf '%s' "$GITHUB_SHA" | cut -c1-7) # Mirrors build-web's tag list and its schedule handling; see # the comments there. - if [ "${GITHUB_EVENT_NAME:-}" = "schedule" ]; then + if [ "${IS_REFRESH:-}" = "true" ]; then echo "tags=git.fabledsword.com/bvandeusen/fabledcurator-ml:latest" >> "$GITHUB_OUTPUT" echo "channel=main" >> "$GITHUB_OUTPUT" elif [ "${GITHUB_REF##*/}" = "main" ]; then @@ -1208,7 +1228,6 @@ jobs: # A scheduled refresh has to bypass reuse by construction: it # rebuilds the SAME source, so fc.revision always matches and the # check would skip every refresh there has ever been. - EVENT: ${{ github.event_name }} run: | set -eu DERIVED=$(sh scripts/artifacts.sh revision ml) @@ -1254,7 +1273,7 @@ jobs: # force/schedule branch below gives: one step decides what this job # does. A promote condition derived independently could disagree with # the tag the build actually wrote. - if [ "${EVENT:-}" = "schedule" ]; then + if [ "${IS_REFRESH:-}" = "true" ]; then echo "build_ref=$IMAGE:refresh-candidate" >> "$GITHUB_OUTPUT" echo "promote=true" >> "$GITHUB_OUTPUT" else @@ -1294,7 +1313,7 @@ jobs: if [ "${FORCE:-false}" = "true" ]; then echo "hit=false" >> "$GITHUB_OUTPUT" echo "reuse: force_build set — building regardless" - elif [ "${EVENT:-}" = "schedule" ]; then + elif [ "${IS_REFRESH:-}" = "true" ]; then echo "hit=false" >> "$GITHUB_OUTPUT" echo "reuse: scheduled base refresh — building regardless" elif [ -n "$PUBLISHED" ] && [ "$PUBLISHED" = "$DERIVED" ]; then @@ -1345,7 +1364,7 @@ jobs: # churn #3265 is about. # # Only on the schedule. An ordinary push wants the cached base. - pull: ${{ github.event_name == 'schedule' }} + pull: ${{ env.IS_REFRESH == 'true' }} # ONE tag, the channel's. Every other tag is written by the step # below, registry-side. buildx here pushes the first tag to the # registry and then re-pushes the rest through the DOCKER driver, @@ -1565,7 +1584,7 @@ jobs: # See sign-extension's copy for why this guard exists. - name: Guard — a scheduled run must have checked out main - if: github.event_name == 'schedule' + if: env.IS_REFRESH == 'true' run: | set -eu BRANCH=$(git rev-parse --abbrev-ref HEAD) @@ -1611,7 +1630,7 @@ jobs: SHORT_SHA=$(printf '%s' "$GITHUB_SHA" | cut -c1-7) # Mirrors build-web's tag list and its schedule handling; see # the comments there. - if [ "${GITHUB_EVENT_NAME:-}" = "schedule" ]; then + if [ "${IS_REFRESH:-}" = "true" ]; then echo "tags=git.fabledsword.com/bvandeusen/fabledcurator-agent:latest" >> "$GITHUB_OUTPUT" echo "channel=main" >> "$GITHUB_OUTPUT" elif [ "${GITHUB_REF##*/}" = "main" ]; then @@ -1694,7 +1713,6 @@ jobs: # A scheduled refresh has to bypass reuse by construction: it # rebuilds the SAME source, so fc.revision always matches and the # check would skip every refresh there has ever been. - EVENT: ${{ github.event_name }} run: | set -eu DERIVED=$(sh scripts/artifacts.sh revision agent) @@ -1740,7 +1758,7 @@ jobs: # force/schedule branch below gives: one step decides what this job # does. A promote condition derived independently could disagree with # the tag the build actually wrote. - if [ "${EVENT:-}" = "schedule" ]; then + if [ "${IS_REFRESH:-}" = "true" ]; then echo "build_ref=$IMAGE:refresh-candidate" >> "$GITHUB_OUTPUT" echo "promote=true" >> "$GITHUB_OUTPUT" else @@ -1780,7 +1798,7 @@ jobs: if [ "${FORCE:-false}" = "true" ]; then echo "hit=false" >> "$GITHUB_OUTPUT" echo "reuse: force_build set — building regardless" - elif [ "${EVENT:-}" = "schedule" ]; then + elif [ "${IS_REFRESH:-}" = "true" ]; then echo "hit=false" >> "$GITHUB_OUTPUT" echo "reuse: scheduled base refresh — building regardless" elif [ -n "$PUBLISHED" ] && [ "$PUBLISHED" = "$DERIVED" ]; then @@ -1831,7 +1849,7 @@ jobs: # churn #3265 is about. # # Only on the schedule. An ordinary push wants the cached base. - pull: ${{ github.event_name == 'schedule' }} + pull: ${{ env.IS_REFRESH == 'true' }} # ONE tag, the channel's. Every other tag is written by the step # below, registry-side. buildx here pushes the first tag to the # registry and then re-pushes the rest through the DOCKER driver, -- 2.54.0