From 0a5bbe81dc1e5483cd98e4c9f59af57b54f68c8b Mon Sep 17 00:00:00 2001 From: Bryan Van Deusen Date: Sun, 30 Aug 2026 12:57:06 -0400 Subject: [PATCH] docs: the scheduled refresh does NOT republish nothing (#3265) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Step 4 asserted that when the base has not moved the refresh is "a ~13s no-op that republishes nothing", and that this no-op was the point. The first half is false and was written without being tested. Run 4934, the first real fire: every content step reported CACHED and both bases resolved to unchanged pinned digests, yet all three :latest tags took a new manifest digest anyway. fabledcurator 4ea5265ba017 -> 380e504de0fa fabledcurator-ml 6e7cfc0c09fd -> 6b2eefc301d8 fabledcurator-agent 44920e0af1f3 -> 54accbeb52ed buildkit mints a fresh image config per run, so identical layers get republished under a new config blob. Storage cost is trivial; the cost that matters is that a :latest digest change stops meaning "something is different", and :c- is handed a new manifest to diverge from every Sunday for no reason. Corrects the workflow comment (x3) and ci-requirements.md to say what actually happens. Filed as #3265 with the candidate fixes; the likely one is a deterministic SOURCE_DATE_EPOCH off the value artifacts.sh already derives, which would make "same source, same version" into "same source, same bytes". The rest of step 4 verified clean on the same run: the guard passed (HEAD is main (499720d), `git checkout -B main`) — so this runner DOES evaluate the env context inside `with:` — the tag list was :latest alone with no :c-, and the repoint step correctly found nothing to write. --- .forgejo/workflows/build.yml | 99 ++++++++++++++++++++++-------------- ci-requirements.md | 23 +++++---- 2 files changed, 73 insertions(+), 49 deletions(-) diff --git a/.forgejo/workflows/build.yml b/.forgejo/workflows/build.yml index 7498f3e..7bd8066 100644 --- a/.forgejo/workflows/build.yml +++ b/.forgejo/workflows/build.yml @@ -773,23 +773,30 @@ jobs: # trusting whatever digest the cache was built against. This is the # whole mechanism of the scheduled refresh (#3154): if the base tag # moved, the FROM layer's cache key changes, every layer above it - # invalidates, and the image genuinely rebuilds. If it did not move, - # the registry cache satisfies the entire graph and the refresh is a - # ~13s no-op that republishes nothing. + # invalidates, and the image genuinely rebuilds. # - # That no-op is the POINT, not a shortfall: :latest should change - # when there is something new in it and not otherwise. A refresh - # that rewrote the image weekly regardless would churn the registry - # and hand :c- a new manifest to diverge from every Sunday, for - # no gain. + # MEASURED on the first real fire, run 4934 (#3265): when the base + # did NOT move, the build is ~13s and every content step reports + # CACHED — but the channel tag STILL gets a new manifest digest. + # buildkit mints a fresh image config each run, so identical layers + # are republished under a new config blob. All three images moved + # that way on 2026-08-30 with nothing whatsoever changed in them. # - # What it therefore does NOT catch: a Debian package update inside + # So a refresh currently rewrites :latest every Sunday whether or + # not there is anything new in it, and :c- is handed a new + # manifest to diverge from on the same cadence. Layers are shared, + # so the storage cost is a config blob; the cost that matters is + # that a digest change no longer MEANS anything. Tracked in #3265 — + # the likely fix is a deterministic SOURCE_DATE_EPOCH, which would + # make "same source, same bytes" true and turn the no-op case into + # a genuine no-op. + # + # What `pull` does NOT catch either: a Debian package update inside # the `apt-get install` layer while the base tag itself stands # still. The official python/cuda images rebuild with those updates - # baked in, so this is a lag rather than a hole — but closing it - # would take `no-cache: true` on the scheduled path, which is the - # weekly-churn trade above. Left as the cheaper of the two on - # purpose. + # baked in, so this is a lag rather than a hole; closing it needs + # `no-cache: true`, which is a much larger version of the same + # churn #3265 is about. # # Only on the schedule. An ordinary push wants the cached base. pull: ${{ github.event_name == 'schedule' }} @@ -1149,23 +1156,30 @@ jobs: # trusting whatever digest the cache was built against. This is the # whole mechanism of the scheduled refresh (#3154): if the base tag # moved, the FROM layer's cache key changes, every layer above it - # invalidates, and the image genuinely rebuilds. If it did not move, - # the registry cache satisfies the entire graph and the refresh is a - # ~13s no-op that republishes nothing. + # invalidates, and the image genuinely rebuilds. # - # That no-op is the POINT, not a shortfall: :latest should change - # when there is something new in it and not otherwise. A refresh - # that rewrote the image weekly regardless would churn the registry - # and hand :c- a new manifest to diverge from every Sunday, for - # no gain. + # MEASURED on the first real fire, run 4934 (#3265): when the base + # did NOT move, the build is ~13s and every content step reports + # CACHED — but the channel tag STILL gets a new manifest digest. + # buildkit mints a fresh image config each run, so identical layers + # are republished under a new config blob. All three images moved + # that way on 2026-08-30 with nothing whatsoever changed in them. # - # What it therefore does NOT catch: a Debian package update inside + # So a refresh currently rewrites :latest every Sunday whether or + # not there is anything new in it, and :c- is handed a new + # manifest to diverge from on the same cadence. Layers are shared, + # so the storage cost is a config blob; the cost that matters is + # that a digest change no longer MEANS anything. Tracked in #3265 — + # the likely fix is a deterministic SOURCE_DATE_EPOCH, which would + # make "same source, same bytes" true and turn the no-op case into + # a genuine no-op. + # + # What `pull` does NOT catch either: a Debian package update inside # the `apt-get install` layer while the base tag itself stands # still. The official python/cuda images rebuild with those updates - # baked in, so this is a lag rather than a hole — but closing it - # would take `no-cache: true` on the scheduled path, which is the - # weekly-churn trade above. Left as the cheaper of the two on - # purpose. + # baked in, so this is a lag rather than a hole; closing it needs + # `no-cache: true`, which is a much larger version of the same + # churn #3265 is about. # # Only on the schedule. An ordinary push wants the cached base. pull: ${{ github.event_name == 'schedule' }} @@ -1511,23 +1525,30 @@ jobs: # trusting whatever digest the cache was built against. This is the # whole mechanism of the scheduled refresh (#3154): if the base tag # moved, the FROM layer's cache key changes, every layer above it - # invalidates, and the image genuinely rebuilds. If it did not move, - # the registry cache satisfies the entire graph and the refresh is a - # ~13s no-op that republishes nothing. + # invalidates, and the image genuinely rebuilds. # - # That no-op is the POINT, not a shortfall: :latest should change - # when there is something new in it and not otherwise. A refresh - # that rewrote the image weekly regardless would churn the registry - # and hand :c- a new manifest to diverge from every Sunday, for - # no gain. + # MEASURED on the first real fire, run 4934 (#3265): when the base + # did NOT move, the build is ~13s and every content step reports + # CACHED — but the channel tag STILL gets a new manifest digest. + # buildkit mints a fresh image config each run, so identical layers + # are republished under a new config blob. All three images moved + # that way on 2026-08-30 with nothing whatsoever changed in them. # - # What it therefore does NOT catch: a Debian package update inside + # So a refresh currently rewrites :latest every Sunday whether or + # not there is anything new in it, and :c- is handed a new + # manifest to diverge from on the same cadence. Layers are shared, + # so the storage cost is a config blob; the cost that matters is + # that a digest change no longer MEANS anything. Tracked in #3265 — + # the likely fix is a deterministic SOURCE_DATE_EPOCH, which would + # make "same source, same bytes" true and turn the no-op case into + # a genuine no-op. + # + # What `pull` does NOT catch either: a Debian package update inside # the `apt-get install` layer while the base tag itself stands # still. The official python/cuda images rebuild with those updates - # baked in, so this is a lag rather than a hole — but closing it - # would take `no-cache: true` on the scheduled path, which is the - # weekly-churn trade above. Left as the cheaper of the two on - # purpose. + # baked in, so this is a lag rather than a hole; closing it needs + # `no-cache: true`, which is a much larger version of the same + # churn #3265 is about. # # Only on the schedule. An ordinary push wants the cached base. pull: ${{ github.event_name == 'schedule' }} diff --git a/ci-requirements.md b/ci-requirements.md index 016fdd3..ccd3081 100644 --- a/ci-requirements.md +++ b/ci-requirements.md @@ -196,16 +196,19 @@ per `docs/process.md`'s "add deps to the image when used by >1 project". push path needed no change for this, because the repoint already excluded the source tag — the same rule that keeps the label readable also keeps a refresh from being undone. - - **`pull: true` on the scheduled path only** is the actual mechanism. If a - base tag moved, the `FROM` layer's cache key changes and everything above - it rebuilds; if it did not, the registry cache satisfies the whole graph - and the refresh is a ~13s no-op that republishes nothing. That no-op is the - point — `:latest` should change when there is something new in it, not - every Sunday. The known lag: a Debian package update inside the `apt-get - install` layer while the base tag stands still is not caught. Closing it - needs `no-cache: true`, which buys weekly churn for it; the official - python/cuda images rebuild with those updates baked in, so this is a lag - rather than a hole. + - **`pull: true` on the scheduled path only** is the mechanism: a moved base + tag changes the `FROM` layer's cache key and everything above it rebuilds. + **It does not currently make the unmoved case free.** Measured on the first + real fire (run 4934, 2026-08-30): every content step reported `CACHED` and + the bases resolved to unchanged digests, yet all three `:latest` tags got a + NEW manifest digest, because buildkit mints a fresh image config per run and + republishes identical layers under it. So `:latest` is rewritten weekly + whether or not anything changed, and `:c-` is handed a new manifest to + diverge from on the same cadence — a digest change stops meaning anything. + Tracked as #3265; the likely fix is a deterministic `SOURCE_DATE_EPOCH`. + Separately not caught: a Debian package update inside the `apt-get install` + layer while the base tag stands still — a lag rather than a hole, since the + official python/cuda images rebuild with those updates baked in. - **`FC_CHANNEL` and `FC_VERSION` are build args, not runtime settings.** `build.yml` passes them to the web image only — the ml and agent images have nothing to report them to. `/api/health` returns both, the foot of Settings