diff --git a/.forgejo/workflows/build.yml b/.forgejo/workflows/build.yml index 7498f3e..7bd8066 100644 --- a/.forgejo/workflows/build.yml +++ b/.forgejo/workflows/build.yml @@ -773,23 +773,30 @@ jobs: # trusting whatever digest the cache was built against. This is the # whole mechanism of the scheduled refresh (#3154): if the base tag # moved, the FROM layer's cache key changes, every layer above it - # invalidates, and the image genuinely rebuilds. If it did not move, - # the registry cache satisfies the entire graph and the refresh is a - # ~13s no-op that republishes nothing. + # invalidates, and the image genuinely rebuilds. # - # That no-op is the POINT, not a shortfall: :latest should change - # when there is something new in it and not otherwise. A refresh - # that rewrote the image weekly regardless would churn the registry - # and hand :c- a new manifest to diverge from every Sunday, for - # no gain. + # MEASURED on the first real fire, run 4934 (#3265): when the base + # did NOT move, the build is ~13s and every content step reports + # CACHED — but the channel tag STILL gets a new manifest digest. + # buildkit mints a fresh image config each run, so identical layers + # are republished under a new config blob. All three images moved + # that way on 2026-08-30 with nothing whatsoever changed in them. # - # What it therefore does NOT catch: a Debian package update inside + # So a refresh currently rewrites :latest every Sunday whether or + # not there is anything new in it, and :c- is handed a new + # manifest to diverge from on the same cadence. Layers are shared, + # so the storage cost is a config blob; the cost that matters is + # that a digest change no longer MEANS anything. Tracked in #3265 — + # the likely fix is a deterministic SOURCE_DATE_EPOCH, which would + # make "same source, same bytes" true and turn the no-op case into + # a genuine no-op. + # + # What `pull` does NOT catch either: a Debian package update inside # the `apt-get install` layer while the base tag itself stands # still. The official python/cuda images rebuild with those updates - # baked in, so this is a lag rather than a hole — but closing it - # would take `no-cache: true` on the scheduled path, which is the - # weekly-churn trade above. Left as the cheaper of the two on - # purpose. + # baked in, so this is a lag rather than a hole; closing it needs + # `no-cache: true`, which is a much larger version of the same + # churn #3265 is about. # # Only on the schedule. An ordinary push wants the cached base. pull: ${{ github.event_name == 'schedule' }} @@ -1149,23 +1156,30 @@ jobs: # trusting whatever digest the cache was built against. This is the # whole mechanism of the scheduled refresh (#3154): if the base tag # moved, the FROM layer's cache key changes, every layer above it - # invalidates, and the image genuinely rebuilds. If it did not move, - # the registry cache satisfies the entire graph and the refresh is a - # ~13s no-op that republishes nothing. + # invalidates, and the image genuinely rebuilds. # - # That no-op is the POINT, not a shortfall: :latest should change - # when there is something new in it and not otherwise. A refresh - # that rewrote the image weekly regardless would churn the registry - # and hand :c- a new manifest to diverge from every Sunday, for - # no gain. + # MEASURED on the first real fire, run 4934 (#3265): when the base + # did NOT move, the build is ~13s and every content step reports + # CACHED — but the channel tag STILL gets a new manifest digest. + # buildkit mints a fresh image config each run, so identical layers + # are republished under a new config blob. All three images moved + # that way on 2026-08-30 with nothing whatsoever changed in them. # - # What it therefore does NOT catch: a Debian package update inside + # So a refresh currently rewrites :latest every Sunday whether or + # not there is anything new in it, and :c- is handed a new + # manifest to diverge from on the same cadence. Layers are shared, + # so the storage cost is a config blob; the cost that matters is + # that a digest change no longer MEANS anything. Tracked in #3265 — + # the likely fix is a deterministic SOURCE_DATE_EPOCH, which would + # make "same source, same bytes" true and turn the no-op case into + # a genuine no-op. + # + # What `pull` does NOT catch either: a Debian package update inside # the `apt-get install` layer while the base tag itself stands # still. The official python/cuda images rebuild with those updates - # baked in, so this is a lag rather than a hole — but closing it - # would take `no-cache: true` on the scheduled path, which is the - # weekly-churn trade above. Left as the cheaper of the two on - # purpose. + # baked in, so this is a lag rather than a hole; closing it needs + # `no-cache: true`, which is a much larger version of the same + # churn #3265 is about. # # Only on the schedule. An ordinary push wants the cached base. pull: ${{ github.event_name == 'schedule' }} @@ -1511,23 +1525,30 @@ jobs: # trusting whatever digest the cache was built against. This is the # whole mechanism of the scheduled refresh (#3154): if the base tag # moved, the FROM layer's cache key changes, every layer above it - # invalidates, and the image genuinely rebuilds. If it did not move, - # the registry cache satisfies the entire graph and the refresh is a - # ~13s no-op that republishes nothing. + # invalidates, and the image genuinely rebuilds. # - # That no-op is the POINT, not a shortfall: :latest should change - # when there is something new in it and not otherwise. A refresh - # that rewrote the image weekly regardless would churn the registry - # and hand :c- a new manifest to diverge from every Sunday, for - # no gain. + # MEASURED on the first real fire, run 4934 (#3265): when the base + # did NOT move, the build is ~13s and every content step reports + # CACHED — but the channel tag STILL gets a new manifest digest. + # buildkit mints a fresh image config each run, so identical layers + # are republished under a new config blob. All three images moved + # that way on 2026-08-30 with nothing whatsoever changed in them. # - # What it therefore does NOT catch: a Debian package update inside + # So a refresh currently rewrites :latest every Sunday whether or + # not there is anything new in it, and :c- is handed a new + # manifest to diverge from on the same cadence. Layers are shared, + # so the storage cost is a config blob; the cost that matters is + # that a digest change no longer MEANS anything. Tracked in #3265 — + # the likely fix is a deterministic SOURCE_DATE_EPOCH, which would + # make "same source, same bytes" true and turn the no-op case into + # a genuine no-op. + # + # What `pull` does NOT catch either: a Debian package update inside # the `apt-get install` layer while the base tag itself stands # still. The official python/cuda images rebuild with those updates - # baked in, so this is a lag rather than a hole — but closing it - # would take `no-cache: true` on the scheduled path, which is the - # weekly-churn trade above. Left as the cheaper of the two on - # purpose. + # baked in, so this is a lag rather than a hole; closing it needs + # `no-cache: true`, which is a much larger version of the same + # churn #3265 is about. # # Only on the schedule. An ordinary push wants the cached base. pull: ${{ github.event_name == 'schedule' }} diff --git a/ci-requirements.md b/ci-requirements.md index 016fdd3..ccd3081 100644 --- a/ci-requirements.md +++ b/ci-requirements.md @@ -196,16 +196,19 @@ per `docs/process.md`'s "add deps to the image when used by >1 project". push path needed no change for this, because the repoint already excluded the source tag — the same rule that keeps the label readable also keeps a refresh from being undone. - - **`pull: true` on the scheduled path only** is the actual mechanism. If a - base tag moved, the `FROM` layer's cache key changes and everything above - it rebuilds; if it did not, the registry cache satisfies the whole graph - and the refresh is a ~13s no-op that republishes nothing. That no-op is the - point — `:latest` should change when there is something new in it, not - every Sunday. The known lag: a Debian package update inside the `apt-get - install` layer while the base tag stands still is not caught. Closing it - needs `no-cache: true`, which buys weekly churn for it; the official - python/cuda images rebuild with those updates baked in, so this is a lag - rather than a hole. + - **`pull: true` on the scheduled path only** is the mechanism: a moved base + tag changes the `FROM` layer's cache key and everything above it rebuilds. + **It does not currently make the unmoved case free.** Measured on the first + real fire (run 4934, 2026-08-30): every content step reported `CACHED` and + the bases resolved to unchanged digests, yet all three `:latest` tags got a + NEW manifest digest, because buildkit mints a fresh image config per run and + republishes identical layers under it. So `:latest` is rewritten weekly + whether or not anything changed, and `:c-` is handed a new manifest to + diverge from on the same cadence — a digest change stops meaning anything. + Tracked as #3265; the likely fix is a deterministic `SOURCE_DATE_EPOCH`. + Separately not caught: a Debian package update inside the `apt-get install` + layer while the base tag stands still — a lag rather than a hole, since the + official python/cuda images rebuild with those updates baked in. - **`FC_CHANNEL` and `FC_VERSION` are build args, not runtime settings.** `build.yml` passes them to the web image only — the ml and agent images have nothing to report them to. `/api/health` returns both, the foot of Settings