.forgejo/workflows/build.yml

@@ -773,23 +773,30 @@ jobs:
           # trusting whatever digest the cache was built against. This is the
           # whole mechanism of the scheduled refresh (#3154): if the base tag
           # moved, the FROM layer's cache key changes, every layer above it
-          # invalidates, and the image genuinely rebuilds. If it did not move,
-          # the registry cache satisfies the entire graph and the refresh is a
-          # ~13s no-op that republishes nothing.
+          # invalidates, and the image genuinely rebuilds.
           #
-          # That no-op is the POINT, not a shortfall: :latest should change
-          # when there is something new in it and not otherwise. A refresh
-          # that rewrote the image weekly regardless would churn the registry
-          # and hand :c-<sha> a new manifest to diverge from every Sunday, for
-          # no gain.
+          # MEASURED on the first real fire, run 4934 (#3265): when the base
+          # did NOT move, the build is ~13s and every content step reports
+          # CACHED — but the channel tag STILL gets a new manifest digest.
+          # buildkit mints a fresh image config each run, so identical layers
+          # are republished under a new config blob. All three images moved
+          # that way on 2026-08-30 with nothing whatsoever changed in them.
           #
-          # What it therefore does NOT catch: a Debian package update inside
+          # So a refresh currently rewrites :latest every Sunday whether or
+          # not there is anything new in it, and :c-<sha> is handed a new
+          # manifest to diverge from on the same cadence. Layers are shared,
+          # so the storage cost is a config blob; the cost that matters is
+          # that a digest change no longer MEANS anything. Tracked in #3265 —
+          # the likely fix is a deterministic SOURCE_DATE_EPOCH, which would
+          # make "same source, same bytes" true and turn the no-op case into
+          # a genuine no-op.
+          #
+          # What `pull` does NOT catch either: a Debian package update inside
           # the `apt-get install` layer while the base tag itself stands
           # still. The official python/cuda images rebuild with those updates
-          # baked in, so this is a lag rather than a hole — but closing it
-          # would take `no-cache: true` on the scheduled path, which is the
-          # weekly-churn trade above. Left as the cheaper of the two on
-          # purpose.
+          # baked in, so this is a lag rather than a hole; closing it needs
+          # `no-cache: true`, which is a much larger version of the same
+          # churn #3265 is about.
           #
           # Only on the schedule. An ordinary push wants the cached base.
           pull: ${{ github.event_name == 'schedule' }}
@@ -1149,23 +1156,30 @@ jobs:
           # trusting whatever digest the cache was built against. This is the
           # whole mechanism of the scheduled refresh (#3154): if the base tag
           # moved, the FROM layer's cache key changes, every layer above it
-          # invalidates, and the image genuinely rebuilds. If it did not move,
-          # the registry cache satisfies the entire graph and the refresh is a
-          # ~13s no-op that republishes nothing.
+          # invalidates, and the image genuinely rebuilds.
           #
-          # That no-op is the POINT, not a shortfall: :latest should change
-          # when there is something new in it and not otherwise. A refresh
-          # that rewrote the image weekly regardless would churn the registry
-          # and hand :c-<sha> a new manifest to diverge from every Sunday, for
-          # no gain.
+          # MEASURED on the first real fire, run 4934 (#3265): when the base
+          # did NOT move, the build is ~13s and every content step reports
+          # CACHED — but the channel tag STILL gets a new manifest digest.
+          # buildkit mints a fresh image config each run, so identical layers
+          # are republished under a new config blob. All three images moved
+          # that way on 2026-08-30 with nothing whatsoever changed in them.
           #
-          # What it therefore does NOT catch: a Debian package update inside
+          # So a refresh currently rewrites :latest every Sunday whether or
+          # not there is anything new in it, and :c-<sha> is handed a new
+          # manifest to diverge from on the same cadence. Layers are shared,
+          # so the storage cost is a config blob; the cost that matters is
+          # that a digest change no longer MEANS anything. Tracked in #3265 —
+          # the likely fix is a deterministic SOURCE_DATE_EPOCH, which would
+          # make "same source, same bytes" true and turn the no-op case into
+          # a genuine no-op.
+          #
+          # What `pull` does NOT catch either: a Debian package update inside
           # the `apt-get install` layer while the base tag itself stands
           # still. The official python/cuda images rebuild with those updates
-          # baked in, so this is a lag rather than a hole — but closing it
-          # would take `no-cache: true` on the scheduled path, which is the
-          # weekly-churn trade above. Left as the cheaper of the two on
-          # purpose.
+          # baked in, so this is a lag rather than a hole; closing it needs
+          # `no-cache: true`, which is a much larger version of the same
+          # churn #3265 is about.
           #
           # Only on the schedule. An ordinary push wants the cached base.
           pull: ${{ github.event_name == 'schedule' }}
@@ -1511,23 +1525,30 @@ jobs:
           # trusting whatever digest the cache was built against. This is the
           # whole mechanism of the scheduled refresh (#3154): if the base tag
           # moved, the FROM layer's cache key changes, every layer above it
-          # invalidates, and the image genuinely rebuilds. If it did not move,
-          # the registry cache satisfies the entire graph and the refresh is a
-          # ~13s no-op that republishes nothing.
+          # invalidates, and the image genuinely rebuilds.
           #
-          # That no-op is the POINT, not a shortfall: :latest should change
-          # when there is something new in it and not otherwise. A refresh
-          # that rewrote the image weekly regardless would churn the registry
-          # and hand :c-<sha> a new manifest to diverge from every Sunday, for
-          # no gain.
+          # MEASURED on the first real fire, run 4934 (#3265): when the base
+          # did NOT move, the build is ~13s and every content step reports
+          # CACHED — but the channel tag STILL gets a new manifest digest.
+          # buildkit mints a fresh image config each run, so identical layers
+          # are republished under a new config blob. All three images moved
+          # that way on 2026-08-30 with nothing whatsoever changed in them.
           #
-          # What it therefore does NOT catch: a Debian package update inside
+          # So a refresh currently rewrites :latest every Sunday whether or
+          # not there is anything new in it, and :c-<sha> is handed a new
+          # manifest to diverge from on the same cadence. Layers are shared,
+          # so the storage cost is a config blob; the cost that matters is
+          # that a digest change no longer MEANS anything. Tracked in #3265 —
+          # the likely fix is a deterministic SOURCE_DATE_EPOCH, which would
+          # make "same source, same bytes" true and turn the no-op case into
+          # a genuine no-op.
+          #
+          # What `pull` does NOT catch either: a Debian package update inside
           # the `apt-get install` layer while the base tag itself stands
           # still. The official python/cuda images rebuild with those updates
-          # baked in, so this is a lag rather than a hole — but closing it
-          # would take `no-cache: true` on the scheduled path, which is the
-          # weekly-churn trade above. Left as the cheaper of the two on
-          # purpose.
+          # baked in, so this is a lag rather than a hole; closing it needs
+          # `no-cache: true`, which is a much larger version of the same
+          # churn #3265 is about.
           #
           # Only on the schedule. An ordinary push wants the cached base.
           pull: ${{ github.event_name == 'schedule' }}

ci-requirements.md

@@ -196,16 +196,19 @@ per `docs/process.md`'s "add deps to the image when used by >1 project".
     push path needed no change for this, because the repoint already excluded
     the source tag — the same rule that keeps the label readable also keeps a
     refresh from being undone.
-  - **`pull: true` on the scheduled path only** is the actual mechanism. If a
-    base tag moved, the `FROM` layer's cache key changes and everything above
-    it rebuilds; if it did not, the registry cache satisfies the whole graph
-    and the refresh is a ~13s no-op that republishes nothing. That no-op is the
-    point — `:latest` should change when there is something new in it, not
-    every Sunday. The known lag: a Debian package update inside the `apt-get
-    install` layer while the base tag stands still is not caught. Closing it
-    needs `no-cache: true`, which buys weekly churn for it; the official
-    python/cuda images rebuild with those updates baked in, so this is a lag
-    rather than a hole.
+  - **`pull: true` on the scheduled path only** is the mechanism: a moved base
+    tag changes the `FROM` layer's cache key and everything above it rebuilds.
+    **It does not currently make the unmoved case free.** Measured on the first
+    real fire (run 4934, 2026-08-30): every content step reported `CACHED` and
+    the bases resolved to unchanged digests, yet all three `:latest` tags got a
+    NEW manifest digest, because buildkit mints a fresh image config per run and
+    republishes identical layers under it. So `:latest` is rewritten weekly
+    whether or not anything changed, and `:c-<sha>` is handed a new manifest to
+    diverge from on the same cadence — a digest change stops meaning anything.
+    Tracked as #3265; the likely fix is a deterministic `SOURCE_DATE_EPOCH`.
+    Separately not caught: a Debian package update inside the `apt-get install`
+    layer while the base tag stands still — a lag rather than a hole, since the
+    official python/cuda images rebuild with those updates baked in.
 - **`FC_CHANNEL` and `FC_VERSION` are build args, not runtime settings.**
   `build.yml` passes them to the web image only — the ml and agent images have
   nothing to report them to. `/api/health` returns both, the foot of Settings