docs: the scheduled refresh does NOT republish nothing (#3265)
Build images / sign-extension (push) Successful in 4s
CI / lint (push) Successful in 4s
CI / extension-version (push) Successful in 5s
Build images / build-ml (push) Successful in 7s
Build images / build-agent (push) Successful in 8s
Build images / build-web (push) Successful in 8s
CI / frontend-build (push) Successful in 20s
extension / lint (push) Successful in 27s
CI / backend-lint-and-test (push) Successful in 52s
CI / integration (push) Successful in 3m58s

Step 4 asserted that when the base has not moved the refresh is "a ~13s
no-op that republishes nothing", and that this no-op was the point. The
first half is false and was written without being tested.

Run 4934, the first real fire: every content step reported CACHED and
both bases resolved to unchanged pinned digests, yet all three :latest
tags took a new manifest digest anyway.

  fabledcurator        4ea5265ba017 -> 380e504de0fa
  fabledcurator-ml     6e7cfc0c09fd -> 6b2eefc301d8
  fabledcurator-agent  44920e0af1f3 -> 54accbeb52ed

buildkit mints a fresh image config per run, so identical layers get
republished under a new config blob. Storage cost is trivial; the cost
that matters is that a :latest digest change stops meaning "something is
different", and :c-<sha> is handed a new manifest to diverge from every
Sunday for no reason.

Corrects the workflow comment (x3) and ci-requirements.md to say what
actually happens. Filed as #3265 with the candidate fixes; the likely one
is a deterministic SOURCE_DATE_EPOCH off the value artifacts.sh already
derives, which would make "same source, same version" into "same source,
same bytes".

The rest of step 4 verified clean on the same run: the guard passed
(HEAD is main (499720d), `git checkout -B main`) — so this runner DOES
evaluate the env context inside `with:` — the tag list was :latest alone
with no :c-<sha>, and the repoint step correctly found nothing to write.
This commit is contained in:
2026-08-30 12:57:06 -04:00
parent 6663e06aa6
commit 0a5bbe81dc
2 changed files with 73 additions and 49 deletions
+60 -39
View File
@@ -773,23 +773,30 @@ jobs:
# trusting whatever digest the cache was built against. This is the
# whole mechanism of the scheduled refresh (#3154): if the base tag
# moved, the FROM layer's cache key changes, every layer above it
# invalidates, and the image genuinely rebuilds. If it did not move,
# the registry cache satisfies the entire graph and the refresh is a
# ~13s no-op that republishes nothing.
# invalidates, and the image genuinely rebuilds.
#
# That no-op is the POINT, not a shortfall: :latest should change
# when there is something new in it and not otherwise. A refresh
# that rewrote the image weekly regardless would churn the registry
# and hand :c-<sha> a new manifest to diverge from every Sunday, for
# no gain.
# MEASURED on the first real fire, run 4934 (#3265): when the base
# did NOT move, the build is ~13s and every content step reports
# CACHED — but the channel tag STILL gets a new manifest digest.
# buildkit mints a fresh image config each run, so identical layers
# are republished under a new config blob. All three images moved
# that way on 2026-08-30 with nothing whatsoever changed in them.
#
# What it therefore does NOT catch: a Debian package update inside
# So a refresh currently rewrites :latest every Sunday whether or
# not there is anything new in it, and :c-<sha> is handed a new
# manifest to diverge from on the same cadence. Layers are shared,
# so the storage cost is a config blob; the cost that matters is
# that a digest change no longer MEANS anything. Tracked in #3265 —
# the likely fix is a deterministic SOURCE_DATE_EPOCH, which would
# make "same source, same bytes" true and turn the no-op case into
# a genuine no-op.
#
# What `pull` does NOT catch either: a Debian package update inside
# the `apt-get install` layer while the base tag itself stands
# still. The official python/cuda images rebuild with those updates
# baked in, so this is a lag rather than a hole — but closing it
# would take `no-cache: true` on the scheduled path, which is the
# weekly-churn trade above. Left as the cheaper of the two on
# purpose.
# baked in, so this is a lag rather than a hole; closing it needs
# `no-cache: true`, which is a much larger version of the same
# churn #3265 is about.
#
# Only on the schedule. An ordinary push wants the cached base.
pull: ${{ github.event_name == 'schedule' }}
@@ -1149,23 +1156,30 @@ jobs:
# trusting whatever digest the cache was built against. This is the
# whole mechanism of the scheduled refresh (#3154): if the base tag
# moved, the FROM layer's cache key changes, every layer above it
# invalidates, and the image genuinely rebuilds. If it did not move,
# the registry cache satisfies the entire graph and the refresh is a
# ~13s no-op that republishes nothing.
# invalidates, and the image genuinely rebuilds.
#
# That no-op is the POINT, not a shortfall: :latest should change
# when there is something new in it and not otherwise. A refresh
# that rewrote the image weekly regardless would churn the registry
# and hand :c-<sha> a new manifest to diverge from every Sunday, for
# no gain.
# MEASURED on the first real fire, run 4934 (#3265): when the base
# did NOT move, the build is ~13s and every content step reports
# CACHED — but the channel tag STILL gets a new manifest digest.
# buildkit mints a fresh image config each run, so identical layers
# are republished under a new config blob. All three images moved
# that way on 2026-08-30 with nothing whatsoever changed in them.
#
# What it therefore does NOT catch: a Debian package update inside
# So a refresh currently rewrites :latest every Sunday whether or
# not there is anything new in it, and :c-<sha> is handed a new
# manifest to diverge from on the same cadence. Layers are shared,
# so the storage cost is a config blob; the cost that matters is
# that a digest change no longer MEANS anything. Tracked in #3265 —
# the likely fix is a deterministic SOURCE_DATE_EPOCH, which would
# make "same source, same bytes" true and turn the no-op case into
# a genuine no-op.
#
# What `pull` does NOT catch either: a Debian package update inside
# the `apt-get install` layer while the base tag itself stands
# still. The official python/cuda images rebuild with those updates
# baked in, so this is a lag rather than a hole — but closing it
# would take `no-cache: true` on the scheduled path, which is the
# weekly-churn trade above. Left as the cheaper of the two on
# purpose.
# baked in, so this is a lag rather than a hole; closing it needs
# `no-cache: true`, which is a much larger version of the same
# churn #3265 is about.
#
# Only on the schedule. An ordinary push wants the cached base.
pull: ${{ github.event_name == 'schedule' }}
@@ -1511,23 +1525,30 @@ jobs:
# trusting whatever digest the cache was built against. This is the
# whole mechanism of the scheduled refresh (#3154): if the base tag
# moved, the FROM layer's cache key changes, every layer above it
# invalidates, and the image genuinely rebuilds. If it did not move,
# the registry cache satisfies the entire graph and the refresh is a
# ~13s no-op that republishes nothing.
# invalidates, and the image genuinely rebuilds.
#
# That no-op is the POINT, not a shortfall: :latest should change
# when there is something new in it and not otherwise. A refresh
# that rewrote the image weekly regardless would churn the registry
# and hand :c-<sha> a new manifest to diverge from every Sunday, for
# no gain.
# MEASURED on the first real fire, run 4934 (#3265): when the base
# did NOT move, the build is ~13s and every content step reports
# CACHED — but the channel tag STILL gets a new manifest digest.
# buildkit mints a fresh image config each run, so identical layers
# are republished under a new config blob. All three images moved
# that way on 2026-08-30 with nothing whatsoever changed in them.
#
# What it therefore does NOT catch: a Debian package update inside
# So a refresh currently rewrites :latest every Sunday whether or
# not there is anything new in it, and :c-<sha> is handed a new
# manifest to diverge from on the same cadence. Layers are shared,
# so the storage cost is a config blob; the cost that matters is
# that a digest change no longer MEANS anything. Tracked in #3265 —
# the likely fix is a deterministic SOURCE_DATE_EPOCH, which would
# make "same source, same bytes" true and turn the no-op case into
# a genuine no-op.
#
# What `pull` does NOT catch either: a Debian package update inside
# the `apt-get install` layer while the base tag itself stands
# still. The official python/cuda images rebuild with those updates
# baked in, so this is a lag rather than a hole — but closing it
# would take `no-cache: true` on the scheduled path, which is the
# weekly-churn trade above. Left as the cheaper of the two on
# purpose.
# baked in, so this is a lag rather than a hole; closing it needs
# `no-cache: true`, which is a much larger version of the same
# churn #3265 is about.
#
# Only on the schedule. An ordinary push wants the cached base.
pull: ${{ github.event_name == 'schedule' }}