test: the smoke runs with egress blocked, on every build, and proves the block (4296)

Milestone 422 step 6's rule-164 half.

THE PROPERTY WAS NEVER TESTED, not even weekly. smoke-web already booted the
built image against real Postgres and Redis — but on the runner's default
network, with the internet one hop away. It proved the image WORKS; it never
proved it works OFFLINE, which is the thing rule 164 is about and the thing
step 6 put at risk by moving download_models out of boot.

Now on an `--internal` docker network, which is the mechanism rule 164's own
verify_with names. `--network none` is explicitly the wrong check here: it
would only prove the app fails without a database, which says nothing about
egress. Internal blocks the default route while leaving container-to-container
traffic and embedded DNS intact, so Postgres and Redis stay reachable and
nothing else does. The service containers are runner-created siblings, so
they are ATTACHED to the network rather than created on it, and their
addresses are re-read on it — the bridge IPs discovered earlier are not
routable from a container that is only on the internal network.

A STEP THAT PROVES THE SANDBOX IS SEALED, before anything depends on it. It
tries to reach 1.1.1.1:443 from inside the candidate and fails the job if it
succeeds. Without it the rest is theatre: if `--internal` silently stopped
working, or the container picked up a second network, every check below would
pass with the internet available and report an offline boot that never
happened. A guard that cannot fail is not a guard (rule 167).

IT RUNS ON EVERY BUILD, not just the weekly refresh. The egress property is
broken by a code or Dockerfile change — a push — so checking it only on the
refresh would test it on the one trigger that changes no source. Addressed by
the DIGEST build-web published rather than by a tag: a tag can move between
the build and the smoke, and then the check reports on bytes nobody built
here. A reuse hit is skipped, because those bytes were smoked when built.

WHAT THIS STILL IS NOT, filed as #4310: on a push it runs AFTER build-web has
written the channel tag, so it detects rather than gates. Rule 164 asks for
the check BETWEEN build and push. Closing that means giving the push path the
candidate-then-promote shape the refresh already has — per-channel candidate
tags, promote learning its channel, and the :c-<sha> repoint moving after the
gate. That is a redesign of the path that publishes production and it is not
something to fold into a test change.

Also filed #4311: retiring the fabledcurator-ml image NAME, gated on the
operator moving their Swarm stack file. Same two-phase shape #406 used for
pixiv, for the same reason — dropping it would not break their deploy, it
would freeze it silently.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR

.forgejo/workflows/build.yml

@@ -502,6 +502,10 @@ jobs:
     # to know — a candidate was published — rather than restating why.
     outputs:
       candidate: ${{ steps.reuse.outputs.promote }}
+      # The manifest THIS run pushed, empty on a reuse hit. smoke-web addresses
+      # it by digest rather than by tag: a tag can move between the build and
+      # the smoke, and then the check reports on bytes nobody built here.
+      digest: ${{ steps.build.outputs.digest }}
     # A plain `needs` — no `always()`. That expression existed to let a
     # SKIPPED sign-extension through on a tag push while still blocking a
     # FAILED one. With no tag trigger, sign-extension always runs, so the
@@ -1152,7 +1156,25 @@ jobs:
   # publish.
   smoke-web:
     needs: [build-web]
-    if: needs.build-web.outputs.candidate == 'true'
+    # Every run that actually BUILT something, not just the weekly refresh.
+    # The egress property (rule 164) is broken by a code or Dockerfile change,
+    # which is a push — checking it only on the refresh would test it on the
+    # one trigger that changes no source.
+    #
+    # A reuse hit is skipped deliberately: those bytes are already published
+    # and were smoked when they were built. Re-smoking them would burn two
+    # minutes to re-learn a fact.
+    #
+    # HONEST LIMIT, and it is the reason #4299 exists: on a push this runs
+    # AFTER build-web has written the channel tag, so it detects rather than
+    # gates. Rule 164's verify_with asks for the check BETWEEN build and push.
+    # Closing that needs the push path to adopt the candidate-then-promote
+    # shape the refresh already has — per-channel candidate tags, promote
+    # learning its channel, and the :c-<sha> repoint moving after the gate.
+    # That is a redesign of the production publish path and is its own task.
+    if: >-
+      needs.build-web.outputs.candidate == 'true'
+      || needs.build-web.outputs.digest != ''
     runs-on: python-ci
     container:
       image: git.fabledsword.com/bvandeusen/ci-python:3.14
@@ -1193,6 +1215,7 @@ jobs:
         env:
           TOKEN: ${{ secrets.RELEASE_TOKEN }}
           ACTOR: ${{ github.actor }}
+          BUILT_DIGEST: ${{ needs.build-web.outputs.digest }}
         run: |
           set -eux
           # Service discovery mirrors ci.yml's integration lane: these jobs run
@@ -1222,10 +1245,59 @@ jobs:
           fi
 
           echo "$TOKEN" | docker login git.fabledsword.com -u "$ACTOR" --password-stdin
-          CANDIDATE="$IMAGE:refresh-candidate"
+          # A refresh publishes to the candidate tag; a push writes the
+          # channel tag directly and hands us its digest. Address the digest
+          # where we have one — it names the exact manifest this run built,
+          # which a tag stops doing the moment anything else moves it.
+          if [ -n "${BUILT_DIGEST:-}" ]; then
+            CANDIDATE="$IMAGE@$BUILT_DIGEST"
+          else
+            CANDIDATE="$IMAGE:refresh-candidate"
+          fi
           docker pull "$CANDIDATE"
 
-          ENVOPTS="-e DB_USER=$DB_USER -e DB_PASSWORD=$DB_PASSWORD -e DB_HOST=$PG_IP"
+          # --- EGRESS BLOCKED from here (rule 164) ---------------------------
+          #
+          # Rule 164 requires a deployed instance to start and serve its full
+          # UI with NO outbound internet, and says to verify it by removing the
+          # network rather than by reading the code. Until now this job proved
+          # the image WORKS; it never proved it works OFFLINE, because every
+          # container below ran on the runner's default network with the
+          # internet one hop away.
+          #
+          # That gap became load-bearing at milestone 422 step 6. The ML role
+          # used to run `download_models` before celery started — a boot that
+          # reached HuggingFace for ~3.5GB — and that fetch moved to a task
+          # enqueued when the lane is enabled. This check is what proves it
+          # actually moved, rather than proving it on the machine that built it
+          # where the model is already cached.
+          #
+          # `--internal` is the mechanism rule 164's own verify_with names, and
+          # `--network none` is explicitly the WRONG check here: it would only
+          # prove the app fails without a database, which proves nothing about
+          # egress. An internal network blocks the default route while leaving
+          # container-to-container traffic and embedded DNS intact, so Postgres
+          # and Redis stay reachable and nothing else is.
+          #
+          # The service containers are SIBLINGS created by the runner, so they
+          # are attached to the internal network rather than created on it.
+          # They keep their original network too — that is fine, since what
+          # must be offline is the APP container, and it is created with only
+          # this network.
+          NET=smoke-noegress-$$
+          docker network create --internal "$NET"
+          trap 'docker network rm "$NET" >/dev/null 2>&1 || true' EXIT
+          docker network connect "$NET" "$PG"
+          docker network connect "$NET" "$RD"
+          # Re-read the addresses ON THIS NETWORK. The IPs discovered above
+          # belong to the runner's default bridge and are not routable from a
+          # container that is only on the internal one.
+          PG_IP=$(docker inspect -f "{{(index .NetworkSettings.Networks \"$NET\").IPAddress}}" "$PG")
+          RD_IP=$(docker inspect -f "{{(index .NetworkSettings.Networks \"$NET\").IPAddress}}" "$RD")
+          test -n "$PG_IP" && test -n "$RD_IP"
+
+          ENVOPTS="--network $NET"
+          ENVOPTS="$ENVOPTS -e DB_USER=$DB_USER -e DB_PASSWORD=$DB_PASSWORD -e DB_HOST=$PG_IP"
           ENVOPTS="$ENVOPTS -e DB_PORT=5432 -e DB_NAME=$DB_NAME -e SECRET_KEY=$SECRET_KEY"
           ENVOPTS="$ENVOPTS -e CELERY_BROKER_URL=redis://$RD_IP:6379/0"
           ENVOPTS="$ENVOPTS -e CELERY_RESULT_BACKEND=redis://$RD_IP:6379/0"
@@ -1238,6 +1310,22 @@ jobs:
           # user-facing file mentions this variable at all.
           ENVOPTS="$ENVOPTS -e CURATOR_BOOTSTRAP_NEW_KEY=1"
 
+          # 0. PROVE the network is actually blocking egress. Without this the
+          #    rest is theatre: if `--internal` silently stopped working, or
+          #    the app container picked up a second network, every check below
+          #    would pass with the internet available and report an offline
+          #    boot that never happened. A guard that cannot fail is not a
+          #    guard (rule 167).
+          echo "smoke: confirming the sandbox has no route out"
+          if docker run --rm --network "$NET" "$CANDIDATE" shell -c \
+               'python3 -c "import socket,sys; s=socket.socket(); s.settimeout(5); sys.exit(0 if s.connect_ex((\"1.1.1.1\", 443)) == 0 else 1)"'; then
+            echo "smoke: FAILED — the sandbox reached 1.1.1.1:443." >&2
+            echo "smoke: the network is NOT internal, so nothing below would" >&2
+            echo "smoke: have tested the offline property (rule 164)." >&2
+            exit 1
+          fi
+          echo "smoke: no route out, as required"
+
           # 1. The schema builds from empty, using the image's OWN libpq and
           #    psycopg. This is the same call entrypoint.sh makes before it
           #    serves anything, so a failure here is a failure to boot.
@@ -1296,7 +1384,7 @@ jobs:
           curl -fsS --max-time 5 "http://$WEB_IP:8080/api/health"
           echo
 
-          echo "smoke: all checks passed against $CANDIDATE"
+          echo "smoke: all checks passed against $CANDIDATE, with egress blocked"
 
   # Move the channel tags — the whole point of the gate.
   #