ci(release): verify a tag release actually shipped its artifacts #123

2026-08-07T08:32:38-04:00

bvandeusen commented

2026-08-07 08:32:38 -04:00

CI-only. Two commits, no server or client changes, no migration.

Arose from v2026.08.07 having to be re-cut. The tag build's android-release job never started — no job log was written at all, so all eight steps reported failure with none executed, and image-release showed skipped.

The run was red, but the release page rendered fine, and main's own push build had already moved :latest, so the code was deployable and nothing looked obviously wrong. What was actually missing — the attached APK and the immutable :v2026.08.07 image — is easy to skim past. An unchanged rerun_run then passed first time.

What this does and doesn't do

It cannot prevent the failure. The cause was a runner failing to launch a container, not anything in this file, and it did not reproduce. What it does is make the consequence legible: an incomplete release now fails with a named error instead of eight mystery step failures, and the message says to re-run the run rather than delete and re-create the tag.

New `verify-release` job

Runs on tag pushes with if: always() — load-bearing, since it has to report precisely when the jobs above did not succeed. Asserts:

a release exists for the tag and has an .apk asset attached
the immutable :vYYYY.MM.DD image was actually pushed (docker manifest inspect)

Checking only the APK was my first version and it was wrong: because the job runs with always(), android-release succeeding while image-release failed would have reported "verified" on a release with no immutable image — half of exactly what went missing. It would have caught the incident we had and waved through its mirror image.

Precondition moved ahead of the expensive work

Attach APK to gitea Release resolves the release by tag and fails if it's absent, but it's the last step — so a bare git push origin vYYYY.MM.DD built an APK for several minutes before discovering it had nowhere to put it. The same check now runs immediately after version computation. Releases created through the API create tag and release together and pass it.

Audit of the existing gating — clean, with one part worth not "fixing"

image-release:
  needs: [android-release]
  if: ${{ !failure() && !cancelled() }}

That reads oddly next to needs, and it's correct. On main pushes android-release is skipped, and a skipped dependency does not satisfy success() — so the obvious-looking if: success() would silently stop main from ever publishing :latest. Left alone deliberately; this note is so nobody tidies it.

Also verified: steps 4/5 (attach the fresh APK) and step 6 (bundle the previous release's APK) are mutually exclusive on the tag context, so a main build can neither do both nor neither; and every image step gates on the Dockerfile + go.mod guard, so steps.tags.outputs.* is never read when it wasn't set.

Correcting the record

I first blamed the failure on the workflow's cancel-in-progress concurrency block. That was wrong. Cancellation requires a newer run in the same group, and there was exactly one run on the tag ref (total_count 632 → 633 on release creation); the main-push runs sit in a different group. A plausible mechanism with an unchecked precondition. The dev-log note has been corrected too.

Validation

No test lane covers workflow YAML, so this was checked by hand: YAML parses (my first attempt did not — a multi-line error string dropped to column 0 and escaped the block scalar), bash -n is clean over every run: block in all three jobs, and the release-asset parsing was exercised against three shapes: a release with an APK, one with no assets, and one with only a non-APK asset.

These jobs only execute on tag pushes, so this release is itself the first live exercise of them.

🤖 Generated with Claude Code

https://claude.ai/code/session_01N6vZoJ4Se5YyaqdtGVkap5

CI-only. Two commits, no server or client changes, no migration. Arose from `v2026.08.07` having to be re-cut. The tag build's `android-release` job **never started** — no job log was written at all, so all eight steps reported `failure` with none executed, and `image-release` showed `skipped`. The run was red, but the **release page rendered fine**, and `main`'s own push build had already moved `:latest`, so the code was deployable and nothing looked obviously wrong. What was actually missing — the attached APK and the immutable `:v2026.08.07` image — is easy to skim past. An unchanged `rerun_run` then passed first time. ## What this does and doesn't do It cannot prevent the failure. The cause was a runner failing to launch a container, not anything in this file, and it did not reproduce. What it does is make the **consequence legible**: an incomplete release now fails with a named error instead of eight mystery step failures, and the message says to re-run the run rather than delete and re-create the tag. ## New `verify-release` job Runs on tag pushes with `if: always()` — load-bearing, since it has to report precisely when the jobs above did *not* succeed. Asserts: 1. a release exists for the tag and has an `.apk` asset attached 2. the immutable `:vYYYY.MM.DD` image was actually pushed (`docker manifest inspect`) Checking only the APK was my first version and it was wrong: because the job runs with `always()`, `android-release` succeeding while `image-release` failed would have reported "verified" on a release with no immutable image — half of exactly what went missing. It would have caught the incident we had and waved through its mirror image. ## Precondition moved ahead of the expensive work `Attach APK to gitea Release` resolves the release by tag and fails if it's absent, but it's the **last** step — so a bare `git push origin vYYYY.MM.DD` built an APK for several minutes before discovering it had nowhere to put it. The same check now runs immediately after version computation. Releases created through the API create tag and release together and pass it. ## Audit of the existing gating — clean, with one part worth not "fixing" ```yaml image-release: needs: [android-release] if: ${{ !failure() && !cancelled() }} ``` That reads oddly next to `needs`, and it's correct. On main pushes `android-release` is **skipped**, and a skipped dependency does not satisfy `success()` — so the obvious-looking `if: success()` would silently stop `main` from ever publishing `:latest`. Left alone deliberately; this note is so nobody tidies it. Also verified: steps 4/5 (attach the fresh APK) and step 6 (bundle the previous release's APK) are mutually exclusive on the tag context, so a main build can neither do both nor neither; and every image step gates on the `Dockerfile + go.mod` guard, so `steps.tags.outputs.*` is never read when it wasn't set. ## Correcting the record I first blamed the failure on the workflow's `cancel-in-progress` concurrency block. **That was wrong.** Cancellation requires a *newer* run in the same group, and there was exactly one run on the tag ref (`total_count` 632 → 633 on release creation); the main-push runs sit in a different group. A plausible mechanism with an unchecked precondition. The dev-log note has been corrected too. ## Validation No test lane covers workflow YAML, so this was checked by hand: YAML parses (my first attempt did **not** — a multi-line error string dropped to column 0 and escaped the block scalar), `bash -n` is clean over every `run:` block in all three jobs, and the release-asset parsing was exercised against three shapes: a release with an APK, one with no assets, and one with only a non-APK asset. These jobs only execute on tag pushes, so this release is itself the first live exercise of them. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01N6vZoJ4Se5YyaqdtGVkap5

bvandeusen added 2 commits 2026-08-07 08:32:39 -04:00

ci(release): fail loudly when a tag release ends up without its APK e368b82f0a

v2026.08.07 had to be re-cut, and the tag build's android-release job
never started — no job log was written at all, so all eight steps reported
`failure` with none executed and image-release showed `skipped`.

The run was red, but the release PAGE rendered fine and main's own push
build had already moved :latest, so the code was deployable and nothing
looked obviously wrong. What was actually missing — the attached APK and
the immutable :vYYYY.MM.DD image — is easy to skim past, and I nearly did.

This cannot prevent that. The cause was a runner failing to launch a
container, not anything in this file, and it did not reproduce on an
unchanged re-run. What this does is make the CONSEQUENCE legible: an
incomplete release now fails with a named error instead of eight mystery
step failures, and the message says to re-run the run rather than delete
and re-create the tag.

`if: always()` is load-bearing — the job has to report precisely when the
jobs above did not succeed.

Correcting the record while here: I first blamed this on the workflow's
`cancel-in-progress` concurrency block. That was wrong. Cancellation needs
a NEWER run in the same group, and there was exactly one run on the tag
ref (total_count 632 -> 633 on release creation); the main-push runs sit
in a different group. Plausible mechanism, unchecked precondition.

Validated locally: YAML parses, `bash -n` clean, and the asset-parsing
logic unit-checked against a release with an APK, one with no assets, and
one with a non-APK asset.

ci(release): close the verify blind spot, check preconditions before the build a254cb2273

Auditing the gating turned up two problems.

verify-release only checked the APK. Because it runs with `always()`, it
runs even when image-release FAILED — so android succeeding while the image
push died would have reported "verified" on a release with no immutable
:vYYYY.MM.DD image. That is exactly half of what was missing when
v2026.08.07 had to be re-cut, so the guard would have caught the incident we
had and waved through its mirror image. Now checks the image too, via
docker manifest inspect.

"Attach APK to gitea Release" resolves the release by tag and fails if it is
absent — but it is the LAST step, so a bare `git push origin vX` built an
APK for several minutes before discovering it had nowhere to put it. Same
check now runs immediately after version computation: seconds, not minutes.
Releases created through the API create tag and release together and pass it.

The rest of the gating audits clean, and one part is worth not "fixing":
image-release's `if: !failure() && !cancelled()` looks odd next to
`needs: [android-release]` but is correct. On main pushes android-release is
SKIPPED, and a skipped dependency is not success() — so the obvious
`if: success()` would silently stop main from ever publishing :latest.
Steps 4/5 vs 6 are mutually exclusive on the tag context, and every image
step gates on the Dockerfile+go.mod guard.

Validated: YAML parses, and `bash -n` over every run: block in all three
jobs is clean.

bvandeusen merged commit 011b4d9a9c into main

2026-08-07 08:32:48 -04:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: bvandeusen/minstrel#123