server: bake the newest Android client into every image (operator call)

Reverses the placement decision made an hour ago. That one put the APK only on
the data volume, reasoning that ~55 MiB should not be charged to installs that
never touch Android. The operator's call is that ending the manual copy is worth
the megabytes, and it is their deployment.

CI now fetches the newest published client into the build context immediately
before the image build, so `:dev`, `:latest` and `:<version>` all ship one and a
`docker compose pull` delivers a new server and a new client together.

**Always the rolling `dev` release — the newest build there is.** A versioned
image therefore carries the newest client rather than one pinned to that
version. Deliberate: the two negotiate a sync protocol version before linking, so
a mismatch is caught by the handshake, and pinning would buy nothing the
handshake does not already provide.

**Fetched by the JOB, never by the Dockerfile.** The release is private, and a
token used inside a build ends up in the context or a layer.

**It cannot fail the image build.** No release yet, a network blip, a first-ever
build — all of them log a warning and produce an image with no client, which is a
state the server already supports. Half a pair is cleaned up rather than shipped:
a sidecar without its APK is worse than neither, because the server would be
describing something it cannot serve.

**The volume still wins.** `DATA_DIR/client/` is checked first and the baked copy
second, so an operator who deliberately drops a build in gets that build — and a
BROKEN drop-in falls through to the image's copy rather than taking the feature
offline, which is what makes the copy-order advice survivable instead of
load-bearing. Three tests cover the precedence, including that last case.

The baked copy lives inside the package, not under DATA_DIR: that path is a
volume mount, and anything the image wrote there would disappear behind it the
moment one is attached.

`client/.keep` is tracked so `COPY client/` cannot fail on a tree where the CI
step never ran; the artifacts themselves are gitignored, since a 55 MiB binary
does not belong in git history and is re-fetched on every build anyway.

.forgejo/workflows/ci.yml

@@ -136,6 +136,43 @@ jobs:
           docker system prune -af || true
           docker builder prune --keep-storage 5g -f || true
 
+      # Bake the Android client in, on EVERY image build, so :dev, :latest and
+      # :<version> all carry one and a `docker compose pull` delivers a new client
+      # along with the new server.
+      #
+      # Always the rolling `dev` release — the newest build there is. A versioned
+      # image therefore carries the newest client rather than one pinned to that
+      # version; the two negotiate a sync protocol version before linking, so
+      # "newest" is safe in a way "matching" would not buy anything over.
+      #
+      # Fetched by the JOB, not by the Dockerfile: the release is private, and a
+      # token used inside a build lands in the context or a layer.
+      #
+      # NEVER fails the build. An image with no Android client advertises none and
+      # hides the download — a supported state, and the only one available before
+      # the first Android build has ever published.
+      - name: Fetch the Android client to bake in
+        env:
+          GITHUB_TOKEN: ${{ github.token }}
+        run: |
+          mkdir -p client
+          base="${{ github.server_url }}/${{ github.repository }}/releases/download/dev"
+          ok=1
+          for f in thoughtsync.apk thoughtsync-android.json; do
+            curl -fsSL -H "Authorization: token $GITHUB_TOKEN" -o "client/$f" "$base/$f" || ok=0
+          done
+          if [ "$ok" = 1 ]; then
+            echo "Baking in:"
+            cat client/thoughtsync-android.json
+            ls -l client/thoughtsync.apk
+          else
+            # Both or neither. Half a pair is worse than none: the server would
+            # read a sidecar describing an APK that isn't there, or an APK it
+            # cannot state a version for.
+            echo "::warning::No Android client on the dev release — this image ships without one."
+            rm -f client/thoughtsync.apk client/thoughtsync-android.json
+          fi
+
       - name: Set up Docker Buildx
         uses: docker/setup-buildx-action@v4
 

.gitignore

@@ -205,3 +205,9 @@ android/local.properties
 *.keystore
 *.p12
 *.b64
+
+# The Android client CI bakes into the server image. Fetched fresh on every image
+# build, so it is never worth 55 MiB of git history. client/.keep IS tracked, so
+# the Dockerfile's COPY always has a directory to copy.
+client/thoughtsync.apk
+client/thoughtsync-android.json

Dockerfile

@@ -24,6 +24,19 @@ COPY --from=build-frontend /build/dist/ src/thoughtsync/static/
 COPY alembic.ini .
 COPY alembic/ alembic/
 
+# The Android client this server hands out. CI fetches the newest published build
+# into ./client immediately before this runs (ci.yml), so every image tag — :dev,
+# :latest and :<version> alike — ships a client, and a `docker compose pull`
+# delivers a new one with no file copying by hand.
+#
+# Fetched by the JOB rather than here on purpose: the release is private, and a
+# token used inside a build ends up in the build context or a layer.
+#
+# The directory is tracked (client/.keep) so this COPY cannot fail on a tree where
+# that step never ran. An image with no APK is a supported state — the server
+# advertises nothing and the web UI hides the download (client_dist.py).
+COPY client/ src/thoughtsync/client/
+
 ENV PYTHONPATH=/app/src
 
 ARG BUILD_VERSION=dev

client/.keep

@@ -0,0 +1,10 @@
+CI drops the Android client here on every image build, and the Dockerfile copies
+the directory into the image (see ci.yml "Fetch the Android client to bake in").
+
+This file exists so the directory does too. `COPY client/ ...` fails outright on a
+missing source, which would break every local `docker build` on a tree that has
+never run that CI step — and an image with no Android client is a supported
+state, not an error.
+
+The artifacts themselves are gitignored: a 55 MiB binary does not belong in git
+history, and it is fetched fresh anyway.

docs/android-distribution.md

@@ -11,28 +11,42 @@ It also keeps the two in step: client and server negotiate a sync protocol versi
 before linking, so a server that serves the client cannot hand out a phone it
 cannot talk to.
 
-## Where it goes
+## Where it comes from
 
-Two files, both required, in `/var/thoughtsync/client/`:
+**Normally: nowhere. It is already in the image.**
+
+CI fetches the newest published Android build into every server image it builds,
+so `:dev`, `:latest` and `:<version>` all ship a client. `docker compose pull &&
+docker compose up -d` delivers a new server and a new client together, and there
+is nothing to copy.
+
+A versioned image therefore carries the *newest* client rather than one pinned to
+that version. That is deliberate: the two negotiate a sync protocol version
+before they link, so a mismatch is caught by the handshake rather than by
+pinning.
+
+## Overriding it
+
+If you want a specific build — testing something, or holding back — drop it in
+`/var/thoughtsync/client/` and it wins over the image's copy.
+
+Two files, both required:
 
 | File | What it is |
 | --- | --- |
 | `thoughtsync.apk` | the client |
 | `thoughtsync-android.json` | `{version_name, version_code, size, sha256}` |
 
-The sidecar exists because an APK keeps its version in a binary manifest that needs
-the Android build tools to read. CI writes it beside the APK, where the real values
-are already known.
+The sidecar exists because an APK keeps its version in a binary manifest that
+needs the Android build tools to read. CI writes it beside the APK, where the
+real values are already known.
 
 `/var/thoughtsync` is the same volume that holds attachments (`Config.DATA_DIR`),
-so a build dropped there survives container recreation. Nothing is baked into the
-image: the APK is ~55 MiB and an install that never touches Android should not
-carry it.
-
-## Putting a build there
+so a build dropped there survives container recreation — and survives an image
+upgrade, which is the point of an override.
 
 Both files are published to the rolling `dev` release on every green Android
-build. From the machine running the server:
+build:
 
 ```sh
 REPO=https://git.fabledsword.com/bvandeusen/thoughtsync
@@ -43,14 +57,14 @@ for f in thoughtsync.apk thoughtsync-android.json; do
     -o "/tmp/$f" "$REPO/releases/download/dev/$f"
 done
 
-# Into the app container's volume. Copy the sidecar LAST: the server treats a
-# sidecar that does not match the APK beside it as "no client at all", so a
-# half-finished copy advertises nothing rather than advertising a lie.
+# Copy the sidecar LAST. A sidecar that does not match the APK beside it is not a
+# client, so a half-finished copy falls back to the image's build rather than
+# advertising a lie.
 docker compose cp /tmp/thoughtsync.apk app:/var/thoughtsync/client/
 docker compose cp /tmp/thoughtsync-android.json app:/var/thoughtsync/client/
 ```
 
-`docker compose cp` creates `/var/thoughtsync/client/` if it does not exist.
+To go back to whatever the image ships, delete both files.
 
 ## Checking it took
 
@@ -64,14 +78,18 @@ offering a button that fails.
 
 ## What happens if you get it wrong
 
-- **Only the APK, no sidecar** — the server reports no client. It cannot state a
-  version it has no way to read.
-- **Mismatched pair** (new APK, old sidecar) — the server reports no client,
-  because the recorded size does not match the file. It will not serve one build
-  while describing another.
-- **Neither** — the server reports no client, the UI hides the card, and
-  `/api/client/android` returns 404. This is the ordinary state of a server whose
-  owner does not use Android, and nothing about it is an error.
+All of these describe an override in `/var/thoughtsync/client/`. A broken
+override does not take the feature away — it falls through to the build the image
+shipped with, which is the whole reason precedence runs in that direction.
+
+- **Only the APK, no sidecar** — not a client. The server cannot state a version
+  it has no way to read.
+- **Mismatched pair** (new APK, old sidecar) — not a client. It will not serve
+  one build while describing another.
+- **Neither, and no baked copy either** — the server reports no client, the UI
+  hides the card, and `/api/client/android` returns 404. That is the state of any
+  image built before the first Android build ever published, and nothing about it
+  is an error.
 
 ## Signing, and why replacing the APK is safe
 

src/thoughtsync/client_dist.py

@@ -13,10 +13,19 @@ the client cannot hand out a phone it is unable to talk to.
 
 ## Where the file comes from
 
-`DATA_DIR/client/` — the same volume that already holds attachments, so an
-operator drops a build there once and container recreation does not lose it.
-Deliberately NOT baked into the image: that would charge ~55 MiB to every
-self-hoster, including everyone who never touches Android.
+Two places, checked in that order:
+
+1. `DATA_DIR/client/` — the mounted volume that already holds attachments. An
+   operator who wants a SPECIFIC build drops it there and it wins.
+2. the copy baked into the image at build time — CI fetches the newest published
+   Android build into every image, so `:dev`, `:latest` and `:<version>` all
+   carry a client and `docker compose pull` delivers a new one with nothing
+   copied by hand.
+
+The precedence is the point: the image is the default, and a person who wants to
+override it should not have to fight it. The baked copy sits inside the package
+rather than under DATA_DIR because DATA_DIR is a volume mount, and anything the
+image wrote there would be hidden the moment one is attached.
 
 Two files, and both must be present:
 
@@ -51,21 +60,23 @@ MANIFEST_NAME = "thoughtsync-android.json"
 DOWNLOAD_PATH = "/api/client/android/download"
 APK_MIMETYPE = "application/vnd.android.package-archive"
 
+# The copy CI bakes into the image. Inside the package, NOT under DATA_DIR: that
+# is a volume mount, and a file the image wrote there would vanish behind it.
+BAKED_ROOT = Path(__file__).resolve().parent / "client"
+
 bp = Blueprint("client_dist", __name__)
 
 
-def android_release() -> dict | None:
-    """What Android build this server holds, or None if it holds none.
+def _read(root: Path) -> dict | None:
+    """The build in one directory, or None.
 
     Never raises. A missing directory, an unreadable sidecar, malformed JSON and a
     sidecar that describes a different file are all the same answer to the only
     question being asked — "is there a client here I can honestly offer?" — and
     that answer is no.
     """
-    root = Path(Config.client_root())
-    apk = root / APK_NAME
     try:
-        size = apk.stat().st_size
+        size = (root / APK_NAME).stat().st_size
         meta = json.loads((root / MANIFEST_NAME).read_text(encoding="utf-8"))
         version = str(meta["version_name"])
         code = int(meta["version_code"])
@@ -94,6 +105,27 @@ def android_release() -> dict | None:
     }
 
 
+def _resolve() -> tuple[Path, dict] | None:
+    """Which directory this server serves from, and what is in it.
+
+    The operator's drop-in beats the baked copy — someone who deliberately put a
+    build on the volume wants that build, not whatever the image happened to ship
+    with. A directory holding a broken or half-copied pair does NOT shadow the
+    image: it simply is not a client, so the search moves on.
+    """
+    for root in (Path(Config.client_root()), BAKED_ROOT):
+        release = _read(root)
+        if release is not None:
+            return root, release
+    return None
+
+
+def android_release() -> dict | None:
+    """What Android build this server holds, or None if it holds none."""
+    resolved = _resolve()
+    return resolved[1] if resolved else None
+
+
 def advertisement() -> dict:
     """The `/api/config` fragment describing this server's Android client.
 
@@ -124,11 +156,13 @@ async def android_download():
     above is public because a client has to be able to ask "is there something
     newer?" cheaply, but the bytes are not for anyone who can reach the port.
     """
-    if android_release() is None:
+    resolved = _resolve()
+    if resolved is None:
         return jsonify({"error": "this server has no Android client"}), 404
-    response = await send_from_directory(
-        Path(Config.client_root()), APK_NAME, mimetype=APK_MIMETYPE
-    )
+    # From the SAME directory the advertisement came from, or a drop-in appearing
+    # between the two calls would serve bytes the metadata does not describe.
+    root, _ = resolved
+    response = await send_from_directory(root, APK_NAME, mimetype=APK_MIMETYPE)
     # Without this some browsers try to render it, and Android's download handler
     # wants a filename to hand to the package installer.
     response.headers["Content-Disposition"] = f'attachment; filename="{APK_NAME}"'

tests/test_client_dist.py

@@ -1,7 +1,9 @@
 import json
+from pathlib import Path
 
 import pytest
 
+from thoughtsync import client_dist
 from thoughtsync.app import create_app
 from thoughtsync.client_dist import APK_NAME, MANIFEST_NAME, advertisement, android_release
 from thoughtsync.config import Config
@@ -18,14 +20,27 @@ from thoughtsync.config import Config
 PAYLOAD = b"not really an apk, but the server only ever stats it"
 
 
+@pytest.fixture(autouse=True)
+def _empty_baked_client(tmp_path, monkeypatch):
+    """Point the baked-in copy at an empty directory.
+
+    In a source checkout `src/thoughtsync/client/` does not exist, so these tests
+    would pass anyway — but only by accident of where they are run. A built image
+    has a real APK there, and a test that silently depends on which tree it is in
+    is one that will eventually lie.
+    """
+    monkeypatch.setattr(client_dist, "BAKED_ROOT", tmp_path / "baked")
+    yield
+
+
 @pytest.fixture
 def app():
     return create_app()
 
 
-def place_client(payload: bytes = PAYLOAD, **overrides) -> dict:
+def place_client(payload: bytes = PAYLOAD, root: Path | None = None, **overrides) -> dict:
     """Put a client + sidecar where the server looks. Overrides corrupt the pair."""
-    root = Config.client_root()
+    root = root if root is not None else Config.client_root()
     root.mkdir(parents=True, exist_ok=True)
     (root / APK_NAME).write_bytes(payload)
     meta = {
@@ -110,3 +125,30 @@ async def test_the_bytes_need_authentication_even_though_the_version_does_not(ap
     place_client()
     resp = await app.test_client().get("/api/client/android/download")
     assert resp.status_code == 401
+
+
+def test_the_baked_in_copy_is_used_when_nothing_was_dropped_in():
+    """The ordinary case for a self-hoster who just pulled the image."""
+    place_client(root=client_dist.BAKED_ROOT, version_name="0.1.300", version_code=300)
+    assert android_release()["version_code"] == 300
+
+
+def test_a_dropped_in_build_beats_the_one_the_image_shipped():
+    """Someone who deliberately put a build on the volume wants that build."""
+    place_client(root=client_dist.BAKED_ROOT, version_name="0.1.300", version_code=300)
+    place_client(version_name="0.1.99", version_code=99)
+    advertised = android_release()
+    # Lower version and all — precedence is about intent, not about newness. An
+    # operator pinning an older client is doing it on purpose.
+    assert advertised["version_code"] == 99
+
+
+def test_a_broken_drop_in_does_not_shadow_the_baked_copy():
+    """A half-finished copy onto the volume must not take the app offline.
+
+    This is the failure the copy-order advice in docs/android-distribution.md is
+    about, and the server should ride it out rather than go dark.
+    """
+    place_client(root=client_dist.BAKED_ROOT, version_name="0.1.300", version_code=300)
+    place_client(size=999_999)  # sidecar describing a different build
+    assert android_release()["version_code"] == 300