M12 — the Android client, end to end #2

Merged
bvandeusen merged 86 commits from dev into main 2026-08-21 08:53:58 -04:00
9 changed files with 457 additions and 3 deletions
Showing only changes of commit d77a79859c - Show all commits
+35
View File
@@ -173,6 +173,41 @@ jobs:
apksigner="$(ls /opt/android-sdk/build-tools/*/apksigner | head -1)"
"$apksigner" verify --print-certs "app/build/outputs/apk/release/app-release.apk"
# Staged with a STABLE name plus the sidecar the server reads its version
# out of — an APK keeps that in a binary manifest Python cannot parse, and
# `aapt` is not on a Quart server. Computed here, where the real values are
# already known.
- name: Stage the client for distribution
if: steps.build.outputs.keystore != ''
run: |
mkdir -p dist
cp "app/build/outputs/apk/release/app-release.apk" dist/thoughtsync.apk
size="$(wc -c < dist/thoughtsync.apk | tr -d ' ')"
sha="$(sha256sum dist/thoughtsync.apk | cut -d' ' -f1)"
cat > dist/thoughtsync-android.json <<JSON
{
"version_name": "${{ steps.build.outputs.name }}",
"version_code": ${{ steps.build.outputs.code }},
"size": $size,
"sha256": "$sha"
}
JSON
cat dist/thoughtsync-android.json
# The rolling dev channel, same fixed-tag release the desktop bundles use.
# CI artifacts are per-run and auth-gated, so they are no use as a fetch
# target; a release asset has a permanent URL. Only ever a SIGNED build —
# publishing an unsigned APK would offer people something they cannot
# install over what they already have.
- name: Publish to the dev channel
if: github.ref == 'refs/heads/dev' && steps.build.outputs.keystore != ''
working-directory: .
env:
GITHUB_TOKEN: ${{ github.token }}
RELEASE_TAG: dev
RELEASE_PRERELEASE: "true"
run: bash desktop/packaging/publish-release.sh
- name: Upload the APK
# Mirrored action, never actions/upload-artifact. @v4+ throws
# GHESNotSupportedError client-side on this hostname, and @v3 is worse —
+10 -2
View File
@@ -58,6 +58,12 @@ shopt -s nullglob
# its signature is one the app will refuse, so they ship together or not at all.
# They only exist when the build ran with a signing key (M10.9); nullglob drops
# them silently otherwise, which is the correct behaviour for an unsigned build.
# The Android client publishes here too, from its own job and its own workspace
# — the same nullglob arrangement that already lets the Linux and Windows jobs
# share one release. Its two files are staged under android/dist by the workflow:
# a STABLY NAMED apk (a fixed name is the whole point of the fixed `dev` tag —
# Forgejo has no /releases/latest/download route) and the sidecar the server reads
# its version out of, because an APK keeps that in a binary manifest.
ASSETS=(
"$BUNDLE_ROOT"/appimage/*.AppImage
"$BUNDLE_ROOT"/appimage/*.AppImage.sig
@@ -65,9 +71,11 @@ ASSETS=(
"$BUNDLE_ROOT"/arch/*.pkg.tar.*
"$WIN_BUNDLE_ROOT"/nsis/*.exe
"$WIN_BUNDLE_ROOT"/nsis/*.exe.sig
"$REPO_ROOT"/android/dist/thoughtsync.apk
"$REPO_ROOT"/android/dist/thoughtsync-android.json
)
if [ ${#ASSETS[@]} -eq 0 ]; then
echo "ERROR: no bundles under $BUNDLE_ROOT — did the tauri build run?" >&2
echo "ERROR: nothing to publish — no desktop bundles under $BUNDLE_ROOT and no APK under $REPO_ROOT/android/dist." >&2
exit 1
fi
echo "==> Publishing release $TAG with ${#ASSETS[@]} asset(s):"
@@ -113,7 +121,7 @@ fi
echo "==> Creating release for $TAG"
BODY=$(cat <<JSON
{"tag_name":"$TAG","name":"ThoughtSync $TAG","draft":false,"prerelease":$RELEASE_PRERELEASE,
"body":"ThoughtSync desktop $TAG.\n\n- **Debian / Ubuntu** — native \`.deb\`\n- **Arch / CachyOS** — native \`.pkg.tar.*\`\n- **everything else** — \`.AppImage\` (de-bundled graphics: renders on any GPU/Wayland setup)\n\nInstall / update — picks the right one for your system:\n\`\`\`\ncurl -fsSL $GITHUB_SERVER_URL/$GITHUB_REPOSITORY/raw/branch/dev/desktop/packaging/install.sh | $INSTALL_TAIL\n\`\`\`$CHANNEL_NOTE"}
"body":"ThoughtSync $TAG.\n\n**Desktop**\n\n- **Debian / Ubuntu** — native \`.deb\`\n- **Arch / CachyOS** — native \`.pkg.tar.*\`\n- **everything else** — \`.AppImage\` (de-bundled graphics: renders on any GPU/Wayland setup)\n\nInstall / update — picks the right one for your system:\n\`\`\`\ncurl -fsSL $GITHUB_SERVER_URL/$GITHUB_REPOSITORY/raw/branch/dev/desktop/packaging/install.sh | $INSTALL_TAIL\n\`\`\`\n\n**Android** — \`thoughtsync.apk\`. Copy it and \`thoughtsync-android.json\` into your server's \`/var/thoughtsync/client/\` and the server will offer it to your devices; see docs/android-distribution.md.$CHANNEL_NOTE"}
JSON
)
# 409 = a release for this tag already exists (re-run) — fall through to lookup.
+82
View File
@@ -0,0 +1,82 @@
# Getting the Android app onto your server
ThoughtSync's server hands out the Android client it syncs with. Once an APK is in
place, anyone with an account on that server can download it from **Account →
Linked devices**, and linked phones can update themselves from it.
This is deliberate rather than incidental. The build is not on an app store and the
Fabled-Git instance is private, so a release page is no use to a self-hoster — but
the server holding their notes is something they already trust and already reach.
It also keeps the two in step: client and server negotiate a sync protocol version
before linking, so a server that serves the client cannot hand out a phone it
cannot talk to.
## Where it goes
Two files, both required, in `/var/thoughtsync/client/`:
| File | What it is |
| --- | --- |
| `thoughtsync.apk` | the client |
| `thoughtsync-android.json` | `{version_name, version_code, size, sha256}` |
The sidecar exists because an APK keeps its version in a binary manifest that needs
the Android build tools to read. CI writes it beside the APK, where the real values
are already known.
`/var/thoughtsync` is the same volume that holds attachments (`Config.DATA_DIR`),
so a build dropped there survives container recreation. Nothing is baked into the
image: the APK is ~55 MiB and an install that never touches Android should not
carry it.
## Putting a build there
Both files are published to the rolling `dev` release on every green Android
build. From the machine running the server:
```sh
REPO=https://git.fabledsword.com/bvandeusen/thoughtsync
TOKEN=... # a Fabled-Git token with read access; the instance is private
for f in thoughtsync.apk thoughtsync-android.json; do
curl -fsSL -H "Authorization: token $TOKEN" \
-o "/tmp/$f" "$REPO/releases/download/dev/$f"
done
# Into the app container's volume. Copy the sidecar LAST: the server treats a
# sidecar that does not match the APK beside it as "no client at all", so a
# half-finished copy advertises nothing rather than advertising a lie.
docker compose cp /tmp/thoughtsync.apk app:/var/thoughtsync/client/
docker compose cp /tmp/thoughtsync-android.json app:/var/thoughtsync/client/
```
`docker compose cp` creates `/var/thoughtsync/client/` if it does not exist.
## Checking it took
```sh
curl -s http://localhost:5000/api/client/android
```
A server with a client answers with the version, size and digest. A server without
one answers `404` — and the download card in the web UI is hidden rather than
offering a button that fails.
## What happens if you get it wrong
- **Only the APK, no sidecar** — the server reports no client. It cannot state a
version it has no way to read.
- **Mismatched pair** (new APK, old sidecar) — the server reports no client,
because the recorded size does not match the file. It will not serve one build
while describing another.
- **Neither** — the server reports no client, the UI hides the card, and
`/api/client/android` returns 404. This is the ordinary state of a server whose
owner does not use Android, and nothing about it is an error.
## Signing, and why replacing the APK is safe
Every release build is signed with the same key, so a phone can install a newer one
straight over the old one and keep its notes. That was not true before August 2026
— builds until then were signed with a throwaway key per CI run, and each install
required uninstalling the last (Scribe #2803). If you are carrying a build from
before that, expect to uninstall once more and sync anything you care about first.
+27 -1
View File
@@ -2,6 +2,17 @@ import { defineStore } from "pinia";
import { ref } from "vue";
import { repo } from "../adapters";
// The Android build this server can hand out. Absent — not null — when it has
// none, so `v-if` on it is the whole test; see client_dist.py.
export interface AndroidClient {
version: string;
// What decides "is this newer". The name is for people and sorts like a string.
version_code: number;
size: number;
sha256: string;
url: string;
}
export interface PublicConfig {
site_name: string;
allow_registration: boolean;
@@ -9,6 +20,7 @@ export interface PublicConfig {
enable_url_unfurl: boolean;
// How many days a note survives in Trash before the server purges it. 0 = forever.
trash_retention_days: number;
android_client?: AndroidClient;
}
// Public, unauthenticated app config (site name, whether signups are open).
@@ -21,6 +33,9 @@ export const useConfigStore = defineStore("config", () => {
// unreachable — and 30 is a safer stand-in than 0, since claiming "kept forever"
// when the server is actually purging is the wrong way to be wrong.
const trashRetentionDays = ref(30);
// Null until proven otherwise: a server with no APK, and an older server that
// never had the field, both correctly show no download.
const androidClient = ref<AndroidClient | null>(null);
const loaded = ref(false);
async function load(): Promise<void> {
@@ -32,6 +47,7 @@ export const useConfigStore = defineStore("config", () => {
version.value = cfg.version;
enableUrlUnfurl.value = cfg.enable_url_unfurl ?? true;
trashRetentionDays.value = cfg.trash_retention_days ?? 30;
androidClient.value = cfg.android_client ?? null;
} catch {
// Keep defaults if the config endpoint is unreachable.
} finally {
@@ -44,5 +60,15 @@ export const useConfigStore = defineStore("config", () => {
await load();
}
return { siteName, allowRegistration, version, enableUrlUnfurl, trashRetentionDays, loaded, load, reload };
return {
siteName,
allowRegistration,
version,
enableUrlUnfurl,
trashRetentionDays,
androidClient,
loaded,
load,
reload,
};
});
+39
View File
@@ -1,5 +1,6 @@
<script setup lang="ts">
import { onMounted, ref } from "vue";
import { useConfigStore } from "../stores/config";
import { useDevicesStore } from "../stores/devices";
import { useUiStore } from "../stores/ui";
import BaseButton from "../components/BaseButton.vue";
@@ -11,6 +12,13 @@ import { isDesktop, desktop as desktopBridge, type IntegrationStatus } from "../
// Android apps authenticate sync with a device bearer token issued here.
const devices = useDevicesStore();
const ui = useUiStore();
// The Android build this server holds, if it holds one. Null on a server with no
// APK — the card below is hidden rather than offering a download that 404s.
const config = useConfigStore();
function readableSize(bytes: number): string {
return `${(bytes / 1024 / 1024).toFixed(0)} MB`;
}
const error = ref("");
const newName = ref("");
@@ -25,6 +33,7 @@ const desktopBusy = ref(false);
async function load() {
error.value = "";
try {
await config.load();
await devices.load();
} catch {
error.value = "Couldn't load your linked devices.";
@@ -151,6 +160,36 @@ onMounted(() => {
</BaseButton>
</section>
<!-- The Android client this server hands out (hidden when it has none) -->
<section
v-if="config.androidClient"
class="mb-6 flex items-center justify-between gap-4 rounded-xl border border-neutral-200 p-4 dark:border-neutral-800"
>
<div class="min-w-0">
<p class="text-sm font-medium text-neutral-800 dark:text-neutral-100">Android app</p>
<p class="mt-0.5 text-xs text-neutral-400">
Version {{ config.androidClient.version }} ·
{{ readableSize(config.androidClient.size) }} · served by this server, so it always
speaks the same sync protocol.
</p>
</div>
<!-- A plain anchor, not BaseButton and not a fetch: this is 55 MB, and the
browser's own download manager handles it better than anything this app
would do with a blob. Styled to match BaseButton's primary variant,
which is a <button> and cannot carry an href. -->
<a
:href="config.androidClient.url"
:download="`thoughtsync-${config.androidClient.version}.apk`"
class="inline-flex shrink-0 items-center justify-center gap-2 rounded-lg bg-brand px-4 py-2.5
text-sm font-semibold text-neutral-900 shadow-sm transition hover:bg-brand-600
active:bg-brand-700 focus:outline-none focus-visible:ring-2 focus-visible:ring-brand
focus-visible:ring-offset-2 focus-visible:ring-offset-neutral-50
dark:focus-visible:ring-offset-neutral-950"
>
Download
</a>
</section>
<!-- One-time token reveal -->
<div
v-if="freshToken"
+6
View File
@@ -12,6 +12,7 @@ from quart.sessions import SecureCookieSessionInterface
from . import __version__
from .auth import bp as auth_bp
from .client_dist import advertisement as client_advertisement, bp as client_bp
from .config import Config
from .db import session_scope
from .graph import bp as graph_bp
@@ -71,6 +72,7 @@ def create_app() -> Quart:
app.register_blueprint(settings_bp)
app.register_blueprint(sync_bp)
app.register_blueprint(saved_filters_bp)
app.register_blueprint(client_bp)
@app.before_serving
async def _bootstrap() -> None:
@@ -113,6 +115,10 @@ def create_app() -> Quart:
# linking — while it still has no token and possibly no account — to decide
# whether it can talk to this server, and which optional features to offer.
data.update(protocol_advertisement())
# Which Android client this server can hand out, if any. Absent rather than
# null when it has none, so the web UI hides the download instead of
# offering a button that 404s.
data.update(client_advertisement())
return jsonify(data)
@app.get("/", defaults={"path": ""})
+135
View File
@@ -0,0 +1,135 @@
"""The server hands out the Android client it is in step with.
## Why the server, and not a release page
The Fabled-Git instance is private (issue 2091), so `install.sh` already cannot
fetch for anyone but the operator — and a self-hoster should not need an account
on someone else's forge to get the app for their own notes. The server they
already trust with the notes is the obvious place to get the client from.
It also keeps the two in step by construction. Client and server already
negotiate a sync protocol version before linking, so a server that also serves
the client cannot hand out a phone it is unable to talk to.
## Where the file comes from
`DATA_DIR/client/` — the same volume that already holds attachments, so an
operator drops a build there once and container recreation does not lose it.
Deliberately NOT baked into the image: that would charge ~55 MiB to every
self-hoster, including everyone who never touches Android.
Two files, and both must be present:
- `thoughtsync.apk` — the client
- `thoughtsync-android.json` — `{version_name, version_code, size, sha256}`
The sidecar exists because an APK's version lives in a binary AXML manifest that
Python cannot read without the Android build tools. CI writes it beside the APK
at publish time, where the real values are already known.
## Absence is normal
A server with no APK advertises nothing, and the web UI hides the download
rather than offering a button that 404s. Same for a mismatched pair: if the
sidecar's recorded size does not match the file on disk, the two did not arrive
together and the server says it has nothing rather than serving one build while
describing another.
"""
from __future__ import annotations
import json
from pathlib import Path
from quart import Blueprint, jsonify, send_from_directory
from .auth import login_required
from .config import Config
APK_NAME = "thoughtsync.apk"
MANIFEST_NAME = "thoughtsync-android.json"
DOWNLOAD_PATH = "/api/client/android/download"
APK_MIMETYPE = "application/vnd.android.package-archive"
bp = Blueprint("client_dist", __name__)
def android_release() -> dict | None:
"""What Android build this server holds, or None if it holds none.
Never raises. A missing directory, an unreadable sidecar, malformed JSON and a
sidecar that describes a different file are all the same answer to the only
question being asked — "is there a client here I can honestly offer?" — and
that answer is no.
"""
root = Path(Config.client_root())
apk = root / APK_NAME
try:
size = apk.stat().st_size
meta = json.loads((root / MANIFEST_NAME).read_text(encoding="utf-8"))
version = str(meta["version_name"])
code = int(meta["version_code"])
recorded = int(meta["size"])
digest = str(meta["sha256"])
except (OSError, ValueError, TypeError, KeyError):
return None
# The pair has to describe one build. A sidecar left behind by a previous
# release would otherwise advertise a version this server cannot serve, and the
# phone would download something other than what it was promised.
if recorded != size:
return None
return {
"version": version,
# What Android actually compares. `version` is for people; a name is a
# string and sorts like one, which is not how "is this newer" works.
"version_code": code,
"size": size,
# Computed by CI over the same bytes it uploaded, so a client can tell a
# truncated download from a complete one BEFORE handing it to the
# installer. Not a trust anchor — the signature is that.
"sha256": digest,
"url": DOWNLOAD_PATH,
}
def advertisement() -> dict:
"""The `/api/config` fragment describing this server's Android client.
An empty dict when there is none, so the key is ABSENT rather than null — a
client testing for the key gets one unambiguous answer instead of having to
distinguish "no client" from "old server that never had this field".
"""
release = android_release()
return {"android_client": release} if release else {}
@bp.get("/api/client/android")
async def android_metadata():
"""Version and digest without the 55 MiB. What an updater polls."""
release = android_release()
if release is None:
return jsonify({"error": "this server has no Android client"}), 404
return jsonify(release)
@bp.get(DOWNLOAD_PATH)
@login_required
async def android_download():
"""The APK itself.
Authenticated — by session cookie from a browser, or by device bearer token
from a client updating itself; `login_required` accepts either. The metadata
above is public because a client has to be able to ask "is there something
newer?" cheaply, but the bytes are not for anyone who can reach the port.
"""
if android_release() is None:
return jsonify({"error": "this server has no Android client"}), 404
response = await send_from_directory(
Path(Config.client_root()), APK_NAME, mimetype=APK_MIMETYPE
)
# Without this some browsers try to render it, and Android's download handler
# wants a filename to hand to the package installer.
response.headers["Content-Disposition"] = f'attachment; filename="{APK_NAME}"'
return response
+11
View File
@@ -33,6 +33,17 @@ class Config:
def media_root(cls) -> Path:
return Path(cls.DATA_DIR) / "media"
@classmethod
def client_root(cls) -> Path:
"""Where the Android APK this server hands out lives.
Under DATA_DIR rather than baked into the image: the APK is ~55 MiB and an
install that never touches Android should not carry it. Being on the same
mounted volume as uploads also means an operator drops a build there once
and container recreation does not lose it. See client_dist.py.
"""
return Path(cls.DATA_DIR) / "client"
@classmethod
def secret_key_env(cls) -> str | None:
"""Optional break-glass override for the cookie-signing secret."""
+112
View File
@@ -0,0 +1,112 @@
import json
import pytest
from thoughtsync.app import create_app
from thoughtsync.client_dist import APK_NAME, MANIFEST_NAME, advertisement, android_release
from thoughtsync.config import Config
# DB-free, like the rest of this suite — the test lane runs no Postgres. That is
# why the advertisement is asserted through `advertisement()` rather than through
# `/api/config`: the route is a one-line merge of this dict into a payload whose
# other half needs a database, and testing it here tests the part that can be wrong.
#
# The two routes below ARE exercised, because neither opens a session: the metadata
# route only stats files, and the download's 401 is returned before any token
# lookup.
PAYLOAD = b"not really an apk, but the server only ever stats it"
@pytest.fixture
def app():
return create_app()
def place_client(payload: bytes = PAYLOAD, **overrides) -> dict:
"""Put a client + sidecar where the server looks. Overrides corrupt the pair."""
root = Config.client_root()
root.mkdir(parents=True, exist_ok=True)
(root / APK_NAME).write_bytes(payload)
meta = {
"version_name": "0.1.216",
"version_code": 216,
"size": len(payload),
"sha256": "ab" * 32,
}
meta.update(overrides)
(root / MANIFEST_NAME).write_text(json.dumps(meta), encoding="utf-8")
return meta
def test_absent_client_is_advertised_as_nothing_at_all():
"""The KEY is missing, not null.
A client testing for it then gets one unambiguous answer rather than having to
tell "this server has no APK" apart from "this server predates the feature".
"""
assert android_release() is None
assert advertisement() == {}
def test_a_present_client_is_advertised_with_what_android_compares():
place_client()
advertised = advertisement()["android_client"]
assert advertised["version"] == "0.1.216"
# The integer is what decides "is this newer", not the name — a name is a
# string and sorts like one.
assert advertised["version_code"] == 216
assert advertised["size"] == len(PAYLOAD)
assert advertised["url"].endswith("/download")
def test_a_sidecar_describing_a_different_build_counts_as_no_client():
"""The likeliest real corruption: a new APK copied over an old sidecar.
Serving one build while advertising another is worse than serving none — the
phone would compare versions against a promise the bytes do not keep.
"""
place_client(size=999_999)
assert android_release() is None
assert advertisement() == {}
def test_an_unreadable_sidecar_counts_as_no_client():
place_client()
(Config.client_root() / MANIFEST_NAME).write_text("{ this is not json", encoding="utf-8")
assert android_release() is None
def test_a_sidecar_missing_a_field_counts_as_no_client():
root = Config.client_root()
root.mkdir(parents=True, exist_ok=True)
(root / APK_NAME).write_bytes(PAYLOAD)
(root / MANIFEST_NAME).write_text(json.dumps({"version_name": "0.1.216"}), encoding="utf-8")
assert android_release() is None
def test_a_sidecar_with_no_apk_beside_it_counts_as_no_client():
root = Config.client_root()
root.mkdir(parents=True, exist_ok=True)
(root / MANIFEST_NAME).write_text(json.dumps({"version_name": "x", "version_code": 1, "size": 1, "sha256": ""}))
assert android_release() is None
async def test_metadata_endpoint_is_public_so_an_updater_can_ask_cheaply(app):
place_client()
resp = await app.test_client().get("/api/client/android")
assert resp.status_code == 200
assert (await resp.get_json())["version_code"] == 216
async def test_metadata_404s_rather_than_describing_a_client_that_is_not_there(app):
resp = await app.test_client().get("/api/client/android")
assert resp.status_code == 404
async def test_the_bytes_need_authentication_even_though_the_version_does_not(app):
"""Anyone who can reach the port may ask what version exists; only an account
or a linked device may pull the 55 MiB."""
place_client()
resp = await app.test_client().get("/api/client/android/download")
assert resp.status_code == 401