AGPL-3.0-only, a source link in the app, and everything on dev since #5 #6

Merged
bvandeusen merged 214 commits from dev into main 2026-10-21 21:40:00 -04:00
7 changed files with 311 additions and 65 deletions
Showing only changes of commit ff6e99eb62 - Show all commits
+24 -36
View File
@@ -323,53 +323,41 @@ jobs:
docker system prune -af || true
docker builder prune --keep-storage 5g -f || true
# Bake the Android client in, on EVERY image build, so :dev, :latest and
# :<version> all carry one and a `docker compose pull` delivers a new client
# along with the new server.
# Bake EVERY client in, on every image build, so a self-hoster gets a working
# app for their machine from the server holding their notes — without an
# account on this forge, which is private (issue 2091) and is why serving them
# from a release page was never an option for anybody but the operator.
#
# Always the rolling `dev` release — the newest build there is. A versioned
# image therefore carries the newest client rather than one pinned to that
# version; the two negotiate a sync protocol version before linking, so
# "newest" is safe in a way "matching" would not buy anything over.
# ~104 MB on top of the ~85 MB image, almost all of it the AppImage. That is
# the price of the product being complete (rule 23), and the AppImage is not
# optional within it: it is the ONLY bundle that can replace itself in place,
# so a server without one cannot serve in-app updates to anyone.
#
# Fetched by the JOB, not by the Dockerfile: the release is private, and a
# Fetched by the JOB, not by the Dockerfile: the releases are private, and a
# token used inside a build lands in the context or a layer.
#
# NEVER fails the build. An image with no Android client advertises none and
# hides the download — a supported state, and the only one available before
# the first Android build has ever published.
- name: Fetch the Android client to bake in
# NEVER fails the build — see the script. A platform with nothing published
# means the server advertises nothing for it and the UI hides that download,
# which is a supported state and the only one available before that platform's
# first build has ever published.
- name: Fetch the clients to bake in
env:
GITHUB_TOKEN: ${{ github.token }}
GITHUB_SERVER_URL: ${{ github.server_url }}
GITHUB_REPOSITORY: ${{ github.repository }}
run: |
mkdir -p client
# THE CHANNEL IS A PROPERTY OF THE IMAGE. A :dev image serves the dev
# client; :latest serves the stable one. This read `download/dev`
# unconditionally until M314 step 3, on every branch — so every stable
# server shipped a dev-channel APK to anyone who downloaded the client
# from it. Not a versioning gap; a plain defect, fixed here because this
# is the step that gave `stable` an APK to point at.
# THE CHANNEL IS A PROPERTY OF THE IMAGE. A :dev image serves dev clients;
# :latest serves stable ones. This read `download/dev` unconditionally
# until M314 step 3, on every branch — so every stable server shipped a
# dev-channel APK to anyone who downloaded the client from it. Not a
# versioning gap; a plain defect, and the reason the channel is chosen here
# rather than inside the script: the caller is what knows which image it is
# building.
case "${{ github.ref_name }}" in
main) channel=stable ;;
*) channel=dev ;;
esac
echo "Baking in the $channel client."
base="${{ github.server_url }}/${{ github.repository }}/releases/download/$channel"
ok=1
for f in thoughtsync.apk thoughtsync-android.json; do
curl -fsSL -H "Authorization: token $GITHUB_TOKEN" -o "client/$f" "$base/$f" || ok=0
done
if [ "$ok" = 1 ]; then
echo "Baking in:"
cat client/thoughtsync-android.json
ls -l client/thoughtsync.apk
else
# Both or neither. Half a pair is worse than none: the server would
# read a sidecar describing an APK that isn't there, or an APK it
# cannot state a version for.
echo "::warning::No Android client on the dev release — this image ships without one."
rm -f client/thoughtsync.apk client/thoughtsync-android.json
fi
sh packaging/fetch-clients.sh "$channel" client
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v4
+8 -1
View File
@@ -463,6 +463,12 @@ jobs:
# match the binary it points at is an updater that never settles. It must
# be `key`: this value is matched against bundle filenames.
version="$(sh packaging/version.sh key desktop)"
# The version a PERSON reads, published beside the manifest as
# `thoughtsync-desktop.json`. The image build reads it to describe the
# bundles it bakes in (packaging/fetch-clients.sh) without re-deriving
# anything from its own checkout — which would be a different commit
# whenever the desktop did not rebuild.
display="$(sh packaging/version.sh display desktop)"
# Both channels are rolling: the manifest lands on the same release that
# holds the bundles, and the previous build's bundles are dropped once it
# points at this one. Nothing can reach them, and they are ~100 MB a push.
@@ -476,4 +482,5 @@ jobs:
export RELEASE_NOTES="Development build from ${GITHUB_SHA}" ;;
esac
export PRUNE_OLD_ASSETS=true
APP_VERSION="$version" bash desktop/packaging/write-manifest.sh
APP_VERSION="$version" DISPLAY_VERSION="$display" \
bash desktop/packaging/write-manifest.sh
+13 -7
View File
@@ -24,17 +24,23 @@ COPY --from=build-frontend /build/dist/ src/thoughtsync/static/
COPY alembic.ini .
COPY alembic/ alembic/
# The Android client this server hands out. CI fetches the newest published build
# into ./client immediately before this runs (ci.yml), so every image tag — :dev,
# :latest and :<version> alike — ships a client, and a `docker compose pull`
# delivers a new one with no file copying by hand.
# The clients this server hands out — the APK and all four desktop bundles. CI
# fetches the newest published build of each into ./client immediately before this
# runs (packaging/fetch-clients.sh), so both image tags ship a full set and a
# `docker compose pull` delivers new ones with no file copying by hand.
#
# Fetched by the JOB rather than here on purpose: the release is private, and a
# ~104 MB of this image is that set, almost all of it the AppImage.
#
# Fetched by the JOB rather than here on purpose: the releases are private, and a
# token used inside a build ends up in the build context or a layer.
#
# LAST of the COPYs, deliberately: this directory changes on every build, so
# putting it above the `pip install` layer would invalidate that layer every time.
#
# The directory is tracked (client/.keep) so this COPY cannot fail on a tree where
# that step never ran. An image with no APK is a supported state — the server
# advertises nothing and the web UI hides the download (client_dist.py).
# that step never ran. An image with no clients — or with some and not others — is
# a supported state: the server advertises what it has and the web UI hides the
# rest (client_dist.py).
COPY client/ src/thoughtsync/client/
ENV PYTHONPATH=/app/src
+42 -17
View File
@@ -24,6 +24,11 @@ set -euo pipefail
: "${GITHUB_REPOSITORY:?GITHUB_REPOSITORY is required (owner/repo)}"
: "${RELEASE_TAG:?RELEASE_TAG is required (the release holding the bundles)}"
: "${APP_VERSION:?APP_VERSION is required (the version the bundles carry)}"
# The version a PERSON reads, published beside the manifest so the image build can
# describe the bundles it bakes in without re-deriving anything. Required rather
# than defaulted: a missing value here would silently publish a sidecar naming the
# wrong build, and there is nothing downstream that could catch it.
: "${DISPLAY_VERSION:?DISPLAY_VERSION is required (the human-readable version)}"
# The manifest is published to the release that HOLDS the bundles. There is no
# second place any more.
@@ -116,27 +121,47 @@ pub_date="$(date -u '+%Y-%m-%dT%H:%M:%SZ')"
echo "==> Manifest:"
cat "$work/latest.json"
# The manifest goes on the same release the bundles were just read from — which is
# also the one `publish-release.sh` created or refreshed moments earlier, so it is
# Both files go on the same release the bundles were just read from — which is also
# the one `publish-release.sh` created or refreshed moments earlier, so it is
# guaranteed to exist by the time this runs.
target_id="$release_id"
target_assets="$assets"
# Replace rather than duplicate: Forgejo rejects a second asset with the same name,
# and this file is rewritten on every publish by design.
old_id="$(printf '%s' "$target_assets" \
| grep -oE "\"id\"[[:space:]]*:[[:space:]]*[0-9]+[^}]*\"name\"[[:space:]]*:[[:space:]]*\"latest\.json\"" \
| head -1 | grep -oE '[0-9]+' | head -1 || true)"
if [ -n "${old_id:-}" ]; then
echo "==> Removing the previous latest.json (id $old_id)"
curl -fsS -X DELETE "${AUTH[@]}" "$API/releases/$target_id/assets/$old_id" >/dev/null
fi
# and these files are rewritten on every publish by design.
replace_asset() {
local path="$1" name="$2" escaped old_id
escaped="${name//./\\.}"
old_id="$(printf '%s' "$assets" \
| grep -oE "\"id\"[[:space:]]*:[[:space:]]*[0-9]+[^}]*\"name\"[[:space:]]*:[[:space:]]*\"$escaped\"" \
| head -1 | grep -oE '[0-9]+' | head -1 || true)"
if [ -n "${old_id:-}" ]; then
echo "==> Removing the previous $name (id $old_id)"
curl -fsS -X DELETE "${AUTH[@]}" "$API/releases/$release_id/assets/$old_id" >/dev/null
fi
echo "==> Uploading $name to $RELEASE_TAG"
curl -fsS -X POST "${AUTH[@]}" "$API/releases/$release_id/assets?name=$name" \
-F "attachment=@$path" >/dev/null
}
echo "==> Uploading latest.json to $RELEASE_TAG"
curl -fsS -X POST "${AUTH[@]}" "$API/releases/$target_id/assets?name=latest.json" \
-F "attachment=@$work/latest.json" >/dev/null
replace_asset "$work/latest.json" "latest.json"
echo "==> Done. $RELEASE_TAG now advertises $APP_VERSION for ${#entries[@]} platform(s)."
# The version pair, for whoever needs to describe these bundles without rebuilding
# them — today the image build, which bakes the desktop clients in and writes each
# one a sidecar (`packaging/fetch-clients.sh`).
#
# It is published HERE, beside the manifest, because this is the step that speaks
# for what the channel serves: both files are written in the same breath from the
# same two values, so they cannot disagree about which build is current. A consumer
# deriving the version from its own checkout instead would describe these bytes
# with whatever commit it happened to be on.
#
# No `size` or `sha256` — those are per-artifact and there are four. Whoever
# downloads a bundle measures the bytes it actually got, which is the only way to
# tell a truncated download from a whole one.
printf '{\n "version_name": "%s",\n "version_code": "%s"\n}\n' \
"$DISPLAY_VERSION" "$APP_VERSION" > "$work/thoughtsync-desktop.json"
replace_asset "$work/thoughtsync-desktop.json" "thoughtsync-desktop.json"
echo "==> Done. $RELEASE_TAG now advertises $DISPLAY_VERSION ($APP_VERSION) for ${#entries[@]} platform(s)."
# --- prune superseded builds from a rolling channel ---------------------------
#
@@ -169,7 +194,7 @@ if [ "${PRUNE_OLD_ASSETS:-false}" = "true" ]; then
# every desktop push regardless. That is exactly what happened on run
# 4098, which swept the APK run 4092 had just published.
case "$asset_name" in
latest.json|thoughtsync.apk|thoughtsync-android.json) continue ;;
latest.json|thoughtsync-desktop.json|thoughtsync.apk|thoughtsync-android.json) continue ;;
*"$APP_VERSION"*) continue ;;
esac
echo " removing $asset_name"
+161
View File
@@ -0,0 +1,161 @@
#!/usr/bin/env sh
#
# Collect every client this image should hand out, into one directory.
#
# fetch-clients.sh <dev|stable> <destdir>
#
# The server serves clients from `DATA_DIR/client/` or from the copy baked into the
# image (`client_dist.py`). This is what fills the second one. It runs in CI, right
# before `docker build`, and writes the FIXED filenames that module looks for.
#
# THE CHANNEL IS A PROPERTY OF THE IMAGE. A `:dev` image serves dev clients;
# `:latest` serves stable ones. Passed in rather than derived here, because the
# caller is the thing that knows which image it is building.
#
# NEVER FAILS. A platform with nothing published means the server advertises
# nothing for it and the UI hides that download — a supported state, and the only
# one available before a platform's first build has ever published. Turning eight
# fetches into eight ways to redden an otherwise fine lane would be strictly worse
# than shipping an image that offers four clients instead of five.
#
# WHY THE VERSION IS FETCHED AND NOT DERIVED. The obvious shortcut is to run
# `version.sh display desktop` here — this job has the checkout, after all. It is
# wrong: this commit may not be the commit the channel is serving. A push touching
# only `src/` does not rebuild the desktop, so the channel still holds an older
# build, and a locally-derived version would describe those bytes with this
# commit's number. The size check in `client_dist.py` would not catch it, because
# the size IS measured from the real file — it would sail through and lie about the
# version only. So the version comes from the channel, beside the bytes it
# describes, and only `size`/`sha256` are measured here.
set -eu
channel="${1:?usage: fetch-clients.sh <dev|stable> <destdir>}"
dest="${2:?usage: fetch-clients.sh <dev|stable> <destdir>}"
case "$channel" in dev|stable) : ;; *)
echo "fetch-clients.sh: unknown channel '$channel'" >&2; exit 2 ;;
esac
SERVER="${GITHUB_SERVER_URL:-https://git.fabledsword.com}"
REPO="${GITHUB_REPOSITORY:-bvandeusen/thoughtsync}"
BASE="$SERVER/$REPO/releases/download/$channel"
mkdir -p "$dest"
# Authenticated when we have a token — these releases are private (issue 2091), so
# on this instance we always do. Anonymous still works against a public fork.
fetch() {
if [ -n "${GITHUB_TOKEN:-}" ]; then
curl -fsSL -H "Authorization: token $GITHUB_TOKEN" -o "$2" "$1"
else
curl -fsSL -o "$2" "$1"
fi
}
# One field out of a small flat JSON object. `grep`/`sed` rather than a parser
# because this runs in the CI image's busybox sh and adding a jq dependency to buy
# one string is not a trade worth making. The sidecars are written by us and are
# one level deep.
field() {
grep -oE "\"$2\"[[:space:]]*:[[:space:]]*\"?[^,\"}]+\"?" "$1" 2>/dev/null \
| head -1 | sed -E 's/.*:[[:space:]]*"?([^"]*)"?[[:space:]]*$/\1/'
}
bytes() { wc -c < "$1" | tr -d ' '; }
digest() { sha256sum "$1" | cut -d' ' -f1; }
# The sidecar shape `client_dist.py` reads. `size` and `sha256` are measured from
# the file that actually landed, so a truncated download cannot be described as a
# whole one.
sidecar() {
_file="$1"; _out="$2"; _name="$3"; _code="$4"
printf '{\n "version_name": "%s",\n "version_code": %s,\n "size": %s,\n "sha256": "%s"\n}\n' \
"$_name" "$_code" "$(bytes "$_file")" "$(digest "$_file")" > "$_out"
}
echo "==> Collecting the $channel clients"
# --- Android -----------------------------------------------------------------
#
# Its sidecar is published whole by the Android lane — an APK keeps its version in
# a binary AXML manifest, so the values are recorded where they were already known.
# Copied verbatim rather than rebuilt here.
if fetch "$BASE/thoughtsync.apk" "$dest/thoughtsync.apk" &&
fetch "$BASE/thoughtsync-android.json" "$dest/thoughtsync-android.json"; then
echo " android $(field "$dest/thoughtsync-android.json" version_name)"
else
# Both or neither. Half a pair is worse than none: the server would read a
# sidecar describing an APK that is not there, or an APK it cannot state a
# version for.
echo "::warning::No Android client on the $channel channel — this image ships without one."
rm -f "$dest/thoughtsync.apk" "$dest/thoughtsync-android.json"
fi
# --- desktop -----------------------------------------------------------------
#
# One sidecar on the channel carries the version PAIR for all four bundles, because
# they are one build: `version_name` is what a person reads, `version_code` is the
# ordering key, and the key is also what the bundle filenames are stamped with.
# Written by `write-manifest.sh`, which is the step that speaks for what the channel
# serves.
bake_desktop() {
desk="$dest/.desktop-release.json"
if ! fetch "$BASE/thoughtsync-desktop.json" "$desk"; then
echo "::warning::No desktop release on the $channel channel — this image ships without desktop clients."
rm -f "$desk"
return 0
fi
name="$(field "$desk" version_name)"
key="$(field "$desk" version_code)"
rm -f "$desk"
if [ -z "$name" ] || [ -z "$key" ]; then
echo "::warning::The $channel desktop sidecar named no version — skipping desktop clients."
return 0
fi
echo " desktop $name (key $key)"
# Bundle filenames are stamped with the ORDERING KEY — what Tauri puts in them,
# and what `write-manifest.sh` already selects on. Constructed rather than
# discovered from the release's asset list: one shape, no JSON walk, and a name
# that does not resolve is caught by the fetch failing rather than by matching
# the wrong file.
#
# `<platform id>|<published name>|<name on disk>`
for row in \
"linux-deb|ThoughtSync_${key}_amd64.deb|thoughtsync.deb" \
"linux-pacman|thoughtsync-${key}-1-x86_64.pkg.tar.zst|thoughtsync.pkg.tar.zst" \
"linux-appimage|ThoughtSync_${key}_amd64.AppImage|thoughtsync.AppImage" \
"windows|ThoughtSync_${key}_x64-setup.exe|thoughtsync-setup.exe"
do
id="${row%%|*}"; rest="${row#*|}"
remote="${rest%%|*}"; local_name="${rest#*|}"
if ! fetch "$BASE/$remote" "$dest/$local_name"; then
echo "::warning::$channel has no $remote — this image ships without the $id client."
rm -f "$dest/$local_name"
continue
fi
# The AppImage is the only bundle that replaces itself in place, so the updater
# verifies a signature before it does. Without one it is not servable as an
# update, and `client_dist.py` treats it as absent rather than offering it
# unverifiable — so drop the bundle too rather than baking 95 MB nothing can use.
if [ "$id" = "linux-appimage" ]; then
if ! fetch "$BASE/$remote.sig" "$dest/$local_name.sig"; then
echo "::warning::$remote has no signature on $channel — dropping the AppImage."
rm -f "$dest/$local_name" "$dest/$local_name.sig"
continue
fi
fi
sidecar "$dest/$local_name" "$dest/thoughtsync-$id.json" "$name" "$key"
echo " $id $(bytes "$dest/$local_name") bytes"
done
}
bake_desktop
echo "==> Baked in:"
ls -l "$dest"
+24 -1
View File
@@ -99,6 +99,18 @@ class Platform:
# the only bundle that can replace itself in place — a package-manager install
# cannot, by design (see the desktop's update.rs).
signed: bool = False
# Whether this platform's ordering key is an INTEGER.
#
# `version_code` is "whatever this platform's comparator reads", and that is not
# one type. Android's is an int because Android's own install gate compares one,
# and it must stay a JSON number — `ClientRelease` in core/src/sync/client.rs
# declares it `i64` and a string would fail to deserialize on every phone in the
# field. The desktop's is Tauri's semver key, `1.0.<minutes>`, which is the value
# its updater compares and is not an integer at all.
#
# Coercing everything to int was inherited from the days when Android was the
# only platform, and would have rejected every desktop sidecar written.
code_is_int: bool = True
@property
def signature(self) -> str:
@@ -130,6 +142,7 @@ PLATFORMS: tuple[Platform, ...] = (
artifact="thoughtsync.deb",
sidecar="thoughtsync-linux-deb.json",
mimetype="application/vnd.debian.binary-package",
code_is_int=False,
),
Platform(
id="linux-pacman",
@@ -137,6 +150,7 @@ PLATFORMS: tuple[Platform, ...] = (
artifact="thoughtsync.pkg.tar.zst",
sidecar="thoughtsync-linux-pacman.json",
mimetype="application/zstd",
code_is_int=False,
),
Platform(
id="linux-appimage",
@@ -148,6 +162,7 @@ PLATFORMS: tuple[Platform, ...] = (
# cannot be wrong.
mimetype="application/octet-stream",
signed=True,
code_is_int=False,
),
Platform(
id="windows",
@@ -155,6 +170,7 @@ PLATFORMS: tuple[Platform, ...] = (
artifact="thoughtsync-setup.exe",
sidecar="thoughtsync-windows.json",
mimetype="application/vnd.microsoft.portable-executable",
code_is_int=False,
),
)
@@ -185,12 +201,19 @@ def _read(root: Path, platform: Platform) -> dict | None:
size = (root / platform.artifact).stat().st_size
meta = json.loads((root / platform.sidecar).read_text(encoding="utf-8"))
version = str(meta["version_name"])
code = int(meta["version_code"])
# See `code_is_int`. Android's must parse as an integer; the desktop's is
# Tauri's semver key and is carried through as written.
code = int(meta["version_code"]) if platform.code_is_int else str(meta["version_code"])
recorded = int(meta["size"])
digest = str(meta["sha256"])
except (OSError, ValueError, TypeError, KeyError):
return None
if not str(code).strip() or not version.strip():
# A sidecar can be well-formed and still say nothing. An empty version is
# not a version, and it would render as a blank on the download card.
return None
# The pair has to describe one build. A sidecar left behind by a previous
# release would otherwise advertise a version this server cannot serve, and the
# client would download something other than what it was promised.
+39 -3
View File
@@ -34,6 +34,18 @@ PAYLOAD = b"not really a client, but the server only ever stats it"
# four would ship untested.
ALL_IDS = [p.id for p in PLATFORMS]
# `version_code` is "whatever this platform's comparator reads", and that is not one
# type. Android's install gate compares an integer; the desktop's updater compares
# Tauri's semver key. The tests carry both shapes for the same reason the module
# does — a suite that only ever wrote integers would pass while every desktop
# sidecar CI writes was being rejected.
ANDROID_CODE = 3503708
DESKTOP_CODE = "1.0.3503707"
def code_for(platform_id: str):
return ANDROID_CODE if BY_ID[platform_id].code_is_int else DESKTOP_CODE
@pytest.fixture(autouse=True)
def _empty_baked_client(tmp_path, monkeypatch):
@@ -71,7 +83,7 @@ def place(
(root / platform.artifact).write_bytes(payload)
meta = {
"version_name": "2026.08.30.0307",
"version_code": 3503708,
"version_code": code_for(platform_id),
"size": len(payload),
"sha256": "ab" * 32,
}
@@ -164,6 +176,30 @@ def test_a_sidecar_with_no_artifact_beside_it_counts_as_no_client(platform_id):
assert release(platform_id) is None
def test_androids_code_must_be_an_integer():
"""`ClientRelease` in core/src/sync/client.rs declares it `i64`. A string here
would fail to deserialize on every phone in the field, so a sidecar carrying one
is not a client this server can honestly offer."""
place("android", version_code="1.0.3503707")
assert release("android") is None
def test_the_desktop_keeps_tauris_semver_key_verbatim():
"""It is not an integer and must not be coerced into one: this is the value the
desktop updater compares, and `1.0.3503707` truncated to `1` orders against
nothing."""
place("linux-deb")
assert release("linux-deb")["version_code"] == "1.0.3503707"
@pytest.mark.parametrize("platform_id", ALL_IDS)
def test_an_empty_version_name_counts_as_no_client(platform_id):
"""A sidecar can be well-formed and still say nothing. A blank would render as
an empty space on the download card, which reads as a layout bug."""
place(platform_id, version_name="")
assert release(platform_id) is None
def test_an_unknown_platform_is_not_a_client():
assert release("blackberry") is None
@@ -178,7 +214,7 @@ def test_a_present_client_reports_what_a_comparator_reads(platform_id):
assert found["version"] == "2026.08.30.0307"
# The integer is what decides "is this newer", not the name — a name is a string
# and sorts like one.
assert found["version_code"] == 3503708
assert found["version_code"] == code_for(platform_id)
assert found["size"] == len(PAYLOAD)
assert found["platform"] == platform_id
# A PATH, not an absolute URL: the client joins it to the base it is already
@@ -317,7 +353,7 @@ async def test_metadata_endpoint_is_public_so_an_updater_can_ask_cheaply(app, pl
place(platform_id)
resp = await app.test_client().get(f"/api/client/{platform_id}")
assert resp.status_code == 200
assert (await resp.get_json())["version_code"] == 3503708
assert (await resp.get_json())["version_code"] == code_for(platform_id)
@pytest.mark.parametrize("platform_id", ALL_IDS)