27 Commits
Author SHA1 Message Date
bvandeusenandClaude Opus 5 02c932260e Updater signing key, a rising dev version, and a production compose
Desktop (Tauri) / Windows installer (cross-compiled) (push) Successful in 2m33s
Desktop (Tauri) / Tauri desktop (Linux) (push) Successful in 4m37s
Desktop (Tauri) / Update manifest (push) Successful in 3s
Three things, all needed before the update loop can be tested.

The public signing key is committed. Verified before trusting it: algorithm
`Ed`, key ID 90E96FEA2F6D9B6A matching its own comment, 32-byte Ed25519 key.

Dev builds now carry a version that RISES. Every build took its version from
Cargo.toml, so each one was 0.1.0 — an installed 0.1.0 would read a manifest
advertising 0.1.0, conclude it was current, and never update. The rolling
channel would have looked broken while working exactly as written. Dev builds
are now 0.1.<ci-run-number>, from one helper shared by both bundle jobs and the
manifest writer, because three separate derivations of "what version is this"
is three chances for the binary and the manifest to disagree.

Plain semver, not a `-dev.N` prerelease: prerelease versions sort BELOW the
release they qualify, so a tagged build would never update to a newer dev one,
and Windows installer metadata wants a numeric X.Y.Z regardless. Bumping the
minor still beats any dev build on the old line — 0.2.0 > 0.1.2932.

The Windows job also gets the signing environment it was missing, so its NSIS
installer is signed too. Without that the manifest would have had a Linux entry
and nothing for the platform actually being tested.

docker-compose.yml is now the production stack, per request: it pulls the
published image instead of building, keeps Postgres OFF the host network, sets
restart policies, health checks and log rotation, and refuses to start without
a POSTGRES_PASSWORD rather than shipping a known one. Volume names are
deliberately unchanged so an existing deployment upgrades in place instead of
silently coming up against an empty database. Development keeps its own
clearly-named file.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SreJkbxB4gx8pPsu8QbLPi
2026-07-27 10:33:17 -04:00
bvandeusenandClaude Opus 5 1f294c4ad8 ci-requirements: record how to format the Rust lane without a local toolchain
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SreJkbxB4gx8pPsu8QbLPi
2026-07-26 19:27:39 -04:00
bvandeusenandClaude Opus 5 2e8717a057 desktop: rustfmt the two new preference helpers
Desktop (Tauri) / Windows installer (cross-compiled) (push) Successful in 2m11s
Desktop (Tauri) / Tauri desktop (Linux) (push) Successful in 4m4s
Desktop (Tauri) / Update manifest (push) Successful in 3s
Verified locally this time rather than in CI. The ci-tauri image is already on
this machine, so `cargo fmt --check` can run in a throwaway container against
the exact toolchain CI uses — no test run, no build, no local stack, just the
formatter. Four consecutive pushes had failed on formatting alone; that class
of failure is now catchable before it costs a cycle.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SreJkbxB4gx8pPsu8QbLPi
2026-07-26 19:27:27 -04:00
bvandeusenandClaude Opus 5 d6734cf7a0 desktop: in-app updates, two channels, signed, fed by fixed-tag releases
CI & Build / Python lint (push) Successful in 3s
CI & Build / TypeScript typecheck (push) Successful in 6s
CI & Build / Python tests (push) Successful in 10s
CI & Build / Build & push image (push) Successful in 30s
Desktop (Tauri) / Tauri desktop (Linux) (push) Failing after 1m59s
Desktop (Tauri) / Windows installer (cross-compiled) (push) Successful in 2m23s
Desktop (Tauri) / Update manifest (push) Has been skipped
There was no in-place update anywhere. The app never checked, downloaded or
applied anything, and the only published release predates the whole sync arc —
so `install.sh` would hand out a build with no sync in it. Installing from
per-run CI artifacts, which is what's been happening, is not something an
updater can point at: ephemeral, auth-gated, no stable URL.

Two channels, switchable in the app: `stable` follows tagged releases, `dev`
follows every green push.

The feed is a Fabled-Git release asset, not a ThoughtSync server route. This
reverses the lean recorded in task 1998, and the reason matters — a
server-hosted feed can only reach a desktop that has linked a server, and
local-first-with-no-server is the whole premise. An unlinked install has to be
able to update itself.

Each channel reads a `latest.json` on a release whose TAG NEVER MOVES.
That's forced, not stylistic: Forgejo has no /releases/latest/download/<asset>
route (verified — it 404s with no redirect), so "newest" cannot be named in a
URL. `dev` carries the rolling bundles; `stable` is a pointer release holding
only the manifest, whose URLs aim at the versioned release's assets, so nothing
is duplicated.

The manifest is written by a third job that runs after both bundle jobs. They
build in separate workspaces and neither can see the other's output, but one
manifest has to describe both platforms — generating it inside either job would
silently omit the other, and a missing platform reads to a user as "no update
available" rather than as a broken feed. It reads what actually landed on the
release, so it can never advertise a bundle that failed to upload.

Signing is gated on the secret existing, in the script rather than an `if:`
(the secrets context isn't reliably available to step conditions). No key means
no updater artifacts and no publish: a feed the app would refuse to verify is
worse than no feed, because it looks like it works. CI stays green until the
key lands.

On Linux the updater can only replace an AppImage — a deb or pacman install is
owned by its package manager and must never be overwritten underneath it. The
app detects that case up front and says so, instead of failing halfway through
with a permissions error nobody can read.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SreJkbxB4gx8pPsu8QbLPi
2026-07-26 19:03:12 -04:00
bvandeusenandClaude Opus 5 b7c0820230 desktop: rustfmt the blob-store literal in the scheme handler
Desktop (Tauri) / Tauri desktop (Linux) (push) Successful in 3m49s
Desktop (Tauri) / Windows installer (cross-compiled) (push) Successful in 1m53s
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SreJkbxB4gx8pPsu8QbLPi
2026-07-26 17:39:12 -04:00
bvandeusenandClaude Opus 5 c40263967d desktop: render synced attachments instead of broken images (task 2114)
Desktop (Tauri) / Tauri desktop (Linux) (push) Failing after 1m31s
Desktop (Tauri) / Windows installer (cross-compiled) (push) Successful in 1m47s
A synced note carried the SERVER's relative attachment path
(/api/notes/<id>/attachments/<aid>). In the webview that resolves against the
app origin and 404s, so every synced image rendered broken even though the
bytes were already on disk from M10.7d. The absolute server URL wouldn't have
worked either: that route wants a bearer token the webview never sends, and it
would put an offline app on the network to show a file it already has.

The bytes now come off disk over a custom URI scheme, served straight from the
content-addressed blob store. The webview caches and range-requests them like
any other resource — which a data: URI would have thrown away — and the URL is
immutable-cacheable because a content address can never describe different
bytes.

Two things worth knowing about the shape of this:

The URL is rewritten in `load_attachments`, the single place the desktop
builds an attachment for the UI. NoteCard and NoteEditor are untouched, so
there's no second render site to drift.

The scheme's URL form is NOT the same on every platform: `scheme://localhost/`
on Linux and macOS, `http://scheme.localhost/` on Windows and Android. Getting
it wrong breaks exactly one channel, silently, and a headless CI runner can
never tell you.

The mime rides in the URL, and this scheme is an origin of its own, so an
attachment claiming to be text/html would run as a document there. Only media
families are echoed back; everything else is served as an opaque download,
which is the right treatment for an arbitrary file anyway. Path safety is
inherited rather than re-implemented — the handler reads through BlobStore,
which already refuses anything that isn't a bare sha256.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SreJkbxB4gx8pPsu8QbLPi
2026-07-26 17:16:44 -04:00
bvandeusenandClaude Opus 5 d634801bd3 desktop: rustfmt the retention query and one assert
Desktop (Tauri) / Windows installer (cross-compiled) (push) Successful in 1m55s
Desktop (Tauri) / Tauri desktop (Linux) (push) Successful in 3m53s
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SreJkbxB4gx8pPsu8QbLPi
2026-07-26 16:46:45 -04:00
bvandeusenandClaude Opus 5 7a77a0e1b9 desktop: fix a retention test that raced the wall clock
Desktop (Tauri) / Tauri desktop (Linux) (push) Failing after 1m31s
Desktop (Tauri) / Windows installer (cross-compiled) (push) Successful in 1m50s
`a_note_exactly_at_the_boundary_survives` stamped a note 30 days ago and then
asked the sweep — which reads `now` microseconds later — whether it was
strictly older than 30 days. It was, by those microseconds. The assertion was
wrong, not the code: an exact tie isn't observable against a wall clock.

Now stamps a note with a minute of its window still to run, which is the
property actually worth pinning: the comparison is strictly-older, so a note
inside the window is kept.

Also rewrote the row scan as plain statements. The `filter_map` over
`query_map` swallowed real rusqlite errors through `.ok()?` on the way to
skipping unparseable timestamps — the two cases deserve different treatment,
and only the second should be silent.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SreJkbxB4gx8pPsu8QbLPi
2026-07-26 16:25:32 -04:00
bvandeusenandClaude Opus 5 e64d67e904 Expire trash after 30 days, and make the deadline something you can see
CI & Build / Python lint (push) Successful in 2s
CI & Build / TypeScript typecheck (push) Successful in 6s
CI & Build / Python tests (push) Successful in 12s
CI & Build / Build & push image (push) Successful in 44s
Desktop (Tauri) / Tauri desktop (Linux) (push) Failing after 1m45s
Desktop (Tauri) / Windows installer (cross-compiled) (push) Successful in 2m12s
Trash had no end. A note sat in /trash until someone emptied it by hand, and
its attachment BYTES sat on disk the whole time — the pile-up the operator
asked about. Nothing purged; there was no scheduler at all.

Retention is server-owned: `trash_retention_days` (default 30, 0 = keep
forever) in the settings registry, so it lands in admin Settings with no
migration and takes effect without a restart. A background sweep started in
before_serving does the work. Clients learn about a purge the way they learn
about any deletion — as a tombstone on the delta feed.

An auto-purge nobody can see coming is data loss on a timer, so the window is
now visible: /api/config publishes it, notes carry `deleted_at`, Trash leads
with the policy, and each card counts down. The countdown rounds DOWN — saying
"1 day left" for a note with ten minutes on the clock is the one error here
that actually costs someone a note.

Three things this turned up on the way:

- `DELETE /api/notes/<id>` hard-deleted the row, leaving no tombstone at all.
  A permanent delete in the web UI never reached a linked device, which would
  keep its copy forever and push it back on the next edit. It now purges
  through the same path as everything else.
- The purge left `note_revisions` and `note_link_previews` behind. A revision
  holds the full body, so the text of a "permanently deleted" note was still
  sitting in the database.
- `deleted_at` now SURVIVES a purge instead of being cleared. It's still true,
  and it means every query that says "not trashed" excludes tombstones for
  free — without it a content-less row reads as a perfectly normal active note
  and shows up on the board as a blank card.

Desktop keeps its own clock only when there's nobody else to keep one: the
sweep runs at startup on an UNLINKED device and refuses otherwise. A linked
client that expired notes on its own schedule could destroy something the
server was deliberately keeping, then push that delete upstream. Local policy
must never outrank the server's — so it also adopts the server's window for
the countdown rather than showing its offline default.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SreJkbxB4gx8pPsu8QbLPi
2026-07-26 16:20:13 -04:00
bvandeusenandClaude Opus 5 6f35e6e6d8 Confirm irreversible deletes, which sync just made far more consequential
CI & Build / Python lint (push) Successful in 3s
CI & Build / TypeScript typecheck (push) Successful in 6s
CI & Build / Python tests (push) Successful in 8s
CI & Build / Build & push image (push) Successful in 31s
The trash model itself was already right and needed no change: notes soft-
delete (`trashed` locally, `deleted_at` server-side), Trash is a real view,
restore works, permanent deletion is a separate second step only offered on
an already-trashed note, `trash()` shows an Undo toast, and nothing auto-
purges — trash persists until someone acts. Sync carries all of it: a trashed
note syncs WITH its content, and only `purged_at` deletes a client's copy.

What was missing is the guard on the irreversible step. "Delete forever" and
label deletion were one click, silent, with no confirmation — and M10.7 has
changed what that costs. Before, a mis-click lost a note on one machine.
Now it pushes a tombstone that deletes it from every linked device, and the
local tombstone survives to make sure it gets there.

Both guards live in the STORE, not the call sites: NoteCard and NoteEditor
both offer delete-forever, and duplicating the copy is how two prompts drift
until one of them stops matching what actually happens.

The copy names the real consequence — "deleted from every device you sync
with" — because that's the part a user cannot infer from a button in a Trash
view. The label prompt also says the notes themselves are kept, since that's
what people actually worry about when deleting a label.

Labels deliberately get a confirmation but NOT a trash of their own. A label
is organization, not content; the reversible middle step notes get would be
ceremony around something that costs nothing to recreate.

Saved-filter deletion already confirmed (AppShell), so these two were the
outliers, not a new convention.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SreJkbxB4gx8pPsu8QbLPi
2026-07-26 15:40:33 -04:00
bvandeusenandClaude Opus 5 810da43f56 desktop: rustfmt the blob-store test
Desktop (Tauri) / Tauri desktop (Linux) (push) Successful in 3m51s
Desktop (Tauri) / Windows installer (cross-compiled) (push) Successful in 1m57s
One hunk from run 2911. Clippy and all 67 tests — including the six new blob
tests and the path-traversal guard — had already passed on the same code.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SreJkbxB4gx8pPsu8QbLPi
2026-07-26 00:46:25 -04:00
bvandeusenandClaude Opus 5 ed623a7bef M10.7d: download attachment bytes into a content-addressed store (task 2107)
CI & Build / Python lint (push) Successful in 3s
CI & Build / TypeScript typecheck (push) Successful in 5s
CI & Build / Python tests (push) Successful in 8s
CI & Build / Build & push image (push) Successful in 35s
Desktop (Tauri) / Tauri desktop (Linux) (push) Failing after 1m29s
Desktop (Tauri) / Windows installer (cross-compiled) (push) Successful in 1m47s
The client half of task 1942's server work. Metadata already rides the delta
feed; this fetches the payload so a synced image exists on the device.

Blobs are filed under their own sha256, so the same image attached to five
notes is stored once and re-downloading it is free — the dedupe the task asks
for falls out of content addressing rather than needing bookkeeping.

The hash is also the integrity check, applied on the way IN. Bytes that don't
hash to what the server advertised are refused rather than filed under a name
that lies about them — and because the blob then still counts as missing, the
next sync simply tries again.

SECURITY: the hash arrives in a server response and becomes a FILENAME, so it
is validated as 64 hex characters before touching the filesystem. Without
that, a hostile or buggy server could send "../../..." and steer a write
outside the blob directory. Tested.

A failed attachment never fails the sync. Notes are the primary data and have
already landed; aborting here would let one unreachable file block every
future sync. Counted, logged, surfaced in the UI as "they'll retry on the
next sync", and retried because the blob is still absent.

sha2 is pure Rust, so the Windows cross-compile lane pays nothing for it —
the constraint recorded in ci-requirements.md.

SPLIT, deliberately: this stores the bytes but does NOT yet render them in
the webview. That half needs a custom URI scheme or the asset protocol, whose
URL form differs by platform (Windows uses http://scheme.localhost/, others
scheme://localhost/) — and CI cannot verify webview rendering at all, being
headless with no webview. Guessing at it here would ship an unverifiable
change on the most fragile lane. Follow-up filed; synced images will show as
broken until it lands.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SreJkbxB4gx8pPsu8QbLPi
2026-07-26 00:43:34 -04:00
bvandeusenandClaude Opus 5 6bef07ff83 desktop: rustfmt the SyncOutcome literal
Desktop (Tauri) / Windows installer (cross-compiled) (push) Successful in 2m2s
Desktop (Tauri) / Tauri desktop (Linux) (push) Successful in 3m56s
One hunk from run 2908. Clippy, all 61 tests, and vue-tsc (run 2907) had
already passed on the same code.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SreJkbxB4gx8pPsu8QbLPi
2026-07-26 00:33:35 -04:00
bvandeusenandClaude Opus 5 fe683595df M10.7e: desktop Sync settings screen (task 2108)
CI & Build / Python lint (push) Successful in 3s
CI & Build / TypeScript typecheck (push) Successful in 6s
CI & Build / Python tests (push) Successful in 11s
CI & Build / Build & push image (push) Successful in 33s
Desktop (Tauri) / Tauri desktop (Linux) (push) Failing after 1m32s
Desktop (Tauri) / Windows installer (cross-compiled) (push) Successful in 2m4s
The surface that turns the engine into a feature (rule 27). Desktop-only —
the web build IS a server's UI, so a "connect a server" screen there would be
nonsense; the route redirects to the board and the nav entry is hidden.

UNLINKED IS THE RESTING STATE, not an incomplete setup. The empty case leads
with "Working offline on this device — everything works without a server",
because a screen that framed the default as a problem would push people into
configuring something they may never need. The app is local-first; this is
opt-in.

Probe before credentials. "Check" shows who actually answered — site name,
version, and the M10.6 verdict — before any password or token is typed. An
incompatible server is shown in red and the sign-in fields never appear, so
you cannot hand a credential to something that can't use it. `degraded` names
the missing capabilities rather than staying quiet and letting a feature
mysteriously do nothing.

Both credential paths, matching the Rust side: email+password (a fresh
install has no session to mint a token from) or a pasted device token (for
anyone who'd rather not type a password into a desktop app). Secrets are
cleared from component state the moment they're exchanged.

Disconnect states plainly that the token stays valid server-side and points
at Account -> Linked devices, rather than implying a remote revoke that
didn't happen (issue 2110). Wording avoids "revoke" for exactly that reason.

Push rejections are surfaced verbatim after a sync, never swallowed — a
duplicate label name is the realistic case and only a person can resolve it.

Adds schema v3: last_sync_at. The cursor can't answer "am I up to date?" —
it's a revision watermark, not a time, and it doesn't move at all when a sync
legitimately finds nothing new, so "synced a moment ago, nothing new" would
be indistinguishable from "never synced". Stamped only after BOTH halves of
the cycle succeed; a stamp after a partial cycle would claim currency the
data doesn't have. Cleared on unlink so a new server can't inherit it.

run_cycle now returns the post-cycle status, so the UI updates from one
round-trip instead of chasing every sync with a status call.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SreJkbxB4gx8pPsu8QbLPi
2026-07-26 00:30:37 -04:00
bvandeusenandClaude Opus 5 75b2d096ec desktop: rustfmt the push module
Desktop (Tauri) / Windows installer (cross-compiled) (push) Successful in 2m6s
Desktop (Tauri) / Tauri desktop (Linux) (push) Successful in 4m12s
Seven hunks, applied verbatim from run 2903's cargo fmt --check diff.

The reordered job already paid off: clippy and all 60 tests ran and passed
in that same run, so this is known to be formatting only.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SreJkbxB4gx8pPsu8QbLPi
2026-07-26 00:23:38 -04:00
bvandeusenandClaude Opus 5 b5f7dc2635 M10.7c: push + the full sync cycle (task 2106)
Desktop (Tauri) / Tauri desktop (Linux) (push) Failing after 1m28s
Desktop (Tauri) / Windows installer (cross-compiled) (push) Successful in 1m46s
Local -> server, then push-then-pull as the only ordering the UI can invoke.

LOCAL TOMBSTONES (schema v2). Found while writing push: delete_forever and
remove_label just DROPPED the row, leaving no record it existed. Offline that
means the delete can never be pushed — and the next pull faithfully
resurrects the note from the server. A deletion that undoes itself is about
the worst thing sync can do, so deletes now record into pending_deletes until
the server acknowledges them. merge_labels had the same hole.

merge_labels also moved memberships without marking the affected notes dirty.
A note's label set only reaches the server via the note itself, so a merge
looked done locally and never synced. Now marked before the delete cascades
the rows away.

Result handling, per status:
  created/applied -> clear dirty, store the returned sync_revision
  noop            -> clear dirty, drop the tombstone (a row the server never
                     saw, created and deleted entirely offline)
  kept            -> clear dirty WITHOUT touching content. Re-pushing would
                     lose the same last-write-wins comparison forever; the
                     following pull adopts the server's version.
  rejected        -> stay dirty and surface the reason. A duplicate label name
                     is the realistic case and only a human can resolve it.

The subtle one is `kept` plus a skewed clock. Normally the server's kept
revision sits above our cursor, so the next pull fetches it anyway. If the
clock makes a genuinely later local edit look older, that revision can be
BELOW the cursor — the pull skips it and the stale local copy stays on screen
with nothing marking it wrong. So a kept result at or below the cursor
rewinds the cursor to re-fetch that note. Both directions tested.

label_ids carries MANUAL memberships only. Tag-sourced ones are re-derived
server-side from the body; sending them would convert them into manual
assignments that no longer disappear when the #tag is deleted from the text.

engine::run_cycle is push-then-pull, and a failed push ABORTS before the
pull — pulling anyway would overwrite the exact rows we just failed to save,
turning a recoverable network error into lost work. sync_pull is removed from
the command surface accordingly: offering a bare pull would hand the UI a way
to discard unsent edits. sync_now and sync_has_pending replace it.

Both loops have anti-spin guards: push stops when a batch clears nothing,
pull stops when the cursor doesn't advance.

15 push tests against an in-memory database.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SreJkbxB4gx8pPsu8QbLPi
2026-07-26 00:20:09 -04:00
bvandeusenandClaude Opus 5 2e32ecda6e desktop: rustfmt the pull tests; run fmt after clippy/test
Desktop (Tauri) / Windows installer (cross-compiled) (push) Successful in 2m20s
Desktop (Tauri) / Tauri desktop (Linux) (push) Successful in 4m8s
Two macro-argument splits and a stray blank line, applied verbatim from run
2900's cargo fmt --check diff.

Also reorders the Linux job so `cargo fmt --check` runs AFTER clippy and the
tests. Fail-fast ordering would normally put the cheapest check first, but
there is no Rust toolchain on the workstation, so this lane is verified
entirely in CI — and a formatting nit failing first SKIPS clippy and the
tests, making a whole cycle teach nothing but whitespace. That has now cost
four cycles in this session alone. It still runs before the 20-40 minute
bundle build, so a fmt failure doesn't burn that either.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SreJkbxB4gx8pPsu8QbLPi
2026-07-26 00:09:07 -04:00
bvandeusenandClaude Opus 5 dc8b2d360d M10.7b: pull the change feed into the local store (task 2105)
Desktop (Tauri) / Tauri desktop (Linux) (push) Failing after 26s
Desktop (Tauri) / Windows installer (cross-compiled) (push) Successful in 1m53s
Server -> local. sync/wire.rs mirrors the delta-feed JSON exactly as
notes/serialize.py sends it; sync/pull.rs applies it.

ATOMICITY IS THE POINT. The cursor is written in the SAME transaction as the
page it describes. A cursor committed ahead of its data would skip those rows
forever while reporting a clean sync — the worst kind of failure, because
nothing looks wrong. A test forces a mid-page failure and asserts the cursor
stayed put.

Every degradation leans toward re-downloading rather than skipping: an
unparseable cursor means full sync, wire fields are all defaulted so a newer
server adding a field (or an older one omitting one) yields a partial note
instead of a rejected page, and a page that fails rolls back whole.

Labels are applied before notes so a membership never references a row that
doesn't exist. A note also carries enough of its labels to materialize them,
because notes and labels page from ONE shared sequence and a note can arrive
referencing a label whose own delta landed in an earlier page.

via_tag is applied verbatim rather than re-deriving #tags from the body. The
server already reconciled them on save, and re-deriving would go through the
local find-or-create path, which marks new labels dirty — pushing them
straight back. Sync churn manufactured out of nothing.

Duplicate-label merge, the subtle one: a label created offline can collide by
name with one the server already had under a different id. Both sides enforce
one label per name, so the server's row has to win — but simply deleting the
local duplicate would CASCADE its note_labels away, stripping the label off
notes this pull never mentions, with no later page to repair it. So we free
the name, insert the server's row, re-point the memberships, then drop the
husk. Tested.

Children (items/attachments/previews/labels) are replaced wholesale rather
than diffed: a delta carries the note's FULL state, so what arrived IS the
complete set, and diffing could strand a row the server no longer has.

The loop trusts the data over the flag — a server claiming has_more without
advancing its cursor stops with an error instead of spinning forever.

Pull can overwrite a row with unpushed local edits. The documented cycle is
push-then-pull (M10.7c), so that should never happen; when it does it's
counted as clobbered_dirty and logged rather than hidden.

17 tests, all against an in-memory database.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SreJkbxB4gx8pPsu8QbLPi
2026-07-26 00:05:27 -04:00
bvandeusenandClaude Opus 5 7d9a6509f3 desktop: rustfmt the M10.7a state tests
Desktop (Tauri) / Windows installer (cross-compiled) (push) Successful in 1m55s
Desktop (Tauri) / Tauri desktop (Linux) (push) Successful in 3m52s
Four macro-argument splits, applied verbatim from run 2895's
cargo fmt --check diff. No logic change.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SreJkbxB4gx8pPsu8QbLPi
2026-07-25 23:38:49 -04:00
bvandeusenandClaude Opus 5 bbb2fd9b1c M10.7a: link/unlink a server — device auth + sync_state (task 2104)
Desktop (Tauri) / Tauri desktop (Linux) (push) Failing after 27s
Desktop (Tauri) / Windows installer (cross-compiled) (push) Successful in 1m55s
The pairing step. Nothing else in the sync arc can move until this works.

sync/state.rs owns the link record in the sync_state row M10.4 already put
in the local schema. Two safety properties are the reason it isn't just
three setters:

- Linking a DIFFERENT server resets the change-feed cursor. A cursor is only
  meaningful against the server that issued it; carrying one across would
  silently skip every change on the new server below that watermark — data
  loss wearing the costume of a successful sync. Re-linking the SAME server
  (a token refresh) keeps it, so a routine re-auth doesn't force a full
  re-download.
- Unlink clears the cursor too, so a later link can't inherit a watermark
  from a server that never issued it.

An unparseable or absent cursor reads as 0 (full sync). That direction is
always safe: a redundant re-sync costs time, a too-high cursor costs notes.
Likewise a half-written row (server but no token) reports NOT linked.

state::Status deliberately has no device_token field — it crosses into the
webview, and a long-lived bearer token has no business reachable from page
scripts. A test asserts the token never appears in its serialization.

Token lives in the app-data SQLite file, not an OS keyring: the keyring
crate needs libsecret/DBus on Linux, which adds a C dependency to a binary
that has to cross-compile and fails outright on headless/minimal-WM setups —
the same class of environment assumption behind the black-window bug.

sync_link runs the M10.6 handshake FIRST and refuses an incompatible server
before any credential is sent. Two credential paths, because neither covers
everyone: device-login (a fresh install has no session to mint a token from)
and a pasted token (some users would rather not type a password into a
desktop app). A pasted token is verified against /api/auth/me before being
stored — auth.py's login_required accepts bearer — since an unverified paste
would turn a copy/paste slip into a failure surfacing at the next sync, far
from its cause.

The store lock is taken only after all network work: a std MutexGuard isn't
Send so it cannot cross an await, and holding the store for a round-trip
would freeze every note operation in the UI.

Unlink is LOCAL only — the token stays valid server-side until revoked under
Account -> Linked devices. A pasted token arrives without its device id, so
a reliable remote revoke isn't possible from here; the UI must say so rather
than imply a revoke that didn't happen. Follow-up filed.

No UI yet — that's M10.7e.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SreJkbxB4gx8pPsu8QbLPi
2026-07-25 23:35:51 -04:00
bvandeusenandClaude Opus 5 9118680bb1 docs: record the CI consequences of the M10.6 TLS dependency
ci-requirements.md is the contract with CI-Runner (rule 39), so the two
things a future image change could silently break belong in it:

libssl-dev + pkg-config in ci-tauri are now load-bearing — native-tls
compiles against OpenSSL on Linux, so a slim-down of that image would fail
the Rust build at openssl-sys rather than anywhere obvious.

The TLS backend choice is a property of the WINDOWS lane, not a dependency
detail: native-tls resolves to schannel on windows-msvc, keeping C/assembly
out of the cross-compile. Swapping to rustls would pull in ring/aws-lc-rs
and their assembler — the same class of dependency that broke that lane
before. Flagged so it's treated as a lane change, not a version bump.

Also documented why libssl3 is left covered TRANSITIVELY rather than
declared. dpkg-shlibdeps now lists it, and verify.sh passes it through
webkit's recursive closure. Declaring it directly would be worse, not
better: the package name is release-dependent (libssl3 on bookworm,
libssl3t64 after the time_t transition), so hardcoding it freezes the .deb
to the build distro, whereas webkit's closure adapts. verify.sh fails loudly
if webkit ever stops pulling OpenSSL, which is what makes that safe.

Docs only — triggers no workflow.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SreJkbxB4gx8pPsu8QbLPi
2026-07-25 22:53:31 -04:00
bvandeusenandClaude Opus 5 4eb92942d0 M10.6: HTTP transport for the handshake (task 1995)
Desktop (Tauri) / Windows installer (cross-compiled) (push) Successful in 2m6s
Desktop (Tauri) / Tauri desktop (Linux) (push) Successful in 4m4s
Adds the client's first outbound call: GET {server}/api/config, carrying
X-ThoughtSync-Client and X-ThoughtSync-Protocol, feeding compat::evaluate.

Deliberately its OWN commit. This introduces the first HTTP+TLS stack into a
crate that cross-compiles to Windows from Linux via cargo-xwin — the lane
that has already broken once on a transitive C dependency (libsqlite3-sys
needing llvm-lib). Landing it alone means a failure here has exactly one
possible cause, instead of surfacing mid-way through M10.7's much larger
change where it would be expensive to bisect.

TLS backend is native-tls, NOT rustls, and that is the whole point of the
choice: on x86_64-pc-windows-msvc native-tls resolves to `schannel`, which
is pure-Rust bindings to the OS TLS stack, so nothing C or assembly has to
cross-compile on the fragile lane. rustls would pull in ring/aws-lc-rs and
their assembler. On Linux native-tls uses OpenSSL, whose headers ci-tauri
already ships (libssl-dev, part of Tauri's own Linux prerequisites).

Verified from run 2884's log rather than assumed: tokio and http are already
in the Windows tree via tauri, but no HTTP client and no TLS stack were —
so this genuinely is new surface there, not a no-op.

probe() distinguishes "never got a usable answer" (Err) from "answered, but
we can't work with it" (Ok + verdict). Those need very different messages:
one is "check what you typed", the other is "update something". Transport
errors are translated out of reqwest's Display, which is accurate but reads
like a stack trace.

Still no UI — M10.7 owns the link/settings surface that calls server_probe.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SreJkbxB4gx8pPsu8QbLPi
2026-07-25 22:47:10 -04:00
bvandeusenandClaude Opus 5 4b4bfe67ad desktop: rustfmt the client-header tuple
Desktop (Tauri) / Windows installer (cross-compiled) (push) Successful in 1m55s
Desktop (Tauri) / Tauri desktop (Linux) (push) Successful in 3m49s
Applied verbatim from run 2886's cargo fmt --check diff.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SreJkbxB4gx8pPsu8QbLPi
2026-07-25 22:42:57 -04:00
bvandeusenandClaude Opus 5 fbbe877c46 M10.6: client↔server sync protocol handshake (task 1995)
Desktop (Tauri) / Tauri desktop (Linux) (push) Failing after 42s
CI & Build / Build & push image (push) Successful in 36s
Desktop (Tauri) / Windows installer (cross-compiled) (push) Successful in 1m34s
CI & Build / Python lint (push) Successful in 3s
CI & Build / TypeScript typecheck (push) Successful in 8s
CI & Build / Python tests (push) Successful in 14s
Version the sync WIRE PROTOCOL separately from either program's release
version, so a self-hosted server and the desktop app can sit on different
releases and still work out whether they can talk.

Each side declares two numbers — what it speaks, and the oldest counterpart
it accepts. Either side can therefore mark a change breaking without the
other shipping in step, which is the whole point: no app↔server lockstep.

Server advertises on the existing public /api/config (a client must be able
to ask "can I talk to you?" before it holds a device token, or even has an
account): sync_protocol_version, min_client_protocol_version, sync_features.

sync_features exists because a version number can only say newer/older. An
ADDITIVE change earns a capability name instead of a minimum bump, so a
newer client meeting an older server drops that one feature and syncs the
rest, rather than refusing. Raising a minimum is reserved for genuinely
breaking changes — it's the switch that hard-blocks the other side.

Client half is pure decision logic (sync/compat.rs), no I/O, so every branch
is unit-testable — there's no live-server lane in CI. Three outcomes: ok /
degraded{unavailable} / incompatible{reason, client_must_update}. The last
names which side can fix it, so the message is actionable. A server that
predates the handshake sends no protocol fields at all; that reads as
"update the server", deliberately not as a parse error, which would look to
the user like they mistyped the URL.

normalize_base_url defaults a bare host to https://, never http:// —
silently downgrading would put a long-lived device token on the wire in
cleartext because someone omitted five characters. Plain HTTP on a trusted
LAN stays supported; the user types http:// and thereby chooses it.

Transport (the actual fetch) lands next, separately: it needs an HTTP/TLS
stack, and that's a real risk to the Windows cross-compile lane, so it gets
its own CI run to bisect against rather than riding along with this.

No UI here by design — the link/settings surface it feeds is M10.7's, per
this task's own sequencing.

Policy documented in docs/sync.md.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SreJkbxB4gx8pPsu8QbLPi
2026-07-25 22:40:25 -04:00
bvandeusenandClaude Opus 5 5b471f5dd4 desktop: generate the Windows icon set in the cross-compile job
Desktop (Tauri) / Windows installer (cross-compiled) (push) Successful in 1m49s
Desktop (Tauri) / Tauri desktop (Linux) (push) Successful in 3m37s
tauri-build needs icons/icon.ico to emit the Windows Resource file, and the
repo only carries the PNG set the Linux bundles use — run 2881 failed with
"icons/icon.ico not found".

Generated in-job from the committed 1024px app-icon.png rather than committing
a hand-made .ico, so there stays one icon of record that can't silently drift
from the brand art. Scoped to the windows job; the Linux bundles don't need it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SreJkbxB4gx8pPsu8QbLPi
2026-07-25 20:37:33 -04:00
bvandeusenandClaude Opus 5 ab961f13ce desktop: cross-compiled Windows NSIS installer lane (task 2015)
Desktop (Tauri) / Windows installer (cross-compiled) (push) Failing after 48s
Desktop (Tauri) / Tauri desktop (Linux) (push) Successful in 3m40s
Adds a `windows` job to desktop.yml on the new ci-tauri-win image, producing a
Windows -setup.exe without any Windows hardware. A Windows container can't run
on a Linux host, so cross-compilation is the only route: --runner cargo-xwin
supplies the MSVC CRT/SDK (pre-warmed into the image) and links with lld-link,
and makensis builds the installer.

NSIS only. .msi needs WiX v3, a Windows program — per Tauri, ".msi installers
can only be created on Windows". It comes back if a Windows node ever exists.

Kept as a separate job so a Windows-side failure can never block the Linux
artifacts, which are the primary product today. publish-release.sh now globs
the windows target root too; nullglob means each job uploads only what its own
workspace contains, and the release is created once and reused via the 409
path, so both jobs can publish to the same release safely.

No app code changes were needed. The AppImage self-integration UI already
gates on is_appimage (AccountView.vue:131, DesktopIntegrationPrompt.vue:22),
and $APPIMAGE is never set on Windows, so the OOBE prompt and Settings toggle
hide themselves.

Recorded plainly in ci-requirements.md that this is the weakest-verified lane
we have: Tauri calls Linux->Windows cross-compilation "not tested as much" and
a last resort, and a Linux runner cannot execute a Windows binary. Green means
it built. A real Windows machine check is mandatory before trusting a release,
and installers are unsigned until a certificate exists.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SreJkbxB4gx8pPsu8QbLPi
2026-07-25 20:14:55 -04:00
bvandeusenandClaude Opus 5 dc68386d1a desktop: point the install command at a branch that exists
Desktop (Tauri) / Tauri desktop (Linux) (push) Successful in 3m27s
The advertised curl URL referenced raw/branch/main, but main has never been
created (creating it is rejected by a branch-protection rule that matches the
name even with no branch behind it), so the one-command install 404'd. dev is
currently the repo's only branch and serves the script fine now that the repo
is public — verified 200, with all three v0.1.0 release assets resolving and
the AppImage downloading in full.

Flagged in the header to move back to main once that branch exists, so the
public install command stops tracking day-to-day work.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SreJkbxB4gx8pPsu8QbLPi
2026-07-25 19:19:05 -04:00
51 changed files with 6066 additions and 73 deletions
+54
View File
@@ -0,0 +1,54 @@
# ThoughtSync production settings. Copy to `.env` and edit:
#
# cp .env.example .env
#
# Only POSTGRES_PASSWORD has no default — compose refuses to start without it.
# Everything else here is optional. Anything NOT in this file (site name, signups,
# attachment limits, trash retention, link previews) is configured in the admin
# Settings UI and stored in the database, not here.
# --- required ---------------------------------------------------------------
# Generate one and keep it: changing it later means also changing it inside the
# database, or Postgres will reject the app's connection.
#
# openssl rand -base64 24 | tr -d '/+=' | head -c 32
#
# Stick to letters and digits. This value goes into a connection URL, so a `@`,
# `/`, `:` or `#` in it will be misparsed as URL structure rather than password.
POSTGRES_PASSWORD=
# --- optional ---------------------------------------------------------------
# Which build to run.
#
# latest tracks the `main` branch — the production line (default)
# dev tracks the `dev` branch — newer, less settled
# <commit sha> pins one exact build; every push publishes one, and this is
# the rollback lever when an upgrade misbehaves
#
# NOTE: `main` can sit well behind `dev`. If a feature you expect is missing,
# check which branch it actually landed on before assuming a bug.
#THOUGHTSYNC_TAG=latest
# The host port the app is published on.
#THOUGHTSYNC_PORT=5000
# Which interface to bind. The default (all interfaces) is what lets desktop
# clients on your network reach the server. Behind a reverse proxy, set this to
# 127.0.0.1 so only the proxy can talk to it.
#THOUGHTSYNC_BIND=0.0.0.0
# Database identity. Changing these AFTER the first start does not rename anything
# that already exists — the volume keeps whatever the first run created.
#POSTGRES_USER=thoughtsync
#POSTGRES_DB=thoughtsync
# --- a note on HTTPS --------------------------------------------------------
#
# The app marks its session cookie Secure automatically when a request arrives over
# HTTPS, directly or via a proxy setting X-Forwarded-Proto — no setting needed.
#
# Worth knowing if you use the desktop app: typing a bare hostname there defaults to
# https://, deliberately, so a device token never crosses the wire in cleartext by
# accident. Serving over plain HTTP means typing the `http://` yourself.
+202 -5
View File
@@ -52,10 +52,6 @@ jobs:
run: npm ci && npm run build
working-directory: frontend
- name: Rust format check
run: cargo fmt --check
working-directory: desktop/src-tauri
- name: Clippy
run: cargo clippy --all-targets -- -D warnings
working-directory: desktop/src-tauri
@@ -64,9 +60,43 @@ jobs:
run: cargo test
working-directory: desktop/src-tauri
# Deliberately AFTER clippy + test, not before.
#
# It's the cheapest check, so fail-fast ordering would normally put it first —
# but there is no Rust toolchain on the workstation (the desktop lane is
# verified entirely here), so a formatting nit failing first SKIPS clippy and
# the tests, and one CI cycle teaches nothing but whitespace. Running it here
# means every push reports its real problems too. Still before the ~20-40 min
# bundle build, so a fmt failure doesn't burn that.
- name: Rust format check
run: cargo fmt --check
working-directory: desktop/src-tauri
# Frontend already built above; skip the beforeBuildCommand rebuild.
#
# createUpdaterArtifacts is applied only when a signing key exists (M10.9):
# tauri FAILS the build if it's asked to produce updater artifacts with no key,
# so making it conditional is what lets the pipeline stay green before the
# operator has added the secret. With the key present, each bundle gets a
# `.sig` beside it — the file the updater actually verifies against.
- name: Tauri build (deb + AppImage)
run: cargo tauri build --config '{"build":{"beforeBuildCommand":""}}'
env:
TAURI_SIGNING_PRIVATE_KEY: ${{ secrets.TAURI_SIGNING_PRIVATE_KEY }}
TAURI_SIGNING_PRIVATE_KEY_PASSWORD: ${{ secrets.TAURI_SIGNING_PRIVATE_KEY_PASSWORD }}
run: |
updater='{}'
if [ -n "${TAURI_SIGNING_PRIVATE_KEY:-}" ]; then
echo "Signing key present — producing updater artifacts."
updater='{"bundle":{"createUpdaterArtifacts":true}}'
else
echo "No TAURI_SIGNING_PRIVATE_KEY — building unsigned, no updater artifacts."
fi
version="$(sh ../packaging/build-version.sh)"
echo "Building version $version"
cargo tauri build \
--config '{"build":{"beforeBuildCommand":""}}' \
--config "{\"version\":\"$version\"}" \
--config "$updater"
working-directory: desktop/src-tauri
# Tauri's AppImage bundles the build host's graphics/display libs
@@ -126,3 +156,170 @@ jobs:
env:
GITHUB_TOKEN: ${{ github.token }}
run: bash desktop/packaging/publish-release.sh
# The rolling DEVELOPMENT channel (M10.9): a release whose tag never moves, so
# the updater has a permanent URL to read — Forgejo has no
# /releases/latest/download/<asset> route, so "newest" can't be named in a URL.
#
# Gated on the signing key INSIDE the script rather than with an `if:`, because
# the secrets context isn't reliably available to step conditions. Publishing
# bundles the app would then refuse to verify is worse than publishing nothing:
# it looks like a working feed.
- name: Publish to the dev channel
if: github.ref == 'refs/heads/dev'
env:
GITHUB_TOKEN: ${{ github.token }}
TAURI_SIGNING_PRIVATE_KEY: ${{ secrets.TAURI_SIGNING_PRIVATE_KEY }}
RELEASE_TAG: dev
RELEASE_PRERELEASE: "true"
run: |
if [ -z "${TAURI_SIGNING_PRIVATE_KEY:-}" ]; then
echo "No TAURI_SIGNING_PRIVATE_KEY — skipping the dev channel publish."
exit 0
fi
bash desktop/packaging/publish-release.sh
# Windows installer, CROSS-COMPILED from Linux — there is no Windows build host.
# A Windows container can't run on a Linux host (containers share the host
# kernel), so cross-compiling is the only route without Windows hardware:
# cargo-xwin + LLVM's lld-link + makensis are Linux programs that emit Windows
# PE output. That toolchain is why this needs its own image rather than ci-tauri.
#
# NSIS only. `.msi` needs WiX v3, which is a Windows program — Tauri: ".msi
# installers can only be created on Windows". It returns if a Windows node does.
#
# A separate job, so a Windows-side failure never blocks the Linux artifacts that
# are the primary product today. Tauri calls this path "not tested as much" and a
# last resort, and nothing here can LAUNCH a Windows binary — green means it
# built, not that it runs. A real-machine check stays mandatory before trusting it.
windows:
name: Windows installer (cross-compiled)
if: github.ref == 'refs/heads/dev' || github.ref == 'refs/heads/main' || startsWith(github.ref, 'refs/tags/v')
runs-on: python-ci
container:
image: git.fabledsword.com/bvandeusen/ci-tauri-win:1.97
steps:
- uses: actions/checkout@v6
# Same reason as the Linux job: generate_context! embeds the built frontend
# at compile time, so it must exist before cargo runs.
- name: Build the shared frontend
run: npm ci && npm run build
working-directory: frontend
# tauri-build generates a Windows Resource file and needs `icons/icon.ico`,
# which the repo doesn't carry — only the PNG set the Linux bundles use.
# Generating it from the committed 1024px source keeps one icon of record
# instead of a hand-made .ico that could silently drift from the brand art.
# Linux doesn't need this step, which is why it lives here and not in `build`.
- name: Generate the Windows icon set
run: cargo tauri icon app-icon.png
working-directory: desktop/src-tauri
# --runner cargo-xwin swaps cargo for the cross-compiling driver (it supplies
# the MSVC CRT/SDK, pre-warmed into the image, and links with lld-link).
# Frontend already built above; skip the beforeBuildCommand rebuild.
- name: Tauri build (NSIS installer)
env:
TAURI_SIGNING_PRIVATE_KEY: ${{ secrets.TAURI_SIGNING_PRIVATE_KEY }}
TAURI_SIGNING_PRIVATE_KEY_PASSWORD: ${{ secrets.TAURI_SIGNING_PRIVATE_KEY_PASSWORD }}
run: |
version="$(sh ../packaging/build-version.sh)"
echo "Building version $version"
updater='{}'
if [ -n "${TAURI_SIGNING_PRIVATE_KEY:-}" ]; then
updater='{"bundle":{"createUpdaterArtifacts":true}}'
fi
cargo tauri build \
--runner cargo-xwin \
--target x86_64-pc-windows-msvc \
--bundles nsis \
--config '{"build":{"beforeBuildCommand":""}}' \
--config "{\"version\":\"$version\"}" \
--config "$updater"
working-directory: desktop/src-tauri
- name: Upload installer
continue-on-error: true
uses: actions/upload-artifact@v3
with:
name: thoughtsync-windows
path: desktop/src-tauri/target/x86_64-pc-windows-msvc/release/bundle/nsis/*.exe
if-no-files-found: warn
# Publishes to the SAME release as the Linux job. Safe to run twice: the
# script reuses an existing release (409) and nullglob means each job uploads
# only the bundles present in its own workspace.
- name: Publish release
if: startsWith(github.ref, 'refs/tags/v')
env:
GITHUB_TOKEN: ${{ github.token }}
run: bash desktop/packaging/publish-release.sh
# The rolling DEVELOPMENT channel (M10.9): a release whose tag never moves, so
# the updater has a permanent URL to read — Forgejo has no
# /releases/latest/download/<asset> route, so "newest" can't be named in a URL.
#
# Gated on the signing key INSIDE the script rather than with an `if:`, because
# the secrets context isn't reliably available to step conditions. Publishing
# bundles the app would then refuse to verify is worse than publishing nothing:
# it looks like a working feed.
- name: Publish to the dev channel
if: github.ref == 'refs/heads/dev'
env:
GITHUB_TOKEN: ${{ github.token }}
TAURI_SIGNING_PRIVATE_KEY: ${{ secrets.TAURI_SIGNING_PRIVATE_KEY }}
RELEASE_TAG: dev
RELEASE_PRERELEASE: "true"
run: |
if [ -z "${TAURI_SIGNING_PRIVATE_KEY:-}" ]; then
echo "No TAURI_SIGNING_PRIVATE_KEY — skipping the dev channel publish."
exit 0
fi
bash desktop/packaging/publish-release.sh
# The updater manifest, written AFTER both bundle jobs — they run in separate
# workspaces and neither can see the other's output, but one latest.json has to
# describe both platforms. Building it inside either job would silently omit the
# other, and a missing platform reads to a user as "no update available" rather
# than as a broken feed.
#
# Reads what actually landed on the channel release, so it can never advertise a
# bundle that failed to upload.
manifest:
name: Update manifest
needs: [build, windows]
if: github.ref == 'refs/heads/dev' || startsWith(github.ref, 'refs/tags/v')
runs-on: python-ci
container:
image: git.fabledsword.com/bvandeusen/ci-tauri:1.97
steps:
- uses: actions/checkout@v6
- name: Write and publish latest.json
env:
GITHUB_TOKEN: ${{ github.token }}
TAURI_SIGNING_PRIVATE_KEY: ${{ secrets.TAURI_SIGNING_PRIVATE_KEY }}
run: |
if [ -z "${TAURI_SIGNING_PRIVATE_KEY:-}" ]; then
echo "No TAURI_SIGNING_PRIVATE_KEY — nothing was signed, so there is no"
echo "manifest to write. Add the secret to enable in-app updates."
exit 0
fi
# The SAME helper the bundles were built with — a second derivation here
# could drift, and a manifest whose version doesn't match the binary it
# points at is an updater that never settles.
version="$(sh desktop/packaging/build-version.sh)"
if [ "${GITHUB_REF_NAME}" = "dev" ]; then
export RELEASE_TAG=dev
export RELEASE_NOTES="Development build from ${GITHUB_SHA}"
APP_VERSION="$version" bash desktop/packaging/write-manifest.sh
else
export RELEASE_TAG="${GITHUB_REF_NAME}"
export RELEASE_NOTES="ThoughtSync ${GITHUB_REF_NAME}"
# Twice: once onto the versioned release itself, and once onto the
# permanent `stable` pointer the app actually reads. Same manifest both
# times — its URLs point at the versioned assets either way.
APP_VERSION="$version" bash desktop/packaging/write-manifest.sh
APP_VERSION="$version" MANIFEST_TAG=stable bash desktop/packaging/write-manifest.sh
fi
+67
View File
@@ -78,9 +78,76 @@ backend/frontend push.
`pacman -Qkk` file verification needs `.MTREE`. Adding `libarchive-tools` +
`zstd` + a docker CLI to `ci-tauri` would upgrade these paths; none of them
block a green build.
- **`libssl-dev` + `pkg-config` are load-bearing** (both already in `ci-tauri`).
Since M10.6 the desktop crate depends on `reqwest` with the **`native-tls`**
backend, which on Linux compiles against OpenSSL. Do NOT drop either package
from `ci-tauri` in a future slim-down — the Rust build fails at `openssl-sys`.
(They're part of Tauri's own documented Linux prerequisites, so they should
stay regardless.)
- **`libssl3` is covered transitively, on purpose — don't "fix" it.** Since
M10.6 `dpkg-shlibdeps` lists `libssl3` among the binary's needs, but the
`.deb` declares only `libwebkit2gtk-4.1-0` + `libgtk-3-0`. `verify.sh` passes
it because webkit's own recursive dependency closure includes OpenSSL, so apt
installs it either way. Declaring it explicitly would be *worse*: the package
name is release-dependent (`libssl3` on bookworm, `libssl3t64` after the
64-bit-time_t transition in trixie/Ubuntu 24.04), so a hardcoded name freezes
the package to the build distro. Leaning on webkit's closure adapts. If webkit
ever stops pulling OpenSSL, `verify.sh` fails the build loudly — that guard is
what makes the indirection safe.
- **Not verifiable in CI:** the runner is Debian, so the pacman package cannot be
`pacman -U`-tested here. That step logs `.PKGINFO` + the full file listing so
the package is auditable from the run log; a real Arch install is the operator's
confirm.
### Windows lane — second job, second image
`desktop.yml` also runs a `windows` job that cross-compiles the NSIS installer.
- **Image:** `git.fabledsword.com/bvandeusen/ci-tauri-win:1.97` (Rust + Node +
`cargo-xwin` + LLVM/`lld` + NSIS). A separate image from `ci-tauri` per
CI-Runner's `docs/process.md` fork rule — the MSVC CRT/SDK cache alone is >1 GB.
Its pins are held in lockstep with `ci-tauri`; bump them together, since both
lanes compile the same source.
- **Why cross-compile:** there is no Windows build host, and a Windows container
cannot run on a Linux host (containers share the host kernel). `cargo-xwin`,
`lld-link` and `makensis` are Linux programs that emit Windows PE output.
- **NSIS only.** `.msi` requires WiX v3, a Windows program — per Tauri, "`.msi`
installers can only be created on Windows."
- **Separate job on purpose:** a Windows failure must not block the Linux
artifacts, which are the primary product today.
- **Weakest verification of any lane.** Tauri documents this path as "not as
straight forward as compiling on Windows directly and is not tested as much",
to be used "only as a last resort" — and a Linux runner cannot execute a
Windows binary. Green means it *built*. A real Windows machine check is
mandatory before trusting a release.
- **Unsigned.** Installers will trip SmartScreen until a code-signing
certificate exists; that is a purchasing decision, not a CI one.
- **TLS backend is chosen for this lane's sake.** The desktop crate pins
`reqwest` to `native-tls`, which on `x86_64-pc-windows-msvc` resolves to
`schannel` — pure-Rust bindings to the OS TLS stack. That keeps C/assembly out
of the cross-compile entirely. Switching to `rustls` would pull in
`ring`/`aws-lc-rs` and their assembler, which is exactly the class of
dependency that broke this lane before (`libsqlite3-sys``llvm-lib`). Treat
a TLS-backend change as a change to *this lane*, not just a dependency bump.
- No Postgres lane (unchanged): the desktop app's local store + sync behavior is
verified on the operator's machine, not in CI.
## Formatting the Rust lane before pushing
`cargo fmt --check` runs in CI and had failed on four consecutive desktop pushes
by itself, each costing a full cycle to learn a whitespace nit. There is no Rust
toolchain on the workstation (rule 10), but the CI image is pullable, and running
a formatter is neither a test run nor a local stack:
```
docker run --rm --user "$(id -u):$(id -g)" -e CARGO_HOME=/tmp/cargo \
-v "$PWD/desktop/src-tauri:/w" -w /w \
git.fabledsword.com/bvandeusen/ci-tauri:1.97 cargo fmt --check
```
Drop `--check` to apply. `--user` keeps the container from leaving root-owned
files behind; `CARGO_HOME` points somewhere writable for that user.
**Don't infer formatting from existing code.** Several lines in `local/store.rs`
exceed 100 characters and survive only because rustfmt cannot break a string
literal — copying that shape caused one of the four failures.
+32
View File
@@ -0,0 +1,32 @@
#!/usr/bin/env sh
#
# Echo the version this build should carry. One definition, used in three places
# (both bundle jobs and the manifest writer) — if they ever disagreed, the app would
# compare its own version against a manifest describing a different build, and the
# updater would either offer nothing or loop forever offering the same thing.
#
# WHY DEV BUILDS NEED THEIR OWN VERSION AT ALL:
# an updater decides by comparing semver. Every dev build carries the version in
# Cargo.toml, so without this they'd all be `0.1.0` — an installed build would see a
# manifest advertising the version it already has, conclude it was current, and never
# update. The rolling channel needs a number that actually rises.
#
# The CI run number is that number: monotonic, already unique per build, and it needs
# no state carried between runs. `0.1.0` + run 2932 becomes `0.1.2932`.
#
# Plain semver on purpose, NOT a `-dev.N` prerelease tag: prerelease versions sort
# BELOW the release they qualify (`0.1.0-dev.5` < `0.1.0`), so a tagged build would
# never update to a newer dev one, and Windows installer metadata wants a numeric
# X.Y.Z anyway. Bumping the minor in Cargo.toml still wins over any dev build on the
# old line, which is the ordering you want: 0.2.0 > 0.1.2932.
set -eu
CARGO_TOML="$(dirname "$0")/../src-tauri/Cargo.toml"
base="$(grep -m1 '^version' "$CARGO_TOML" | sed -E 's/.*"([^"]+)".*/\1/')"
# Dev builds only. Anything else (a v* tag, main) ships the version as written.
if [ "${GITHUB_REF_NAME:-}" = "dev" ] && [ -n "${GITHUB_RUN_NUMBER:-}" ]; then
printf '%s.%s\n' "${base%.*}" "$GITHUB_RUN_NUMBER"
else
printf '%s\n' "$base"
fi
+5 -1
View File
@@ -2,7 +2,11 @@
#
# ThoughtSync desktop — one-command Linux installer.
#
# curl -fsSL https://git.fabledsword.com/bvandeusen/thoughtsync/raw/branch/main/desktop/packaging/install.sh | sh
# curl -fsSL https://git.fabledsword.com/bvandeusen/thoughtsync/raw/branch/dev/desktop/packaging/install.sh | sh
#
# Points at `dev` because that is currently the repo's only branch — `main` does
# not exist yet, so a main URL 404s. Move this to `main` once that branch is
# created, so the public install command stops tracking day-to-day work.
#
# Fetches the LATEST published release for this machine's architecture and
# installs it, ending with a working app + menu entry. Native-first:
+24 -3
View File
@@ -17,6 +17,7 @@
# desktop/src-tauri/target/release/bundle/appimage/*.AppImage (de-bundled)
# desktop/src-tauri/target/release/bundle/deb/*.deb
# desktop/src-tauri/target/release/bundle/arch/*.pkg.tar.* (prebuilt pacman)
# desktop/src-tauri/target/x86_64-pc-windows-msvc/release/bundle/nsis/*.exe
#
# Instance-agnostic: server + repo come from the runner's github.* context
# (Forgejo populates them for compatibility), so nothing is hardcoded to one host.
@@ -30,20 +31,40 @@ set -euo pipefail
: "${GITHUB_REPOSITORY:?GITHUB_REPOSITORY is required (owner/repo)}"
: "${GITHUB_REF_NAME:?GITHUB_REF_NAME is required (the tag, e.g. v0.1.0)}"
# The release to publish to. Defaults to the pushed tag (the versioned, stable
# case). M10.9 also calls this with RELEASE_TAG=dev to maintain the rolling
# development channel — a release whose tag never moves, because Forgejo has no
# `/releases/latest/download/<asset>` route for an updater to point at.
RELEASE_TAG="${RELEASE_TAG:-$GITHUB_REF_NAME}"
RELEASE_PRERELEASE="${RELEASE_PRERELEASE:-false}"
API="$GITHUB_SERVER_URL/api/v1/repos/$GITHUB_REPOSITORY"
TAG="$GITHUB_REF_NAME"
TAG="$RELEASE_TAG"
AUTH=(-H "Authorization: token $GITHUB_TOKEN")
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
REPO_ROOT="$(cd "$SCRIPT_DIR/../.." && pwd)"
BUNDLE_ROOT="$REPO_ROOT/desktop/src-tauri/target/release/bundle"
# Cross-compiled Windows output lands under the target triple, not the host root.
WIN_BUNDLE_ROOT="$REPO_ROOT/desktop/src-tauri/target/x86_64-pc-windows-msvc/release/bundle"
# --- collect the assets to upload -------------------------------------------
shopt -s nullglob
# nullglob (set above) drops the patterns that didn't match, which is what lets the
# Linux job and the Windows job each run this script against the SAME release and
# upload only what they actually built — they run in separate workspaces, so neither
# can see the other's bundles. The release is created once and reused (409 path).
# The `.sig` files are the updater's whole trust story — a bundle published without
# its signature is one the app will refuse, so they ship together or not at all.
# They only exist when the build ran with a signing key (M10.9); nullglob drops
# them silently otherwise, which is the correct behaviour for an unsigned build.
ASSETS=(
"$BUNDLE_ROOT"/appimage/*.AppImage
"$BUNDLE_ROOT"/appimage/*.AppImage.sig
"$BUNDLE_ROOT"/deb/*.deb
"$BUNDLE_ROOT"/arch/*.pkg.tar.*
"$WIN_BUNDLE_ROOT"/nsis/*.exe
"$WIN_BUNDLE_ROOT"/nsis/*.exe.sig
)
if [ ${#ASSETS[@]} -eq 0 ]; then
echo "ERROR: no bundles under $BUNDLE_ROOT — did the tauri build run?" >&2
@@ -75,8 +96,8 @@ first_id() { grep -oE '"id"[[:space:]]*:[[:space:]]*[0-9]+' | head -1 | grep -oE
# --- create (or reuse) the release for the tag ------------------------------
echo "==> Creating release for $TAG"
BODY=$(cat <<JSON
{"tag_name":"$TAG","name":"ThoughtSync $TAG","draft":false,"prerelease":false,
"body":"ThoughtSync desktop $TAG.\n\n- **Debian / Ubuntu** — native \`.deb\`\n- **Arch / CachyOS** — native \`.pkg.tar.*\`\n- **everything else** — \`.AppImage\` (de-bundled graphics: renders on any GPU/Wayland setup)\n\nInstall / update — picks the right one for your system:\n\`\`\`\ncurl -fsSL $GITHUB_SERVER_URL/$GITHUB_REPOSITORY/raw/branch/main/desktop/packaging/install.sh | sh\n\`\`\`"}
{"tag_name":"$TAG","name":"ThoughtSync $TAG","draft":false,"prerelease":$RELEASE_PRERELEASE,
"body":"ThoughtSync desktop $TAG.\n\n- **Debian / Ubuntu** — native \`.deb\`\n- **Arch / CachyOS** — native \`.pkg.tar.*\`\n- **everything else** — \`.AppImage\` (de-bundled graphics: renders on any GPU/Wayland setup)\n\nInstall / update — picks the right one for your system:\n\`\`\`\ncurl -fsSL $GITHUB_SERVER_URL/$GITHUB_REPOSITORY/raw/branch/dev/desktop/packaging/install.sh | sh\n\`\`\`"}
JSON
)
# 409 = a release for this tag already exists (re-run) — fall through to lookup.
+142
View File
@@ -0,0 +1,142 @@
#!/usr/bin/env bash
#
# Write the updater manifest (`latest.json`) for one channel and attach it to that
# channel's release.
#
# WHY A SEPARATE STEP: the Linux and Windows bundles are built by two jobs in two
# workspaces, and neither can see the other's output — but ONE manifest has to
# describe both platforms. So this runs after both, reads what actually landed on
# the release, and writes the manifest from that. Building it inside either job
# would produce a manifest that silently omits the other platform, and a missing
# platform reads to a user as "no update available" rather than as a broken feed.
#
# WHAT IT READS: the release's own asset list. The signature for each bundle is a
# `.sig` asset published beside it (see publish-release.sh); its CONTENT is what
# goes in the manifest, which is why each one is downloaded rather than linked.
#
# Tauri's expected shape:
# { "version": "0.1.0", "pub_date": "...", "notes": "...",
# "platforms": { "<target>-<arch>": { "signature": "...", "url": "..." } } }
set -euo pipefail
: "${GITHUB_TOKEN:?GITHUB_TOKEN is required}"
: "${GITHUB_SERVER_URL:?GITHUB_SERVER_URL is required}"
: "${GITHUB_REPOSITORY:?GITHUB_REPOSITORY is required (owner/repo)}"
: "${RELEASE_TAG:?RELEASE_TAG is required (the release holding the bundles)}"
: "${APP_VERSION:?APP_VERSION is required (the version the bundles carry)}"
# Where the manifest is PUBLISHED, which need not be where the bundles live.
#
# That split is what makes the stable channel work at all. A versioned release
# (`v0.2.0`) holds the real assets, but the app can only read a URL that never
# changes — so the same manifest is also attached to a `stable` release whose tag is
# permanent and whose only content is this file. It points back at the versioned
# assets, so nothing is duplicated.
MANIFEST_TAG="${MANIFEST_TAG:-$RELEASE_TAG}"
API="$GITHUB_SERVER_URL/api/v1/repos/$GITHUB_REPOSITORY"
AUTH=(-H "Authorization: token $GITHUB_TOKEN")
NOTES="${RELEASE_NOTES:-}"
work="$(mktemp -d)"
trap 'rm -rf "$work"' EXIT INT TERM
echo "==> Reading assets on release $RELEASE_TAG"
release="$(curl -sS "${AUTH[@]}" "$API/releases/tags/$RELEASE_TAG")"
release_id="$(printf '%s' "$release" | grep -oE '"id"[[:space:]]*:[[:space:]]*[0-9]+' | head -1 | grep -oE '[0-9]+')"
[ -n "$release_id" ] || { echo "ERROR: no release tagged $RELEASE_TAG" >&2; exit 1; }
assets="$(curl -sS "${AUTH[@]}" "$API/releases/$release_id/assets")"
# Asset names, one per line. The API returns them in a single JSON blob; this is
# the only field needed, and grep beats adding a jq dependency to the CI image.
names="$(printf '%s' "$assets" | grep -oE '"name"[[:space:]]*:[[:space:]]*"[^"]+"' | sed -E 's/.*"([^"]+)"$/\1/')"
download_url() { printf '%s/%s/releases/download/%s/%s' "$GITHUB_SERVER_URL" "$GITHUB_REPOSITORY" "$RELEASE_TAG" "$1"; }
# One platform entry, or nothing if that platform's bundle or signature is absent.
# Emitting a partial entry would be worse than emitting none: the app would try to
# install something it can't verify.
platform_entry() {
local target="$1" pattern="$2" bundle sig_name
bundle="$(printf '%s\n' "$names" | grep -E "$pattern" | head -1 || true)"
[ -n "$bundle" ] || { echo " no bundle matching $pattern — skipping $target" >&2; return; }
sig_name="$bundle.sig"
if ! printf '%s\n' "$names" | grep -qxF "$sig_name"; then
echo " $bundle has no $sig_name — skipping $target (was the build signed?)" >&2
return
fi
curl -fsSL "${AUTH[@]}" -o "$work/sig" "$(download_url "$sig_name")"
# The signature is base64 on one line already; strip any stray newline so it
# can't break the JSON string it's about to become.
local signature
signature="$(tr -d '\r\n' < "$work/sig")"
printf ' "%s": { "signature": "%s", "url": "%s" }' "$target" "$signature" "$(download_url "$bundle")"
}
echo "==> Building the manifest"
entries=()
# `.AppImage` only on Linux: the updater replaces the running bundle in place, which
# a package-manager install (deb/pacman) must never have done to it.
if entry="$(platform_entry "linux-x86_64" '\.AppImage$')" && [ -n "$entry" ]; then entries+=("$entry"); fi
if entry="$(platform_entry "windows-x86_64" '\.exe$')" && [ -n "$entry" ]; then entries+=("$entry"); fi
if [ ${#entries[@]} -eq 0 ]; then
echo "ERROR: no signed bundle on $RELEASE_TAG — refusing to publish an empty manifest." >&2
echo " (An empty manifest would tell every client it is up to date.)" >&2
exit 1
fi
# No `date -u -Is` — busybox date in the CI image doesn't take it.
pub_date="$(date -u '+%Y-%m-%dT%H:%M:%SZ')"
{
printf '{\n'
printf ' "version": "%s",\n' "$APP_VERSION"
printf ' "pub_date": "%s",\n' "$pub_date"
printf ' "notes": "%s",\n' "$NOTES"
printf ' "platforms": {\n'
for i in "${!entries[@]}"; do
[ "$i" -eq 0 ] || printf ',\n'
printf '%s' "${entries[$i]}"
done
printf '\n }\n'
printf '}\n'
} > "$work/latest.json"
echo "==> Manifest:"
cat "$work/latest.json"
# --- resolve the release the manifest is published TO ------------------------
if [ "$MANIFEST_TAG" = "$RELEASE_TAG" ]; then
target_id="$release_id"
target_assets="$assets"
else
echo "==> Resolving the $MANIFEST_TAG channel release"
target="$(curl -sS "${AUTH[@]}" "$API/releases/tags/$MANIFEST_TAG")"
target_id="$(printf '%s' "$target" | grep -oE '"id"[[:space:]]*:[[:space:]]*[0-9]+' | head -1 | grep -oE '[0-9]+' || true)"
if [ -z "${target_id:-}" ]; then
# First publish to this channel. A pointer release: no bundles of its own, just
# a permanent tag for the manifest to live under.
echo " creating it (pointer release, manifest only)"
body="{\"tag_name\":\"$MANIFEST_TAG\",\"name\":\"ThoughtSync ($MANIFEST_TAG channel)\",\"draft\":false,\"prerelease\":false,\"body\":\"Update channel pointer. The installable builds live on the versioned releases; this holds only the updater manifest.\"}"
target="$(curl -sS -X POST "${AUTH[@]}" -H "Content-Type: application/json" -d "$body" "$API/releases")"
target_id="$(printf '%s' "$target" | grep -oE '"id"[[:space:]]*:[[:space:]]*[0-9]+' | head -1 | grep -oE '[0-9]+')"
fi
[ -n "${target_id:-}" ] || { echo "ERROR: could not resolve the $MANIFEST_TAG release" >&2; exit 1; }
target_assets="$(curl -sS "${AUTH[@]}" "$API/releases/$target_id/assets")"
fi
# Replace rather than duplicate: Forgejo rejects a second asset with the same name,
# and this file is rewritten on every publish by design.
old_id="$(printf '%s' "$target_assets" \
| grep -oE "\"id\"[[:space:]]*:[[:space:]]*[0-9]+[^}]*\"name\"[[:space:]]*:[[:space:]]*\"latest\.json\"" \
| head -1 | grep -oE '[0-9]+' | head -1 || true)"
if [ -n "${old_id:-}" ]; then
echo "==> Removing the previous latest.json (id $old_id)"
curl -fsS -X DELETE "${AUTH[@]}" "$API/releases/$target_id/assets/$old_id" >/dev/null
fi
echo "==> Uploading latest.json to $MANIFEST_TAG"
curl -fsS -X POST "${AUTH[@]}" "$API/releases/$target_id/assets?name=latest.json" \
-F "attachment=@$work/latest.json" >/dev/null
echo "==> Done. $MANIFEST_TAG now advertises $APP_VERSION for ${#entries[@]} platform(s)."
+17
View File
@@ -28,6 +28,23 @@ chrono = { version = "0.4", default-features = false, features = ["clock"] }
# issues are diagnosable from any environment. `log` is the facade the code uses.
tauri-plugin-log = "2"
log = "0.4"
# In-app updates (M10.9). Signature verification is minisign; the public half lives
# in tauri.conf.json and the private half only ever as a CI secret.
tauri-plugin-updater = "2"
# HTTP for the opt-in server handshake (M10.6) and, next, the sync engine (M10.7).
#
# native-tls, NOT rustls, deliberately: on x86_64-pc-windows-msvc native-tls
# resolves to `schannel` — pure-Rust bindings to the OS TLS stack — so nothing C or
# assembly has to cross-compile on the Windows lane, which is the fragile one (it
# builds on Linux via cargo-xwin, and a C dependency there is what broke it before).
# rustls would instead pull in ring/aws-lc-rs and their assembler. On Linux
# native-tls uses OpenSSL, whose headers (libssl-dev) ci-tauri already ships.
# default-features off drops http2/charset we don't need for a JSON API.
reqwest = { version = "0.12", default-features = false, features = ["json", "native-tls"] }
# Verifying downloaded attachment bytes against the sha256 the server advertised.
# Pure Rust (no C/asm beyond optional cpufeatures), so it costs the Windows
# cross-compile lane nothing — see ci-requirements.md on why that matters here.
sha2 = "0.10"
# Tauri's default release profile: smaller, faster shipped binaries.
[profile.release]
+70
View File
@@ -8,6 +8,11 @@
mod integration;
mod local;
// `pub` (unlike the modules above) because parts of it have no in-crate caller yet —
// the engine that will consume them is M10.7b/c, and a private module's unreachable
// items read as dead code.
pub mod sync;
mod update;
#[cfg_attr(mobile, tauri::mobile_entry_point)]
pub fn run() {
@@ -29,6 +34,32 @@ pub fn run() {
])
.build(),
)
// In-app updates (M10.9). Registering the plugin is inert on its own — it
// reads its config only when `update_check`/`update_install` ask it to, so a
// build without a signing key still starts normally and simply reports that
// updates aren't configured.
.plugin(tauri_plugin_updater::Builder::new().build())
// Attachment bytes are served to the webview from the local blob store
// (M10.7f). Registered on the BUILDER because a scheme has to exist before
// the webview is created; the directory it reads from arrives later, in
// `setup`, via `blobs::publish_root`.
.register_uri_scheme_protocol(sync::blobs::BLOB_SCHEME, |_ctx, request| {
let (status, content_type, body) =
sync::blobs::serve(request.uri().path(), request.uri().query());
tauri::http::Response::builder()
.status(status)
.header("Content-Type", content_type)
// The bytes are content-addressed: a given URL can never describe
// different bytes, so the webview may keep them indefinitely.
.header("Cache-Control", "public, max-age=31536000, immutable")
.body(body)
.unwrap_or_else(|_| {
tauri::http::Response::builder()
.status(500)
.body(Vec::new())
.expect("a bodiless 500 always builds")
})
})
.setup(|app| {
use tauri::Manager;
log_environment(app);
@@ -41,7 +72,16 @@ pub fn run() {
log::info!("opening local store: {}", db_path.display());
let db = local::open(&db_path)?;
log::info!("local store ready — {}", local::summary(&db));
sweep_local_trash(&db);
app.manage(db);
// Attachment bytes live beside the database, filed by content hash, so a
// synced image is readable with no network (M10.7d).
let blobs = sync::blobs::BlobStore::new(dir.join("blobs"))?;
log::info!("attachment store ready: {}", blobs.root().display());
// Hand the directory to the URI-scheme handler registered below, which
// was built before this path could be resolved.
sync::blobs::publish_root(blobs.root().to_path_buf());
app.manage(blobs);
Ok(())
})
.invoke_handler(tauri::generate_handler![
@@ -85,11 +125,41 @@ pub fn run() {
local::commands::saved_filters_create,
local::commands::saved_filters_remove,
local::commands::saved_filters_rename,
sync::commands::sync_probe,
sync::commands::sync_link,
sync::commands::sync_unlink,
sync::commands::sync_status,
sync::commands::sync_now,
sync::commands::sync_has_pending,
update::update_channel_get,
update::update_channel_set,
update::update_check,
update::update_install,
])
.run(tauri::generate_context!())
.expect("error while running the ThoughtSync desktop app");
}
/// Expire old trash at startup, on an unlinked device only (see `local::retention`).
///
/// At startup rather than on a timer: a desktop app isn't a server, and a sweep the
/// user is present for is one they can see the result of. A failure here is logged and
/// stepped over — housekeeping must never be the reason the app won't open.
fn sweep_local_trash(db: &local::Db) {
let conn = match db.0.lock() {
Ok(conn) => conn,
Err(_) => {
log::warn!("skipping the trash sweep: store lock poisoned");
return;
}
};
match local::retention::sweep_if_unlinked(&conn) {
Ok(Some(0)) | Ok(None) => {}
Ok(Some(n)) => log::info!("trash retention: purged {n} expired note(s)"),
Err(e) => log::warn!("trash sweep failed: {e}"),
}
}
/// Frontend logging bridge: routes boot milestones and errors from the webview into
/// the same stdout + file log as the Rust side (see frontend/src/desktop/bridge.ts).
#[tauri::command]
+14 -1
View File
@@ -7,20 +7,33 @@ use serde_json::Value;
use tauri::State;
use crate::local::models::*;
use crate::local::retention;
use crate::local::store;
use crate::local::Db;
use crate::sync::state;
// A macro would hide the (very regular) locking; kept explicit so each command reads
// as an obvious lock -> delegate -> stringify.
#[tauri::command]
pub fn config_get() -> PublicConfig {
pub fn config_get(db: State<'_, Db>) -> PublicConfig {
// What the Trash view counts down against: the linked server's window if we know
// it, else this device's own. Reading it here rather than hard-coding the offline
// default is what keeps the deadline on screen equal to the one that will actually
// be enforced. A store error falls back to the default rather than failing the
// call — the app must still boot.
let fallback = retention::LOCAL_RETENTION_DAYS;
let retention_days = match db.0.lock() {
Ok(conn) => state::effective_retention_days(&conn, fallback).unwrap_or(fallback),
Err(_) => fallback,
};
// Offline defaults: no signups, no server-side URL unfurling (needs network).
PublicConfig {
site_name: "ThoughtSync".to_string(),
allow_registration: false,
version: env!("CARGO_PKG_VERSION").to_string(),
enable_url_unfurl: false,
trash_retention_days: retention_days.max(0) as u32,
}
}
+1
View File
@@ -5,6 +5,7 @@
pub mod commands;
pub mod derive;
pub mod models;
pub mod retention;
pub mod schema;
pub mod store;
+4
View File
@@ -19,6 +19,9 @@ pub struct Note {
pub pinned: bool,
pub archived: bool,
pub trashed: bool,
/// When it was trashed (null unless trashed). Named for the server's field so the
/// shared frontend counts down the retention window identically either way.
pub deleted_at: Option<String>,
pub remind_at: Option<String>,
pub recurrence: Option<String>,
pub labels: Vec<NoteLabel>,
@@ -112,6 +115,7 @@ pub struct PublicConfig {
pub allow_registration: bool,
pub version: String,
pub enable_url_unfurl: bool,
pub trash_retention_days: u32,
}
/// The synthetic single user the offline core reports, so the app's auth-gated
+228
View File
@@ -0,0 +1,228 @@
//! Trash retention for a device with no server (M11.3).
//!
//! The server owns this policy whenever there IS one: a linked client learns about
//! every permanent deletion from the delta feed, as a tombstone, and does exactly
//! what it's told. This module exists for the case the server can't cover — an
//! offline-only install, where trash would otherwise sit forever and the attachment
//! bytes with it.
//!
//! Which is why the sweep refuses to run while linked. If it didn't, a device could
//! decide on its own that a note had expired, destroy it, and then push that delete
//! upstream — overruling a server that was deliberately keeping it (retention off, or
//! a longer window than this constant). A client's local policy must never outrank
//! the server's.
use chrono::{DateTime, Duration, Utc};
use rusqlite::Connection;
use super::store;
use crate::sync::state;
/// The window an unlinked device uses. Matches the server's default so a device that
/// later links doesn't see its trash behave differently from one that always was.
pub const LOCAL_RETENTION_DAYS: i64 = 30;
/// Purge trash older than `retention_days`. Returns how many notes went.
///
/// `now` is a parameter so the window arithmetic is testable without waiting a month.
pub fn sweep_expired_trash(
conn: &Connection,
retention_days: i64,
now: DateTime<Utc>,
) -> rusqlite::Result<usize> {
if retention_days <= 0 {
return Ok(0);
}
let cutoff = now - Duration::days(retention_days);
let mut expired: Vec<String> = Vec::new();
{
let mut stmt = conn.prepare(
"SELECT id, trashed_at FROM notes WHERE trashed = 1 AND trashed_at IS NOT NULL",
)?;
let mut rows = stmt.query([])?;
while let Some(row) = rows.next()? {
let id: String = row.get(0)?;
let stamped: String = row.get(1)?;
// PARSED, not string-compared. The server writes `+00:00` offsets and this
// client writes `Z`, so two timestamps for the same instant don't sort
// against each other as text — and the failure would be silent.
//
// An unparseable stamp means "age unknown", and the only safe reading of
// that is to keep the note. Deleting on a guess is the one outcome nobody
// can undo.
let Ok(trashed_at) = DateTime::parse_from_rfc3339(&stamped) else {
continue;
};
if trashed_at.with_timezone(&Utc) < cutoff {
expired.push(id);
}
}
}
for id in &expired {
// Through delete_forever, so a `pending_deletes` tombstone is recorded. That's
// right even here: while unlinked this device holds the only copy, so if it
// links later the server should learn the note was deleted, not re-send it.
store::delete_forever(conn, id)?;
}
Ok(expired.len())
}
/// The startup sweep: runs only on an unlinked device (see the module note).
/// Returns `None` when it didn't run because the device is linked.
pub fn sweep_if_unlinked(conn: &Connection) -> rusqlite::Result<Option<usize>> {
if state::read(conn)?.server_url.is_some() {
return Ok(None);
}
sweep_expired_trash(conn, LOCAL_RETENTION_DAYS, Utc::now()).map(Some)
}
#[cfg(test)]
mod tests {
use super::*;
use crate::local::schema;
fn db() -> Connection {
let conn = Connection::open_in_memory().expect("in-memory db");
schema::migrate(&conn).expect("migrate");
conn
}
/// A trashed note of a given age, stamped in the format the CLIENT writes
/// (`...Z`, millisecond precision — see `store::now`).
fn trashed_note_aged(conn: &Connection, id: &str, age: Duration) {
let when = Utc::now() - age;
let stamped = when.to_rfc3339_opts(chrono::SecondsFormat::Millis, true);
conn.execute(
"INSERT INTO notes (id, title, body, created_at, updated_at, trashed, trashed_at)
VALUES (?1, 'T', 'B', ?2, ?2, 1, ?2)",
rusqlite::params![id, stamped],
)
.expect("insert");
}
fn trashed_note(conn: &Connection, id: &str, days_ago: i64) {
trashed_note_aged(conn, id, Duration::days(days_ago));
}
fn sweep(conn: &Connection, days: i64) -> usize {
sweep_expired_trash(conn, days, Utc::now()).expect("sweep")
}
fn note_count(conn: &Connection) -> i64 {
conn.query_row("SELECT COUNT(*) FROM notes", [], |r| r.get(0))
.expect("count")
}
#[test]
fn purges_trash_past_the_window_and_keeps_the_rest() {
let conn = db();
trashed_note(&conn, "old", 40);
trashed_note(&conn, "fresh", 3);
let purged = sweep(&conn, 30);
assert_eq!(purged, 1);
assert_eq!(note_count(&conn), 1, "only the expired note should go");
}
#[test]
fn a_note_just_inside_the_window_survives() {
// The comparison is STRICTLY older than the cutoff, so a note with a minute
// of its 30 days still to run is kept. An exact tie isn't testable against a
// wall clock — the sweep reads `now` microseconds after the row is stamped,
// which is precisely how the first version of this test failed.
let conn = db();
let almost = Duration::days(30) - Duration::minutes(1);
trashed_note_aged(&conn, "boundary", almost);
assert_eq!(sweep(&conn, 30), 0);
assert_eq!(note_count(&conn), 1);
}
#[test]
fn retention_off_purges_nothing() {
let conn = db();
trashed_note(&conn, "ancient", 4000);
assert_eq!(sweep(&conn, 0), 0);
assert_eq!(sweep(&conn, -1), 0);
assert_eq!(note_count(&conn), 1);
}
#[test]
fn an_untrashed_note_is_never_swept() {
let conn = db();
conn.execute(
"INSERT INTO notes (id, title, body, created_at, updated_at, trashed)
VALUES ('live', 'T', 'B', '2020-01-01T00:00:00.000Z', '2020-01-01T00:00:00.000Z', 0)",
[],
)
.expect("insert");
assert_eq!(sweep(&conn, 30), 0);
assert_eq!(note_count(&conn), 1);
}
#[test]
fn an_unparseable_timestamp_keeps_the_note() {
// "Age unknown" must never resolve to "delete it".
let conn = db();
conn.execute(
"INSERT INTO notes (id, title, body, created_at, updated_at, trashed, trashed_at)
VALUES ('weird', 'T', 'B', '2020-01-01T00:00:00.000Z', '2020-01-01T00:00:00.000Z', 1, 'not a date')",
[],
)
.expect("insert");
assert_eq!(sweep(&conn, 30), 0);
assert_eq!(note_count(&conn), 1);
}
#[test]
fn a_server_style_offset_timestamp_is_understood() {
// The server serializes with a `+00:00` offset, not `Z`. Comparing those as
// strings would quietly never match — this is the case that catches it.
let conn = db();
let stamped = (Utc::now() - Duration::days(40)).to_rfc3339();
conn.execute(
"INSERT INTO notes (id, title, body, created_at, updated_at, trashed, trashed_at)
VALUES ('server', 'T', 'B', ?1, ?1, 1, ?1)",
rusqlite::params![stamped],
)
.expect("insert");
assert_eq!(sweep(&conn, 30), 1);
}
#[test]
fn a_purged_note_leaves_a_pending_delete_behind() {
// Without the tombstone, linking this device later would let the server
// re-send a note the user already destroyed here.
let conn = db();
trashed_note(&conn, "old", 40);
sweep(&conn, 30);
let pending: i64 = conn
.query_row(
"SELECT COUNT(*) FROM pending_deletes WHERE entity = 'note' AND id = 'old'",
[],
|r| r.get(0),
)
.expect("count");
assert_eq!(pending, 1);
}
#[test]
fn a_linked_device_does_not_sweep() {
// The whole safety rule: with a server present, purging is the server's call.
let conn = db();
trashed_note(&conn, "old", 400);
state::set_link(&conn, "https://notes.example", "token").expect("link");
assert_eq!(sweep_if_unlinked(&conn).expect("sweep"), None);
assert_eq!(
note_count(&conn),
1,
"the note must survive on a linked device"
);
}
#[test]
fn an_unlinked_device_sweeps() {
let conn = db();
trashed_note(&conn, "old", 400);
assert_eq!(sweep_if_unlinked(&conn).expect("sweep"), Some(1));
assert_eq!(note_count(&conn), 0);
}
}
+70
View File
@@ -105,6 +105,60 @@ CREATE TABLE sync_state (
INSERT INTO sync_state (id) VALUES (1);
"#;
// v2 (M10.7c): local tombstones.
//
// A permanent delete previously just dropped the row, which left NO record that it
// ever existed. Offline, that means the delete can never be pushed — and the next
// pull would faithfully resurrect the note from the server. A deletion that undoes
// itself is about the worst outcome sync can produce, so deletes are now recorded
// here until they've been acknowledged by the server and cleared.
const SCHEMA_V2: &str = r#"
CREATE TABLE pending_deletes (
entity TEXT NOT NULL, -- 'note' | 'label'
id TEXT NOT NULL,
deleted_at TEXT NOT NULL,
PRIMARY KEY (entity, id)
);
"#;
// v3 (M10.7e): when the last successful sync finished.
//
// The cursor alone can't answer "is this up to date?" — it's a revision watermark,
// not a time, and it doesn't move at all when a sync legitimately finds nothing new.
// The UI needs a timestamp to say anything honest.
const SCHEMA_V3: &str = r#"
ALTER TABLE sync_state ADD COLUMN last_sync_at TEXT;
"#;
// v4 (M11.3): WHEN a note was trashed.
//
// The table only ever recorded THAT a note was trashed, which is enough to draw a
// Trash view and nothing else. Retention needs an age: without a timestamp there is
// no way to tell a note trashed this morning from one trashed last spring, so an
// offline device could never expire its own trash — and the UI couldn't warn anyone
// before it did.
// It also records the LINKED server's retention window, captured from /api/config.
// Once linked, the server's policy is the one that actually applies, so showing this
// device's offline default would put a countdown on screen that doesn't match what
// happens — a wrong deadline is worse than none.
const SCHEMA_V4: &str = r#"
ALTER TABLE notes ADD COLUMN trashed_at TEXT;
UPDATE notes SET trashed_at = updated_at WHERE trashed = 1;
ALTER TABLE sync_state ADD COLUMN server_retention_days INTEGER;
"#;
// v5 (M10.9): small key/value app preferences.
//
// The first entry is the update channel, which is neither note data nor part of the
// server link — so it belongs in neither `notes` nor `sync_state`. Generic on
// purpose: the next device-local preference shouldn't need another migration.
const SCHEMA_V5: &str = r#"
CREATE TABLE prefs (
key TEXT PRIMARY KEY,
value TEXT NOT NULL
);
"#;
/// Bring the database up to the latest schema. Idempotent.
pub fn migrate(conn: &Connection) -> rusqlite::Result<()> {
conn.execute_batch("PRAGMA foreign_keys = ON;")?;
@@ -113,5 +167,21 @@ pub fn migrate(conn: &Connection) -> rusqlite::Result<()> {
conn.execute_batch(SCHEMA_V1)?;
conn.execute_batch("PRAGMA user_version = 1;")?;
}
if version < 2 {
conn.execute_batch(SCHEMA_V2)?;
conn.execute_batch("PRAGMA user_version = 2;")?;
}
if version < 3 {
conn.execute_batch(SCHEMA_V3)?;
conn.execute_batch("PRAGMA user_version = 3;")?;
}
if version < 4 {
conn.execute_batch(SCHEMA_V4)?;
conn.execute_batch("PRAGMA user_version = 4;")?;
}
if version < 5 {
conn.execute_batch(SCHEMA_V5)?;
conn.execute_batch("PRAGMA user_version = 5;")?;
}
Ok(())
}
+78 -6
View File
@@ -92,13 +92,28 @@ fn load_attachments(conn: &Connection, note_id: &str) -> rusqlite::Result<Vec<At
"SELECT id, url, filename, mime, size, sha256 FROM attachments WHERE note_id = ?1 ORDER BY position ASC",
)?;
let rows = stmt.query_map([note_id], |r| {
let server_url: String = r.get(1)?;
let mime: String = r.get(3)?;
let sha256: Option<String> = r.get(5)?;
Ok(Attachment {
id: r.get(0)?,
url: r.get(1)?,
// Point at the LOCAL bytes, not the server's route. The stored url is the
// server's relative path, which resolves against the app origin in the
// webview and 404s — and even absolute it would need a bearer token the
// webview never sends. Rewriting here rather than at each render site
// means NoteCard and NoteEditor stay untouched and can't drift.
//
// Without a hash there's nothing to address the blob by (an older server
// that predates the sha256 column), so the original url is left alone:
// still broken, but no more broken than it already was.
url: match sha256.as_deref() {
Some(hash) if !hash.is_empty() => crate::sync::blobs::url_for(hash, &mime),
_ => server_url,
},
filename: r.get(2)?,
mime: r.get(3)?,
mime,
size: r.get(4)?,
sha256: r.get(5)?,
sha256,
})
})?;
rows.collect()
@@ -123,7 +138,7 @@ fn load_previews(conn: &Connection, note_id: &str) -> rusqlite::Result<Vec<LinkP
fn load_note(conn: &Connection, id: &str) -> rusqlite::Result<Note> {
let mut note = conn.query_row(
"SELECT id, title, body, color, kind, position, pinned, archived, trashed, remind_at, recurrence, created_at, updated_at
"SELECT id, title, body, color, kind, position, pinned, archived, trashed, remind_at, recurrence, created_at, updated_at, trashed_at
FROM notes WHERE id = ?1",
[id],
|r| {
@@ -141,6 +156,7 @@ fn load_note(conn: &Connection, id: &str) -> rusqlite::Result<Note> {
pinned: r.get(6)?,
archived: r.get(7)?,
trashed: r.get(8)?,
deleted_at: r.get(13)?,
remind_at: r.get(9)?,
recurrence: r.get(10)?,
labels: Vec::new(),
@@ -621,22 +637,67 @@ pub fn reorder(conn: &Connection, ordered_ids: &[String]) -> rusqlite::Result<()
}
pub fn trash(conn: &Connection, id: &str) -> rusqlite::Result<Note> {
conn.execute("UPDATE notes SET trashed = 1 WHERE id = ?1", [id])?;
// COALESCE, so trashing an already-trashed note doesn't restart its retention
// clock. The server keeps its `deleted_at` the same way — a note shouldn't earn
// another 30 days because something touched it twice.
conn.execute(
"UPDATE notes SET trashed = 1, trashed_at = COALESCE(trashed_at, ?1) WHERE id = ?2",
params![now(), id],
)?;
touch(conn, id)?;
load_note(conn, id)
}
pub fn restore(conn: &Connection, id: &str) -> rusqlite::Result<Note> {
conn.execute("UPDATE notes SET trashed = 0 WHERE id = ?1", [id])?;
conn.execute(
"UPDATE notes SET trashed = 0, trashed_at = NULL WHERE id = ?1",
[id],
)?;
touch(conn, id)?;
load_note(conn, id)
}
pub fn delete_forever(conn: &Connection, id: &str) -> rusqlite::Result<()> {
record_pending_delete(conn, "note", id)?;
conn.execute("DELETE FROM notes WHERE id = ?1", [id])?;
Ok(())
}
/// Remember that a row was permanently deleted, so the sync engine can tell the
/// server. Without this the deleted row leaves no trace at all, and the next pull
/// would resurrect it — a delete that quietly undoes itself.
///
/// Harmless when the app is unlinked: the row is simply never read, and a later push
/// gets a `noop` for an id the server never had.
pub fn record_pending_delete(conn: &Connection, entity: &str, id: &str) -> rusqlite::Result<()> {
conn.execute(
"INSERT OR REPLACE INTO pending_deletes (entity, id, deleted_at) VALUES (?1, ?2, ?3)",
params![entity, id, now()],
)?;
Ok(())
}
// ---- device-local preferences (schema v5) -----------------------------------
/// A stored preference, or `None` if it was never set. Callers supply their own
/// default rather than one being invented here — the meaning of "unset" belongs
/// with the setting, not with the storage.
pub fn pref(conn: &Connection, key: &str) -> rusqlite::Result<Option<String>> {
conn.query_row("SELECT value FROM prefs WHERE key = ?1", [key], |r| {
r.get(0)
})
.optional()
}
pub fn set_pref(conn: &Connection, key: &str, value: &str) -> rusqlite::Result<()> {
conn.execute(
"INSERT INTO prefs (key, value) VALUES (?1, ?2)
ON CONFLICT(key) DO UPDATE SET value = excluded.value",
params![key, value],
)?;
Ok(())
}
pub fn revisions(conn: &Connection, id: &str) -> rusqlite::Result<Vec<NoteRevision>> {
let mut stmt = conn
.prepare("SELECT id, title, body, created_at FROM note_revisions WHERE note_id = ?1 ORDER BY created_at DESC")?;
@@ -727,6 +788,7 @@ pub fn set_label_color(conn: &Connection, id: &str, color: &str) -> rusqlite::Re
}
pub fn remove_label(conn: &Connection, id: &str) -> rusqlite::Result<()> {
record_pending_delete(conn, "label", id)?;
conn.execute("DELETE FROM labels WHERE id = ?1", [id])?;
Ok(())
}
@@ -741,6 +803,16 @@ pub fn merge_labels(
SELECT note_id, ?2, 0 FROM note_labels WHERE label_id = ?1",
params![source_id, target_id],
)?;
// The notes that carried the source now have a different label set, and that set
// only reaches the server via the note itself (push sends label_ids per note).
// Without this the merge would look done locally and never sync. Marked BEFORE
// the delete, which cascades the membership rows away.
conn.execute(
"UPDATE notes SET dirty = 1
WHERE id IN (SELECT note_id FROM note_labels WHERE label_id = ?1)",
[source_id],
)?;
record_pending_delete(conn, "label", source_id)?;
conn.execute("DELETE FROM labels WHERE id = ?1", [source_id])?;
load_label(conn, target_id)
}
+318
View File
@@ -0,0 +1,318 @@
//! Local storage for attachment bytes (M10.7d).
//!
//! Content-addressed: a blob is filed under its own sha256, so the same image
//! attached to five notes is stored once and re-downloading it is free. The hash is
//! also the integrity check — bytes that don't hash to what the server advertised
//! are refused rather than filed under a name that lies about them.
//!
//! Attachment METADATA rides the delta feed; only the bytes come through here
//! (docs/sync.md).
use std::fs;
use std::path::{Path, PathBuf};
use std::sync::OnceLock;
/// A sha256 in lowercase hex, and nothing else.
///
/// This is a **path-safety** check, not a formatting nicety: the hash is taken
/// straight from a server response and used as a filename. Without it, a hostile or
/// buggy server could send `../../…` and steer a write outside the blob directory.
fn is_hash(candidate: &str) -> bool {
candidate.len() == 64 && candidate.bytes().all(|b| b.is_ascii_hexdigit())
}
fn digest(bytes: &[u8]) -> String {
use sha2::{Digest, Sha256};
let mut hasher = Sha256::new();
hasher.update(bytes);
hasher
.finalize()
.iter()
.map(|b| format!("{b:02x}"))
.collect()
}
pub struct BlobStore {
root: PathBuf,
}
impl BlobStore {
/// Open (creating if needed) the blob directory.
pub fn new(root: PathBuf) -> std::io::Result<Self> {
fs::create_dir_all(&root)?;
Ok(Self { root })
}
pub fn root(&self) -> &Path {
&self.root
}
/// Where a blob lives, or `None` if the hash isn't one.
pub fn path(&self, sha256: &str) -> Option<PathBuf> {
let lower = sha256.to_ascii_lowercase();
is_hash(&lower).then(|| self.root.join(lower))
}
/// Whether we already hold these bytes. Drives the "don't download it twice"
/// skip, which is the entire point of keying by content.
pub fn has(&self, sha256: &str) -> bool {
self.path(sha256).is_some_and(|p| p.is_file())
}
/// File bytes under `expected`, refusing them if they don't hash to it.
///
/// Verifying on the way IN rather than on the way out means a corrupted transfer
/// can never be served later as if it were genuine — and the next sync simply
/// tries again, because the blob still counts as missing.
pub fn store(&self, expected: &str, bytes: &[u8]) -> Result<PathBuf, String> {
let path = self
.path(expected)
.ok_or_else(|| format!("refusing an attachment with a malformed hash: {expected}"))?;
let actual = digest(bytes);
if actual != expected.to_ascii_lowercase() {
return Err(format!(
"attachment failed its integrity check (expected {expected}, got {actual})"
));
}
fs::write(&path, bytes).map_err(|e| format!("couldn't save an attachment: {e}"))?;
Ok(path)
}
pub fn read(&self, sha256: &str) -> Option<Vec<u8>> {
fs::read(self.path(sha256)?).ok()
}
}
// --- Serving blobs to the webview (M10.7f) -----------------------------------
//
// A synced note's attachment `url` is the SERVER's relative path
// (`/api/notes/<id>/attachments/<aid>`). In the desktop webview that resolves
// against the app origin and 404s, and swapping in the absolute server URL wouldn't
// help either — that route needs a bearer token the webview won't send, and it would
// make an offline app fetch over the network to show a file it already has on disk.
//
// So the bytes are served locally, over a custom URI scheme, straight out of this
// store. The webview then caches and range-requests them like any other resource,
// which a `data:` URI would have thrown away.
/// The scheme the webview fetches attachment bytes over.
pub const BLOB_SCHEME: &str = "tsblob";
/// The blob directory, published once the app has resolved its data dir.
///
/// A `OnceLock` rather than Tauri's managed state because the scheme handler is
/// registered on the BUILDER, before `setup` has computed that path — and because
/// reading it this way keeps the handler independent of which Tauri 2.x minor
/// changed the handler's context argument.
static SERVE_ROOT: OnceLock<PathBuf> = OnceLock::new();
pub fn publish_root(root: PathBuf) {
let _ = SERVE_ROOT.set(root);
}
/// The URL an `<img>`/`<audio>`/`<a href>` should point at for these bytes.
///
/// **The two forms are not interchangeable.** A custom scheme is reachable as
/// `scheme://localhost/<path>` on Linux and macOS, but Windows and Android map it
/// onto `http://scheme.localhost/<path>`. Getting this wrong breaks exactly one
/// platform, silently, and CI cannot catch it — the runner is headless.
pub fn url_for(sha256: &str, mime: &str) -> String {
let query = urlencode(mime);
if cfg!(any(windows, target_os = "android")) {
format!("http://{BLOB_SCHEME}.localhost/{sha256}?mime={query}")
} else {
format!("{BLOB_SCHEME}://localhost/{sha256}?mime={query}")
}
}
/// Percent-encode the few characters a mime type can contain that don't belong in a
/// query value. Hand-rolled rather than adding a dependency for `/` and `+`.
fn urlencode(value: &str) -> String {
let mut out = String::with_capacity(value.len() + 8);
for b in value.bytes() {
match b {
b'a'..=b'z' | b'A'..=b'Z' | b'0'..=b'9' | b'-' | b'.' | b'_' | b'~' => {
out.push(b as char)
}
_ => out.push_str(&format!("%{b:02X}")),
}
}
out
}
fn urldecode(value: &str) -> String {
let bytes = value.as_bytes();
let mut out: Vec<u8> = Vec::with_capacity(bytes.len());
let mut i = 0;
while i < bytes.len() {
if bytes[i] == b'%' && i + 2 < bytes.len() {
let hex = std::str::from_utf8(&bytes[i + 1..i + 3]).unwrap_or("");
if let Ok(byte) = u8::from_str_radix(hex, 16) {
out.push(byte);
i += 3;
continue;
}
}
out.push(bytes[i]);
i += 1;
}
String::from_utf8_lossy(&out).into_owned()
}
/// The Content-Type to serve for a claimed mime.
///
/// The mime rides in the URL and this scheme is an origin of its own, so echoing an
/// arbitrary type would let an attachment claiming `text/html` run as a document
/// there. Echoing is safe only because of the FAMILY check: nothing starting with
/// `image/` can name a scriptable type. Everything else is served as an opaque
/// download — the right treatment for an arbitrary file regardless.
fn content_type_for(mime: &str) -> String {
const RENDERABLE: &[&str] = &["image/", "audio/", "video/"];
let familiar = RENDERABLE.iter().any(|p| mime.starts_with(p)) || mime == "application/pdf";
// A header value can't carry control characters, and a mime type has no business
// being long — both would only arrive from a malformed or hostile feed.
let printable = mime.len() <= 100 && mime.bytes().all(|b| b.is_ascii_graphic());
if familiar && printable {
mime.to_string()
} else {
"application/octet-stream".to_string()
}
}
/// Serve one request from the blob store. `path` is the URI path, `query` its query.
pub fn serve(path: &str, query: Option<&str>) -> (u16, String, Vec<u8>) {
let requested = path.trim_start_matches('/');
let Some(root) = SERVE_ROOT.get() else {
// A request before the store was published — nothing to serve yet.
return (503, "text/plain".into(), Vec::new());
};
let store = BlobStore { root: root.clone() };
// `read` goes through `path`, which rejects anything that isn't a bare sha256 —
// so this handler inherits the traversal guard rather than re-implementing it.
let Some(bytes) = store.read(requested) else {
return (404, "text/plain".into(), Vec::new());
};
let claimed = query
.and_then(|q| q.split('&').find_map(|p| p.strip_prefix("mime=")))
.map(urldecode)
.unwrap_or_default();
(200, content_type_for(&claimed), bytes)
}
#[cfg(test)]
mod tests {
use super::*;
/// A blob store in a throwaway directory. No tempfile dependency for one test
/// fixture — the process id keeps concurrent runs apart.
fn store(tag: &str) -> BlobStore {
let dir = std::env::temp_dir().join(format!("ts-blobs-{}-{tag}", std::process::id()));
let _ = fs::remove_dir_all(&dir);
BlobStore::new(dir).expect("store")
}
/// sha256("hello") — a fixed vector, so a broken digest can't agree with itself.
const HELLO: &str = "2cf24dba5fb0a30e26e83b2ac5b9e29e1b161e5c1fa7425e73043362938b9824";
#[test]
fn digest_matches_a_known_vector() {
assert_eq!(digest(b"hello"), HELLO);
}
#[test]
fn stores_and_reads_back() {
let store = store("roundtrip");
assert!(!store.has(HELLO));
store.store(HELLO, b"hello").expect("store");
assert!(store.has(HELLO));
assert_eq!(store.read(HELLO).as_deref(), Some(&b"hello"[..]));
}
#[test]
fn refuses_bytes_that_dont_match_the_hash() {
// A corrupted or substituted transfer must never be filed under a name that
// claims it's genuine.
let store = store("mismatch");
let err = store.store(HELLO, b"goodbye").expect_err("must reject");
assert!(err.contains("integrity"), "got {err}");
assert!(!store.has(HELLO), "nothing should have been written");
}
#[test]
fn rejects_a_hash_that_could_escape_the_directory() {
// The hash arrives from a server response and becomes a filename.
let store = store("traversal");
assert!(store.path("../../etc/passwd").is_none());
assert!(store.store("../../etc/passwd", b"x").is_err());
assert!(store.path("").is_none());
assert!(store.path("nothex!!").is_none());
}
#[test]
fn accepts_an_uppercase_hash() {
// The wire format isn't guaranteed to be lowercase; the filename is.
let store = store("case");
store
.store(&HELLO.to_ascii_uppercase(), b"hello")
.expect("store");
assert!(store.has(HELLO), "should be found under the lowercase name");
}
#[test]
fn the_blob_url_carries_the_hash_and_the_mime() {
let url = url_for(HELLO, "image/png");
assert!(url.contains(HELLO), "the hash addresses the bytes: {url}");
assert!(url.contains("mime=image%2Fpng"), "mime encoded: {url}");
// The platform split is the whole risk of this feature, and CI is headless,
// so at least pin that the right branch was taken for THIS build.
if cfg!(any(windows, target_os = "android")) {
assert!(url.starts_with("http://tsblob.localhost/"), "{url}");
} else {
assert!(url.starts_with("tsblob://localhost/"), "{url}");
}
}
#[test]
fn url_encoding_round_trips_a_mime() {
assert_eq!(urldecode(&urlencode("image/svg+xml")), "image/svg+xml");
assert_eq!(urldecode(&urlencode("audio/mpeg")), "audio/mpeg");
// A malformed escape is left alone rather than eaten — the value still has to
// survive intact enough for `content_type_for` to reject it.
assert_eq!(urldecode("not-an-escape%ZZ"), "not-an-escape%ZZ");
}
#[test]
fn media_types_are_echoed_back() {
assert_eq!(content_type_for("image/png"), "image/png");
assert_eq!(content_type_for("audio/mpeg"), "audio/mpeg");
assert_eq!(content_type_for("application/pdf"), "application/pdf");
}
#[test]
fn a_scriptable_type_is_served_as_a_download() {
// This scheme is an origin of its own. An attachment claiming to be HTML
// must not be handed back as a document that can run there.
let opaque = "application/octet-stream";
assert_eq!(content_type_for("text/html"), opaque);
assert_eq!(content_type_for("application/javascript"), opaque);
assert_eq!(content_type_for(""), opaque);
// A control character can't reach a header value even under a safe family.
assert_eq!(content_type_for("image/png\r\nX-Evil: 1"), opaque);
}
#[test]
fn serving_refuses_a_path_that_isnt_a_hash() {
// Delegated to `path`, so the traversal guard is the same one `store` uses.
publish_root(std::env::temp_dir().join("ts-blobs-serve-guard"));
let (status, _, body) = serve("/../../etc/passwd", None);
assert_eq!(status, 404);
assert!(body.is_empty());
}
#[test]
fn missing_blob_reads_as_none() {
let store = store("missing");
assert!(store.read(HELLO).is_none());
assert!(!store.has(HELLO));
}
}
+353
View File
@@ -0,0 +1,353 @@
//! HTTP transport to a ThoughtSync server.
//!
//! Covers the compatibility handshake (M10.6) and device-token auth (M10.7a). The
//! engine that moves notes — push, pull, cursor — grows on top of the same client,
//! which is why the timeout, identity headers and error vocabulary live here rather
//! than inline at each call site.
//!
//! Nothing here runs unless the user has linked a server; the app is local-first and
//! fully usable with no network at all.
use std::time::Duration;
use reqwest::{RequestBuilder, StatusCode};
use serde::{Deserialize, Serialize};
use super::compat::{self, Compatibility, ServerInfo};
use super::wire;
/// Timeout for the short request/response calls in this module. Kept tight because a
/// user is watching a button while they run, and the most common mistake — a wrong
/// host on a LAN — fails by hanging rather than refusing, so an unbounded wait would
/// just look frozen. The sync engine's bulk transfers will need their own, longer one.
const REQUEST_TIMEOUT: Duration = Duration::from_secs(10);
/// Bulk transfers get much longer: a first full sync can be thousands of notes, and
/// failing one at ten seconds would make a large store impossible to ever pull.
const SYNC_TIMEOUT: Duration = Duration::from_secs(120);
/// Shared by every call that presents a token, so a revoked one reads the same way
/// wherever it surfaces.
const TOKEN_REJECTED: &str = "This server rejected the device token — it may have been \
revoked. Unlink and link again to issue a new one.";
/// What the link UI needs after a handshake: where we ended up (the normalized URL,
/// which may differ from what was typed), who answered, and whether we can work
/// with them.
#[derive(Debug, Serialize)]
pub struct ProbeResult {
pub base_url: String,
pub server: ServerInfo,
pub compatibility: Compatibility,
}
/// The account a device token belongs to. Surfaced after linking so the user can
/// confirm they linked the account they meant to — easy to get wrong on a server
/// hosting more than one.
#[derive(Debug, Clone, Deserialize, Serialize)]
pub struct Identity {
pub id: String,
pub email: String,
#[serde(default)]
pub display_name: String,
}
#[derive(Deserialize)]
struct DeviceLoginResponse {
token: String,
user: Identity,
}
fn http_with(timeout: Duration) -> Result<reqwest::Client, String> {
reqwest::Client::builder()
.timeout(timeout)
.build()
.map_err(|e| format!("Could not start the network client: {e}"))
}
fn http() -> Result<reqwest::Client, String> {
http_with(REQUEST_TIMEOUT)
}
/// Attach the client-identity headers every request carries, plus a bearer token
/// when we hold one.
fn prepare(builder: RequestBuilder, token: Option<&str>) -> RequestBuilder {
let mut builder = builder;
for (name, value) in compat::client_headers() {
builder = builder.header(name, value);
}
match token {
Some(t) => builder.bearer_auth(t),
None => builder,
}
}
fn unexpected_status(base_url: &str, status: StatusCode) -> String {
format!(
"{base_url} answered with HTTP {}. Check the address — a reverse proxy or a \
different site may be answering there.",
status.as_u16()
)
}
/// Ask a server who it is and whether we can sync with it.
///
/// `Err` means we never got a usable answer (bad address, unreachable, not a
/// ThoughtSync server). A server that answers but is *incompatible* comes back `Ok`
/// with a verdict — that distinction matters, because the two need very different
/// messages: one is "check what you typed", the other is "update something".
pub async fn probe(raw_url: &str) -> Result<ProbeResult, String> {
let base_url = compat::normalize_base_url(raw_url)
.ok_or("Enter a server address, like https://notes.example.com")?;
let request = prepare(http()?.get(config_url(&base_url)), None);
let response = request
.send()
.await
.map_err(|e| describe_transport_error(&base_url, &e))?;
let status = response.status();
if !status.is_success() {
return Err(unexpected_status(&base_url, status));
}
// Something answered 200 that isn't a ThoughtSync server (a router login page, a
// captive portal). Report the address, not the parse error, which would mean
// nothing to the person reading it.
let server: ServerInfo = response.json().await.map_err(|_| {
format!(
"{base_url} responded, but not with ThoughtSync's configuration. \
Is that the right address?"
)
})?;
let compatibility = compat::evaluate(&server);
Ok(ProbeResult {
base_url,
server,
compatibility,
})
}
/// Exchange email + password for a device bearer token.
///
/// The fresh-install path: it needs no existing session, which is what lets a brand
/// new desktop install link without visiting the web app first.
pub async fn device_login(
base_url: &str,
email: &str,
password: &str,
device_name: &str,
) -> Result<(String, Identity), String> {
let body = serde_json::json!({
"email": email,
"password": password,
"name": device_name,
});
let request = prepare(http()?.post(device_login_url(base_url)), None).json(&body);
let response = request
.send()
.await
.map_err(|e| describe_transport_error(base_url, &e))?;
let status = response.status();
if status == StatusCode::UNAUTHORIZED {
return Err("That email and password didn't match an account on this server.".to_string());
}
if !status.is_success() {
return Err(unexpected_status(base_url, status));
}
let parsed: DeviceLoginResponse = response
.json()
.await
.map_err(|_| format!("{base_url} signed us in but sent an unexpected reply."))?;
Ok((parsed.token, parsed.user))
}
/// Validate a token by asking whom it belongs to.
///
/// Used when the user pastes a token issued from the web app. Storing it unverified
/// would turn a copy/paste slip into a failure that only surfaces at the next sync,
/// far from the thing that caused it.
pub async fn fetch_identity(base_url: &str, token: &str) -> Result<Identity, String> {
let request = prepare(http()?.get(me_url(base_url)), Some(token));
let response = request
.send()
.await
.map_err(|e| describe_transport_error(base_url, &e))?;
let status = response.status();
if status == StatusCode::UNAUTHORIZED {
let message = "That token isn't valid on this server — it may have been revoked. \
Issue a new one from the web app under Account → Linked devices.";
return Err(message.to_string());
}
if !status.is_success() {
return Err(unexpected_status(base_url, status));
}
response
.json()
.await
.map_err(|_| format!("{base_url} accepted the token but sent an unexpected reply."))
}
/// Fetch one page of the change feed, starting after `since`.
///
/// The caller loops until `has_more` is false (see `pull::run`); paging lives there
/// rather than here so the transport stays a single request/response.
pub async fn fetch_changes(
base_url: &str,
token: &str,
since: i64,
) -> Result<wire::ChangesPage, String> {
let url = format!("{base_url}/api/sync/changes?since={since}");
let request = prepare(http_with(SYNC_TIMEOUT)?.get(url), Some(token));
let response = request
.send()
.await
.map_err(|e| describe_transport_error(base_url, &e))?;
let status = response.status();
if status == StatusCode::UNAUTHORIZED {
return Err(TOKEN_REJECTED.to_string());
}
if !status.is_success() {
return Err(unexpected_status(base_url, status));
}
response
.json()
.await
.map_err(|e| format!("Couldn't read the change feed from {base_url}: {e}"))
}
/// Download one attachment's bytes.
///
/// Metadata already arrived on the delta feed; this is only the payload, fetched
/// over the same route the web app uses (owner/shared scoped server-side).
pub async fn fetch_attachment(
base_url: &str,
token: &str,
note_id: &str,
attachment_id: &str,
) -> Result<Vec<u8>, String> {
let url = format!("{base_url}/api/notes/{note_id}/attachments/{attachment_id}");
let request = prepare(http_with(SYNC_TIMEOUT)?.get(url), Some(token));
let response = request
.send()
.await
.map_err(|e| describe_transport_error(base_url, &e))?;
let status = response.status();
if status == StatusCode::UNAUTHORIZED {
return Err(TOKEN_REJECTED.to_string());
}
if !status.is_success() {
return Err(unexpected_status(base_url, status));
}
response
.bytes()
.await
.map(|b| b.to_vec())
.map_err(|e| format!("Couldn't download an attachment from {base_url}: {e}"))
}
/// Send a batch of changes and hand back the raw reply.
///
/// Returns text rather than parsed results so this module stays pure transport —
/// `push::parse_results` owns the result shapes, and keeping them there is what lets
/// the parsing be unit-tested without a server.
pub async fn push_changes<T: Serialize>(
base_url: &str,
token: &str,
changes: &[T],
) -> Result<String, String> {
let body = serde_json::json!({ "changes": changes });
let url = format!("{base_url}/api/sync/push");
let request = prepare(http_with(SYNC_TIMEOUT)?.post(url), Some(token)).json(&body);
let response = request
.send()
.await
.map_err(|e| describe_transport_error(base_url, &e))?;
let status = response.status();
if status == StatusCode::UNAUTHORIZED {
return Err(TOKEN_REJECTED.to_string());
}
if !status.is_success() {
return Err(unexpected_status(base_url, status));
}
response
.text()
.await
.map_err(|e| format!("Couldn't read the push reply from {base_url}: {e}"))
}
/// The public, unauthenticated endpoint carrying the handshake.
fn config_url(base_url: &str) -> String {
format!("{base_url}/api/config")
}
fn device_login_url(base_url: &str) -> String {
format!("{base_url}/api/auth/device-login")
}
fn me_url(base_url: &str) -> String {
format!("{base_url}/api/auth/me")
}
/// Turn a transport failure into something a person can act on. reqwest's own
/// Display is accurate but reads like a stack trace.
fn describe_transport_error(base_url: &str, err: &reqwest::Error) -> String {
if err.is_timeout() {
// No specific duration here: these calls run under two different budgets
// (interactive vs bulk sync), and naming the wrong one is worse than naming
// none.
format!(
"{base_url} didn't respond in time. It may be offline, or unreachable \
from this network."
)
} else if err.is_connect() {
format!(
"Couldn't reach {base_url}. Check the address and that the server is \
running. If it uses plain HTTP, include http:// explicitly."
)
} else {
format!("Couldn't reach {base_url}: {err}")
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn urls_join_without_doubling_slashes() {
// normalize_base_url has already stripped any trailing slash, so plain
// concatenation is correct — this pins that assumption.
assert_eq!(
config_url("https://notes.example.com"),
"https://notes.example.com/api/config"
);
assert_eq!(
device_login_url("https://notes.example.com"),
"https://notes.example.com/api/auth/device-login"
);
assert_eq!(
me_url("https://notes.example.com"),
"https://notes.example.com/api/auth/me"
);
}
#[test]
fn urls_preserve_a_port_and_subpath() {
assert_eq!(
config_url("http://192.168.1.10:8000/thoughtsync"),
"http://192.168.1.10:8000/thoughtsync/api/config"
);
}
}
+166
View File
@@ -0,0 +1,166 @@
//! Tauri commands for pairing with a server (M10.7a).
//!
//! Linking is opt-in and reversible; the app is fully usable having never touched
//! any of this. The Settings UI (M10.7e) drives these.
use serde::{Deserialize, Serialize};
use tauri::State;
use crate::local::Db;
use crate::sync::blobs::BlobStore;
use crate::sync::client::{self, Identity, ProbeResult};
use crate::sync::compat::Compatibility;
use crate::sync::engine;
use crate::sync::push;
use crate::sync::state;
/// Ask a server who it is, without committing to anything. The UI calls this as the
/// user finishes typing an address, so they see what answered before handing over
/// credentials.
#[tauri::command]
pub async fn sync_probe(url: String) -> Result<ProbeResult, String> {
client::probe(&url).await
}
/// Either a password login or a token pasted from the web app. Both are offered
/// because neither covers everyone: a fresh install has no session to mint a token
/// from, while someone using a password manager or SSO may prefer not to type a
/// password into a desktop app at all.
#[derive(Deserialize)]
pub struct LinkInput {
pub url: String,
#[serde(default)]
pub email: Option<String>,
#[serde(default)]
pub password: Option<String>,
#[serde(default)]
pub token: Option<String>,
/// How this device is labelled in the server's device list.
#[serde(default)]
pub name: Option<String>,
}
#[derive(Serialize)]
pub struct LinkResult {
pub status: state::Status,
pub identity: Identity,
/// Carried through so the UI can warn about a `degraded` server right after
/// linking, instead of staying silent until a feature quietly does nothing.
pub compatibility: Compatibility,
}
/// A recognizable default, so a server's device list doesn't fill up with "Device".
fn default_device_name() -> String {
format!("ThoughtSync desktop ({})", std::env::consts::OS)
}
fn trimmed(value: &Option<String>) -> Option<&str> {
value.as_deref().map(str::trim).filter(|s| !s.is_empty())
}
#[tauri::command]
pub async fn sync_link(input: LinkInput, db: State<'_, Db>) -> Result<LinkResult, String> {
// 1. Handshake FIRST. Never hand credentials to a server we've established we
// can't sync with — and an incompatible server is exactly the case where a
// later failure would be hardest to attribute.
let probe = client::probe(&input.url).await?;
if let Compatibility::Incompatible { reason, .. } = &probe.compatibility {
return Err(reason.clone());
}
let base_url = probe.base_url;
// 2. Obtain a credential.
let (token, identity) = match trimmed(&input.token) {
Some(token) => {
// Verify before storing: an unverified paste turns a copy/paste slip
// into a failure that only surfaces at the next sync.
let identity = client::fetch_identity(&base_url, token).await?;
(token.to_string(), identity)
}
None => {
let (Some(email), Some(password)) = (trimmed(&input.email), trimmed(&input.password))
else {
return Err("Enter your email and password, or paste a device token.".to_string());
};
let name = trimmed(&input.name)
.map(str::to_string)
.unwrap_or_else(default_device_name);
client::device_login(&base_url, email, password, &name).await?
}
};
// 3. Persist. The lock is taken only now, for two reasons: a std MutexGuard
// isn't Send so it cannot be held across an await, and holding the store
// locked for a network round-trip would freeze every note operation in the UI.
let status = {
let conn = db.0.lock().map_err(|e| e.to_string())?;
state::set_link(&conn, &base_url, &token).map_err(|e| e.to_string())?;
// Adopt the server's trash-retention window immediately, so the Trash view
// stops counting down against this device's offline default the moment it's
// no longer the policy in force.
if let Some(days) = probe.server.trash_retention_days {
state::set_server_retention(&conn, days as i64).map_err(|e| e.to_string())?;
}
state::status(&conn).map_err(|e| e.to_string())?
};
log::info!("linked to {} as {}", base_url, identity.email);
Ok(LinkResult {
status,
identity,
compatibility: probe.compatibility,
})
}
/// Stop syncing and forget the server.
///
/// Local only: the device token remains valid on the SERVER until revoked there
/// (Account → Linked devices). We can't reliably revoke it from here — a pasted
/// token arrives without its device id — so the UI must say so rather than imply a
/// remote revoke that didn't happen. Tracked for follow-up.
#[tauri::command]
pub fn sync_unlink(db: State<'_, Db>) -> Result<state::Status, String> {
let conn = db.0.lock().map_err(|e| e.to_string())?;
state::clear_link(&conn).map_err(|e| e.to_string())?;
log::info!("unlinked from server");
state::status(&conn).map_err(|e| e.to_string())
}
#[tauri::command]
pub fn sync_status(db: State<'_, Db>) -> Result<state::Status, String> {
let conn = db.0.lock().map_err(|e| e.to_string())?;
state::status(&conn).map_err(|e| e.to_string())
}
/// The server URL + token, or a plain "not linked" error. Every networked sync
/// command needs exactly this, and none of them may hold the lock past it.
fn credentials(db: &State<'_, Db>) -> Result<(String, String), String> {
let conn = db.0.lock().map_err(|e| e.to_string())?;
let current = state::read(&conn).map_err(|e| e.to_string())?;
match (current.server_url, current.device_token) {
(Some(url), Some(token)) => Ok((url, token)),
_ => Err("This app isn't linked to a server yet.".to_string()),
}
}
/// Run one full sync: push local changes, then pull the server's.
///
/// The only sync entry point exposed to the UI, on purpose. Push and pull exist
/// separately inside the crate, but offering a bare "pull" would let the UI overwrite
/// unsent local edits — the ordering isn't a suggestion, it's what keeps them.
#[tauri::command]
pub async fn sync_now(
db: State<'_, Db>,
blobs: State<'_, BlobStore>,
) -> Result<engine::SyncOutcome, String> {
let (base_url, token) = credentials(&db)?;
engine::run_cycle(db.inner(), blobs.inner(), &base_url, &token).await
}
/// Whether anything is waiting to be sent. Lets the UI show an honest "unsynced
/// changes" state without running a sync to find out.
#[tauri::command]
pub fn sync_has_pending(db: State<'_, Db>) -> Result<bool, String> {
let conn = db.0.lock().map_err(|e| e.to_string())?;
push::has_pending(&conn).map_err(|e| e.to_string())
}
+393
View File
@@ -0,0 +1,393 @@
//! Client<->server compatibility handshake (M10.6).
//!
//! The desktop app is local-first: it never *needs* a server. When the user links
//! one, this module decides whether the two can actually talk — before a single
//! note moves. The sync engine (M10.7) consults it on link and on every sync.
//!
//! The contract is two integers per side, versioning the WIRE PROTOCOL separately
//! from either program's release version:
//!
//! | | this client | the server advertises |
//! |---|---|---|
//! | speaks | `CLIENT_PROTOCOL_VERSION` | `sync_protocol_version` |
//! | accepts down to | `MIN_SERVER_PROTOCOL_VERSION` | `min_client_protocol_version` |
//!
//! Each side declaring its own floor is what avoids app<->server lockstep: either
//! side can mark a change breaking without the other needing to ship in step. See
//! `docs/sync.md` for the policy that governs when those numbers move.
use serde::{Deserialize, Serialize};
/// The sync wire protocol this client speaks.
pub const CLIENT_PROTOCOL_VERSION: u32 = 1;
/// The oldest server protocol this client can drive — the symmetric half of the
/// server's `min_client_protocol_version`.
pub const MIN_SERVER_PROTOCOL_VERSION: u32 = 1;
/// Capabilities without which syncing is meaningless, so their absence BLOCKS the
/// link rather than degrading it.
pub const REQUIRED_FEATURES: &[&str] = &["notes", "labels"];
/// Capabilities whose absence costs a feature but not the link. Listing these
/// explicitly (rather than diffing against whatever the server happens to send) is
/// what lets the UI name exactly what the user will be missing.
pub const OPTIONAL_FEATURES: &[&str] = &["attachments", "tombstones", "revisions"];
/// The handshake fields of `GET /api/config`.
///
/// Every protocol field is optional because a server predating M10.6 simply won't
/// send them. That case has to read as "this server is too old to sync", not as a
/// parse failure — which would look to the user like they mistyped the URL.
#[derive(Debug, Clone, Default, Deserialize, Serialize)]
pub struct ServerInfo {
#[serde(default)]
pub site_name: Option<String>,
/// The server's release version, for display only — never gate on it.
#[serde(default)]
pub version: Option<String>,
#[serde(default)]
pub sync_protocol_version: Option<u32>,
#[serde(default)]
pub min_client_protocol_version: Option<u32>,
#[serde(default)]
pub sync_features: Vec<String>,
/// How long the SERVER keeps a trashed note before purging it (0 = forever).
/// Once linked this is the window that actually applies, so the desktop's Trash
/// countdown has to come from here rather than from its own offline default.
#[serde(default)]
pub trash_retention_days: Option<u32>,
}
impl ServerInfo {
fn has_feature(&self, name: &str) -> bool {
self.sync_features.iter().any(|f| f.as_str() == name)
}
fn missing(&self, from: &[&str]) -> Vec<String> {
from.iter()
.copied()
.filter(|f| !self.has_feature(f))
.map(String::from)
.collect()
}
}
/// The verdict the link/settings UI renders and the sync engine obeys.
///
/// Serialized tagged so the frontend can `switch` on `status` directly.
#[derive(Debug, Clone, PartialEq, Eq, Serialize)]
#[serde(tag = "status", rename_all = "snake_case")]
pub enum Compatibility {
/// Full parity — sync everything.
Ok,
/// Safe to sync, but these named capabilities aren't available here.
Degraded { unavailable: Vec<String> },
/// Do not sync. `client_must_update` points the user at the side that can fix
/// it, so the message can be actionable instead of just "incompatible".
Incompatible {
reason: String,
client_must_update: bool,
},
}
fn incompatible(reason: &str, client_must_update: bool) -> Compatibility {
Compatibility::Incompatible {
reason: reason.to_string(),
client_must_update,
}
}
/// Decide whether this client can sync with the described server.
///
/// Pure: the transport fetches `ServerInfo`, this decides what it means. Keeping
/// the decision free of I/O is what makes every branch below unit-testable, which
/// matters because there is no Postgres/live-server lane in CI.
pub fn evaluate(info: &ServerInfo) -> Compatibility {
// Ordered most-fundamental first, so the user sees the root problem rather than
// a downstream symptom of it.
let Some(server_proto) = info.sync_protocol_version else {
return incompatible(
"This server doesn't support device sync — it predates the sync protocol. \
Update the server, then link again.",
false,
);
};
if server_proto < MIN_SERVER_PROTOCOL_VERSION {
return incompatible(
&format!(
"This server speaks sync protocol v{server_proto}, but this app needs \
at least v{MIN_SERVER_PROTOCOL_VERSION}. Update the server."
),
false,
);
}
// The server's floor is what hard-blocks an old client. Absent => no floor: a
// server that advertises a protocol but no minimum accepts anything.
let floor = info.min_client_protocol_version.unwrap_or(0);
if CLIENT_PROTOCOL_VERSION < floor {
return incompatible(
&format!(
"This server requires client protocol v{floor} or newer; this app \
speaks v{CLIENT_PROTOCOL_VERSION}. Update ThoughtSync."
),
true,
);
}
// A version match still isn't enough: a server can speak the protocol with a
// core capability compiled out or disabled.
let missing_required = info.missing(REQUIRED_FEATURES);
if !missing_required.is_empty() {
return incompatible(
&format!(
"This server is missing sync capabilities this app requires: {}.",
missing_required.join(", ")
),
false,
);
}
let unavailable = info.missing(OPTIONAL_FEATURES);
if unavailable.is_empty() {
Compatibility::Ok
} else {
Compatibility::Degraded { unavailable }
}
}
/// Headers this client puts on every request to a linked server, so the server can
/// log or gate on client identity without a separate handshake round-trip.
pub fn client_headers() -> [(&'static str, String); 2] {
let agent = format!("thoughtsync-desktop/{}", env!("CARGO_PKG_VERSION"));
[
("X-ThoughtSync-Client", agent),
(
"X-ThoughtSync-Protocol",
CLIENT_PROTOCOL_VERSION.to_string(),
),
]
}
/// Turn what a user typed into a base URL we can build request paths on, or `None`
/// if there's nothing usable in it.
///
/// A bare host gets **`https://`**, never `http://`. Silently downgrading would put
/// a long-lived device token on the wire in cleartext because someone omitted five
/// characters. Plain HTTP on a trusted LAN stays fully supported — the user just
/// has to type `http://` and thereby choose it.
pub fn normalize_base_url(raw: &str) -> Option<String> {
let trimmed = raw.trim();
if trimmed.is_empty() {
return None;
}
// Resolve the scheme BEFORE touching trailing slashes — stripping them first
// turns a bare "https://" into "https:", which then reads as a hostname.
let with_scheme = match trimmed.split_once("://") {
Some((scheme, rest)) => {
// Anything that isn't HTTP(S) (ftp://, file://, a stray "foo://") can't
// be a ThoughtSync server; reject rather than fail confusingly later.
let scheme = scheme.to_ascii_lowercase();
if scheme != "http" && scheme != "https" {
return None;
}
format!("{scheme}://{rest}")
}
None => format!("https://{trimmed}"),
};
let (scheme, rest) = with_scheme.split_once("://")?;
let rest = rest.trim_end_matches('/');
// Reject a scheme with no authority ("https://", "http:///path").
if rest.split(['/', '?', '#']).next().unwrap_or("").is_empty() {
return None;
}
Some(format!("{scheme}://{rest}"))
}
#[cfg(test)]
mod tests {
use super::*;
/// A server matching this client exactly, which each test then degrades.
fn current_server() -> ServerInfo {
ServerInfo {
site_name: Some("ThoughtSync".into()),
version: Some("0.1.0".into()),
sync_protocol_version: Some(CLIENT_PROTOCOL_VERSION),
min_client_protocol_version: Some(CLIENT_PROTOCOL_VERSION),
sync_features: REQUIRED_FEATURES
.iter()
.chain(OPTIONAL_FEATURES.iter())
.copied()
.map(String::from)
.collect(),
trash_retention_days: Some(30),
}
}
#[test]
fn current_server_is_fully_compatible() {
assert_eq!(evaluate(&current_server()), Compatibility::Ok);
}
#[test]
fn server_without_protocol_fields_is_too_old() {
// A pre-M10.6 server: /api/config parses, but carries no protocol block.
let info = ServerInfo {
site_name: Some("ThoughtSync".into()),
version: Some("0.0.9".into()),
..Default::default()
};
match evaluate(&info) {
Compatibility::Incompatible {
client_must_update, ..
} => assert!(!client_must_update, "the SERVER is the old side here"),
other => panic!("expected incompatible, got {other:?}"),
}
}
#[test]
fn client_older_than_the_servers_floor_must_update() {
let info = ServerInfo {
sync_protocol_version: Some(CLIENT_PROTOCOL_VERSION + 5),
min_client_protocol_version: Some(CLIENT_PROTOCOL_VERSION + 5),
..current_server()
};
match evaluate(&info) {
Compatibility::Incompatible {
client_must_update, ..
} => assert!(client_must_update),
other => panic!("expected incompatible, got {other:?}"),
}
}
#[test]
fn newer_server_within_our_floor_still_works() {
// The whole point of the two-number contract: a server can move ahead
// additively without locking out a client that predates the change.
let info = ServerInfo {
sync_protocol_version: Some(CLIENT_PROTOCOL_VERSION + 3),
min_client_protocol_version: Some(CLIENT_PROTOCOL_VERSION),
..current_server()
};
assert_eq!(evaluate(&info), Compatibility::Ok);
}
#[test]
fn server_with_no_declared_floor_accepts_us() {
let info = ServerInfo {
min_client_protocol_version: None,
..current_server()
};
assert_eq!(evaluate(&info), Compatibility::Ok);
}
#[test]
fn missing_optional_feature_degrades_rather_than_blocks() {
let info = ServerInfo {
sync_features: current_server()
.sync_features
.into_iter()
.filter(|f| f.as_str() != "attachments")
.collect(),
..current_server()
};
assert_eq!(
evaluate(&info),
Compatibility::Degraded {
unavailable: vec!["attachments".to_string()]
}
);
}
#[test]
fn missing_required_feature_blocks() {
let info = ServerInfo {
sync_features: vec!["labels".to_string()],
..current_server()
};
match evaluate(&info) {
Compatibility::Incompatible { reason, .. } => assert!(reason.contains("notes")),
other => panic!("expected incompatible, got {other:?}"),
}
}
#[test]
fn version_mismatch_outranks_a_missing_feature() {
// Both wrong → report the version, the root cause of the missing feature.
let info = ServerInfo {
sync_protocol_version: Some(CLIENT_PROTOCOL_VERSION + 2),
min_client_protocol_version: Some(CLIENT_PROTOCOL_VERSION + 2),
sync_features: vec![],
..current_server()
};
match evaluate(&info) {
Compatibility::Incompatible {
client_must_update, ..
} => assert!(client_must_update),
other => panic!("expected incompatible, got {other:?}"),
}
}
#[test]
fn verdict_serializes_tagged_for_the_frontend() {
let verdict = Compatibility::Degraded {
unavailable: vec!["attachments".into()],
};
let json = serde_json::to_string(&verdict).expect("verdict serializes");
assert!(json.contains("\"status\":\"degraded\""), "got {json}");
}
#[test]
fn server_info_tolerates_unknown_and_absent_fields() {
// Forward compatibility: a NEWER server sending fields we've never heard of
// must not break the handshake.
let info: ServerInfo = serde_json::from_str(
r#"{"site_name":"S","sync_protocol_version":1,
"min_client_protocol_version":1,
"sync_features":["notes","labels","attachments","tombstones","revisions"],
"some_future_field":{"nested":true}}"#,
)
.expect("unknown fields are ignored");
assert_eq!(evaluate(&info), Compatibility::Ok);
}
#[test]
fn client_headers_identify_app_and_protocol() {
let headers = client_headers();
assert_eq!(headers[0].0, "X-ThoughtSync-Client");
assert!(headers[0].1.starts_with("thoughtsync-desktop/"));
assert_eq!(headers[1].1, CLIENT_PROTOCOL_VERSION.to_string());
}
#[test]
fn base_url_defaults_to_https_and_trims() {
assert_eq!(
normalize_base_url(" notes.example.com/ "),
Some("https://notes.example.com".to_string())
);
assert_eq!(
normalize_base_url("https://notes.example.com///"),
Some("https://notes.example.com".to_string())
);
}
#[test]
fn base_url_keeps_an_explicit_http_choice() {
// Plain HTTP on a LAN is supported — the user just has to ask for it.
assert_eq!(
normalize_base_url("http://192.168.1.10:8000"),
Some("http://192.168.1.10:8000".to_string())
);
}
#[test]
fn base_url_rejects_junk() {
assert_eq!(normalize_base_url(""), None);
assert_eq!(normalize_base_url(" "), None);
assert_eq!(normalize_base_url("https://"), None);
assert_eq!(normalize_base_url("ftp://files.example.com"), None);
}
}
+77
View File
@@ -0,0 +1,77 @@
//! The sync cycle (M10.7c).
//!
//! Deliberately the ONLY way the UI can sync. Push and pull are each usable on their
//! own inside this crate, but exposing them separately would let a caller pull
//! without pushing, which quietly overwrites unsent local edits.
use chrono::{SecondsFormat, Utc};
use serde::Serialize;
use super::blobs::BlobStore;
use super::pull;
use super::push;
use super::state;
use crate::local::Db;
#[derive(Debug, Serialize)]
pub struct SyncOutcome {
pub push: push::PushSummary,
pub pull: pull::PullSummary,
/// The state after the cycle, so the UI updates from one round-trip instead of
/// following every sync with a status call.
pub status: state::Status,
}
/// Push, then pull — in that order, always.
///
/// Pull writes the server's version straight over the local row, so anything not yet
/// sent would be lost to it. Pushing first is what puts the local edit in front of
/// the server's last-write-wins comparison, and it's the reason
/// `PullSummary::clobbered_dirty` should be zero on every healthy cycle.
///
/// A failed push aborts before the pull. Pulling anyway would take the exact rows we
/// just failed to save and overwrite them — turning a recoverable network error into
/// lost work.
pub async fn run_cycle(
db: &Db,
blobs: &BlobStore,
base_url: &str,
token: &str,
) -> Result<SyncOutcome, String> {
let push = push::run(db, base_url, token).await?;
let pull = pull::run(db, blobs, base_url, token).await?;
if pull.clobbered_dirty > 0 {
// Push ran first and reported success, so nothing should still have been
// dirty. Reaching here means something wrote to the store mid-cycle, or a
// change never got collected — worth a loud line either way.
log::warn!(
"sync cycle overwrote {} locally-edited note(s) despite pushing first",
pull.clobbered_dirty
);
}
// While we're already talking to this server, re-read what it says about itself.
// Today that's the trash-retention window the Trash view counts down against, and
// it can change under us whenever an admin edits the setting. Best-effort on
// purpose: a config blip must not fail a cycle whose actual work already
// succeeded, and the stored value simply stays as it was.
let retention = super::client::probe(base_url)
.await
.ok()
.and_then(|p| p.server.trash_retention_days);
let status = {
let conn = db.0.lock().map_err(|e| e.to_string())?;
if let Some(days) = retention {
state::set_server_retention(&conn, days as i64).map_err(|e| e.to_string())?;
}
// Stamped only here, after BOTH halves succeeded. A timestamp written after a
// partial cycle would tell the user they're up to date when they aren't.
let now = Utc::now().to_rfc3339_opts(SecondsFormat::Millis, true);
state::mark_synced(&conn, &now).map_err(|e| e.to_string())?;
state::status(&conn).map_err(|e| e.to_string())?
};
Ok(SyncOutcome { push, pull, status })
}
+23
View File
@@ -0,0 +1,23 @@
//! Talking to a ThoughtSync server — entirely opt-in.
//!
//! The app is local-first: `local` is the source of truth and everything works
//! unlinked. Nothing in here runs until the user links a server.
//!
//! - `compat` — the version/capability handshake (M10.6): whether a given server can
//! be talked to at all. Pure decision logic, no I/O.
//! - `client` — HTTP transport: the handshake call and device-token auth.
//! - `state` — the persisted link record (server, token, change-feed cursor).
//! - `commands` — the Tauri surface the Settings UI drives.
//!
//! The engine that moves notes — push, pull, last-write-wins — lands in M10.7b/c and
//! consults `compat` before it does anything.
pub mod blobs;
pub mod client;
pub mod commands;
pub mod compat;
pub mod engine;
pub mod pull;
pub mod push;
pub mod state;
pub mod wire;
+840
View File
@@ -0,0 +1,840 @@
//! Pull: bring a server's changes into the local store (M10.7b).
//!
//! The feed is a single monotonic sequence shared by notes and labels, so one
//! integer cursor is a total-order watermark over both (docs/sync.md). We loop pages
//! until the server says there are no more, persisting the cursor **in the same
//! transaction** as the page it describes — a cursor committed ahead of its data
//! would silently skip those rows forever, which reads as a clean sync.
use chrono::{SecondsFormat, Utc};
use rusqlite::{params, Connection, OptionalExtension};
use serde::Serialize;
use super::blobs::BlobStore;
use super::client;
use super::state;
use super::wire;
use crate::local::Db;
/// Backstop against a server that never stops saying `has_more`. At the server's
/// 1000-row page cap this is 10M rows — far past any real store, so hitting it means
/// something is wrong, not that someone has a lot of notes.
const MAX_PAGES: usize = 10_000;
/// What a pull did — for the UI, and for the log when something looks off.
#[derive(Debug, Clone, Default, Serialize, PartialEq, Eq)]
pub struct PullSummary {
pub pages: usize,
pub notes_applied: usize,
pub notes_deleted: usize,
pub labels_applied: usize,
pub labels_deleted: usize,
pub cursor: i64,
/// Rows that still held unpushed local edits when the server's version landed on
/// top. Should be 0 in the normal cycle, because push runs first; anything higher
/// means local work was overwritten, which is worth saying out loud.
pub clobbered_dirty: usize,
pub blobs_downloaded: usize,
/// Attachments whose bytes couldn't be fetched or failed verification. Counted
/// rather than fatal — see `download_missing_blobs`.
pub blobs_failed: usize,
}
impl PullSummary {
fn absorb(&mut self, other: PullSummary) {
self.pages += other.pages;
self.notes_applied += other.notes_applied;
self.notes_deleted += other.notes_deleted;
self.labels_applied += other.labels_applied;
self.labels_deleted += other.labels_deleted;
self.clobbered_dirty += other.clobbered_dirty;
self.blobs_downloaded += other.blobs_downloaded;
self.blobs_failed += other.blobs_failed;
self.cursor = other.cursor;
}
}
/// `(note_id, attachment_id, sha256)` for every attachment that advertises a hash.
/// The caller filters against the blob store — which blobs we hold isn't a SQL
/// question.
pub fn hashed_attachments(conn: &Connection) -> rusqlite::Result<Vec<(String, String, String)>> {
let mut stmt = conn.prepare(
"SELECT note_id, id, sha256 FROM attachments
WHERE sha256 IS NOT NULL AND sha256 <> ''",
)?;
let rows = stmt.query_map([], |r| Ok((r.get(0)?, r.get(1)?, r.get(2)?)))?;
rows.collect()
}
/// Fetch the bytes for any attachment we have metadata for but no blob.
///
/// A failed attachment NEVER fails the sync. Notes are the primary data and they've
/// already landed; an image that didn't arrive is retried on the next cycle simply
/// because its blob still counts as missing. Aborting here would mean one unreachable
/// file could block every future sync.
async fn download_missing_blobs(
db: &Db,
blobs: &BlobStore,
base_url: &str,
token: &str,
) -> Result<(usize, usize), String> {
let wanted = {
let conn = db.0.lock().map_err(|e| e.to_string())?;
hashed_attachments(&conn).map_err(|e| e.to_string())?
};
let mut downloaded = 0;
let mut failed = 0;
for (note_id, attachment_id, sha256) in wanted {
// Content-addressed, so this skips blobs we already hold — including the same
// image attached to a different note.
if blobs.has(&sha256) {
continue;
}
match client::fetch_attachment(base_url, token, &note_id, &attachment_id).await {
Ok(bytes) => match blobs.store(&sha256, &bytes) {
Ok(_) => downloaded += 1,
Err(e) => {
log::warn!("attachment {attachment_id}: {e}");
failed += 1;
}
},
Err(e) => {
log::warn!("attachment {attachment_id}: {e}");
failed += 1;
}
}
}
Ok((downloaded, failed))
}
fn now() -> String {
Utc::now().to_rfc3339_opts(SecondsFormat::Millis, true)
}
/// Apply one page and advance the cursor, atomically.
///
/// Labels are applied before notes so a membership never references a label row that
/// doesn't exist yet.
pub fn apply_page(conn: &Connection, page: &wire::ChangesPage) -> rusqlite::Result<PullSummary> {
let tx = conn.unchecked_transaction()?;
let mut summary = PullSummary {
pages: 1,
cursor: page.cursor,
..Default::default()
};
for label in &page.labels {
if label.is_tombstone() {
tx.execute("DELETE FROM labels WHERE id = ?1", params![label.id])?;
summary.labels_deleted += 1;
} else {
upsert_label(&tx, label)?;
summary.labels_applied += 1;
}
}
for note in &page.notes {
if note.is_tombstone() {
// A purge tombstone carries no content — its only job is to say "delete
// your copy". Children go with it via ON DELETE CASCADE.
tx.execute("DELETE FROM notes WHERE id = ?1", params![note.id])?;
summary.notes_deleted += 1;
continue;
}
if is_dirty(&tx, &note.id)? {
summary.clobbered_dirty += 1;
}
upsert_note(&tx, note)?;
summary.notes_applied += 1;
}
state::set_cursor(&tx, page.cursor)?;
tx.commit()?;
Ok(summary)
}
fn is_dirty(conn: &Connection, note_id: &str) -> rusqlite::Result<bool> {
let dirty: Option<i64> = conn
.query_row(
"SELECT dirty FROM notes WHERE id = ?1",
params![note_id],
|r| r.get(0),
)
.optional()?;
Ok(dirty == Some(1))
}
fn upsert_label(conn: &Connection, label: &wire::Label) -> rusqlite::Result<()> {
// One label per name is enforced on both sides (locally a UNIQUE index on
// lower(name); on the server, per owner). A label created offline can therefore
// collide with one the server already had under a different id — "work" typed on
// this machine and "work" that already existed.
//
// The server's row wins, but its MEMBERSHIPS have to survive the swap. Just
// deleting the local duplicate would cascade its note_labels away, stripping the
// label off notes that this pull never even mentions — silent loss that no later
// page would repair. So: free the name, insert the server's row, re-point the
// memberships onto it, then drop the husk.
let duplicates: Vec<String> = {
let mut stmt =
conn.prepare("SELECT id FROM labels WHERE lower(name) = lower(?1) AND id <> ?2")?;
let rows = stmt.query_map(params![label.name, label.id], |r| r.get::<_, String>(0))?;
rows.collect::<rusqlite::Result<Vec<String>>>()?
};
// Renaming first is what makes the insert possible at all — the unique index
// would otherwise reject the server's row before anything could be merged.
for old in &duplicates {
conn.execute(
"UPDATE labels SET name = name || ' (superseded ' || id || ')' WHERE id = ?1",
params![old],
)?;
}
let created = label.created_at.clone().unwrap_or_else(now);
conn.execute(
"INSERT INTO labels (id, name, color, created_at, updated_at, sync_revision, dirty)
VALUES (?1, ?2, ?3, ?4, ?4, ?5, 0)
ON CONFLICT(id) DO UPDATE SET
name = excluded.name,
color = excluded.color,
sync_revision = excluded.sync_revision,
dirty = 0",
params![
label.id,
label.name,
label.color,
created,
label.sync_revision
],
)?;
for old in &duplicates {
// OR IGNORE guards a (note_id, label_id) collision. Today the unique index on
// lower(name) makes that unreachable — two same-name labels can't coexist
// locally — so this is belt-and-braces against that index changing, not a
// case we've seen. Anything it skips cascades away with the husk below, which
// is correct: those are duplicates of a membership that now exists.
conn.execute(
"UPDATE OR IGNORE note_labels SET label_id = ?1 WHERE label_id = ?2",
params![label.id, old],
)?;
conn.execute("DELETE FROM labels WHERE id = ?1", params![old])?;
}
Ok(())
}
fn upsert_note(conn: &Connection, note: &wire::Note) -> rusqlite::Result<()> {
let created = note.created_at.clone().unwrap_or_else(now);
let updated = note.updated_at.clone().unwrap_or_else(|| created.clone());
// The server's `deleted_at` is the authority on trash AGE. Taking it from the feed
// rather than stamping "now" locally is what keeps a note trashed three weeks ago
// from looking brand-new to a device that only just heard about it — otherwise
// every fresh install would silently reset the whole retention clock. Falls back
// to the note's updated_at only if an older server omits the field.
let trashed_at = if note.trashed {
note.deleted_at.clone().or_else(|| Some(updated.clone()))
} else {
None
};
// `created_at` is deliberately absent from the UPDATE clause: a note's birth time
// never changes, and the server's copy is the same value anyway.
conn.execute(
"INSERT INTO notes (id, title, body, color, kind, position, pinned, archived,
trashed, remind_at, recurrence, created_at, updated_at,
sync_revision, trashed_at, dirty)
VALUES (?1, ?2, ?3, ?4, ?5, ?6, ?7, ?8, ?9, ?10, ?11, ?12, ?13, ?14, ?15, 0)
ON CONFLICT(id) DO UPDATE SET
title = excluded.title,
body = excluded.body,
color = excluded.color,
kind = excluded.kind,
position = excluded.position,
pinned = excluded.pinned,
archived = excluded.archived,
trashed = excluded.trashed,
remind_at = excluded.remind_at,
recurrence = excluded.recurrence,
updated_at = excluded.updated_at,
sync_revision = excluded.sync_revision,
trashed_at = excluded.trashed_at,
dirty = 0",
params![
note.id,
note.title,
note.body,
note.color,
note.kind,
note.position,
note.pinned,
note.archived,
note.trashed,
note.remind_at,
note.recurrence,
created,
updated,
note.sync_revision,
trashed_at,
],
)?;
// Children are replaced wholesale: a delta carries the note's FULL current state,
// so "what the server sent" IS the complete set. Diffing would be more code and
// could leave behind a row the server no longer has.
replace_items(conn, note)?;
replace_attachments(conn, note)?;
replace_previews(conn, note)?;
replace_labels(conn, note)?;
Ok(())
}
fn replace_items(conn: &Connection, note: &wire::Note) -> rusqlite::Result<()> {
conn.execute(
"DELETE FROM checklist_items WHERE note_id = ?1",
params![note.id],
)?;
for (index, item) in note.items.iter().enumerate() {
conn.execute(
"INSERT INTO checklist_items (id, note_id, text, checked, position)
VALUES (?1, ?2, ?3, ?4, ?5)",
params![
item.id,
note.id,
item.text,
item.checked,
position_of(item.position, index)
],
)?;
}
Ok(())
}
fn replace_attachments(conn: &Connection, note: &wire::Note) -> rusqlite::Result<()> {
conn.execute(
"DELETE FROM attachments WHERE note_id = ?1",
params![note.id],
)?;
for (index, att) in note.attachments.iter().enumerate() {
// The feed carries no explicit position for attachments — they arrive in
// creation order, so the index preserves it.
conn.execute(
"INSERT INTO attachments (id, note_id, url, filename, mime, size, sha256, position)
VALUES (?1, ?2, ?3, ?4, ?5, ?6, ?7, ?8)",
params![
att.id,
note.id,
att.url,
att.filename,
att.mime,
att.size,
att.sha256,
index as i64
],
)?;
}
Ok(())
}
fn replace_previews(conn: &Connection, note: &wire::Note) -> rusqlite::Result<()> {
conn.execute(
"DELETE FROM link_previews WHERE note_id = ?1",
params![note.id],
)?;
for (index, preview) in note.previews.iter().enumerate() {
conn.execute(
"INSERT INTO link_previews (id, note_id, url, title, description, image_url,
site_name, position)
VALUES (?1, ?2, ?3, ?4, ?5, ?6, ?7, ?8)",
params![
preview.id,
note.id,
preview.url,
preview.title,
preview.description,
preview.image_url,
preview.site_name,
index as i64
],
)?;
}
Ok(())
}
fn replace_labels(conn: &Connection, note: &wire::Note) -> rusqlite::Result<()> {
conn.execute(
"DELETE FROM note_labels WHERE note_id = ?1",
params![note.id],
)?;
for label in &note.labels {
ensure_label_stub(conn, label)?;
// `via_tag` is applied verbatim rather than re-derived from the body. The
// server already reconciled tags when it saved the note, and re-deriving here
// would call the local find-or-create path, which marks new labels dirty and
// would push them straight back — sync churn out of nothing.
conn.execute(
"INSERT OR IGNORE INTO note_labels (note_id, label_id, via_tag)
VALUES (?1, ?2, ?3)",
params![note.id, label.id, label.via_tag],
)?;
}
Ok(())
}
/// Materialize a label referenced by a note, if we don't have it yet.
///
/// Notes and labels page from one shared sequence, so a note can reference a label
/// whose own delta landed in an earlier page — or, right at a page boundary, hasn't
/// landed. The note carries enough of the label to create it, so a membership never
/// fails on a missing row. `OR IGNORE` because the label's real delta (later in this
/// page or a future one) is the authority on its name and color.
fn ensure_label_stub(conn: &Connection, label: &wire::NoteLabel) -> rusqlite::Result<()> {
let ts = now();
conn.execute(
"INSERT OR IGNORE INTO labels (id, name, color, created_at, updated_at, dirty)
VALUES (?1, ?2, ?3, ?4, ?4, 0)",
params![label.id, label.name, label.color, ts],
)?;
Ok(())
}
/// Trust an explicit position; fall back to arrival order when the server sent 0 for
/// everything (which is what an unordered list looks like on the wire).
fn position_of(explicit: i64, index: usize) -> i64 {
if explicit > 0 {
explicit
} else {
index as i64
}
}
/// Loop the feed to exhaustion, starting from the persisted cursor.
///
/// NOTE ON ORDERING: the full cycle is push-then-pull (docs/sync.md). Running this
/// against a store with unpushed edits lets the server's version land on top of them
/// — counted as `clobbered_dirty` and logged, rather than hidden.
pub async fn run(
db: &Db,
blobs: &BlobStore,
base_url: &str,
token: &str,
) -> Result<PullSummary, String> {
let mut total = PullSummary::default();
loop {
let since = {
let conn = db.0.lock().map_err(|e| e.to_string())?;
state::read(&conn).map_err(|e| e.to_string())?.last_cursor
};
let page = client::fetch_changes(base_url, token, since).await?;
// Trust the data over the flag: a server that claims more pages without
// advancing the cursor would spin this loop forever.
if page.has_more && page.cursor <= since {
return Err(format!(
"The server reported more changes but its cursor didn't advance past \
{since}. Stopping rather than looping forever."
));
}
let has_more = page.has_more;
let applied = {
let conn = db.0.lock().map_err(|e| e.to_string())?;
apply_page(&conn, &page).map_err(|e| e.to_string())?
};
total.absorb(applied);
if !has_more {
break;
}
if total.pages >= MAX_PAGES {
return Err(format!(
"Stopped after {MAX_PAGES} pages without reaching the end of the \
server's changes. Something is wrong with the feed."
));
}
}
// Notes first, bytes after: the metadata is what makes the attachments knowable,
// and knowing one is missing is what lets the next cycle retry it.
let (downloaded, failed) = download_missing_blobs(db, blobs, base_url, token).await?;
total.blobs_downloaded = downloaded;
total.blobs_failed = failed;
if total.clobbered_dirty > 0 {
log::warn!(
"pull overwrote {} note(s) that still had unpushed local edits",
total.clobbered_dirty
);
}
if total.blobs_failed > 0 {
log::warn!(
"pull: {} attachment(s) couldn't be downloaded; will retry next sync",
total.blobs_failed
);
}
log::info!(
"pull complete: {} page(s), {} note(s) applied, {} deleted, {} label(s) applied, cursor {}",
total.pages,
total.notes_applied,
total.notes_deleted,
total.labels_applied,
total.cursor
);
Ok(total)
}
#[cfg(test)]
mod tests {
use super::*;
use crate::local::schema;
fn db() -> Connection {
let conn = Connection::open_in_memory().expect("in-memory db");
schema::migrate(&conn).expect("migrate");
conn
}
fn note(id: &str, revision: i64) -> wire::Note {
wire::Note {
id: id.to_string(),
title: Some("Title".into()),
body: "Body".into(),
color: "default".into(),
kind: "text".into(),
position: 0,
pinned: false,
archived: false,
trashed: false,
deleted_at: None,
remind_at: None,
recurrence: None,
created_at: Some("2026-07-26T00:00:00.000Z".into()),
updated_at: Some("2026-07-26T00:00:00.000Z".into()),
sync_revision: revision,
purged_at: None,
labels: vec![],
items: vec![],
attachments: vec![],
previews: vec![],
}
}
fn page(notes: Vec<wire::Note>, labels: Vec<wire::Label>, cursor: i64) -> wire::ChangesPage {
wire::ChangesPage {
notes,
labels,
cursor,
has_more: false,
}
}
fn count(conn: &Connection, sql: &str) -> i64 {
conn.query_row(sql, [], |r| r.get(0)).expect("count")
}
fn trash_stamp(conn: &Connection, id: &str) -> Option<String> {
let sql = "SELECT trashed_at FROM notes WHERE id = ?1";
conn.query_row(sql, [id], |r| r.get(0)).expect("stamp")
}
#[test]
fn applies_a_note_and_advances_the_cursor() {
let conn = db();
let summary = apply_page(&conn, &page(vec![note("n1", 7)], vec![], 7)).expect("apply");
assert_eq!(summary.notes_applied, 1);
assert_eq!(count(&conn, "SELECT COUNT(*) FROM notes"), 1);
assert_eq!(state::read(&conn).expect("state").last_cursor, 7);
}
#[test]
fn pulled_rows_are_not_dirty() {
// They came FROM the server, so pushing them back would be pure churn.
let conn = db();
apply_page(&conn, &page(vec![note("n1", 1)], vec![], 1)).expect("apply");
assert_eq!(count(&conn, "SELECT dirty FROM notes WHERE id = 'n1'"), 0);
}
#[test]
fn tombstone_deletes_the_local_note() {
let conn = db();
apply_page(&conn, &page(vec![note("n1", 1)], vec![], 1)).expect("apply");
let mut dead = note("n1", 2);
dead.purged_at = Some("2026-07-26T01:00:00.000Z".into());
let summary = apply_page(&conn, &page(vec![dead], vec![], 2)).expect("apply");
assert_eq!(summary.notes_deleted, 1);
assert_eq!(count(&conn, "SELECT COUNT(*) FROM notes"), 0);
}
#[test]
fn trashed_is_not_a_tombstone() {
// `trashed` is ordinary state that keeps syncing; only `purged_at` deletes.
let conn = db();
let mut trashed = note("n1", 1);
trashed.trashed = true;
apply_page(&conn, &page(vec![trashed], vec![], 1)).expect("apply");
assert_eq!(count(&conn, "SELECT COUNT(*) FROM notes"), 1);
assert_eq!(count(&conn, "SELECT trashed FROM notes WHERE id = 'n1'"), 1);
}
#[test]
fn trash_age_comes_from_the_server_not_from_now() {
// The retention countdown runs off this timestamp. Stamping it locally would
// hand every note a fresh 30 days on any device that syncs it for the first
// time — a note trashed last month would never expire anywhere.
let conn = db();
let mut trashed = note("n1", 1);
trashed.trashed = true;
trashed.deleted_at = Some("2026-06-01T09:30:00+00:00".into());
apply_page(&conn, &page(vec![trashed], vec![], 1)).expect("apply");
let stamped = trash_stamp(&conn, "n1");
assert_eq!(stamped.as_deref(), Some("2026-06-01T09:30:00+00:00"));
}
#[test]
fn restoring_a_note_server_side_clears_its_trash_stamp() {
let conn = db();
let mut trashed = note("n1", 1);
trashed.trashed = true;
trashed.deleted_at = Some("2026-06-01T09:30:00+00:00".into());
apply_page(&conn, &page(vec![trashed], vec![], 1)).expect("apply");
apply_page(&conn, &page(vec![note("n1", 2)], vec![], 2)).expect("apply");
let stamped = trash_stamp(&conn, "n1");
assert_eq!(stamped, None, "an untrashed note keeps no trash stamp");
}
#[test]
fn an_older_server_without_deleted_at_still_ages_the_trash() {
// Falls back to updated_at rather than leaving the stamp null, which would
// make the note un-expirable and its countdown blank.
let conn = db();
let mut trashed = note("n1", 1);
trashed.trashed = true;
trashed.deleted_at = None;
apply_page(&conn, &page(vec![trashed], vec![], 1)).expect("apply");
let stamped = trash_stamp(&conn, "n1");
assert_eq!(stamped.as_deref(), Some("2026-07-26T00:00:00.000Z"));
}
#[test]
fn children_are_replaced_not_merged() {
let conn = db();
let mut first = note("n1", 1);
first.items = vec![
wire::Item {
id: "i1".into(),
text: "one".into(),
checked: false,
position: 0,
},
wire::Item {
id: "i2".into(),
text: "two".into(),
checked: false,
position: 1,
},
];
apply_page(&conn, &page(vec![first], vec![], 1)).expect("apply");
assert_eq!(count(&conn, "SELECT COUNT(*) FROM checklist_items"), 2);
// The server dropped an item; the local copy must drop it too.
let mut second = note("n1", 2);
second.items = vec![wire::Item {
id: "i1".into(),
text: "one".into(),
checked: true,
position: 0,
}];
apply_page(&conn, &page(vec![second], vec![], 2)).expect("apply");
assert_eq!(count(&conn, "SELECT COUNT(*) FROM checklist_items"), 1);
}
#[test]
fn note_label_membership_materializes_a_missing_label() {
// The label's own delta may have landed in an earlier page, or not yet.
let conn = db();
let mut n = note("n1", 1);
n.labels = vec![wire::NoteLabel {
id: "l1".into(),
name: "work".into(),
color: "blue".into(),
via_tag: true,
}];
apply_page(&conn, &page(vec![n], vec![], 1)).expect("apply");
assert_eq!(count(&conn, "SELECT COUNT(*) FROM labels"), 1);
assert_eq!(
count(
&conn,
"SELECT via_tag FROM note_labels WHERE note_id = 'n1'"
),
1,
"via_tag is applied verbatim, not re-derived"
);
}
#[test]
fn server_label_replaces_a_local_duplicate_by_name() {
let conn = db();
conn.execute(
"INSERT INTO labels (id, name, color, created_at, updated_at, dirty)
VALUES ('local-id', 'Work', 'default', '2026-01-01', '2026-01-01', 1)",
[],
)
.expect("seed local label");
let server = wire::Label {
id: "server-id".into(),
name: "work".into(),
color: "blue".into(),
sync_revision: 5,
purged_at: None,
created_at: Some("2026-07-26T00:00:00.000Z".into()),
};
apply_page(&conn, &page(vec![], vec![server], 5)).expect("apply");
assert_eq!(count(&conn, "SELECT COUNT(*) FROM labels"), 1);
let id: String = conn
.query_row("SELECT id FROM labels", [], |r| r.get(0))
.expect("label");
assert_eq!(id, "server-id", "the server's row wins on pull");
}
#[test]
fn merging_a_duplicate_label_keeps_its_note_memberships() {
// The notes carrying the local label may not be in this page at all, so a
// plain delete would strip the label off them with nothing to repair it.
let conn = db();
apply_page(&conn, &page(vec![note("n1", 1)], vec![], 1)).expect("seed note");
conn.execute(
"INSERT INTO labels (id, name, color, created_at, updated_at, dirty)
VALUES ('local-id', 'Work', 'default', '2026-01-01', '2026-01-01', 1)",
[],
)
.expect("seed local label");
conn.execute(
"INSERT INTO note_labels (note_id, label_id, via_tag)
VALUES ('n1', 'local-id', 0)",
[],
)
.expect("seed membership");
let server = wire::Label {
id: "server-id".into(),
name: "work".into(),
color: "blue".into(),
sync_revision: 5,
purged_at: None,
created_at: None,
};
apply_page(&conn, &page(vec![], vec![server], 5)).expect("apply");
assert_eq!(count(&conn, "SELECT COUNT(*) FROM labels"), 1);
let label_id: String = conn
.query_row(
"SELECT label_id FROM note_labels WHERE note_id = 'n1'",
[],
|r| r.get(0),
)
.expect("membership survived");
assert_eq!(label_id, "server-id", "membership re-pointed, not dropped");
}
#[test]
fn label_tombstone_deletes_and_cascades_memberships() {
let conn = db();
let mut n = note("n1", 1);
n.labels = vec![wire::NoteLabel {
id: "l1".into(),
name: "work".into(),
color: "blue".into(),
via_tag: false,
}];
apply_page(&conn, &page(vec![n], vec![], 1)).expect("apply");
assert_eq!(count(&conn, "SELECT COUNT(*) FROM note_labels"), 1);
let dead = wire::Label {
id: "l1".into(),
name: "work".into(),
color: "blue".into(),
sync_revision: 2,
purged_at: Some("2026-07-26T01:00:00.000Z".into()),
created_at: None,
};
apply_page(&conn, &page(vec![], vec![dead], 2)).expect("apply");
assert_eq!(count(&conn, "SELECT COUNT(*) FROM labels"), 0);
assert_eq!(
count(&conn, "SELECT COUNT(*) FROM note_labels"),
0,
"membership should cascade with the label"
);
}
#[test]
fn overwriting_a_dirty_note_is_counted() {
let conn = db();
conn.execute(
"INSERT INTO notes (id, body, created_at, updated_at, dirty)
VALUES ('n1', 'local edit', '2026-01-01', '2026-01-01', 1)",
[],
)
.expect("seed dirty note");
let summary = apply_page(&conn, &page(vec![note("n1", 9)], vec![], 9)).expect("apply");
assert_eq!(summary.clobbered_dirty, 1);
}
#[test]
fn applying_a_fresh_note_reports_no_clobber() {
let conn = db();
let summary = apply_page(&conn, &page(vec![note("n1", 1)], vec![], 1)).expect("apply");
assert_eq!(summary.clobbered_dirty, 0);
}
#[test]
fn empty_page_still_advances_the_cursor() {
// The server can page past rows that were trimmed to the shared watermark.
let conn = db();
apply_page(&conn, &page(vec![], vec![], 42)).expect("apply");
assert_eq!(state::read(&conn).expect("state").last_cursor, 42);
}
#[test]
fn note_upsert_preserves_the_original_created_at() {
let conn = db();
apply_page(&conn, &page(vec![note("n1", 1)], vec![], 1)).expect("apply");
let mut later = note("n1", 2);
later.created_at = Some("2099-01-01T00:00:00.000Z".into());
apply_page(&conn, &page(vec![later], vec![], 2)).expect("apply");
let created: String = conn
.query_row("SELECT created_at FROM notes WHERE id = 'n1'", [], |r| {
r.get(0)
})
.expect("created_at");
assert_eq!(created, "2026-07-26T00:00:00.000Z");
}
#[test]
fn a_page_that_fails_leaves_the_cursor_untouched() {
// Atomicity is the whole resumability story: a cursor committed ahead of its
// data would skip those rows forever. Force a failure with a duplicate
// checklist-item id inside one page.
let conn = db();
let mut n = note("n1", 3);
n.items = vec![
wire::Item {
id: "dup".into(),
text: "one".into(),
checked: false,
position: 0,
},
wire::Item {
id: "dup".into(),
text: "two".into(),
checked: false,
position: 1,
},
];
assert!(apply_page(&conn, &page(vec![n], vec![], 3)).is_err());
assert_eq!(state::read(&conn).expect("state").last_cursor, 0);
assert_eq!(count(&conn, "SELECT COUNT(*) FROM notes"), 0);
}
}
+751
View File
@@ -0,0 +1,751 @@
//! Push: send local changes to the server and apply what it says (M10.7c).
//!
//! Two sources feed a push: rows flagged `dirty` (created or edited locally) and rows
//! in `pending_deletes` (permanently deleted locally — see `local::schema` v2 for why
//! a delete needs its own record).
//!
//! Sync is **whole-note**: an upsert carries the client's full current state, not a
//! patch (docs/sync.md). The server resolves conflicts last-write-wins by the client's
//! `edited_at`, snapshotting anything it overwrites into the note's version history.
use rusqlite::{params, Connection, OptionalExtension};
use serde::{Deserialize, Serialize};
use super::client;
use super::state;
use crate::local::Db;
/// The server rejects a batch larger than this (`MAX_PUSH` in `sync.py`).
const BATCH: usize = 500;
/// Backstop: a batch whose results never clear `dirty` would loop forever.
const MAX_BATCHES: usize = 10_000;
#[derive(Debug, Clone, Default, Serialize, PartialEq, Eq)]
pub struct PushSummary {
pub batches: usize,
pub sent: usize,
pub created: usize,
pub applied: usize,
/// The server had a newer edit and kept it. Not a failure — the local row stops
/// being dirty and the following pull adopts the server's version.
pub kept: usize,
pub noop: usize,
/// Still dirty, and surfaced: these need a human (a duplicate label name is the
/// realistic case). Silently retrying forever would be the wrong shape.
pub rejected: usize,
pub errors: Vec<String>,
}
impl PushSummary {
fn absorb(&mut self, other: PushSummary) {
self.batches += other.batches;
self.sent += other.sent;
self.created += other.created;
self.applied += other.applied;
self.kept += other.kept;
self.noop += other.noop;
self.rejected += other.rejected;
self.errors.extend(other.errors);
}
}
// --- outgoing shapes ---------------------------------------------------------
/// One entry in the `changes` array. Notes and labels share the envelope; serde skips
/// the fields that don't apply, so the server sees exactly the shape docs/sync.md
/// describes for each entity.
#[derive(Debug, Serialize)]
pub struct Change {
pub entity: &'static str,
pub id: String,
pub op: &'static str,
pub edited_at: String,
#[serde(skip_serializing_if = "Option::is_none")]
pub title: Option<String>,
#[serde(skip_serializing_if = "Option::is_none")]
pub body: Option<String>,
#[serde(skip_serializing_if = "Option::is_none")]
pub color: Option<String>,
#[serde(skip_serializing_if = "Option::is_none")]
pub kind: Option<String>,
#[serde(skip_serializing_if = "Option::is_none")]
pub pinned: Option<bool>,
#[serde(skip_serializing_if = "Option::is_none")]
pub archived: Option<bool>,
#[serde(skip_serializing_if = "Option::is_none")]
pub trashed: Option<bool>,
#[serde(skip_serializing_if = "Option::is_none")]
pub remind_at: Option<String>,
#[serde(skip_serializing_if = "Option::is_none")]
pub recurrence: Option<String>,
#[serde(skip_serializing_if = "Option::is_none")]
pub position: Option<i64>,
#[serde(skip_serializing_if = "Option::is_none")]
pub items: Option<Vec<ItemOut>>,
#[serde(skip_serializing_if = "Option::is_none")]
pub label_ids: Option<Vec<String>>,
#[serde(skip_serializing_if = "Option::is_none")]
pub created_at: Option<String>,
#[serde(skip_serializing_if = "Option::is_none")]
pub name: Option<String>,
}
impl Change {
fn delete(entity: &'static str, id: String, edited_at: String) -> Self {
Change {
entity,
id,
op: "delete",
edited_at,
title: None,
body: None,
color: None,
kind: None,
pinned: None,
archived: None,
trashed: None,
remind_at: None,
recurrence: None,
position: None,
items: None,
label_ids: None,
created_at: None,
name: None,
}
}
}
#[derive(Debug, Serialize)]
pub struct ItemOut {
pub text: String,
pub checked: bool,
}
// --- incoming results --------------------------------------------------------
#[derive(Debug, Deserialize)]
struct PushResponse {
#[serde(default)]
results: Vec<PushResult>,
}
#[derive(Debug, Clone, Deserialize)]
pub struct PushResult {
#[serde(default)]
pub id: Option<String>,
#[serde(default)]
pub entity: Option<String>,
#[serde(default)]
pub status: String,
#[serde(default)]
pub sync_revision: Option<i64>,
#[serde(default)]
pub error: Option<String>,
}
// --- collecting --------------------------------------------------------------
/// Everything waiting to go up, oldest edit first so a truncated batch still makes
/// forward progress in a sensible order.
pub fn collect(conn: &Connection, limit: usize) -> rusqlite::Result<Vec<Change>> {
let mut out = Vec::new();
collect_deletes(conn, &mut out, limit)?;
if out.len() < limit {
collect_labels(conn, &mut out, limit)?;
}
if out.len() < limit {
collect_notes(conn, &mut out, limit)?;
}
Ok(out)
}
fn collect_deletes(conn: &Connection, out: &mut Vec<Change>, limit: usize) -> rusqlite::Result<()> {
let mut stmt = conn.prepare(
"SELECT entity, id, deleted_at FROM pending_deletes ORDER BY deleted_at LIMIT ?1",
)?;
let rows = stmt.query_map(params![limit as i64], |r| {
Ok((
r.get::<_, String>(0)?,
r.get::<_, String>(1)?,
r.get::<_, String>(2)?,
))
})?;
for row in rows {
let (entity, id, deleted_at) = row?;
// Only 'note' and 'label' exist on the wire; anything else is a bug in a
// writer, and shipping it would earn a blanket rejection for the batch.
let entity: &'static str = match entity.as_str() {
"note" => "note",
"label" => "label",
_ => continue,
};
out.push(Change::delete(entity, id, deleted_at));
}
Ok(())
}
fn collect_labels(conn: &Connection, out: &mut Vec<Change>, limit: usize) -> rusqlite::Result<()> {
let remaining = limit.saturating_sub(out.len());
let mut stmt = conn.prepare(
"SELECT id, name, color, updated_at FROM labels
WHERE dirty = 1 ORDER BY updated_at LIMIT ?1",
)?;
let rows = stmt.query_map(params![remaining as i64], |r| {
Ok(Change {
entity: "label",
id: r.get(0)?,
op: "upsert",
name: Some(r.get(1)?),
color: Some(r.get(2)?),
edited_at: r.get(3)?,
title: None,
body: None,
kind: None,
pinned: None,
archived: None,
trashed: None,
remind_at: None,
recurrence: None,
position: None,
items: None,
label_ids: None,
created_at: None,
})
})?;
for row in rows {
out.push(row?);
}
Ok(())
}
fn collect_notes(conn: &Connection, out: &mut Vec<Change>, limit: usize) -> rusqlite::Result<()> {
let remaining = limit.saturating_sub(out.len());
let ids: Vec<String> = {
let mut stmt =
conn.prepare("SELECT id FROM notes WHERE dirty = 1 ORDER BY updated_at LIMIT ?1")?;
let rows = stmt.query_map(params![remaining as i64], |r| r.get::<_, String>(0))?;
rows.collect::<rusqlite::Result<Vec<String>>>()?
};
for id in ids {
out.push(note_change(conn, &id)?);
}
Ok(())
}
/// The note's own columns. A named struct rather than a twelve-wide tuple so the
/// field-to-column mapping stays readable at the call site.
struct NoteRow {
title: Option<String>,
body: String,
color: String,
kind: String,
position: i64,
pinned: bool,
archived: bool,
trashed: bool,
remind_at: Option<String>,
recurrence: Option<String>,
created_at: String,
updated_at: String,
}
fn note_row(conn: &Connection, id: &str) -> rusqlite::Result<NoteRow> {
conn.query_row(
"SELECT title, body, color, kind, position, pinned, archived, trashed,
remind_at, recurrence, created_at, updated_at
FROM notes WHERE id = ?1",
params![id],
|r| {
Ok(NoteRow {
title: r.get(0)?,
body: r.get(1)?,
color: r.get(2)?,
kind: r.get(3)?,
position: r.get(4)?,
pinned: r.get::<_, i64>(5)? != 0,
archived: r.get::<_, i64>(6)? != 0,
trashed: r.get::<_, i64>(7)? != 0,
remind_at: r.get(8)?,
recurrence: r.get(9)?,
created_at: r.get(10)?,
updated_at: r.get(11)?,
})
},
)
}
fn note_change(conn: &Connection, id: &str) -> rusqlite::Result<Change> {
let row = note_row(conn, id)?;
let items = {
let mut stmt = conn.prepare(
"SELECT text, checked FROM checklist_items WHERE note_id = ?1 ORDER BY position",
)?;
let rows = stmt.query_map(params![id], |r| {
Ok(ItemOut {
text: r.get(0)?,
checked: r.get::<_, i64>(1)? != 0,
})
})?;
rows.collect::<rusqlite::Result<Vec<ItemOut>>>()?
};
// MANUAL memberships only. Tag-sourced ones (`via_tag = 1`) are re-derived by the
// server from the body; sending them as label_ids would convert them into manual
// assignments that no longer disappear when the #tag is removed from the text.
let label_ids = {
let mut stmt =
conn.prepare("SELECT label_id FROM note_labels WHERE note_id = ?1 AND via_tag = 0")?;
let rows = stmt.query_map(params![id], |r| r.get::<_, String>(0))?;
rows.collect::<rusqlite::Result<Vec<String>>>()?
};
Ok(Change {
entity: "note",
id: id.to_string(),
op: "upsert",
// The local `updated_at` IS the client's edit time, which is what the
// server's last-write-wins comparison runs against.
edited_at: row.updated_at,
title: row.title,
body: Some(row.body),
color: Some(row.color),
kind: Some(row.kind),
pinned: Some(row.pinned),
archived: Some(row.archived),
trashed: Some(row.trashed),
remind_at: row.remind_at,
recurrence: row.recurrence,
position: Some(row.position),
items: Some(items),
label_ids: Some(label_ids),
created_at: Some(row.created_at),
name: None,
})
}
// --- applying results --------------------------------------------------------
/// Fold one batch's results back into the local store, atomically.
pub fn apply_results(
conn: &Connection,
sent: &[Change],
results: &[PushResult],
) -> rusqlite::Result<PushSummary> {
let tx = conn.unchecked_transaction()?;
let mut summary = PushSummary {
batches: 1,
sent: sent.len(),
..Default::default()
};
// The server answers positionally, one result per change. Zip rather than trust
// the echoed id: a rejected malformed entry may carry no id at all.
let mut lowest_kept: Option<i64> = None;
for (change, result) in sent.iter().zip(results.iter()) {
match result.status.as_str() {
"created" | "applied" => {
clear_dirty(&tx, change, result.sync_revision)?;
if result.status == "created" {
summary.created += 1;
} else {
summary.applied += 1;
}
if change.op == "delete" {
forget_pending_delete(&tx, change)?;
}
}
"noop" => {
// The server had nothing to do — typically a delete for a row it
// never saw (created and deleted while offline).
clear_dirty(&tx, change, result.sync_revision)?;
forget_pending_delete(&tx, change)?;
summary.noop += 1;
}
"kept" => {
// The server's version is newer. Stop being dirty — re-pushing would
// lose to the same comparison forever — and let the next pull bring
// the server's copy down.
clear_dirty(&tx, change, None)?;
if change.op == "delete" {
// Our delete lost to a newer server edit; the note lives on, and
// the pull will restore it locally. Drop the tombstone so we
// don't keep trying to delete a note the user has since edited.
forget_pending_delete(&tx, change)?;
}
if let Some(revision) = result.sync_revision {
lowest_kept = Some(lowest_kept.map_or(revision, |c: i64| c.min(revision)));
}
summary.kept += 1;
}
_ => {
// "rejected" and anything unrecognized: leave the row dirty so it is
// retried, and surface the reason. A duplicate label name is the
// realistic case and only a human can resolve it.
summary.rejected += 1;
let reason = result
.error
.clone()
.unwrap_or_else(|| result.status.clone());
summary
.errors
.push(format!("{} {}: {reason}", change.entity, change.id));
}
}
}
// A `kept` result means the server holds a version we have not seen. Normally its
// revision is above our cursor and the next pull fetches it anyway. If it is NOT
// — which happens when a skewed clock makes a genuinely later local edit look
// older — rewind so that note is re-fetched. Without this the local edit is
// dropped from sync and the stale copy stays on screen with nothing marking it.
if let Some(revision) = lowest_kept {
let current = state::read(&tx)?.last_cursor;
if revision <= current {
state::set_cursor(&tx, (revision - 1).max(0))?;
}
}
tx.commit()?;
Ok(summary)
}
fn clear_dirty(conn: &Connection, change: &Change, revision: Option<i64>) -> rusqlite::Result<()> {
// A delete has no local row left to update.
if change.op == "delete" {
return Ok(());
}
let table = match change.entity {
"label" => "labels",
_ => "notes",
};
match revision {
Some(rev) => conn.execute(
&format!("UPDATE {table} SET dirty = 0, sync_revision = ?2 WHERE id = ?1"),
params![change.id, rev],
)?,
None => conn.execute(
&format!("UPDATE {table} SET dirty = 0 WHERE id = ?1"),
params![change.id],
)?,
};
Ok(())
}
fn forget_pending_delete(conn: &Connection, change: &Change) -> rusqlite::Result<()> {
if change.op != "delete" {
return Ok(());
}
conn.execute(
"DELETE FROM pending_deletes WHERE entity = ?1 AND id = ?2",
params![change.entity, change.id],
)?;
Ok(())
}
/// True when anything is waiting to go up. Cheap enough to call before a cycle.
pub fn has_pending(conn: &Connection) -> rusqlite::Result<bool> {
let pending: Option<i64> = conn
.query_row(
"SELECT 1 FROM notes WHERE dirty = 1
UNION ALL SELECT 1 FROM labels WHERE dirty = 1
UNION ALL SELECT 1 FROM pending_deletes LIMIT 1",
[],
|r| r.get(0),
)
.optional()?;
Ok(pending.is_some())
}
/// Send everything pending, in batches, applying each batch's results before the
/// next is collected.
pub async fn run(db: &Db, base_url: &str, token: &str) -> Result<PushSummary, String> {
let mut total = PushSummary::default();
loop {
let batch = {
let conn = db.0.lock().map_err(|e| e.to_string())?;
collect(&conn, BATCH).map_err(|e| e.to_string())?
};
if batch.is_empty() {
break;
}
let raw = client::push_changes(base_url, token, &batch).await?;
let results = parse_results(&raw)?;
let applied = {
let conn = db.0.lock().map_err(|e| e.to_string())?;
apply_results(&conn, &batch, &results).map_err(|e| e.to_string())?
};
// Everything rejected clears nothing, so the same batch would be collected
// again forever. Stop and report instead.
let progressed = applied.rejected < applied.sent;
total.absorb(applied);
if !progressed {
break;
}
if total.batches >= MAX_BATCHES {
return Err(format!(
"Stopped after {MAX_BATCHES} push batches without draining the queue."
));
}
}
if total.rejected > 0 {
log::warn!(
"push: {} change(s) rejected by the server: {}",
total.rejected,
total.errors.join("; ")
);
}
log::info!(
"push complete: {} sent ({} created, {} applied, {} kept, {} noop, {} rejected)",
total.sent,
total.created,
total.applied,
total.kept,
total.noop,
total.rejected
);
Ok(total)
}
/// Parse the server's reply. Kept next to the shapes it produces.
pub fn parse_results(raw: &str) -> Result<Vec<PushResult>, String> {
let parsed: PushResponse =
serde_json::from_str(raw).map_err(|e| format!("Couldn't read the push response: {e}"))?;
Ok(parsed.results)
}
#[cfg(test)]
mod tests {
use super::*;
use crate::local::schema;
use crate::local::store;
fn db() -> Connection {
let conn = Connection::open_in_memory().expect("in-memory db");
schema::migrate(&conn).expect("migrate");
conn
}
fn seed_note(conn: &Connection, id: &str, dirty: i64) {
conn.execute(
"INSERT INTO notes (id, title, body, color, kind, position, pinned, archived,
trashed, created_at, updated_at, sync_revision, dirty)
VALUES (?1, 'T', 'B', 'default', 'text', 0, 0, 0, 0,
'2026-07-26T00:00:00.000Z', '2026-07-26T00:00:00.000Z', 3, ?2)",
params![id, dirty],
)
.expect("seed note");
}
fn ok(status: &str, revision: Option<i64>) -> PushResult {
PushResult {
id: None,
entity: None,
status: status.to_string(),
sync_revision: revision,
error: None,
}
}
fn dirty_count(conn: &Connection) -> i64 {
conn.query_row("SELECT COUNT(*) FROM notes WHERE dirty = 1", [], |r| {
r.get(0)
})
.expect("count")
}
#[test]
fn collects_only_dirty_notes() {
let conn = db();
seed_note(&conn, "clean", 0);
seed_note(&conn, "dirty", 1);
let batch = collect(&conn, 100).expect("collect");
assert_eq!(batch.len(), 1);
assert_eq!(batch[0].id, "dirty");
assert_eq!(batch[0].op, "upsert");
}
#[test]
fn sends_only_manual_label_memberships() {
// Tag-sourced labels are re-derived server-side. Sending them as label_ids
// would convert them to manual assignments that survive removing the #tag.
let conn = db();
seed_note(&conn, "n1", 1);
for (id, name, via_tag) in [("manual", "Manual", 0), ("tagged", "Tagged", 1)] {
conn.execute(
"INSERT INTO labels (id, name, color, created_at, updated_at, dirty)
VALUES (?1, ?2, 'default', '2026-01-01', '2026-01-01', 0)",
params![id, name],
)
.expect("seed label");
conn.execute(
"INSERT INTO note_labels (note_id, label_id, via_tag) VALUES ('n1', ?1, ?2)",
params![id, via_tag],
)
.expect("seed membership");
}
let batch = collect(&conn, 100).expect("collect");
let note = batch.iter().find(|c| c.entity == "note").expect("note");
assert_eq!(note.label_ids.as_deref(), Some(&["manual".to_string()][..]));
}
#[test]
fn a_local_delete_becomes_a_delete_change() {
let conn = db();
seed_note(&conn, "n1", 0);
store::delete_forever(&conn, "n1").expect("delete");
let batch = collect(&conn, 100).expect("collect");
assert_eq!(batch.len(), 1);
assert_eq!(batch[0].op, "delete");
assert_eq!(batch[0].entity, "note");
assert_eq!(batch[0].id, "n1");
}
#[test]
fn applied_clears_dirty_and_records_the_revision() {
let conn = db();
seed_note(&conn, "n1", 1);
let batch = collect(&conn, 100).expect("collect");
apply_results(&conn, &batch, &[ok("applied", Some(42))]).expect("apply");
assert_eq!(dirty_count(&conn), 0);
let rev: i64 = conn
.query_row("SELECT sync_revision FROM notes WHERE id = 'n1'", [], |r| {
r.get(0)
})
.expect("revision");
assert_eq!(rev, 42);
}
#[test]
fn kept_clears_dirty_so_it_is_not_pushed_forever() {
// The server has a newer edit. Re-pushing would lose the same comparison
// every time; the following pull adopts the server's version instead.
let conn = db();
seed_note(&conn, "n1", 1);
let batch = collect(&conn, 100).expect("collect");
let summary = apply_results(&conn, &batch, &[ok("kept", Some(99))]).expect("apply");
assert_eq!(summary.kept, 1);
assert_eq!(dirty_count(&conn), 0);
}
#[test]
fn kept_rewinds_the_cursor_when_the_server_version_is_already_behind_it() {
// Clock skew: a genuinely later local edit can look older, so the server
// keeps its copy at a revision we have ALREADY consumed. Without a rewind the
// next pull skips it and the stale local copy stays on screen silently.
let conn = db();
seed_note(&conn, "n1", 1);
state::set_cursor(&conn, 100).expect("cursor");
let batch = collect(&conn, 100).expect("collect");
apply_results(&conn, &batch, &[ok("kept", Some(40))]).expect("apply");
assert_eq!(state::read(&conn).expect("state").last_cursor, 39);
}
#[test]
fn kept_leaves_the_cursor_alone_when_the_server_version_is_ahead() {
let conn = db();
seed_note(&conn, "n1", 1);
state::set_cursor(&conn, 10).expect("cursor");
let batch = collect(&conn, 100).expect("collect");
apply_results(&conn, &batch, &[ok("kept", Some(40))]).expect("apply");
assert_eq!(
state::read(&conn).expect("state").last_cursor,
10,
"the pending pull already covers it"
);
}
#[test]
fn rejected_stays_dirty_and_is_reported() {
let conn = db();
seed_note(&conn, "n1", 1);
let batch = collect(&conn, 100).expect("collect");
let mut bad = ok("rejected", None);
bad.error = Some("name in use".into());
let summary = apply_results(&conn, &batch, &[bad]).expect("apply");
assert_eq!(summary.rejected, 1);
assert_eq!(dirty_count(&conn), 1, "a rejected change must be retried");
assert!(summary.errors[0].contains("name in use"));
}
#[test]
fn an_acknowledged_delete_drops_its_tombstone() {
let conn = db();
seed_note(&conn, "n1", 0);
store::delete_forever(&conn, "n1").expect("delete");
let batch = collect(&conn, 100).expect("collect");
apply_results(&conn, &batch, &[ok("applied", Some(7))]).expect("apply");
assert!(!has_pending(&conn).expect("pending"));
}
#[test]
fn a_noop_delete_also_drops_its_tombstone() {
// Created and deleted entirely offline: the server never saw it.
let conn = db();
seed_note(&conn, "n1", 1);
store::delete_forever(&conn, "n1").expect("delete");
let batch = collect(&conn, 100).expect("collect");
apply_results(&conn, &batch, &[ok("noop", None)]).expect("apply");
assert!(!has_pending(&conn).expect("pending"));
}
#[test]
fn merging_labels_marks_the_affected_notes_dirty() {
// The membership change only reaches the server through the note itself.
let conn = db();
seed_note(&conn, "n1", 0);
for (id, name) in [("src", "Source"), ("dst", "Target")] {
conn.execute(
"INSERT INTO labels (id, name, color, created_at, updated_at, dirty)
VALUES (?1, ?2, 'default', '2026-01-01', '2026-01-01', 0)",
params![id, name],
)
.expect("seed label");
}
conn.execute(
"INSERT INTO note_labels (note_id, label_id, via_tag) VALUES ('n1', 'src', 0)",
[],
)
.expect("seed membership");
store::merge_labels(&conn, "src", "dst").expect("merge");
assert_eq!(dirty_count(&conn), 1, "the note's label set changed");
}
#[test]
fn has_pending_is_false_on_a_clean_store() {
let conn = db();
seed_note(&conn, "n1", 0);
assert!(!has_pending(&conn).expect("pending"));
}
#[test]
fn parse_results_reads_the_documented_shape() {
let results = parse_results(
r#"{"results":[{"id":"a","entity":"note","status":"created","sync_revision":44},
{"id":"b","entity":"label","status":"rejected","error":"name in use"}]}"#,
)
.expect("parse");
assert_eq!(results.len(), 2);
assert_eq!(results[0].status, "created");
assert_eq!(results[1].error.as_deref(), Some("name in use"));
}
#[test]
fn a_delete_change_serializes_without_note_fields() {
let change = Change::delete("note", "n1".into(), "2026-07-26T00:00:00.000Z".into());
let json = serde_json::to_string(&change).expect("serialize");
assert!(json.contains("\"op\":\"delete\""), "got {json}");
assert!(
!json.contains("body"),
"a delete carries no content: {json}"
);
}
}
+342
View File
@@ -0,0 +1,342 @@
//! The link record: which server this app is paired with, the device token that
//! authenticates to it, and how far it has consumed that server's change feed.
//!
//! One row, enforced by `CHECK (id = 1)` and seeded during migration, so every
//! operation here is an UPDATE — there is no create-or-missing case to handle.
//!
//! The token lives in the app-data SQLite file rather than an OS keyring on purpose:
//! the `keyring` crate needs libsecret/DBus on Linux, which adds a C dependency to a
//! binary that has to cross-compile, and fails outright on headless or minimal-WM
//! setups. Protecting the database file is the portable trade.
use rusqlite::{params, Connection};
use serde::Serialize;
/// The full link record, token included. Internal to the Rust side.
#[derive(Debug, Clone, Default, PartialEq, Eq)]
pub struct SyncState {
pub server_url: Option<String>,
pub device_token: Option<String>,
pub last_cursor: i64,
pub last_sync_at: Option<String>,
/// The linked server's trash-retention window, as it last advertised it. `None`
/// until a probe or sync has learned it.
pub server_retention_days: Option<i64>,
}
impl SyncState {
/// Linked means BOTH a server and a credential for it. Either one alone is a
/// half-written link that nothing can act on, so it must not read as linked.
pub fn is_linked(&self) -> bool {
self.server_url.is_some() && self.device_token.is_some()
}
}
/// What the UI is allowed to see.
///
/// Deliberately has no `device_token` field: this crosses into the webview, and a
/// long-lived bearer token has no business being reachable from page scripts.
#[derive(Debug, Clone, Serialize, PartialEq, Eq)]
pub struct Status {
pub linked: bool,
pub server_url: Option<String>,
pub last_cursor: i64,
pub last_sync_at: Option<String>,
}
impl From<&SyncState> for Status {
fn from(s: &SyncState) -> Self {
Status {
linked: s.is_linked(),
server_url: s.server_url.clone(),
last_cursor: s.last_cursor,
last_sync_at: s.last_sync_at.clone(),
}
}
}
/// Treat a blank string as absent, so a half-cleared row can't masquerade as linked.
fn present(value: Option<String>) -> Option<String> {
value.filter(|s| !s.trim().is_empty())
}
pub fn read(conn: &Connection) -> rusqlite::Result<SyncState> {
conn.query_row(
"SELECT server_url, device_token, last_cursor, last_sync_at, server_retention_days
FROM sync_state WHERE id = 1",
[],
|row| {
let cursor: Option<String> = row.get(2)?;
Ok(SyncState {
last_sync_at: present(row.get(3)?),
server_retention_days: row.get(4)?,
server_url: present(row.get(0)?),
device_token: present(row.get(1)?),
// Stored TEXT (schema) but used as an integer watermark. Absent or
// unparseable means "start from the beginning" — always the safe
// reading, because a redundant full sync costs time, never data,
// whereas a too-high cursor silently skips changes.
last_cursor: cursor.and_then(|c| c.trim().parse().ok()).unwrap_or(0),
})
},
)
}
/// Record a link.
///
/// Resets the change-feed cursor whenever the server differs from the one previously
/// linked. A cursor is only meaningful against the server that issued it; carrying
/// one across would silently skip every change on the new server below that
/// watermark — data loss that looks like a successful sync. Re-linking the SAME
/// server (after a token refresh, say) keeps the cursor, so a routine re-auth doesn't
/// force a full re-download.
pub fn set_link(conn: &Connection, server_url: &str, device_token: &str) -> rusqlite::Result<()> {
let keep_cursor = read(conn)?.server_url.as_deref() == Some(server_url);
conn.execute(
"UPDATE sync_state
SET server_url = ?1,
device_token = ?2,
last_cursor = CASE WHEN ?3 THEN last_cursor ELSE NULL END
WHERE id = 1",
params![server_url, device_token, keep_cursor],
)?;
Ok(())
}
/// Forget the server entirely.
///
/// Clears the cursor as well as the credentials: a cursor left behind would, on the
/// next link, be interpreted against a server that never issued it.
pub fn clear_link(conn: &Connection) -> rusqlite::Result<()> {
conn.execute(
"UPDATE sync_state
SET server_url = NULL, device_token = NULL, last_cursor = NULL,
last_sync_at = NULL, server_retention_days = NULL
WHERE id = 1",
[],
)?;
Ok(())
}
/// Remember the linked server's trash-retention window (0 = it never purges).
///
/// Refreshed on every sync rather than only at link time, so changing the setting on
/// the server reaches the desktop's Trash countdown on the next cycle instead of
/// waiting for someone to re-link.
pub fn set_server_retention(conn: &Connection, days: i64) -> rusqlite::Result<()> {
conn.execute(
"UPDATE sync_state SET server_retention_days = ?1 WHERE id = 1",
params![days],
)?;
Ok(())
}
/// The retention window in force on THIS device: the linked server's if we know it,
/// otherwise the caller's offline default. A linked device must never enforce or
/// advertise its own window over the server's.
pub fn effective_retention_days(conn: &Connection, offline_default: i64) -> rusqlite::Result<i64> {
let state = read(conn)?;
if !state.is_linked() {
return Ok(offline_default);
}
// Linked but the server hasn't told us yet (linked by an older build, or no sync
// has completed). Fall back to the default rather than claiming "kept forever".
Ok(state.server_retention_days.unwrap_or(offline_default))
}
/// Stamp a completed sync. The cursor can't stand in for this: it's a revision
/// watermark, and it doesn't move at all when a sync correctly finds nothing new —
/// so "synced a moment ago, no changes" would be indistinguishable from "never
/// synced" without it.
pub fn mark_synced(conn: &Connection, when: &str) -> rusqlite::Result<()> {
conn.execute(
"UPDATE sync_state SET last_sync_at = ?1 WHERE id = 1",
params![when],
)?;
Ok(())
}
/// Advance the consumed-change watermark. Called by the pull loop (M10.7b) only
/// after a page has been fully applied.
pub fn set_cursor(conn: &Connection, cursor: i64) -> rusqlite::Result<()> {
conn.execute(
"UPDATE sync_state SET last_cursor = ?1 WHERE id = 1",
params![cursor.to_string()],
)?;
Ok(())
}
pub fn status(conn: &Connection) -> rusqlite::Result<Status> {
Ok(Status::from(&read(conn)?))
}
#[cfg(test)]
mod tests {
use super::*;
use crate::local::schema;
fn db() -> Connection {
let conn = Connection::open_in_memory().expect("in-memory db");
schema::migrate(&conn).expect("migrate");
conn
}
#[test]
fn fresh_store_is_unlinked() {
let conn = db();
let state = read(&conn).expect("read");
assert_eq!(state, SyncState::default());
assert!(!state.is_linked());
assert_eq!(state.last_cursor, 0);
}
#[test]
fn link_round_trips() {
let conn = db();
set_link(&conn, "https://notes.example.com", "tok-1").expect("link");
let state = read(&conn).expect("read");
assert!(state.is_linked());
assert_eq!(
state.server_url.as_deref(),
Some("https://notes.example.com")
);
assert_eq!(state.device_token.as_deref(), Some("tok-1"));
}
#[test]
fn an_unlinked_device_uses_its_own_retention_window() {
let conn = db();
assert_eq!(effective_retention_days(&conn, 30).expect("read"), 30);
}
#[test]
fn a_linked_device_adopts_the_servers_window() {
// Including 0 — a server that keeps trash forever must not have this device
// showing a 30-day countdown that will never fire.
let conn = db();
set_link(&conn, "https://notes.example.com", "tok-1").expect("link");
set_server_retention(&conn, 0).expect("retention");
assert_eq!(effective_retention_days(&conn, 30).expect("read"), 0);
set_server_retention(&conn, 90).expect("retention");
assert_eq!(effective_retention_days(&conn, 30).expect("read"), 90);
}
#[test]
fn a_linked_device_that_hasnt_heard_yet_falls_back() {
// Linked by an older build, or no cycle has completed. The default is a
// safer guess than "forever", which would promise a note is being kept.
let conn = db();
set_link(&conn, "https://notes.example.com", "tok-1").expect("link");
assert_eq!(effective_retention_days(&conn, 30).expect("read"), 30);
}
#[test]
fn unlinking_forgets_the_servers_window() {
let conn = db();
set_link(&conn, "https://notes.example.com", "tok-1").expect("link");
set_server_retention(&conn, 90).expect("retention");
clear_link(&conn).expect("unlink");
assert_eq!(read(&conn).expect("read").server_retention_days, None);
assert_eq!(effective_retention_days(&conn, 30).expect("read"), 30);
}
#[test]
fn relinking_the_same_server_keeps_the_cursor() {
let conn = db();
set_link(&conn, "https://a.example.com", "tok-1").expect("link");
set_cursor(&conn, 4242).expect("cursor");
// e.g. the token was revoked and the user re-authenticated.
set_link(&conn, "https://a.example.com", "tok-2").expect("relink");
let state = read(&conn).expect("read");
assert_eq!(
state.last_cursor, 4242,
"a re-auth shouldn't force a full re-sync"
);
assert_eq!(state.device_token.as_deref(), Some("tok-2"));
}
#[test]
fn linking_a_different_server_resets_the_cursor() {
let conn = db();
set_link(&conn, "https://a.example.com", "tok-1").expect("link");
set_cursor(&conn, 4242).expect("cursor");
set_link(&conn, "https://b.example.com", "tok-2").expect("relink");
assert_eq!(
read(&conn).expect("read").last_cursor,
0,
"a cursor from another server would skip everything below it"
);
}
#[test]
fn unlink_clears_the_cursor_too() {
let conn = db();
set_link(&conn, "https://a.example.com", "tok-1").expect("link");
set_cursor(&conn, 99).expect("cursor");
clear_link(&conn).expect("unlink");
let state = read(&conn).expect("read");
assert!(!state.is_linked());
assert_eq!(state.last_cursor, 0);
assert!(state.server_url.is_none());
assert!(state.device_token.is_none());
}
#[test]
fn unlink_clears_the_last_sync_stamp() {
// Otherwise a freshly-linked server would claim it synced at a time that
// belonged to a different one.
let conn = db();
set_link(&conn, "https://a.example.com", "tok-1").expect("link");
mark_synced(&conn, "2026-07-26T04:00:00.000Z").expect("stamp");
assert!(read(&conn).expect("read").last_sync_at.is_some());
clear_link(&conn).expect("unlink");
assert!(read(&conn).expect("read").last_sync_at.is_none());
}
#[test]
fn half_written_link_is_not_linked() {
let conn = db();
conn.execute(
"UPDATE sync_state SET server_url = 'https://a.example.com' WHERE id = 1",
[],
)
.expect("partial write");
assert!(!read(&conn).expect("read").is_linked());
}
#[test]
fn blank_strings_count_as_absent() {
let conn = db();
conn.execute(
"UPDATE sync_state SET server_url = ' ', device_token = '' WHERE id = 1",
[],
)
.expect("blank write");
let state = read(&conn).expect("read");
assert!(!state.is_linked());
assert!(state.server_url.is_none());
}
#[test]
fn unparseable_cursor_falls_back_to_a_full_sync() {
let conn = db();
conn.execute(
"UPDATE sync_state SET last_cursor = 'garbage' WHERE id = 1",
[],
)
.expect("bad cursor");
assert_eq!(read(&conn).expect("read").last_cursor, 0);
}
#[test]
fn status_never_carries_the_token() {
let conn = db();
set_link(&conn, "https://a.example.com", "super-secret").expect("link");
let json = serde_json::to_string(&status(&conn).expect("status")).expect("serialize");
assert!(
!json.contains("super-secret"),
"token leaked to the webview: {json}"
);
assert!(json.contains("\"linked\":true"), "got {json}");
}
}
+166
View File
@@ -0,0 +1,166 @@
//! The delta-feed JSON shapes, exactly as `GET /api/sync/changes` sends them.
//!
//! Mirrors the server's serializers (`notes/serialize.py` + `serialize.py`) — see
//! `docs/sync.md` for the contract. Every field is `#[serde(default)]` or `Option`
//! so a NEWER server adding fields, or an older one omitting one, degrades to a
//! partial note rather than failing the whole page. Losing one attribute is
//! recoverable; refusing a page stalls sync permanently at that cursor.
use serde::Deserialize;
#[derive(Debug, Clone, Deserialize, Default)]
pub struct ChangesPage {
#[serde(default)]
pub notes: Vec<Note>,
#[serde(default)]
pub labels: Vec<Label>,
#[serde(default)]
pub cursor: i64,
#[serde(default)]
pub has_more: bool,
}
#[derive(Debug, Clone, Deserialize)]
pub struct Note {
pub id: String,
#[serde(default)]
pub title: Option<String>,
#[serde(default)]
pub body: String,
#[serde(default = "default_color")]
pub color: String,
#[serde(default = "default_kind")]
pub kind: String,
#[serde(default)]
pub position: i64,
#[serde(default)]
pub pinned: bool,
#[serde(default)]
pub archived: bool,
/// The server derives this from `deleted_at` — trash, NOT a tombstone.
#[serde(default)]
pub trashed: bool,
/// WHEN it was trashed. The trash-retention clock runs from here, so it has to be
/// the server's timestamp rather than anything this device invents. Absent from an
/// older server, which is why it's optional rather than required.
#[serde(default)]
pub deleted_at: Option<String>,
#[serde(default)]
pub remind_at: Option<String>,
#[serde(default)]
pub recurrence: Option<String>,
#[serde(default)]
pub created_at: Option<String>,
#[serde(default)]
pub updated_at: Option<String>,
#[serde(default)]
pub sync_revision: i64,
/// Set means the row was permanently purged: a content-less tombstone whose only
/// job is to tell clients to delete their copy.
#[serde(default)]
pub purged_at: Option<String>,
#[serde(default)]
pub labels: Vec<NoteLabel>,
#[serde(default)]
pub items: Vec<Item>,
#[serde(default)]
pub attachments: Vec<Attachment>,
#[serde(default)]
pub previews: Vec<Preview>,
}
impl Note {
pub fn is_tombstone(&self) -> bool {
self.purged_at.is_some()
}
}
/// A label as it appears attached to a note. Carries enough to materialize the label
/// row itself, which is what lets a membership be applied even if the label's own
/// delta hasn't arrived (see `pull::apply_page`).
#[derive(Debug, Clone, Deserialize)]
pub struct NoteLabel {
pub id: String,
#[serde(default)]
pub name: String,
#[serde(default = "default_color")]
pub color: String,
/// True when the membership came from a `#tag` in the body rather than a manual
/// assignment. Applied verbatim rather than re-derived — see `pull::apply_page`.
#[serde(default)]
pub via_tag: bool,
}
#[derive(Debug, Clone, Deserialize)]
pub struct Item {
pub id: String,
#[serde(default)]
pub text: String,
#[serde(default)]
pub checked: bool,
#[serde(default)]
pub position: i64,
}
#[derive(Debug, Clone, Deserialize)]
pub struct Attachment {
pub id: String,
#[serde(default)]
pub url: String,
#[serde(default)]
pub filename: Option<String>,
#[serde(default = "default_mime")]
pub mime: String,
#[serde(default)]
pub size: Option<i64>,
#[serde(default)]
pub sha256: Option<String>,
}
#[derive(Debug, Clone, Deserialize)]
pub struct Preview {
pub id: String,
#[serde(default)]
pub url: String,
#[serde(default)]
pub title: Option<String>,
#[serde(default)]
pub description: Option<String>,
#[serde(default)]
pub image_url: Option<String>,
#[serde(default)]
pub site_name: Option<String>,
}
#[derive(Debug, Clone, Deserialize)]
pub struct Label {
pub id: String,
#[serde(default)]
pub name: String,
#[serde(default = "default_color")]
pub color: String,
#[serde(default)]
pub sync_revision: i64,
#[serde(default)]
pub purged_at: Option<String>,
#[serde(default)]
pub created_at: Option<String>,
}
impl Label {
pub fn is_tombstone(&self) -> bool {
self.purged_at.is_some()
}
}
fn default_color() -> String {
"default".to_string()
}
fn default_kind() -> String {
"text".to_string()
}
fn default_mime() -> String {
"application/octet-stream".to_string()
}
+277
View File
@@ -0,0 +1,277 @@
//! In-app updates (M10.9).
//!
//! Two channels, because two audiences: `stable` follows tagged `v*` releases,
//! `dev` follows every green push. Each reads a `latest.json` published as an asset
//! on a release whose TAG NEVER CHANGES — verified necessary, because Forgejo has no
//! `/releases/latest/download/<asset>` route (it 404s), so "newest" cannot be named
//! in a URL. A fixed tag can.
//!
//! The feed lives on Fabled-Git rather than on a ThoughtSync server, deliberately:
//! this app is usable having never linked a server, and an install that can't reach
//! its own updates because it isn't paired with anything would contradict the whole
//! local-first premise.
//!
//! Updates are signed. The public half is baked into `tauri.conf.json`; the private
//! half exists only as a CI secret, and is generated by the operator — a release
//! signing key that has passed through anyone else's hands is not a signing key.
use serde::{Deserialize, Serialize};
use tauri::State;
use tauri_plugin_updater::UpdaterExt;
use crate::local::{store, Db};
/// Where the manifests live. Fixed tags, so these URLs are permanent.
const FEED_BASE: &str = "https://git.fabledsword.com/bvandeusen/thoughtsync/releases/download";
const CHANNEL_PREF: &str = "update_channel";
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
#[serde(rename_all = "snake_case")]
pub enum Channel {
Stable,
Dev,
}
impl Channel {
fn as_str(self) -> &'static str {
match self {
Channel::Stable => "stable",
Channel::Dev => "dev",
}
}
/// Anything unrecognized reads as `stable`. A corrupted or hand-edited value
/// must not silently opt someone into pre-release builds.
fn parse(raw: &str) -> Self {
match raw.trim() {
"dev" => Channel::Dev,
_ => Channel::Stable,
}
}
fn feed_url(self) -> String {
format!("{FEED_BASE}/{}/latest.json", self.as_str())
}
}
/// What the UI needs to describe the update situation without a second call.
#[derive(Debug, Serialize)]
pub struct UpdateStatus {
pub channel: Channel,
pub current_version: String,
/// The newer version on offer, or `None` when already up to date.
pub available: Option<String>,
pub notes: Option<String>,
/// False when this install can't apply an update to itself (see
/// `self_update_blocker`). The UI must not offer a button that cannot work.
pub can_install: bool,
/// Why not, in words meant for the person reading them.
pub blocked_reason: Option<String>,
}
/// Whether this install can replace itself, and if not, why.
///
/// The updater rewrites the running bundle in place, which only works for formats
/// that ARE a single self-contained file. On Linux that means the AppImage and
/// nothing else: a `.deb` or pacman install is owned by the package manager, and
/// silently overwriting files it tracks would corrupt its database. Tauri detects
/// the AppImage case by the `APPIMAGE` env var the runtime sets.
fn self_update_blocker() -> Option<String> {
if cfg!(target_os = "linux") && std::env::var_os("APPIMAGE").is_none() {
return Some(
"This copy was installed by your package manager, so it updates the same \
way — `apt upgrade`, `pacman -Syu`, or re-running the install script. \
In-app updates work on the AppImage build."
.to_string(),
);
}
None
}
fn read_channel(db: &State<'_, Db>) -> Result<Channel, String> {
let conn = db.0.lock().map_err(|e| e.to_string())?;
let raw = store::pref(&conn, CHANNEL_PREF).map_err(|e| e.to_string())?;
Ok(raw
.as_deref()
.map(Channel::parse)
.unwrap_or(Channel::Stable))
}
#[tauri::command]
pub fn update_channel_get(db: State<'_, Db>) -> Result<Channel, String> {
read_channel(&db)
}
#[tauri::command]
pub fn update_channel_set(channel: Channel, db: State<'_, Db>) -> Result<Channel, String> {
let conn = db.0.lock().map_err(|e| e.to_string())?;
store::set_pref(&conn, CHANNEL_PREF, channel.as_str()).map_err(|e| e.to_string())?;
log::info!("update channel set to {}", channel.as_str());
Ok(channel)
}
/// Ask the feed whether there's something newer. Never installs anything.
#[tauri::command]
pub async fn update_check(
app: tauri::AppHandle,
db: State<'_, Db>,
) -> Result<UpdateStatus, String> {
let channel = read_channel(&db)?;
let current_version = app.package_info().version.to_string();
let blocked_reason = self_update_blocker();
let url = channel
.feed_url()
.parse()
.map_err(|e| format!("the update feed address is malformed: {e}"))?;
let updater = app
.updater_builder()
.endpoints(vec![url])
.map_err(|e| e.to_string())?
.build()
.map_err(|e| format!("updates aren't configured for this build: {e}"))?;
// A missing manifest is the ordinary state of a channel nobody has published to
// yet — report it as "nothing available" rather than as a failure to act on.
// Matched on the message rather than an error variant so this doesn't break on a
// plugin minor that renames one.
let found = match updater.check().await {
Ok(found) => found,
Err(e) => {
let detail = e.to_string();
if is_missing_manifest(&detail) {
None
} else {
return Err(describe_check_error(&detail));
}
}
};
Ok(UpdateStatus {
channel,
current_version,
available: found.as_ref().map(|u| u.version.clone()),
notes: found.as_ref().and_then(|u| u.body.clone()),
can_install: found.is_some() && blocked_reason.is_none(),
blocked_reason,
})
}
/// Download, verify and apply the update, then relaunch.
///
/// Refuses up front on an install that can't replace itself, rather than failing
/// halfway through with a permissions error nobody can interpret.
#[tauri::command]
pub async fn update_install(app: tauri::AppHandle, db: State<'_, Db>) -> Result<(), String> {
if let Some(reason) = self_update_blocker() {
return Err(reason);
}
let channel = read_channel(&db)?;
let url = channel
.feed_url()
.parse()
.map_err(|e| format!("the update feed address is malformed: {e}"))?;
let updater = app
.updater_builder()
.endpoints(vec![url])
.map_err(|e| e.to_string())?
.build()
.map_err(|e| format!("updates aren't configured for this build: {e}"))?;
let found = updater
.check()
.await
.map_err(|e| describe_check_error(&e.to_string()))?;
let Some(update) = found else {
return Err("There's no update to install — this is already the newest build.".to_string());
};
log::info!(
"installing update {} over {} ({} channel)",
update.version,
app.package_info().version,
channel.as_str()
);
update
.download_and_install(|_chunk, _total| {}, || {})
.await
.map_err(|e| format!("the update couldn't be installed: {e}"))?;
// Only reached if the install succeeded. `restart` diverges, so it's the tail.
log::info!("update installed; restarting");
app.restart()
}
/// Whether a check failure just means "this channel has nothing published yet".
fn is_missing_manifest(detail: &str) -> bool {
let lower = detail.to_lowercase();
lower.contains("404") || lower.contains("not found")
}
/// Turn a check failure into something worth reading. The default rendering of a
/// transport error names the URL and nothing else, which tells a user nothing about
/// what they could do next.
fn describe_check_error(detail: &str) -> String {
let lower = detail.to_lowercase();
if lower.contains("error sending request") || lower.contains("dns") || lower.contains("connect")
{
return "Couldn't reach the update server — check your connection.".to_string();
}
format!("Couldn't check for updates: {detail}")
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn each_channel_has_its_own_fixed_feed() {
assert!(Channel::Stable.feed_url().ends_with("/stable/latest.json"));
assert!(Channel::Dev.feed_url().ends_with("/dev/latest.json"));
assert_ne!(Channel::Stable.feed_url(), Channel::Dev.feed_url());
}
#[test]
fn the_feed_is_https() {
// The manifest names the URL and signature of a binary that is about to
// replace this one. The signature is what actually protects it, but there's
// no reason to hand an attacker the manifest to tamper with in the first place.
assert!(Channel::Stable.feed_url().starts_with("https://"));
assert!(Channel::Dev.feed_url().starts_with("https://"));
}
#[test]
fn an_unknown_channel_falls_back_to_stable() {
// A corrupted or hand-edited pref must never silently opt someone into
// pre-release builds — the safe default is the conservative one.
assert_eq!(Channel::parse("dev"), Channel::Dev);
assert_eq!(Channel::parse("stable"), Channel::Stable);
assert_eq!(Channel::parse("nightly"), Channel::Stable);
assert_eq!(Channel::parse(""), Channel::Stable);
}
#[test]
fn an_unpublished_channel_is_not_reported_as_a_failure() {
// Before the first publish, or on a channel nobody uses, the feed simply
// isn't there. That's "you're up to date", not something to alarm anyone with.
assert!(is_missing_manifest("http status: 404 Not Found"));
assert!(is_missing_manifest("Release Not Found"));
assert!(!is_missing_manifest("invalid signature"));
}
#[test]
fn an_unreachable_server_reads_as_a_connection_problem() {
let msg = describe_check_error("error sending request for url (https://…)");
assert!(msg.contains("connection"), "got {msg}");
// Anything unrecognized still surfaces its detail rather than being swallowed.
assert!(describe_check_error("invalid signature").contains("invalid signature"));
}
#[test]
fn the_channel_name_round_trips() {
for channel in [Channel::Stable, Channel::Dev] {
assert_eq!(Channel::parse(channel.as_str()), channel);
}
}
}
+11
View File
@@ -27,6 +27,17 @@
"csp": null
}
},
"plugins": {
"updater": {
"endpoints": [
"https://git.fabledsword.com/bvandeusen/thoughtsync/releases/download/stable/latest.json"
],
"pubkey": "dW50cnVzdGVkIGNvbW1lbnQ6IG1pbmlzaWduIHB1YmxpYyBrZXk6IDkwRTk2RkVBMkY2RDlCNkEKUldScW0yMHY2bS9wa0VBdWFpM3c1d2trQnlNVUJXUUtwZXBzQjduM3FRVzdGa3dXNGxObkZFV28K",
"windows": {
"installMode": "passive"
}
}
},
"bundle": {
"active": true,
"targets": ["deb", "appimage"],
+5 -1
View File
@@ -5,7 +5,11 @@
# docker compose -f docker-compose.dev.yml up
#
# Then open http://localhost:5173 (the Vite dev server proxies /api to the backend).
# (docker-compose.yml, by contrast, builds + runs the production image on :5000.)
#
# docker-compose.yml — the DEFAULT file — is the production stack instead: it pulls
# the published image, keeps Postgres off the host network, and expects a .env. This
# one builds nothing and is deliberately insecure-by-convenience (weak password,
# Postgres published on 5432) because it is meant for a laptop, not a deployment.
services:
db:
image: postgres:16-alpine
+76 -12
View File
@@ -1,34 +1,98 @@
# Local two-service stack (app + Postgres). Provided for convenience — per family
# rule 12 the agent does NOT start this; run it yourself with `docker compose up`.
# ThoughtSync — PRODUCTION stack (app + Postgres).
#
# This is the default compose file: `docker compose up -d` runs a real deployment
# from the published image. Development lives in docker-compose.dev.yml (hot-reload,
# builds from source).
#
# cp .env.example .env # then set POSTGRES_PASSWORD
# docker compose up -d
#
# Per family rule 12 the agent does NOT start this — run it yourself.
#
# Upgrades: docker compose pull && docker compose up -d
# Rollback: set THOUGHTSYNC_TAG to a commit sha in .env, then the same two commands.
# Every push publishes an immutable :<sha> image for exactly this.
#
# Schema migrations run automatically at container start (see the Dockerfile CMD),
# so an upgrade is just a pull and a restart. Take a backup first anyway:
#
# docker compose exec -T db pg_dump -U thoughtsync thoughtsync > backup.sql
#
# Attachments are files, not rows — they live in the `thoughtsync-data` volume and
# a pg_dump does NOT contain them. Back up both or you'll restore notes whose images
# are gone.
services:
db:
image: postgres:16-alpine
restart: unless-stopped
environment:
POSTGRES_USER: thoughtsync
POSTGRES_PASSWORD: thoughtsync
POSTGRES_DB: thoughtsync
POSTGRES_USER: ${POSTGRES_USER:-thoughtsync}
# No default on purpose. A production compose that ships a known password is
# how self-hosted databases end up in search engines; compose fails fast here
# instead, with the message below.
POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:?set POSTGRES_PASSWORD in .env — see .env.example}
POSTGRES_DB: ${POSTGRES_DB:-thoughtsync}
volumes:
# Volume names kept from the previous compose file so an existing deployment
# upgrades in place. Renaming them would silently start against an empty
# database while the old one sat there, orphaned and looking like data loss.
- thoughtsync-db:/var/lib/postgresql/data
ports:
- "5432:5432"
# Deliberately NOT published to the host. The app reaches Postgres over the
# compose network; exposing 5432 only widens the attack surface. If you need
# psql, `docker compose exec db psql -U thoughtsync` gets you there without it.
healthcheck:
test: ["CMD-SHELL", "pg_isready -U thoughtsync"]
interval: 5s
test: ["CMD-SHELL", "pg_isready -U ${POSTGRES_USER:-thoughtsync} -d ${POSTGRES_DB:-thoughtsync}"]
interval: 10s
timeout: 5s
retries: 10
start_period: 30s
logging: &logging
driver: json-file
options:
# Unbounded container logs are a slow-motion disk-full outage on a
# long-running self-hosted box.
max-size: "10m"
max-file: "3"
app:
build: .
# :latest tracks `main`. Set THOUGHTSYNC_TAG=dev in .env to follow the
# development line instead, or a commit sha to pin exactly.
image: git.fabledsword.com/bvandeusen/thoughtsync:${THOUGHTSYNC_TAG:-latest}
restart: unless-stopped
depends_on:
db:
condition: service_healthy
environment:
THOUGHTSYNC_DATABASE_URL: postgresql+asyncpg://thoughtsync:thoughtsync@db:5432/thoughtsync
# The only required application setting. Everything else a person might want
# to tune lives in the admin Settings UI, backed by the database (rule 25).
THOUGHTSYNC_DATABASE_URL: postgresql+asyncpg://${POSTGRES_USER:-thoughtsync}:${POSTGRES_PASSWORD}@db:5432/${POSTGRES_DB:-thoughtsync}
volumes:
# Uploaded attachments. /var/thoughtsync is fixed in the app (Config.DATA_DIR),
# not configurable — mount it or lose every image on container recreation.
- thoughtsync-data:/var/thoughtsync
ports:
- "5000:5000"
# Default binds every interface, which is what lets desktop clients on the LAN
# reach it. Behind a reverse proxy, set THOUGHTSYNC_BIND=127.0.0.1 so only the
# proxy can talk to it.
- "${THOUGHTSYNC_BIND:-0.0.0.0}:${THOUGHTSYNC_PORT:-5000}:5000"
healthcheck:
# python rather than curl: the runtime image is python:3.12-slim and carries no
# HTTP client binary. Hits the app's own /api/health.
test:
- CMD
- python
- -c
- |
import sys, urllib.request
sys.exit(0 if urllib.request.urlopen("http://127.0.0.1:5000/api/health", timeout=5).status == 200 else 1)
interval: 30s
timeout: 10s
retries: 5
# Generous: the container waits for Postgres and runs migrations before it
# serves anything, and a first boot builds the whole schema.
start_period: 60s
logging: *logging
volumes:
thoughtsync-db:
+88 -7
View File
@@ -13,6 +13,68 @@ All sync endpoints live under `/api/sync`. Everything is **owner-scoped** and
> operator on deploy, not in CI. Pure logic (LWW comparator, paging cursor,
> token hashing) is unit-tested.
## Protocol versioning — the compatibility handshake
Clients and servers update on their own schedules; a self-hosted server can sit on
an older release than the desktop app for months. So the wire protocol is
versioned **separately from either program's release version**, and each side
declares two numbers: what it speaks, and the oldest counterpart it accepts.
| | server (`src/thoughtsync/sync.py`) | client (`desktop/src-tauri/src/sync/compat.rs`) |
|---|---|---|
| speaks | `SYNC_PROTOCOL_VERSION` | `CLIENT_PROTOCOL_VERSION` |
| accepts down to | `MIN_CLIENT_PROTOCOL_VERSION` | `MIN_SERVER_PROTOCOL_VERSION` |
The server publishes its half on the **public, unauthenticated** `GET /api/config`
— a client must be able to ask "can I talk to you?" before it holds a device
token, or even has an account:
```json
{ "site_name": "...", "version": "0.1.0",
"sync_protocol_version": 1,
"min_client_protocol_version": 1,
"sync_features": ["notes", "labels", "attachments", "tombstones", "revisions"],
"trash_retention_days": 30 }
```
The client identifies itself on every request with
`X-ThoughtSync-Client: thoughtsync-desktop/<app version>` and
`X-ThoughtSync-Protocol: <n>`.
### `sync_features` — why versions alone aren't enough
A version number can only say "newer" or "older". `sync_features` names
capabilities, so a client tests for the one it needs instead of inferring it from
a number. That is what keeps an **additive** change from forcing a lockstep
upgrade: a newer client meeting an older server drops the missing feature and
syncs everything else.
### The policy
- **Any wire change** → bump `SYNC_PROTOCOL_VERSION`.
- **Additive change** (a new field, a new capability) → add a `sync_features`
name. Do **not** raise a minimum. Old clients keep working.
- **Breaking change only** → raise `MIN_CLIENT_PROTOCOL_VERSION` (or the client's
`MIN_SERVER_PROTOCOL_VERSION`). This is the switch that hard-blocks the other
side, so it is the one to be stingy with.
- Never gate behavior on the *release* version (`version`) — it's for display.
### The three outcomes
The client evaluates the advertisement (`compat::evaluate`) and gets exactly one
of:
- **ok** — full parity; sync everything.
- **degraded** — safe to sync, but named capabilities are unavailable here; the UI
says which.
- **incompatible** — do not sync. Carries `client_must_update` so the message can
point at the side that can actually fix it, rather than just saying
"incompatible".
A server that predates this handshake sends no protocol fields at all. That is
treated as **incompatible (update the server)** — deliberately not as a parse
error, which would look to the user like they mistyped the URL.
## Authentication — device bearer tokens
Native clients authenticate with a long-lived **device token**, not a session
@@ -60,13 +122,31 @@ as `?since=`. `since=0` (or absent) is a **full initial sync**.
Two levels, both propagate:
- **Trash** — `deleted_at` is a normal field. A trashed note still syncs with its
content; the client shows it in its Trash. Restoring clears `deleted_at`.
- **Trash** — `deleted_at` is a normal field, and it's **on the wire**: a trashed
note still syncs with its content, the client shows it in its Trash, and the
timestamp is what the client counts the retention window against. Restoring
clears it.
- **Purge (permanent delete)** — becomes a **content-less tombstone**: `purged_at`
is set, title/body/items/labels/attachments are cleared/removed, and the row is
kept. A client seeing `purged_at != null` deletes the row from its local store.
Tombstones are retained indefinitely (cheap for a personal store); revisit if
they ever grow large.
is set, title/body/items/labels/attachments/previews/revisions are cleared or
removed, and the row is kept. A client seeing `purged_at != null` deletes the row
from its local store. `deleted_at` deliberately SURVIVES a purge, so ordinary
server-side queries (`deleted_at IS NULL`) never see a tombstone as a live note.
Tombstones themselves are retained indefinitely (cheap for a personal store);
revisit if they ever grow large.
### Retention — trash expires
A trashed note is purged automatically once it is older than the server's
`trash_retention_days` setting (default **30**, `0` = keep forever), advertised on
`/api/config` so a client can show the countdown. A background sweep on the server
does the work; clients learn about it as ordinary tombstones and need no special
handling.
**A linked client must not run its own expiry.** The server owns the policy — one
clock, one window. A client that purged on its own schedule could destroy a note
the server was deliberately keeping and then push that delete upstream. An
*unlinked* client (offline-only, no server to defer to) expires its own trash on
its own default, which is the only case where nothing else can.
## Pull — `GET /api/sync/changes`
@@ -76,7 +156,8 @@ Response:
```json
{
"notes": [ { "...full note...", "sync_revision": 42, "purged_at": null } ],
"notes": [ { "...full note...", "trashed": false, "deleted_at": null,
"sync_revision": 42, "purged_at": null } ],
"labels": [ { "id": "...", "name": "...", "color": "...",
"sync_revision": 43, "purged_at": null, "created_at": "..." } ],
"cursor": 43,
+12
View File
@@ -12,6 +12,7 @@ import CommandPalette from "./CommandPalette.vue";
import Icon from "./Icon.vue";
import ImportNotes from "./ImportNotes.vue";
import LabelsModal from "./LabelsModal.vue";
import { isDesktop } from "../desktop/bridge";
import { facetsToQuery } from "../notes/facets";
import { NOTE_SWATCH_CLASSES, type NoteColor } from "../notes/colors";
@@ -23,6 +24,8 @@ const labels = useLabelsStore();
const savedFilters = useSavedFiltersStore();
const reminders = useReminderStore();
const ui = useUiStore();
// Sync is a desktop-app concern: the web build already IS the server's UI.
const desktopApp = isDesktop();
async function removeView(f: SavedFilter) {
if (!window.confirm(`Delete the "${f.name}" view?`)) return;
@@ -260,6 +263,15 @@ async function signOut() {
<span class="hidden text-sm text-neutral-500 md:inline dark:text-neutral-400">{{
session.user?.display_name
}}</span>
<RouterLink
v-if="desktopApp"
to="/sync"
class="icon-btn"
title="Sync"
aria-label="Sync"
>
<Icon name="sync" />
</RouterLink>
<RouterLink to="/account" class="icon-btn" title="Linked devices" aria-label="Linked devices">
<Icon name="device" />
</RouterLink>
+1
View File
@@ -26,6 +26,7 @@ const paths: Record<string, string> = {
download: '<path d="M21 15v4a2 2 0 0 1-2 2H5a2 2 0 0 1-2-2v-4"/><polyline points="7 10 12 15 17 10"/><line x1="12" x2="12" y1="15" y2="3"/>',
upload: '<path d="M21 15v4a2 2 0 0 1-2 2H5a2 2 0 0 1-2-2v-4"/><polyline points="17 8 12 3 7 8"/><line x1="12" x2="12" y1="3" y2="15"/>',
device: '<rect width="14" height="20" x="5" y="2" rx="2" ry="2"/><path d="M12 18h.01"/>',
sync: '<path d="M3 12a9 9 0 0 1 9-9 9.75 9.75 0 0 1 6.74 2.74L21 8"/><path d="M21 3v5h-5"/><path d="M21 12a9 9 0 0 1-9 9 9.75 9.75 0 0 1-6.74-2.74L3 16"/><path d="M3 21v-5h5"/>',
paperclip: '<path d="m21.44 11.05-9.19 9.19a6 6 0 0 1-8.49-8.49l8.57-8.57A4 4 0 1 1 18 8.84l-8.59 8.57a2 2 0 0 1-2.83-2.83l8.49-8.48"/>',
link: '<path d="M10 13a5 5 0 0 0 7.54.54l3-3a5 5 0 0 0-7.07-7.07l-1.72 1.71"/><path d="M14 11a5 5 0 0 0-7.54-.54l-3 3a5 5 0 0 0 7.07 7.07l1.71-1.71"/>',
filter: '<polygon points="22 3 2 3 10 12.46 10 19 14 21 14 12.46 22 3"/>',
+39 -1
View File
@@ -14,7 +14,8 @@ import Icon from "./Icon.vue";
import LinkPreview from "./LinkPreview.vue";
import MarkdownText from "./MarkdownText.vue";
import NoteChecklist from "./NoteChecklist.vue";
import { formatReminder, isOverdue } from "../notes/datetime";
import { formatReminder, formatTrashCountdown, isOverdue, trashDaysLeft } from "../notes/datetime";
import { useConfigStore } from "../stores/config";
const props = defineProps<{ note: Note; reorderable?: boolean; active?: boolean }>();
const emit = defineEmits<{
@@ -24,6 +25,17 @@ const emit = defineEmits<{
(e: "drop", note: Note): void;
}>();
const notes = useNotesStore();
const config = useConfigStore();
// --- Retention countdown. A note in Trash is on a clock, and the card is the only
// place someone browsing Trash would ever find that out in time to restore it.
// Null whenever nothing is going to happen: not trashed, or retention turned off. ---
const trashDays = computed(() =>
props.note.trashed ? trashDaysLeft(props.note.deleted_at, config.trashRetentionDays) : null,
);
const trashCountdown = computed(() => formatTrashCountdown(trashDays.value));
// Same red the overdue reminder uses — the last few days are worth noticing.
const trashUrgent = computed(() => trashDays.value !== null && trashDays.value <= 3);
// The card previews the first image inline; non-image files show as compact chips.
const firstImage = computed(() => props.note.attachments.find((a) => a.mime.startsWith("image/")));
@@ -236,6 +248,32 @@ onBeforeUnmount(() => document.removeEventListener("mousedown", onDocMousedown))
</span>
</div>
<div v-if="trashCountdown" class="mt-2">
<span
class="inline-flex items-center gap-1 rounded-full px-2 py-0.5 text-xs"
:class="
trashUrgent
? 'bg-red-100 text-red-700 dark:bg-red-950/50 dark:text-red-300'
: 'bg-black/5 text-neutral-600 dark:bg-white/10 dark:text-neutral-300'
"
:title="`Permanently deleted ${config.trashRetentionDays} days after it was trashed`"
>
<svg
class="h-3 w-3"
viewBox="0 0 24 24"
fill="none"
stroke="currentColor"
stroke-width="2"
stroke-linecap="round"
stroke-linejoin="round"
>
<circle cx="12" cy="12" r="10" />
<polyline points="12 6 12 12 16 14" />
</svg>
{{ trashCountdown }}
</span>
</div>
<!-- Toolbar overlays the card's top-right on hover/focus as a floating pill
(window-control style) instead of reserving a permanent row so at rest
the card is content-sized with even padding, not text pinned to the top
+1
View File
@@ -65,6 +65,7 @@ const draftNote = computed<Note>(() => ({
pinned: false,
archived: false,
trashed: false,
deleted_at: null,
remind_at: null,
recurrence: null,
labels: labelList.value,
+128
View File
@@ -47,3 +47,131 @@ export const desktop = {
integrate: () => invoke<IntegrationStatus>("integrate_desktop"),
unintegrate: () => invoke<IntegrationStatus>("unintegrate_desktop"),
};
// --- opt-in server sync (M10.7) ----------------------------------------------
// The desktop app is local-first: none of this runs unless the user links a server,
// and the app is fully usable having never done so.
/** Never carries the device token — that stays on the Rust side, out of the webview. */
export interface SyncStatus {
linked: boolean;
server_url: string | null;
last_cursor: number;
last_sync_at: string | null;
}
export interface ServerInfo {
site_name: string | null;
version: string | null;
sync_protocol_version: number | null;
min_client_protocol_version: number | null;
sync_features: string[];
}
/**
* The M10.6 handshake verdict. `incompatible` carries `client_must_update` so the
* message can name which side has to change rather than just saying "incompatible".
*/
export type Compatibility =
| { status: "ok" }
| { status: "degraded"; unavailable: string[] }
| { status: "incompatible"; reason: string; client_must_update: boolean };
export interface ProbeResult {
base_url: string;
server: ServerInfo;
compatibility: Compatibility;
}
export interface Identity {
id: string;
email: string;
display_name: string;
}
export interface LinkResult {
status: SyncStatus;
identity: Identity;
compatibility: Compatibility;
}
export interface PushSummary {
batches: number;
sent: number;
created: number;
applied: number;
kept: number;
noop: number;
rejected: number;
errors: string[];
}
export interface PullSummary {
pages: number;
notes_applied: number;
notes_deleted: number;
labels_applied: number;
labels_deleted: number;
cursor: number;
clobbered_dirty: number;
blobs_downloaded: number;
/** Attachments whose bytes didn't arrive. Retried next sync, never fatal. */
blobs_failed: number;
}
export interface SyncOutcome {
push: PushSummary;
pull: PullSummary;
status: SyncStatus;
}
/** Either a password login or a token pasted from the web app's Linked devices. */
export interface LinkInput {
url: string;
email?: string;
password?: string;
token?: string;
name?: string;
}
export const sync = {
/** Ask who's at an address without committing to anything. */
probe: (url: string) => invoke<ProbeResult>("sync_probe", { url }),
link: (input: LinkInput) => invoke<LinkResult>("sync_link", { input }),
unlink: () => invoke<SyncStatus>("sync_unlink"),
status: () => invoke<SyncStatus>("sync_status"),
/**
* One full cycle: push, then pull. There is deliberately no bare "pull" — pulling
* without pushing first overwrites unsent local edits.
*/
now: () => invoke<SyncOutcome>("sync_now"),
hasPending: () => invoke<boolean>("sync_has_pending"),
};
// --- In-app updates (M10.9) --------------------------------------------------
/** `stable` follows tagged releases; `dev` follows every green build. */
export type UpdateChannel = "stable" | "dev";
export interface UpdateStatus {
channel: UpdateChannel;
current_version: string;
/** The newer version on offer, or null when already up to date. */
available: string | null;
notes: string | null;
/** False when this install can't replace itself — see `blocked_reason`. */
can_install: boolean;
blocked_reason: string | null;
}
export const updates = {
channel: () => invoke<UpdateChannel>("update_channel_get"),
setChannel: (channel: UpdateChannel) => invoke<UpdateChannel>("update_channel_set", { channel }),
/** Ask the feed what's out there. Never installs anything. */
check: () => invoke<UpdateStatus>("update_check"),
/**
* Download, verify, apply, relaunch. Resolves only on failure — a success
* restarts the app out from under the caller.
*/
install: () => invoke<void>("update_install"),
};
+33
View File
@@ -49,3 +49,36 @@ export function formatLocalDay(d: Date): string {
const pad = (n: number) => String(n).padStart(2, "0");
return `${d.getFullYear()}-${pad(d.getMonth() + 1)}-${pad(d.getDate())}`;
}
// --- Trash retention. The server permanently deletes a trashed note once it's older
// than `trash_retention_days` (0 = keep forever). Counting down from the note's own
// deleted_at is what turns that from a surprise into a policy: a card in Trash can
// say how long it has left while there's still time to restore it. ---
const MS_PER_DAY = 24 * 60 * 60 * 1000;
// Whole days a trashed note has left, or null when nothing will happen to it
// (retention off, or the note isn't trashed).
//
// Rounds DOWN deliberately. Rounding up would report "1 day left" for a note with
// ten minutes on the clock — overstating the time remaining is the one error here
// that actually costs someone a note.
export function trashDaysLeft(
deletedAt: string | null | undefined,
retentionDays: number,
now: number = Date.now(),
): number | null {
if (!deletedAt || retentionDays <= 0) return null;
const trashedAt = new Date(deletedAt).getTime();
if (Number.isNaN(trashedAt)) return null;
const remaining = trashedAt + retentionDays * MS_PER_DAY - now;
return remaining <= 0 ? 0 : Math.floor(remaining / MS_PER_DAY);
}
// The countdown as the card shows it. "" when there's nothing to say.
export function formatTrashCountdown(daysLeft: number | null): string {
if (daysLeft === null) return "";
if (daysLeft <= 0) return "Deletes today";
if (daysLeft === 1) return "1 day left";
return `${daysLeft} days left`;
}
+12 -1
View File
@@ -1,7 +1,7 @@
import { createRouter, createWebHistory } from "vue-router";
import { useSessionStore } from "../stores/session";
import { useConfigStore } from "../stores/config";
import { logEvent } from "../desktop/bridge";
import { isDesktop, logEvent } from "../desktop/bridge";
// One-time boot diagnostic: the first navigation is where config + session resolve,
// so it's the moment that tells us whether the app got past its startup gate.
@@ -32,6 +32,14 @@ const router = createRouter({
component: () => import("../views/SettingsView.vue"),
meta: { requiresAuth: true, requiresAdmin: true },
},
{
// Desktop only: connect this app to a server. Meaningless in the web build,
// which IS a server's UI — there's nothing for it to link to.
path: "/sync",
name: "sync",
component: () => import("../views/SyncView.vue"),
meta: { requiresAuth: true, requiresDesktop: true },
},
{
// Per-user account: linked devices (native-client sync tokens). Any user.
path: "/account",
@@ -74,6 +82,9 @@ router.beforeEach(async (to) => {
if (to.meta.requiresAdmin && !session.user?.is_admin) {
return { name: "board" };
}
if (to.meta.requiresDesktop && !isDesktop()) {
return { name: "board" };
}
if (to.name === "register" && !config.allowRegistration) {
return { name: "login" };
}
+8 -1
View File
@@ -7,6 +7,8 @@ export interface PublicConfig {
allow_registration: boolean;
version: string;
enable_url_unfurl: boolean;
// How many days a note survives in Trash before the server purges it. 0 = forever.
trash_retention_days: number;
}
// Public, unauthenticated app config (site name, whether signups are open).
@@ -15,6 +17,10 @@ export const useConfigStore = defineStore("config", () => {
const allowRegistration = ref(true);
const version = ref("");
const enableUrlUnfurl = ref(true);
// Mirrors the server default (settings.REGISTRY). Only used if /api/config is
// unreachable — and 30 is a safer stand-in than 0, since claiming "kept forever"
// when the server is actually purging is the wrong way to be wrong.
const trashRetentionDays = ref(30);
const loaded = ref(false);
async function load(): Promise<void> {
@@ -25,6 +31,7 @@ export const useConfigStore = defineStore("config", () => {
allowRegistration.value = cfg.allow_registration;
version.value = cfg.version;
enableUrlUnfurl.value = cfg.enable_url_unfurl ?? true;
trashRetentionDays.value = cfg.trash_retention_days ?? 30;
} catch {
// Keep defaults if the config endpoint is unreachable.
} finally {
@@ -37,5 +44,5 @@ export const useConfigStore = defineStore("config", () => {
await load();
}
return { siteName, allowRegistration, version, enableUrlUnfurl, loaded, load, reload };
return { siteName, allowRegistration, version, enableUrlUnfurl, trashRetentionDays, loaded, load, reload };
});
+13
View File
@@ -47,6 +47,19 @@ export const useLabelsStore = defineStore("labels", () => {
}
async function remove(id: string): Promise<void> {
// Labels have no trash of their own — a label is organization, not content, so
// the reversible middle step notes get would be ceremony. But deleting one is
// still irreversible and now reaches every synced device, so it asks first. The
// notes themselves survive; only the membership goes, which is the part people
// most need reassuring about.
const label = items.value.find((lb) => lb.id === id);
const subject = label ? `the label "${label.name}"` : "this label";
const confirmed = window.confirm(
`Delete ${subject}?\n\n` +
"It will be removed from every note that uses it, on every device you sync " +
"with. The notes themselves are kept.",
);
if (!confirmed) return;
await repo.labels.remove(id);
items.value = items.value.filter((lb) => lb.id !== id);
}
+15
View File
@@ -76,6 +76,9 @@ export interface Note {
pinned: boolean;
archived: boolean;
trashed: boolean;
// When it was trashed (null unless trashed). The Trash view counts the retention
// window from here to show how long the note has left before it's purged.
deleted_at: string | null;
remind_at: string | null;
recurrence: string | null;
labels: NoteLabel[];
@@ -249,6 +252,18 @@ export const useNotesStore = defineStore("notes", () => {
}
async function deleteForever(id: string): Promise<void> {
// Guarded HERE rather than at the call sites (NoteCard and NoteEditor both offer
// it) so the two can't drift on the one action with no undo. Trash is the
// reversible step and already offers Undo; this is the point of no return — and
// since sync propagates a tombstone, it reaches every linked device too.
const note = items.value.find((n) => n.id === id);
const title = note?.display_title.trim();
const subject = title ? `"${title}"` : "this note";
const confirmed = window.confirm(
`Permanently delete ${subject}?\n\n` +
"This can't be undone, and it will be deleted from every device you sync with.",
);
if (!confirmed) return;
await repo.notes.deleteForever(id);
const idx = items.value.findIndex((n) => n.id === id);
if (idx >= 0) items.value.splice(idx, 1);
+20
View File
@@ -2,6 +2,7 @@
import { computed, onBeforeUnmount, onMounted, ref, watch } from "vue";
import { useRoute, useRouter } from "vue-router";
import { useNotesStore, type Note, type NoteView } from "../stores/notes";
import { useConfigStore } from "../stores/config";
import { useUiStore } from "../stores/ui";
import { facetCount, facetsFromQuery, facetsToQuery } from "../notes/facets";
import { useNoteEditor } from "../composables/useNoteEditor";
@@ -12,6 +13,7 @@ import NoteCard from "../components/NoteCard.vue";
import NoteEditor from "../components/NoteEditor.vue";
const notes = useNotesStore();
const config = useConfigStore();
const route = useRoute();
const ui = useUiStore();
const router = useRouter();
@@ -144,6 +146,17 @@ watch(
);
watch([currentView, currentLabel, facetKey], () => (focusedIndex.value = -1));
// The retention policy, said out loud at the top of Trash. Empty when retention is
// off — promising a deletion that never comes is its own kind of lie.
const retentionNotice = computed(() => {
if (currentView.value !== "trash") return "";
const days = config.trashRetentionDays;
if (days <= 0) return "Notes stay in Trash until you delete them.";
return days === 1
? "Notes here are permanently deleted 1 day after you trash them. Restore one to keep it."
: `Notes here are permanently deleted ${days} days after you trash them. Restore one to keep it.`;
});
const emptyState = computed(() => {
if (filtered.value) return { title: "No notes match these filters", subtitle: "Try clearing or loosening a facet." };
if (currentView.value === "trash") return { title: "Trash is empty", subtitle: "Notes you delete land here first." };
@@ -208,6 +221,13 @@ async function onDrop(target: Note) {
</button>
<FilterBar v-if="isMainBoard" />
<p
v-if="retentionNotice"
class="mx-auto mb-4 max-w-xl rounded-xl bg-black/5 px-4 py-2.5 text-center text-sm text-neutral-600 dark:bg-white/10 dark:text-neutral-300"
>
{{ retentionNotice }}
</p>
<AsyncState
:loading="notes.loading"
:error="loadError || undefined"
+500
View File
@@ -0,0 +1,500 @@
<script setup lang="ts">
import { computed, onMounted, ref } from "vue";
import { useUiStore } from "../stores/ui";
import BaseButton from "../components/BaseButton.vue";
import BaseInput from "../components/BaseInput.vue";
import Icon from "../components/Icon.vue";
import {
sync as syncBridge,
updates as updateBridge,
type Compatibility,
type ProbeResult,
type SyncStatus,
type UpdateChannel,
type UpdateStatus,
} from "../desktop/bridge";
// Opt-in server sync for the desktop app. Being UNLINKED is the normal resting
// state, not an incomplete setup — the app is local-first and fully usable having
// never touched this screen. The copy has to carry that, or every new user will
// think something is broken.
const ui = useUiStore();
const status = ref<SyncStatus | null>(null);
const pending = ref(false);
const loading = ref(true);
// Connect form
const url = ref("");
const mode = ref<"password" | "token">("password");
const email = ref("");
const password = ref("");
const token = ref("");
const deviceName = ref("");
const probing = ref(false);
const probe = ref<ProbeResult | null>(null);
const probeError = ref("");
const linking = ref(false);
const linkError = ref("");
const linkedAs = ref("");
const degraded = ref<string[]>([]);
const syncing = ref(false);
const syncError = ref("");
const lastResult = ref("");
const linked = computed(() => status.value?.linked === true);
/** Only offer to connect once a probe has said the server is usable. */
const canLink = computed(() => {
if (!probe.value || probe.value.compatibility.status === "incompatible") return false;
return mode.value === "token" ? token.value.trim().length > 0 : email.value.trim().length > 0 && password.value.length > 0;
});
function describe(c: Compatibility): string {
if (c.status === "ok") return "Fully compatible.";
if (c.status === "degraded") {
return `Compatible, but these features aren't available on this server: ${c.unavailable.join(", ")}.`;
}
return c.reason;
}
// --- App updates (M10.9). Independent of sync: an unlinked, server-less install
// still updates itself, which is why the feed is the release host and not a
// ThoughtSync server. ---
const channel = ref<UpdateChannel>("stable");
const update = ref<UpdateStatus | null>(null);
const checking = ref(false);
const installing = ref(false);
const updateError = ref("");
const checkedOnce = ref(false);
const updateAvailable = computed(() => !!update.value?.available);
async function checkUpdates() {
checking.value = true;
updateError.value = "";
try {
update.value = await updateBridge.check();
channel.value = update.value.channel;
} catch (e) {
updateError.value = String((e as Error)?.message ?? e);
} finally {
checking.value = false;
checkedOnce.value = true;
}
}
async function switchChannel(next: UpdateChannel) {
if (next === channel.value) return;
updateError.value = "";
try {
channel.value = await updateBridge.setChannel(next);
// The previous answer described the OTHER channel, so it's meaningless now.
update.value = null;
checkedOnce.value = false;
await checkUpdates();
} catch (e) {
updateError.value = String((e as Error)?.message ?? e);
}
}
async function installUpdate() {
installing.value = true;
updateError.value = "";
try {
// On success the app restarts and this never returns; reaching the next line
// means it failed.
await updateBridge.install();
} catch (e) {
updateError.value = String((e as Error)?.message ?? e);
} finally {
installing.value = false;
}
}
async function refresh() {
try {
channel.value = await updateBridge.channel();
} catch {
// An older build without the update commands — leave the default showing
// rather than blocking the whole Sync screen on it.
}
try {
status.value = await syncBridge.status();
pending.value = await syncBridge.hasPending();
} catch {
status.value = null;
} finally {
loading.value = false;
}
}
async function runProbe() {
probing.value = true;
probeError.value = "";
probe.value = null;
try {
probe.value = await syncBridge.probe(url.value);
} catch (e) {
probeError.value = String((e as Error)?.message ?? e);
} finally {
probing.value = false;
}
}
async function connect() {
linking.value = true;
linkError.value = "";
try {
const result = await syncBridge.link({
url: probe.value?.base_url ?? url.value,
email: mode.value === "password" ? email.value.trim() : undefined,
password: mode.value === "password" ? password.value : undefined,
token: mode.value === "token" ? token.value.trim() : undefined,
name: deviceName.value.trim() || undefined,
});
status.value = result.status;
linkedAs.value = result.identity.email;
degraded.value =
result.compatibility.status === "degraded" ? result.compatibility.unavailable : [];
// Never keep the secrets around after they've been exchanged for a token.
password.value = "";
token.value = "";
probe.value = null;
ui.showToast(`Connected to ${result.status.server_url}.`);
await syncNow();
} catch (e) {
linkError.value = String((e as Error)?.message ?? e);
} finally {
linking.value = false;
}
}
async function syncNow() {
syncing.value = true;
syncError.value = "";
try {
const outcome = await syncBridge.now();
status.value = outcome.status;
pending.value = await syncBridge.hasPending();
const received = outcome.pull.notes_applied + outcome.pull.notes_deleted;
const sent = outcome.push.created + outcome.push.applied;
const blobs = outcome.pull.blobs_downloaded;
const parts: string[] = [];
if (sent > 0) parts.push(`sent ${sent}`);
if (received > 0) parts.push(`received ${received}`);
if (blobs > 0) parts.push(`${blobs} attachment${blobs === 1 ? "" : "s"}`);
lastResult.value = parts.length ? `Synced — ${parts.join(", ")}.` : "Already up to date.";
// Attachments that didn't arrive are retried next sync, so this is a note, not
// an error — but saying nothing would leave a missing image unexplained.
if (outcome.pull.blobs_failed > 0) {
lastResult.value += ` ${outcome.pull.blobs_failed} attachment(s) didn't download — they'll retry on the next sync.`;
}
// Rejections are the server refusing a specific change — surfaced, never
// swallowed, because only the person can resolve them.
if (outcome.push.rejected > 0) {
syncError.value = `${outcome.push.rejected} change(s) the server wouldn't accept: ${outcome.push.errors.join("; ")}`;
}
} catch (e) {
syncError.value = String((e as Error)?.message ?? e);
} finally {
syncing.value = false;
}
}
async function disconnect() {
if (
!window.confirm(
"Stop syncing with this server?\n\nYour notes stay on this device, and the copy on the server is left alone. The device token remains valid until you revoke it on the server under Account → Linked devices.",
)
) {
return;
}
try {
status.value = await syncBridge.unlink();
linkedAs.value = "";
degraded.value = [];
lastResult.value = "";
ui.showToast("Disconnected. This device now works offline only.");
} catch (e) {
ui.showToast(String((e as Error)?.message ?? e));
}
}
function fmt(iso: string | null): string {
if (!iso) return "never";
return new Date(iso).toLocaleString();
}
onMounted(refresh);
</script>
<template>
<div class="mx-auto min-h-full max-w-2xl px-4 py-8">
<header class="mb-8 flex items-center gap-3">
<RouterLink to="/" class="icon-btn" title="Back to board" aria-label="Back to board">
<svg
class="h-[18px] w-[18px]"
viewBox="0 0 24 24"
fill="none"
stroke="currentColor"
stroke-width="2"
stroke-linecap="round"
stroke-linejoin="round"
aria-hidden="true"
>
<path d="m15 18-6-6 6-6" />
</svg>
</RouterLink>
<h1 class="text-xl font-bold tracking-tight">Sync</h1>
</header>
<div v-if="loading" class="py-10 text-center text-sm text-neutral-400">Loading</div>
<!-- Linked -->
<template v-else-if="linked">
<section class="mb-6 rounded-xl border border-neutral-200 p-4 dark:border-neutral-800">
<div class="flex items-start justify-between gap-4">
<div class="min-w-0">
<p class="text-sm font-medium text-neutral-800 dark:text-neutral-100">
Connected to
<span class="font-mono text-xs">{{ status?.server_url }}</span>
</p>
<p v-if="linkedAs" class="mt-0.5 text-xs text-neutral-400">as {{ linkedAs }}</p>
<p class="mt-1 text-xs text-neutral-400">
Last synced {{ fmt(status?.last_sync_at ?? null) }}
<span v-if="pending"> · unsent changes on this device</span>
</p>
</div>
<div class="flex shrink-0 gap-2">
<BaseButton :loading="syncing" @click="syncNow">Sync now</BaseButton>
<BaseButton variant="ghost" @click="disconnect">Disconnect</BaseButton>
</div>
</div>
<p v-if="lastResult && !syncError" class="mt-3 text-xs text-neutral-500 dark:text-neutral-400">
{{ lastResult }}
</p>
<p v-if="degraded.length" class="mt-3 text-xs text-amber-600 dark:text-amber-400">
This server doesn't support: {{ degraded.join(", ") }}. Everything else syncs normally.
</p>
<p v-if="syncError" class="mt-3 text-sm text-red-600 dark:text-red-400">{{ syncError }}</p>
</section>
<p class="text-xs text-neutral-400">
Your notes live on this device either way syncing just keeps a server copy in step, so
other devices can catch up.
</p>
</template>
<!-- Not linked: the normal resting state, deliberately not framed as a problem -->
<template v-else>
<section
class="mb-6 rounded-xl border border-neutral-200 p-4 dark:border-neutral-800"
aria-live="polite"
>
<p class="text-sm font-medium text-neutral-800 dark:text-neutral-100">
Working offline on this device
</p>
<p class="mt-1 text-sm text-neutral-500 dark:text-neutral-400">
Everything works without a server your notes are stored on this machine. Connect a
ThoughtSync server if you want them to reach your other devices.
</p>
</section>
<form class="flex flex-col gap-4" @submit.prevent="probe ? connect() : runProbe()">
<div class="flex items-end gap-3">
<BaseInput
id="server-url"
v-model="url"
label="Server address"
placeholder="notes.example.com"
autocomplete="url"
class="flex-1"
/>
<BaseButton type="button" variant="ghost" :loading="probing" @click="runProbe">
Check
</BaseButton>
</div>
<p class="-mt-2 text-xs text-neutral-400">
Uses https unless you type http:// yourself.
</p>
<p v-if="probeError" class="text-sm text-red-600 dark:text-red-400">{{ probeError }}</p>
<!-- What answered, BEFORE any credentials are handed over -->
<div
v-if="probe"
class="rounded-xl border p-3 text-sm"
:class="
probe.compatibility.status === 'incompatible'
? 'border-red-300 bg-red-50 dark:border-red-900 dark:bg-red-950/30'
: 'border-neutral-200 dark:border-neutral-800'
"
>
<p class="font-medium text-neutral-800 dark:text-neutral-100">
{{ probe.server.site_name || "ThoughtSync server" }}
<span v-if="probe.server.version" class="text-xs font-normal text-neutral-400">
v{{ probe.server.version }}
</span>
</p>
<p
class="mt-1 text-xs"
:class="
probe.compatibility.status === 'incompatible'
? 'text-red-700 dark:text-red-300'
: 'text-neutral-500 dark:text-neutral-400'
"
>
{{ describe(probe.compatibility) }}
</p>
</div>
<template v-if="probe && probe.compatibility.status !== 'incompatible'">
<fieldset class="flex flex-col gap-3">
<legend class="mb-1 text-sm font-medium text-neutral-800 dark:text-neutral-100">
Sign in
</legend>
<div class="flex gap-4 text-sm">
<label class="flex items-center gap-2">
<input v-model="mode" type="radio" value="password" class="accent-brand" />
Email and password
</label>
<label class="flex items-center gap-2">
<input v-model="mode" type="radio" value="token" class="accent-brand" />
Paste a device token
</label>
</div>
<template v-if="mode === 'password'">
<BaseInput
id="sync-email"
v-model="email"
label="Email"
type="email"
autocomplete="username"
/>
<BaseInput
id="sync-password"
v-model="password"
label="Password"
type="password"
autocomplete="current-password"
/>
</template>
<template v-else>
<BaseInput
id="sync-token"
v-model="token"
label="Device token"
placeholder="Paste the token from Account → Linked devices"
/>
</template>
<BaseInput
id="sync-device-name"
v-model="deviceName"
label="Name for this device (optional)"
placeholder="e.g. My laptop"
/>
</fieldset>
<p v-if="linkError" class="text-sm text-red-600 dark:text-red-400">{{ linkError }}</p>
<div>
<BaseButton type="submit" :loading="linking" :disabled="!canLink">
<Icon name="sync" /> Connect and sync
</BaseButton>
</div>
</template>
</form>
</template>
<!-- Updates sit outside the linked/unlinked split on purpose: an install that
has never touched a server still updates itself. -->
<section class="mt-10 border-t border-neutral-200 pt-8 dark:border-neutral-800">
<h2 class="text-sm font-semibold">App updates</h2>
<p class="mt-1 text-sm text-neutral-500 dark:text-neutral-400">
This is version {{ update?.current_version ?? "—" }}.
</p>
<fieldset class="mt-4">
<legend class="text-xs font-semibold uppercase tracking-wide text-neutral-400">
Channel
</legend>
<div class="mt-2 flex flex-col gap-2">
<label class="flex items-start gap-2 text-sm">
<input
type="radio"
class="mt-1"
name="update-channel"
value="stable"
:checked="channel === 'stable'"
@change="switchChannel('stable')"
/>
<span>
<span class="font-medium">Stable</span>
<span class="block text-neutral-500 dark:text-neutral-400">
Released versions only.
</span>
</span>
</label>
<label class="flex items-start gap-2 text-sm">
<input
type="radio"
class="mt-1"
name="update-channel"
value="dev"
:checked="channel === 'dev'"
@change="switchChannel('dev')"
/>
<span>
<span class="font-medium">Development</span>
<span class="block text-neutral-500 dark:text-neutral-400">
Every build that passes CI. Newer, and less tested.
</span>
</span>
</label>
</div>
</fieldset>
<p
v-if="update?.blocked_reason"
class="mt-4 rounded-lg bg-black/5 px-3 py-2 text-sm text-neutral-600 dark:bg-white/10 dark:text-neutral-300"
>
{{ update.blocked_reason }}
</p>
<div v-if="updateAvailable" class="mt-4 rounded-lg bg-brand/10 px-3 py-2.5">
<p class="text-sm font-medium">Version {{ update?.available }} is available.</p>
<p v-if="update?.notes" class="mt-1 whitespace-pre-line text-sm text-neutral-600 dark:text-neutral-300">
{{ update.notes }}
</p>
</div>
<p
v-else-if="checkedOnce && !updateError"
class="mt-4 text-sm text-neutral-500 dark:text-neutral-400"
>
You're on the newest {{ channel === "dev" ? "development" : "stable" }} build.
</p>
<p v-if="updateError" class="mt-4 text-sm text-red-600 dark:text-red-400">{{ updateError }}</p>
<div class="mt-4 flex flex-wrap gap-2">
<BaseButton variant="ghost" :loading="checking" @click="checkUpdates">
Check for updates
</BaseButton>
<BaseButton
v-if="updateAvailable && update?.can_install"
:loading="installing"
@click="installUpdate"
>
Install and restart
</BaseButton>
</div>
</section>
</div>
</template>
+23 -1
View File
@@ -1,8 +1,10 @@
from __future__ import annotations
import asyncio
import mimetypes
import os
import secrets
from contextlib import suppress
from datetime import timedelta
from quart import Quart, has_request_context, jsonify, request, send_from_directory
@@ -15,10 +17,11 @@ from .db import session_scope
from .graph import bp as graph_bp
from .labels import bp as labels_bp
from .notes import bp as notes_bp
from .retention import run_sweeper
from .saved_filters import bp as saved_filters_bp
from .settings import get_public_config, get_setting, load_or_create_secret_key
from .settings_api import bp as settings_bp
from .sync import bp as sync_bp
from .sync import bp as sync_bp, protocol_advertisement
STATIC_DIR = os.path.join(os.path.dirname(__file__), "static")
@@ -80,6 +83,21 @@ def create_app() -> Quart:
app.config["PERMANENT_SESSION_LIFETIME"] = timedelta(days=days)
except (ValueError, TypeError, KeyError):
pass
# Expire old trash in the background (retention.py). One task per process is
# correct because the image serves with a single hypercorn worker (Dockerfile);
# if that ever gains `--workers`, this needs a lock so N workers don't each
# sweep. Duplicate sweeps would be harmless but wasteful — a purged row is
# skipped by `purged_at IS NULL` — so this is about load, not correctness.
app.config["TRASH_SWEEPER"] = asyncio.create_task(run_sweeper())
@app.after_serving
async def _shutdown() -> None:
task = app.config.get("TRASH_SWEEPER")
if task is not None:
task.cancel()
# Await the cancellation so shutdown doesn't race a sweep mid-transaction.
with suppress(asyncio.CancelledError):
await task
@app.get("/api/health")
async def health():
@@ -91,6 +109,10 @@ def create_app() -> Quart:
async with session_scope() as db:
data = await get_public_config(db)
data["version"] = app.config["APP_VERSION"]
# The sync-protocol handshake (M10.6). A native client reads this BEFORE
# linking — while it still has no token and possibly no account — to decide
# whether it can talk to this server, and which optional features to offer.
data.update(protocol_advertisement())
return jsonify(data)
@app.get("/", defaults={"path": ""})
+3
View File
@@ -83,6 +83,9 @@ class Note(Base):
"pinned": self.pinned,
"archived": self.archived,
"trashed": self.deleted_at is not None,
# WHEN it was trashed, not just that it was: clients count the retention
# window from here to show how long a note has left before it's purged.
"deleted_at": iso(self.deleted_at),
"remind_at": iso(self.remind_at),
"recurrence": self.recurrence,
"created_at": iso(self.created_at),
+6 -1
View File
@@ -36,6 +36,7 @@ from ..models.note_link import NoteLink
from ..models.note_link_preview import NoteLinkPreview
from ..models.note_revision import NoteRevision
from ..responses import json_error, not_found, parse_uuid
from ..retention import purge_note
from ..settings import get_setting
from ..unfurl import UnfurlError, unfurl
from ._bp import bp
@@ -971,6 +972,10 @@ async def delete_note(note_id: str):
return not_found()
if note.deleted_at is None:
return json_error("note must be trashed before permanent delete", 409)
await db.delete(note)
# A tombstone, not a dropped row. Deleting the row outright would leave the
# server with no record the note ever existed, so a linked device that was
# offline at the time would keep its copy forever — and push it back the
# next time it was edited. The delete has to be something clients can LEARN.
await purge_note(db, note)
await db.commit()
return jsonify({"ok": True})
+17 -4
View File
@@ -45,20 +45,33 @@ def parse_list_items(raw: object) -> list[str]:
def apply_filter(stmt, filter_name: str):
"""Narrow a notes query to one board view."""
"""Narrow a notes query to one board view.
Every branch excludes purge tombstones — content-less rows kept only so the sync
feed can tell offline clients a note is gone (see `retention.purge_note`). The
active/archived branches get that for free from `deleted_at IS NULL`, since a
tombstone keeps the timestamp; Trash is the one view that has to say so.
"""
if filter_name == "archived":
return stmt.where(Note.deleted_at.is_(None), Note.archived.is_(True))
if filter_name == "trash":
return stmt.where(Note.deleted_at.is_not(None))
return stmt.where(Note.deleted_at.is_not(None), Note.purged_at.is_(None))
return stmt.where(Note.deleted_at.is_(None), Note.archived.is_(False))
async def _get_owned(db, note_id: str) -> Note | None:
"""Fetch a note the current user OWNS (mutations are owner-only in M1/M2)."""
"""Fetch a note the current user OWNS (mutations are owner-only in M1/M2).
A purged note reads as absent: the REST API must treat it as gone, so opening,
editing or restoring one 404s. The sync push path looks rows up directly rather
than through here, which is what still lets a client re-create an id it owns.
"""
nid = parse_uuid(note_id)
if nid is None:
return None
return await db.scalar(select(Note).where(Note.id == nid, Note.owner_id == g.user_id))
return await db.scalar(
select(Note).where(Note.id == nid, Note.owner_id == g.user_id, Note.purged_at.is_(None))
)
def _escape_like(s: str) -> str:
+170
View File
@@ -0,0 +1,170 @@
"""Trash retention — what "permanently deleted" means, and when it happens by itself.
Two things live here, deliberately together:
**`purge_note`** — the single definition of destroying a note. Three callers reach
permanent deletion by different routes (the user's Delete forever in the web UI,
a client's `op=delete` over sync, and the sweeper below), and if each had its own
idea of what to tear down they would drift — one would forget the files, another
the revision history, and "permanently deleted" would quietly mean three different
things depending on how you got there.
**The sweeper** — trash that nobody empties is not free: a trashed note keeps its
attachment BYTES on disk for as long as it sits there. So trash expires. The window
is the `trash_retention_days` setting (default 30, `0` = keep forever), re-read on
every pass so a change in admin Settings takes effect without a restart.
A purged note is not a deleted ROW — it's a content-less tombstone. That's what lets
an offline client that reappears next month learn the note is gone instead of
faithfully resurrecting it on the next push.
"""
from __future__ import annotations
import asyncio
import logging
from datetime import datetime, timedelta, timezone
from sqlalchemy import delete as sa_delete
from sqlalchemy import select
from .config import Config
from .db import session_scope
from .models.label import NoteLabel
from .models.note import Note
from .models.note_attachment import NoteAttachment
from .models.note_item import NoteItem
from .models.note_link import NoteLink
from .models.note_link_preview import NoteLinkPreview
from .models.note_revision import NoteRevision
from .settings import get_setting
logger = logging.getLogger(__name__)
# How often the sweeper wakes. Retention is measured in days, so anything under
# "a few times a day" buys nothing but load — a note trashed at 09:00 expiring at
# 14:00 rather than 09:00 thirty days later is not a difference anyone can feel.
SWEEP_INTERVAL_SECONDS = 6 * 60 * 60
# Let the app finish booting (migrations, first requests) before the first sweep.
SWEEP_STARTUP_DELAY_SECONDS = 60
# Rows purged per transaction. A long-neglected install could have thousands of
# expired notes on the first sweep; committing in batches keeps that from becoming
# one enormous transaction holding locks while it deletes files.
SWEEP_BATCH = 200
def expired_before(now: datetime, retention_days: int) -> datetime | None:
"""The cutoff: trash older than this has expired. `None` = retention is off.
Kept separate from the query so the window arithmetic — including the two ways
to say "never" (0 and negative, the latter reachable by typing a stray minus in
Settings) — is testable without a database.
"""
if retention_days <= 0:
return None
return now - timedelta(days=retention_days)
async def purge_note(db, note: Note, edited_at: datetime | None = None) -> None:
"""Turn a note into a content-less tombstone: delete its children (and the
attachment files on disk), clear its content, stamp `purged_at`.
The row survives on purpose — offline clients read it off the delta feed and
learn the note is gone. Everything that carries the note's CONTENT goes, and
that includes history: a revision row holds the full body, so leaving revisions
behind would mean the text of a "permanently deleted" note is still on the
server, recoverable by anyone who can read the table.
"""
atts = (await db.scalars(select(NoteAttachment).where(NoteAttachment.note_id == note.id))).all()
for a in atts:
try:
(Config.media_root() / a.path).unlink(missing_ok=True)
except OSError:
# A missing or unreadable file must not strand the row: the DB record is
# what the user asked us to destroy, and a failed unlink leaving it in
# place would make the note reappear whole on the next sweep.
logger.warning("couldn't remove attachment file %s during purge", a.path, exc_info=True)
await db.execute(sa_delete(NoteAttachment).where(NoteAttachment.note_id == note.id))
await db.execute(sa_delete(NoteItem).where(NoteItem.note_id == note.id))
await db.execute(sa_delete(NoteLabel).where(NoteLabel.note_id == note.id))
await db.execute(sa_delete(NoteLink).where(NoteLink.source_id == note.id))
await db.execute(sa_delete(NoteLinkPreview).where(NoteLinkPreview.note_id == note.id))
await db.execute(sa_delete(NoteRevision).where(NoteRevision.note_id == note.id))
note.title = None
note.body = ""
note.display_title = ""
# `deleted_at` deliberately SURVIVES. It's still true — that is when the note was
# deleted — and keeping it means every ordinary query, present and future, that
# says "not trashed" (`deleted_at IS NULL`) excludes tombstones for free. Clearing
# it would leave a content-less row looking like a perfectly normal active note,
# and it would surface on the board as a blank card. Only the Trash view, which
# asks for `deleted_at IS NOT NULL`, has to name `purged_at` explicitly.
note.remind_at = None
note.purged_at = datetime.now(timezone.utc)
if edited_at is not None:
note.updated_at = edited_at
async def sweep_expired_trash(db, retention_days: int, *, now: datetime | None = None) -> int:
"""Purge every note whose trash has expired. Returns how many were purged.
Runs across ALL owners — it's a server-wide policy, not a per-user action, and
the sweeper has no session to scope it by (rule 47 is about honoring the ACL on
user-initiated reads, not about exempting rows from server maintenance).
"""
cutoff = expired_before(now or datetime.now(timezone.utc), retention_days)
if cutoff is None:
return 0
total = 0
while True:
expired = (
await db.scalars(
select(Note)
.where(
Note.deleted_at.is_not(None),
Note.deleted_at < cutoff,
# Already a tombstone. Without this the purge would re-run on
# every sweep forever, bumping sync_revision each time and
# handing clients an endless stream of "news" about one note.
Note.purged_at.is_(None),
)
.order_by(Note.deleted_at)
.limit(SWEEP_BATCH)
)
).all()
if not expired:
return total
for note in expired:
await purge_note(db, note)
await db.commit()
total += len(expired)
async def sweep_once() -> int:
"""One sweep against the live retention setting, in its own session."""
async with session_scope() as db:
try:
days = int(await get_setting(db, "trash_retention_days"))
except (KeyError, TypeError, ValueError):
return 0
return await sweep_expired_trash(db, days)
async def run_sweeper() -> None:
"""The background loop. Started in `before_serving`, cancelled on shutdown.
A sweep failure (DB blip, unreadable media directory) must never take the loop
down with it — the next pass simply finds the same expired rows and tries again.
"""
await asyncio.sleep(SWEEP_STARTUP_DELAY_SECONDS)
while True:
try:
purged = await sweep_once()
if purged:
logger.info("trash retention: purged %d expired note(s)", purged)
except asyncio.CancelledError:
raise
except Exception:
logger.exception("trash retention sweep failed; will retry next pass")
await asyncio.sleep(SWEEP_INTERVAL_SECONDS)
+15 -1
View File
@@ -43,6 +43,15 @@ REGISTRY: list[SettingDef] = [
"How long a signed-in session stays valid before another login is required.",
"Access",
),
SettingDef(
"trash_retention_days",
"int",
30,
"Trash retention (days)",
"How long a note stays in Trash before it's permanently deleted, freeing its "
"attachments from disk. Set to 0 to keep trashed notes until they're deleted by hand.",
"Notes",
),
SettingDef(
"max_attachment_mb",
"int",
@@ -117,11 +126,16 @@ async def get_setting(db, key: str) -> Any:
async def get_public_config(db) -> dict:
"""Non-sensitive settings the unauthenticated login/register screen needs."""
"""Non-sensitive settings every client reads — the login/register screen before
sign-in, and the app itself afterwards. Nothing here is owner-scoped."""
return {
"site_name": await get_setting(db, "site_name"),
"allow_registration": await get_setting(db, "allow_registration"),
"enable_url_unfurl": await get_setting(db, "enable_url_unfurl"),
# Server policy, not user data: clients need it to say how long a note has
# left in Trash. A native client also reads it BEFORE linking, which is why
# it belongs on the unauthenticated config rather than behind login.
"trash_retention_days": await get_setting(db, "trash_retention_days"),
}
+44 -27
View File
@@ -20,14 +20,11 @@ from sqlalchemy import func, select
from .auth import login_required
from .common import iso, parse_dt
from .config import Config
from .db import session_scope
from .labeling import reconcile_manual_labels, resolve_owned_label_ids
from .models.label import Label, NoteLabel
from .models.note import Note
from .models.note_attachment import NoteAttachment
from .models.note_item import NoteItem
from .models.note_link import NoteLink
from .models.note_revision import NoteRevision
from .notes import (
_reconcile_tags,
@@ -38,6 +35,7 @@ from .notes import (
normalize_color,
normalize_recurrence,
)
from .retention import purge_note
from .serialize import serialize_label_sync
bp = Blueprint("sync", __name__, url_prefix="/api/sync")
@@ -46,6 +44,48 @@ DEFAULT_LIMIT = 500
MAX_LIMIT = 1000
MAX_PUSH = 1000 # per-batch change cap
# --- protocol versioning (M10.6) --------------------------------------------
#
# The client<->server compatibility contract. These integers version the WIRE
# PROTOCOL, deliberately separate from the app's release version, so a client and
# server on different releases can still work out whether they can talk. Without
# that separation every protocol change would force app<->server lockstep.
#
# SYNC_PROTOCOL_VERSION what this server speaks.
# MIN_CLIENT_PROTOCOL_VERSION the oldest client protocol it still accepts.
#
# Bump SYNC_PROTOCOL_VERSION for ANY wire change. Raise
# MIN_CLIENT_PROTOCOL_VERSION only for a genuinely BREAKING one: it is the switch
# that hard-blocks older clients, so additive changes must leave it alone.
SYNC_PROTOCOL_VERSION = 1
MIN_CLIENT_PROTOCOL_VERSION = 1
# Named capabilities beyond the base protocol. An ADDITIVE change earns a name
# here rather than a min-version bump, so a newer client meeting an older server
# can degrade to "some features unavailable" instead of refusing to sync. Clients
# test for the name, never infer a capability from a version number — that's what
# keeps feature gating independent of release lockstep.
SYNC_FEATURES: tuple[str, ...] = (
"notes", # note delta sync (pull + push)
"labels", # the label catalog as its own entity
"attachments", # blob upload/download, deduped by sha256
"tombstones", # purge propagates as a content-less row
"revisions", # an overwritten version snapshots into note history
)
def protocol_advertisement() -> dict:
"""What the server publishes about the sync protocol, merged into `/api/config`.
DB-free and unauthenticated on purpose: a client has to be able to ask "can I
talk to you at all?" before it holds a device token — or even has an account.
"""
return {
"sync_protocol_version": SYNC_PROTOCOL_VERSION,
"min_client_protocol_version": MIN_CLIENT_PROTOCOL_VERSION,
"sync_features": list(SYNC_FEATURES),
}
def _parse_since(raw: str | None) -> int:
"""The pull cursor: a non-negative revision watermark. Bad/absent → 0 (full sync)."""
@@ -203,29 +243,6 @@ async def _apply_note_manual_labels(db, note: Note, ch: dict) -> None:
await reconcile_manual_labels(db, note, owned)
async def _purge_note(db, note: Note, edited_at: datetime | None) -> None:
"""Turn a note into a content-less tombstone: delete children (+ attachment files),
clear content, set purged_at. Kept so offline clients learn it's gone."""
atts = (await db.scalars(select(NoteAttachment).where(NoteAttachment.note_id == note.id))).all()
for a in atts:
try:
(Config.media_root() / a.path).unlink(missing_ok=True)
except OSError:
pass
await db.execute(sa_delete(NoteAttachment).where(NoteAttachment.note_id == note.id))
await db.execute(sa_delete(NoteItem).where(NoteItem.note_id == note.id))
await db.execute(sa_delete(NoteLabel).where(NoteLabel.note_id == note.id))
await db.execute(sa_delete(NoteLink).where(NoteLink.source_id == note.id))
note.title = None
note.body = ""
note.display_title = ""
note.deleted_at = None
note.remind_at = None
note.purged_at = datetime.now(timezone.utc)
if edited_at is not None:
note.updated_at = edited_at
async def _apply_note(db, ch: dict) -> dict:
raw_id = ch.get("id")
try:
@@ -248,7 +265,7 @@ async def _apply_note(db, ch: dict) -> dict:
return {"id": str(nid), "entity": "note", "status": "noop"}
if not client_wins(edited_at, note.updated_at):
return {"id": str(nid), "entity": "note", "status": "kept", "sync_revision": note.sync_revision}
await _purge_note(db, note, edited_at)
await purge_note(db, note, edited_at)
await db.flush()
await db.refresh(note, ["sync_revision"])
return {"id": str(nid), "entity": "note", "status": "applied", "sync_revision": note.sync_revision}
+83
View File
@@ -0,0 +1,83 @@
from datetime import datetime, timedelta, timezone
import pytest
from thoughtsync.retention import (
SWEEP_BATCH,
SWEEP_INTERVAL_SECONDS,
SWEEP_STARTUP_DELAY_SECONDS,
expired_before,
sweep_expired_trash,
)
from thoughtsync.settings import REGISTRY, get_public_config, validate_updates
NOW = datetime(2026, 7, 26, 12, 0, tzinfo=timezone.utc)
def test_expired_before_is_the_window_ago():
assert expired_before(NOW, 30) == NOW - timedelta(days=30)
assert expired_before(NOW, 1) == NOW - timedelta(days=1)
def test_zero_means_keep_forever():
# The opt-out. A user who wants Trash to be an indefinite archive gets one, and
# `None` is what stops the sweep before it builds a query at all.
assert expired_before(NOW, 0) is None
def test_a_negative_window_also_means_never():
# Reachable by typing a stray minus into the Settings field. The dangerous reading
# of -1 would be "expired a day in the FUTURE", which purges the entire trash on
# the next sweep; refusing to run is the only safe interpretation.
assert expired_before(NOW, -1) is None
assert expired_before(NOW, -3650) is None
async def test_sweep_is_a_noop_when_retention_is_off():
# Passing None as the session proves it: retention off must return before it so
# much as touches the database.
assert await sweep_expired_trash(None, 0) == 0
assert await sweep_expired_trash(None, -1) == 0
def test_retention_setting_is_registered_with_a_30_day_default():
defn = next((d for d in REGISTRY if d.key == "trash_retention_days"), None)
assert defn is not None, "the setting must appear in the admin Settings UI"
assert defn.type == "int"
assert defn.default == 30
# The operator has to be able to tell what it does without reading the code.
assert "0" in defn.description, "the keep-forever escape hatch must be documented"
def test_retention_setting_accepts_an_int_and_rejects_nonsense():
clean, err = validate_updates({"trash_retention_days": "7"})
assert err is None
assert clean == {"trash_retention_days": 7}
_, err = validate_updates({"trash_retention_days": "soon"})
assert err is not None
async def test_public_config_publishes_the_window():
# Clients need it to say how long a note has left in Trash, and a native client
# reads it before it holds any credential — so it rides the unauthenticated
# config. A stub session stands in for the DB: no row set => registry default.
class _NoRows:
async def get(self, *_args):
return None
cfg = await get_public_config(_NoRows())
assert cfg["trash_retention_days"] == 30
@pytest.mark.parametrize(
"value", [SWEEP_INTERVAL_SECONDS, SWEEP_STARTUP_DELAY_SECONDS, SWEEP_BATCH]
)
def test_sweeper_pacing_constants_are_positive(value):
# A zero interval would turn the background loop into a busy spin against the DB.
assert value > 0
def test_sweep_interval_is_well_under_a_day():
# Retention is measured in days, but the sweep still has to run often enough that
# "30 days" doesn't quietly become 31.
assert SWEEP_INTERVAL_SECONDS <= 12 * 60 * 60
+29
View File
@@ -6,10 +6,14 @@ from thoughtsync.app import create_app
from thoughtsync.sync import (
DEFAULT_LIMIT,
MAX_LIMIT,
MIN_CLIENT_PROTOCOL_VERSION,
SYNC_FEATURES,
SYNC_PROTOCOL_VERSION,
_clamp_limit,
_page_cursor,
_parse_since,
client_wins,
protocol_advertisement,
)
@@ -82,3 +86,28 @@ def test_page_cursor_both_full_uses_min_boundary():
cursor, more = _page_cursor([1, 2, 10], [3, 4, 5], since=0, limit=3)
assert cursor == 5
assert more is True
# --- protocol handshake (M10.6) ---------------------------------------------
def test_protocol_advertisement_shape():
ad = protocol_advertisement()
assert ad["sync_protocol_version"] == SYNC_PROTOCOL_VERSION
assert ad["min_client_protocol_version"] == MIN_CLIENT_PROTOCOL_VERSION
# A list, not a tuple — it has to survive jsonify as a JSON array.
assert isinstance(ad["sync_features"], list)
assert ad["sync_features"] == list(SYNC_FEATURES)
def test_protocol_floor_never_exceeds_current():
# A server can't demand a client protocol newer than the one it speaks itself —
# that would lock out every client, including a perfectly current one.
assert MIN_CLIENT_PROTOCOL_VERSION <= SYNC_PROTOCOL_VERSION
def test_protocol_features_are_unique_nonempty_names():
# Clients match capabilities by exact name, so duplicates or blanks would make
# a feature check silently meaningless.
assert all(f and f.strip() == f for f in SYNC_FEATURES)
assert len(set(SYNC_FEATURES)) == len(SYNC_FEATURES)