# ThoughtSync sync protocol The contract the local-first native clients (Tauri desktop, Android) implement against. The server is the **sync hub**: each client keeps a full local store (SQLite), works fully offline, and reconciles with the server when linked. The web app does **not** use this API — it stays on the live REST API (`/api/notes`, …) as an online-only stopgap. All sync endpoints live under `/api/sync`. Everything is **owner-scoped** and **deterministic** (no AI). > **No Postgres CI lane.** Trigger/migration/sync behavior is verified by the > operator on deploy, not in CI. Pure logic (LWW comparator, paging cursor, > token hashing) is unit-tested. ## Protocol versioning — the compatibility handshake Clients and servers update on their own schedules; a self-hosted server can sit on an older release than the desktop app for months. So the wire protocol is versioned **separately from either program's release version**, and each side declares two numbers: what it speaks, and the oldest counterpart it accepts. | | server (`src/thoughtsync/sync.py`) | client (`desktop/src-tauri/src/sync/compat.rs`) | |---|---|---| | speaks | `SYNC_PROTOCOL_VERSION` | `CLIENT_PROTOCOL_VERSION` | | accepts down to | `MIN_CLIENT_PROTOCOL_VERSION` | `MIN_SERVER_PROTOCOL_VERSION` | The server publishes its half on the **public, unauthenticated** `GET /api/config` — a client must be able to ask "can I talk to you?" before it holds a device token, or even has an account: ```json { "site_name": "...", "version": "0.1.0", "sync_protocol_version": 1, "min_client_protocol_version": 1, "sync_features": ["notes", "labels", "attachments", "tombstones", "revisions"], "trash_retention_days": 30 } ``` The client identifies itself on every request with `X-ThoughtSync-Client: thoughtsync-desktop/` and `X-ThoughtSync-Protocol: `. ### `sync_features` — why versions alone aren't enough A version number can only say "newer" or "older". `sync_features` names capabilities, so a client tests for the one it needs instead of inferring it from a number. That is what keeps an **additive** change from forcing a lockstep upgrade: a newer client meeting an older server drops the missing feature and syncs everything else. ### The policy - **Any wire change** → bump `SYNC_PROTOCOL_VERSION`. - v2 (M13): `kind` and `title` left the wire; **floor raised**, because a v1 client kept pushing both and read back notes carrying neither — and `title` was the note's NAME, so an old client showed nameless notes. - v3: attachments/tombstones/revisions. - v4 (M315): `color` left the note; **floor NOT raised**. Both directions degrade in silence and neither loses anything visible — an old client reading a v4 note falls back to the colour it derives locally, and one pushing `color` has the key ignored. The test is not "did a field leave" but "does either side end up showing something wrong". - **Additive change** (a new field, a new capability) → add a `sync_features` name. Do **not** raise a minimum. Old clients keep working. - **Breaking change only** → raise `MIN_CLIENT_PROTOCOL_VERSION` (or the client's `MIN_SERVER_PROTOCOL_VERSION`). This is the switch that hard-blocks the other side, so it is the one to be stingy with. - Never gate behavior on the *release* version (`version`) — it's for display. ### The three outcomes The client evaluates the advertisement (`compat::evaluate`) and gets exactly one of: - **ok** — full parity; sync everything. - **degraded** — safe to sync, but named capabilities are unavailable here; the UI says which. - **incompatible** — do not sync. Carries `client_must_update` so the message can point at the side that can actually fix it, rather than just saying "incompatible". A server that predates this handshake sends no protocol fields at all. That is treated as **incompatible (update the server)** — deliberately not as a parse error, which would look to the user like they mistyped the URL. ## Authentication — device bearer tokens Native clients authenticate with a long-lived **device token**, not a session cookie. Only the token's SHA-256 hash is stored server-side; the plaintext is shown once at creation. - **First link (no session):** `POST /api/auth/device-login` with `{email, password, name}` → `{token, device, user}`. Store `token`; send it as `Authorization: Bearer ` on every subsequent call. - **From the web app (already signed in):** the user creates a token at `/account` ("Linked devices"); `POST /api/auth/devices` `{name}` → `{token, device}`. They paste it into the native app. - **Manage:** `GET /api/auth/devices` (list), `DELETE /api/auth/devices/` (revoke). A revoked token stops authenticating immediately. - **Self-revoke:** `DELETE /api/auth/devices/self` retires the token presented in the `Authorization` header. This is what a native client calls when the user unlinks. It exists because a client can't use the id-keyed route: a token pasted from the web app arrives without a device id, and `/api/auth/me` describes the user, not the device row. A caller authenticated by session cookie gets `400` — it holds no device token, so there is nothing for it to mean. Unlinking is never blocked on this call. If the server is unreachable or too old to have the route, the client still unlinks locally and tells the user the token is still live and where to revoke it. Every authenticated request (sync or otherwise) accepts the bearer token in place of the session cookie. ## The revision cursor Every syncable row carries a monotonic **`sync_revision`** (bigint), assigned by a database trigger from a single shared sequence (`sync_revision_seq`) on every insert/update. Because it's a shared sequence, one integer is a total-order watermark across **all** of a user's notes and labels. It is not a timestamp — it is immune to clock skew and is only ever compared with `>`. The client persists the highest cursor it has fully consumed and passes it back as `?since=`. `since=0` (or absent) is a **full initial sync**. ## Entities and what syncs - **Note** — the primary sync unit. It travels with its **checklist items, label memberships, and attachment metadata inline** (same shape as the REST serialization). A change to any child bumps the parent note's `sync_revision`, so pulling the note re-syncs the whole thing. - **Label** — the label *catalog* (name + color) syncs as its own entity so a rename/recolor/delete propagates independently of notes. - **Derived, NOT synced:** `#tags` are parsed from the note body. Clients recompute them locally; the server recomputes them on push. They never travel over the wire. (`[[wiki-links]]` were derived the same way until they were removed — note 2897.) - **Attachment blobs** sync by id over the existing upload/download routes (see Attachments below); only their metadata rides the delta feed. ## Tombstones (deletes) Two levels, both propagate: - **Trash** — `deleted_at` is a normal field, and it's **on the wire**: a trashed note still syncs with its content, the client shows it in its Trash, and the timestamp is what the client counts the retention window against. Restoring clears it. - **Purge (permanent delete)** — becomes a **content-less tombstone**: `purged_at` is set, title/body/items/labels/attachments/previews/revisions are cleared or removed, and the row is kept. A client seeing `purged_at != null` deletes the row from its local store. `deleted_at` deliberately SURVIVES a purge, so ordinary server-side queries (`deleted_at IS NULL`) never see a tombstone as a live note. Tombstones themselves are retained indefinitely (cheap for a personal store); revisit if they ever grow large. ### Retention — trash expires A trashed note is purged automatically once it is older than the server's `trash_retention_days` setting (default **30**, `0` = keep forever), advertised on `/api/config` so a client can show the countdown. A background sweep on the server does the work; clients learn about it as ordinary tombstones and need no special handling. **A linked client must not run its own expiry.** The server owns the policy — one clock, one window. A client that purged on its own schedule could destroy a note the server was deliberately keeping and then push that delete upstream. An *unlinked* client (offline-only, no server to defer to) expires its own trash on its own default, which is the only case where nothing else can. ## Pull — `GET /api/sync/changes` Query: `?since=&limit=` (limit default 500, max 1000). Response: ```json { "notes": [ { "...full note...", "trashed": false, "deleted_at": null, "sync_revision": 42, "purged_at": null } ], "labels": [ { "id": "...", "name": "...", "color": "...", "sync_revision": 43, "purged_at": null, "created_at": "..." } ], "cursor": 43, "has_more": false } ``` Returns **all** of the caller's notes + labels (any state — active, archived, trash, purged) whose `sync_revision > since`, ascending by revision. Notes and labels share the sequence, so paging merges the two streams: when either stream fills a page, `cursor` advances only to the **smaller** of the two page boundaries, so nothing between `cursor` and the next pull is skipped. Loop while `has_more` is true, advancing `since = cursor` each time. Note attachment metadata carries `{id, url, mime, size, sha256}`. ## Push — `POST /api/sync/push` Body: `{ "changes": [ ... ] }` (max 1000 per batch). Each change: ```json { "entity": "note", "id": "", "op": "upsert", "edited_at": "", "body": "...", "pinned": false, "archived": false, "trashed": false, "remind_at": null, "recurrence": null, "position": 0, "label_ids": ["", ...], "created_at": "" } ``` - **Client-generated ids.** Notes/labels are UUIDs; the client mints the id when it creates the row offline and sends it here. Create-if-absent, else update. - **Whole-note semantics.** A note upsert carries the client's *full* current state (not a partial patch) — the server overwrites all scalar fields and sets manual label memberships from `label_ids` (tag-sourced labels are re-derived from the body). `#tags` are recomputed server-side. A checklist is `- [ ] ` lines inside `body` (M304), so there is no separate `items` array. - **Fields a change may still carry, and the server reads past.** `title` and `kind` (removed in v2), `items` (M304) and `color` (v4, M315). The server reads its payload key by key and never validates the shape, which is exactly what lets an older client keep pushing a field this one has stopped storing — see the version policy above for why none of those needed a floor raise on their own. - **`op: "delete"`** purges (tombstones) the row. Trashing is just an upsert with `trashed: true`. - **Labels:** `{entity: "label", op: "upsert"|"delete", id, edited_at, name, color}`. A per-owner name clash on a *different* id is rejected (fix locally and retry). ### Conflict resolution — last-write-wins + history On a clash the **most-recently-edited version wins**, by `edited_at` (client wall-clock) compared against the server row's last-edit time (`updated_at`). The client applies iff `client.edited_at >= server.updated_at`. When a winning upsert overwrites an existing title/body, the server first **snapshots the overwritten version into the note's version history** (`note_revisions`) — so a "lost" edit is never truly lost; it's one Restore away. A stale delete loses to a newer server edit. Response — per item, so the client can mark its local rows synced: ```json { "results": [ { "id": "...", "entity": "note", "status": "created", "sync_revision": 44 }, { "id": "...", "entity": "note", "status": "kept", "sync_revision": 40 }, { "id": "...", "entity": "label","status": "rejected","error": "name in use" } ] } ``` `status` ∈ `created | applied | kept | noop | rejected`. `kept` means the server had a newer edit and the client should adopt the server version on its next pull. ## Attachments (blobs) Metadata rides the delta feed (`id, url, mime, size, sha256`); the bytes move over the existing routes: - **Upload:** `POST /api/notes//attachments` (multipart, field `file`; optional field `id` to keep a client-minted attachment id). Re-uploading an id the note already has is an idempotent no-op. Server stores + hashes the bytes. - **Download:** `GET /api/notes//attachments/` (owner/shared scoped). - The client uses `sha256` to skip blobs it already holds and to verify integrity after download. (Currently image mimes only; broadening to any file is tracked separately.) ## A sync cycle 1. **Push** local changes since the last sync (batched). Apply the per-item results (mark synced, adopt server version where `kept`). 2. **Pull** from the stored `since` cursor until `has_more` is false. Upsert notes/labels into the local store; delete rows whose `purged_at` is set; download any attachment blobs referenced by a new/changed `sha256`. 3. Persist the new `cursor`. Initial sync is the same with `since=0`. Because everything is keyed by stable ids and a monotonic cursor, the cycle is **idempotent and resumable** — a client can crash mid-sync and simply resume from its last persisted cursor.