thoughtsync/docs/sync.md

ThoughtSync sync protocol

The contract the local-first native clients (Tauri desktop, Android) implement against. The server is the sync hub: each client keeps a full local store (SQLite), works fully offline, and reconciles with the server when linked. The web app does not use this API — it stays on the live REST API (/api/notes, …) as an online-only stopgap.

All sync endpoints live under /api/sync. Everything is owner-scoped and deterministic (no AI).

No Postgres CI lane. Trigger/migration/sync behavior is verified by the operator on deploy, not in CI. Pure logic (LWW comparator, paging cursor, token hashing) is unit-tested.

Protocol versioning — the compatibility handshake

Clients and servers update on their own schedules; a self-hosted server can sit on an older release than the desktop app for months. So the wire protocol is versioned separately from either program's release version, and each side declares two numbers: what it speaks, and the oldest counterpart it accepts.

	server (src/thoughtsync/sync.py)	client (desktop/src-tauri/src/sync/compat.rs)
speaks	SYNC_PROTOCOL_VERSION	CLIENT_PROTOCOL_VERSION
accepts down to	MIN_CLIENT_PROTOCOL_VERSION	MIN_SERVER_PROTOCOL_VERSION

The server publishes its half on the public, unauthenticated GET /api/config — a client must be able to ask "can I talk to you?" before it holds a device token, or even has an account:

{ "site_name": "...", "version": "0.1.0",
  "sync_protocol_version": 1,
  "min_client_protocol_version": 1,
  "sync_features": ["notes", "labels", "attachments", "tombstones", "revisions"],
  "trash_retention_days": 30 }

The client identifies itself on every request with X-ThoughtSync-Client: thoughtsync-desktop/<app version> and X-ThoughtSync-Protocol: <n>.

sync_features — why versions alone aren't enough

A version number can only say "newer" or "older". sync_features names capabilities, so a client tests for the one it needs instead of inferring it from a number. That is what keeps an additive change from forcing a lockstep upgrade: a newer client meeting an older server drops the missing feature and syncs everything else.

The policy

• Any wire change → bump SYNC_PROTOCOL_VERSION.
  • v2 (M13): kind and title left the wire; floor raised, because a v1 client kept pushing both and read back notes carrying neither — and title was the note's NAME, so an old client showed nameless notes.
  • v3: attachments/tombstones/revisions.
  • v4 (M315): color left the note; floor NOT raised. Both directions degrade in silence and neither loses anything visible — an old client reading a v4 note falls back to the colour it derives locally, and one pushing color has the key ignored. The test is not "did a field leave" but "does either side end up showing something wrong".
• Additive change (a new field, a new capability) → add a sync_features name. Do not raise a minimum. Old clients keep working.
• Breaking change only → raise MIN_CLIENT_PROTOCOL_VERSION (or the client's MIN_SERVER_PROTOCOL_VERSION). This is the switch that hard-blocks the other side, so it is the one to be stingy with.
• Never gate behavior on the release version (version) — it's for display.

The three outcomes

The client evaluates the advertisement (compat::evaluate) and gets exactly one of:

• ok — full parity; sync everything.
• degraded — safe to sync, but named capabilities are unavailable here; the UI says which.
• incompatible — do not sync. Carries client_must_update so the message can point at the side that can actually fix it, rather than just saying "incompatible".

A server that predates this handshake sends no protocol fields at all. That is treated as incompatible (update the server) — deliberately not as a parse error, which would look to the user like they mistyped the URL.

Authentication — device bearer tokens

Native clients authenticate with a long-lived device token, not a session cookie. Only the token's SHA-256 hash is stored server-side; the plaintext is shown once at creation.

• First link (no session): POST /api/auth/device-login with {email, password, name} → {token, device, user}. Store token; send it as Authorization: Bearer <token> on every subsequent call.

• From the web app (already signed in): the user creates a token at /account ("Linked devices"); POST /api/auth/devices {name} → {token, device}. They paste it into the native app.

• Manage: GET /api/auth/devices (list), DELETE /api/auth/devices/<id> (revoke). A revoked token stops authenticating immediately.

• Self-revoke: DELETE /api/auth/devices/self retires the token presented in the Authorization header. This is what a native client calls when the user unlinks. It exists because a client can't use the id-keyed route: a token pasted from the web app arrives without a device id, and /api/auth/me describes the user, not the device row. A caller authenticated by session cookie gets 400 — it holds no device token, so there is nothing for it to mean.

  Unlinking is never blocked on this call. If the server is unreachable or too old to have the route, the client still unlinks locally and tells the user the token is still live and where to revoke it.

Every authenticated request (sync or otherwise) accepts the bearer token in place of the session cookie.

The revision cursor

Every syncable row carries a monotonic sync_revision (bigint), assigned by a database trigger from a single shared sequence (sync_revision_seq) on every insert/update. Because it's a shared sequence, one integer is a total-order watermark across all of a user's notes and labels. It is not a timestamp — it is immune to clock skew and is only ever compared with >.

The client persists the highest cursor it has fully consumed and passes it back as ?since=. since=0 (or absent) is a full initial sync.

Entities and what syncs

• Note — the primary sync unit. It travels with its checklist items, label memberships, and attachment metadata inline (same shape as the REST serialization). A change to any child bumps the parent note's sync_revision, so pulling the note re-syncs the whole thing.
• Label — the label catalog (name + color) syncs as its own entity so a rename/recolor/delete propagates independently of notes.
• Derived, NOT synced: #tags are parsed from the note body. Clients recompute them locally; the server recomputes them on push. They never travel over the wire. ([[wiki-links]] were derived the same way until they were removed — note 2897.)
• Attachment blobs sync by id over the existing upload/download routes (see Attachments below); only their metadata rides the delta feed.

Tombstones (deletes)

Two levels, both propagate:

• Trash — deleted_at is a normal field, and it's on the wire: a trashed note still syncs with its content, the client shows it in its Trash, and the timestamp is what the client counts the retention window against. Restoring clears it.
• Purge (permanent delete) — becomes a content-less tombstone: purged_at is set, title/body/items/labels/attachments/previews/revisions are cleared or removed, and the row is kept. A client seeing purged_at != null deletes the row from its local store. deleted_at deliberately SURVIVES a purge, so ordinary server-side queries (deleted_at IS NULL) never see a tombstone as a live note. Tombstones themselves are retained indefinitely (cheap for a personal store); revisit if they ever grow large.

Retention — trash expires

A trashed note is purged automatically once it is older than the server's trash_retention_days setting (default 30, 0 = keep forever), advertised on /api/config so a client can show the countdown. A background sweep on the server does the work; clients learn about it as ordinary tombstones and need no special handling.

A linked client must not run its own expiry. The server owns the policy — one clock, one window. A client that purged on its own schedule could destroy a note the server was deliberately keeping and then push that delete upstream. An unlinked client (offline-only, no server to defer to) expires its own trash on its own default, which is the only case where nothing else can.

Pull — GET /api/sync/changes

Query: ?since=<cursor>&limit=<n> (limit default 500, max 1000).

Response:

{
  "notes":  [ { "...full note...", "trashed": false, "deleted_at": null,
               "sync_revision": 42, "purged_at": null } ],
  "labels": [ { "id": "...", "name": "...", "color": "...",
               "sync_revision": 43, "purged_at": null, "created_at": "..." } ],
  "cursor": 43,
  "has_more": false
}

Returns all of the caller's notes + labels (any state — active, archived, trash, purged) whose sync_revision > since, ascending by revision. Notes and labels share the sequence, so paging merges the two streams: when either stream fills a page, cursor advances only to the smaller of the two page boundaries, so nothing between cursor and the next pull is skipped. Loop while has_more is true, advancing since = cursor each time.

Note attachment metadata carries {id, url, mime, size, sha256}.

Push — POST /api/sync/push

Body: { "changes": [ ... ] } (max 1000 per batch). Each change:

{ "entity": "note", "id": "<uuid>", "op": "upsert", "edited_at": "<iso8601>",
  "body": "...",
  "pinned": false, "archived": false, "trashed": false, "remind_at": null,
  "recurrence": null, "position": 0,
  "label_ids": ["<uuid>", ...], "created_at": "<iso8601, on create>" }

• Client-generated ids. Notes/labels are UUIDs; the client mints the id when it creates the row offline and sends it here. Create-if-absent, else update.
• Whole-note semantics. A note upsert carries the client's full current state (not a partial patch) — the server overwrites all scalar fields and sets manual label memberships from label_ids (tag-sourced labels are re-derived from the body). #tags are recomputed server-side. A checklist is - [ ] lines inside body (M304), so there is no separate items array.
• Fields a change may still carry, and the server reads past. title and kind (removed in v2), items (M304) and color (v4, M315). The server reads its payload key by key and never validates the shape, which is exactly what lets an older client keep pushing a field this one has stopped storing — see the version policy above for why none of those needed a floor raise on their own.
• op: "delete" purges (tombstones) the row. Trashing is just an upsert with trashed: true.
• Labels: {entity: "label", op: "upsert"|"delete", id, edited_at, name, color}. A per-owner name clash on a different id is rejected (fix locally and retry).

Conflict resolution — last-write-wins + history

On a clash the most-recently-edited version wins, by edited_at (client wall-clock) compared against the server row's last-edit time (updated_at). The client applies iff client.edited_at >= server.updated_at. When a winning upsert overwrites an existing title/body, the server first snapshots the overwritten version into the note's version history (note_revisions) — so a "lost" edit is never truly lost; it's one Restore away. A stale delete loses to a newer server edit.

Response — per item, so the client can mark its local rows synced:

{ "results": [
  { "id": "...", "entity": "note", "status": "created", "sync_revision": 44 },
  { "id": "...", "entity": "note", "status": "kept",    "sync_revision": 40 },
  { "id": "...", "entity": "label","status": "rejected","error": "name in use" }
] }

status ∈ created | applied | kept | noop | rejected. kept means the server had a newer edit and the client should adopt the server version on its next pull.

Attachments (blobs)

Metadata rides the delta feed (id, url, mime, size, sha256); the bytes move over the existing routes:

• Upload: POST /api/notes/<note_id>/attachments (multipart, field file; optional field id to keep a client-minted attachment id). Re-uploading an id the note already has is an idempotent no-op. Server stores + hashes the bytes.
• Download: GET /api/notes/<note_id>/attachments/<id> (owner/shared scoped).
• The client uses sha256 to skip blobs it already holds and to verify integrity after download. (Currently image mimes only; broadening to any file is tracked separately.)

A sync cycle

1. Push local changes since the last sync (batched). Apply the per-item results (mark synced, adopt server version where kept).
2. Pull from the stored since cursor until has_more is false. Upsert notes/labels into the local store; delete rows whose purged_at is set; download any attachment blobs referenced by a new/changed sha256.
3. Persist the new cursor.

Initial sync is the same with since=0. Because everything is keyed by stable ids and a monotonic cursor, the cycle is idempotent and resumable — a client can crash mid-sync and simply resume from its last persisted cursor.