thoughtsync/docs/sync.md

# ThoughtSync sync protocol

The contract the local-first native clients (Tauri desktop, Android) implement
against. The server is the **sync hub**: each client keeps a full local store
(SQLite), works fully offline, and reconciles with the server when linked. The
web app does **not** use this API — it stays on the live REST API (`/api/notes`,
…) as an online-only stopgap.

All sync endpoints live under `/api/sync`. Everything is **owner-scoped** and
**deterministic** (no AI).

> **No Postgres CI lane.** Trigger/migration/sync behavior is verified by the
> operator on deploy, not in CI. Pure logic (LWW comparator, paging cursor,
> token hashing) is unit-tested.

## Protocol versioning — the compatibility handshake

Clients and servers update on their own schedules; a self-hosted server can sit on
an older release than the desktop app for months. So the wire protocol is
versioned **separately from either program's release version**, and each side
declares two numbers: what it speaks, and the oldest counterpart it accepts.

| | server (`src/thoughtsync/sync.py`) | client (`desktop/src-tauri/src/sync/compat.rs`) |
|---|---|---|
| speaks | `SYNC_PROTOCOL_VERSION` | `CLIENT_PROTOCOL_VERSION` |
| accepts down to | `MIN_CLIENT_PROTOCOL_VERSION` | `MIN_SERVER_PROTOCOL_VERSION` |

The server publishes its half on the **public, unauthenticated** `GET /api/config`
— a client must be able to ask "can I talk to you?" before it holds a device
token, or even has an account:

```json
{ "site_name": "...", "version": "0.1.0",
  "sync_protocol_version": 1,
  "min_client_protocol_version": 1,
  "sync_features": ["notes", "labels", "attachments", "tombstones", "revisions"],
  "trash_retention_days": 30 }
```

The client identifies itself on every request with
`X-ThoughtSync-Client: thoughtsync-desktop/<app version>` and
`X-ThoughtSync-Protocol: <n>`.

### `sync_features` — why versions alone aren't enough

A version number can only say "newer" or "older". `sync_features` names
capabilities, so a client tests for the one it needs instead of inferring it from
a number. That is what keeps an **additive** change from forcing a lockstep
upgrade: a newer client meeting an older server drops the missing feature and
syncs everything else.

### The policy

- **Any wire change** → bump `SYNC_PROTOCOL_VERSION`.
  - v2 (M13): `kind` and `title` left the wire; **floor raised**, because a v1
    client kept pushing both and read back notes carrying neither — and `title` was
    the note's NAME, so an old client showed nameless notes.
  - v3: attachments/tombstones/revisions.
  - v4 (M315): `color` left the note; **floor NOT raised**. Both directions degrade
    in silence and neither loses anything visible — an old client reading a v4 note
    falls back to the colour it derives locally, and one pushing `color` has the key
    ignored. The test is not "did a field leave" but "does either side end up
    showing something wrong".
- **Additive change** (a new field, a new capability) → add a `sync_features`
  name. Do **not** raise a minimum. Old clients keep working.
- **Breaking change only** → raise `MIN_CLIENT_PROTOCOL_VERSION` (or the client's
  `MIN_SERVER_PROTOCOL_VERSION`). This is the switch that hard-blocks the other
  side, so it is the one to be stingy with.
- Never gate behavior on the *release* version (`version`) — it's for display.

### The three outcomes

The client evaluates the advertisement (`compat::evaluate`) and gets exactly one
of:

- **ok** — full parity; sync everything.
- **degraded** — safe to sync, but named capabilities are unavailable here; the UI
  says which.
- **incompatible** — do not sync. Carries `client_must_update` so the message can
  point at the side that can actually fix it, rather than just saying
  "incompatible".

A server that predates this handshake sends no protocol fields at all. That is
treated as **incompatible (update the server)** — deliberately not as a parse
error, which would look to the user like they mistyped the URL.

## Authentication — device bearer tokens

Native clients authenticate with a long-lived **device token**, not a session
cookie. Only the token's SHA-256 hash is stored server-side; the plaintext is
shown once at creation.

- **First link (no session):** `POST /api/auth/device-login` with
  `{email, password, name}` → `{token, device, user}`. Store `token`; send it as
  `Authorization: Bearer <token>` on every subsequent call.
- **From the web app (already signed in):** the user creates a token at
  `/account` ("Linked devices"); `POST /api/auth/devices` `{name}` → `{token,
  device}`. They paste it into the native app.
- **Manage:** `GET /api/auth/devices` (list), `DELETE /api/auth/devices/<id>`
  (revoke). A revoked token stops authenticating immediately.
- **Self-revoke:** `DELETE /api/auth/devices/self` retires the token presented in
  the `Authorization` header. This is what a native client calls when the user
  unlinks. It exists because a client can't use the id-keyed route: a token pasted
  from the web app arrives without a device id, and `/api/auth/me` describes the
  user, not the device row. A caller authenticated by session cookie gets `400` —
  it holds no device token, so there is nothing for it to mean.

  Unlinking is never blocked on this call. If the server is unreachable or too old
  to have the route, the client still unlinks locally and tells the user the token
  is still live and where to revoke it.

Every authenticated request (sync or otherwise) accepts the bearer token in
place of the session cookie.

## The revision cursor

Every syncable row carries a monotonic **`sync_revision`** (bigint), assigned by
a database trigger from a single shared sequence (`sync_revision_seq`) on every
insert/update. Because it's a shared sequence, one integer is a total-order
watermark across **all** of a user's notes and labels. It is not a timestamp —
it is immune to clock skew and is only ever compared with `>`.

The client persists the highest cursor it has fully consumed and passes it back
as `?since=`. `since=0` (or absent) is a **full initial sync**.

## Entities and what syncs

- **Note** — the primary sync unit. It travels with its **checklist items,
  label memberships, and attachment metadata inline** (same shape as the REST
  serialization). A change to any child bumps the parent note's `sync_revision`,
  so pulling the note re-syncs the whole thing.
- **Label** — the label *catalog* (name + color) syncs as its own entity so a
  rename/recolor/delete propagates independently of notes.
- **Derived, NOT synced:** `#tags` are parsed from the note body. Clients recompute
  them locally; the server recomputes them on push. They never travel over the wire.
  (`[[wiki-links]]` were derived the same way until they were removed — note 2897.)
- **Attachment blobs** sync by id over the existing upload/download routes (see
  Attachments below); only their metadata rides the delta feed.

## Tombstones (deletes)

Two levels, both propagate:

- **Trash** — `deleted_at` is a normal field, and it's **on the wire**: a trashed
  note still syncs with its content, the client shows it in its Trash, and the
  timestamp is what the client counts the retention window against. Restoring
  clears it.
- **Purge (permanent delete)** — becomes a **content-less tombstone**: `purged_at`
  is set, title/body/items/labels/attachments/previews/revisions are cleared or
  removed, and the row is kept. A client seeing `purged_at != null` deletes the row
  from its local store. `deleted_at` deliberately SURVIVES a purge, so ordinary
  server-side queries (`deleted_at IS NULL`) never see a tombstone as a live note.
  Tombstones themselves are retained indefinitely (cheap for a personal store);
  revisit if they ever grow large.

### Retention — trash expires

A trashed note is purged automatically once it is older than the server's
`trash_retention_days` setting (default **30**, `0` = keep forever), advertised on
`/api/config` so a client can show the countdown. A background sweep on the server
does the work; clients learn about it as ordinary tombstones and need no special
handling.

**A linked client must not run its own expiry.** The server owns the policy — one
clock, one window. A client that purged on its own schedule could destroy a note
the server was deliberately keeping and then push that delete upstream. An
*unlinked* client (offline-only, no server to defer to) expires its own trash on
its own default, which is the only case where nothing else can.

## Pull — `GET /api/sync/changes`

Query: `?since=<cursor>&limit=<n>` (limit default 500, max 1000).

Response:

```json
{
  "notes":  [ { "...full note...", "trashed": false, "deleted_at": null,
               "sync_revision": 42, "purged_at": null } ],
  "labels": [ { "id": "...", "name": "...", "color": "...",
               "sync_revision": 43, "purged_at": null, "created_at": "..." } ],
  "cursor": 43,
  "has_more": false
}
```

Returns **all** of the caller's notes + labels (any state — active, archived,
trash, purged) whose `sync_revision > since`, ascending by revision. Notes and
labels share the sequence, so paging merges the two streams: when either stream
fills a page, `cursor` advances only to the **smaller** of the two page
boundaries, so nothing between `cursor` and the next pull is skipped. Loop while
`has_more` is true, advancing `since = cursor` each time.

Note attachment metadata carries `{id, url, mime, size, sha256}`.

## Push — `POST /api/sync/push`

Body: `{ "changes": [ ... ] }` (max 1000 per batch). Each change:

```json
{ "entity": "note", "id": "<uuid>", "op": "upsert", "edited_at": "<iso8601>",
  "body": "...",
  "pinned": false, "archived": false, "trashed": false, "remind_at": null,
  "recurrence": null, "position": 0,
  "label_ids": ["<uuid>", ...], "created_at": "<iso8601, on create>" }
```

- **Client-generated ids.** Notes/labels are UUIDs; the client mints the id when
  it creates the row offline and sends it here. Create-if-absent, else update.
- **Whole-note semantics.** A note upsert carries the client's *full* current
  state (not a partial patch) — the server overwrites all scalar fields and sets
  manual label memberships from `label_ids` (tag-sourced labels are re-derived
  from the body). `#tags` are recomputed server-side. A checklist is `- [ ] ` lines
  inside `body` (M304), so there is no separate `items` array.
- **Fields a change may still carry, and the server reads past.** `title` and
  `kind` (removed in v2), `items` (M304) and `color` (v4, M315). The server reads
  its payload key by key and never validates the shape, which is exactly what lets
  an older client keep pushing a field this one has stopped storing — see the
  version policy above for why none of those needed a floor raise on their own.
- **`op: "delete"`** purges (tombstones) the row. Trashing is just an upsert with
  `trashed: true`.
- **Labels:** `{entity: "label", op: "upsert"|"delete", id, edited_at, name,
  color}`. A per-owner name clash on a *different* id is rejected (fix locally
  and retry).

### Conflict resolution — last-write-wins + history

On a clash the **most-recently-edited version wins**, by `edited_at`
(client wall-clock) compared against the server row's last-edit time
(`updated_at`). The client applies iff `client.edited_at >= server.updated_at`.
When a winning upsert overwrites an existing title/body, the server first
**snapshots the overwritten version into the note's version history**
(`note_revisions`) — so a "lost" edit is never truly lost; it's one Restore away.
A stale delete loses to a newer server edit.

Response — per item, so the client can mark its local rows synced:

```json
{ "results": [
  { "id": "...", "entity": "note", "status": "created", "sync_revision": 44 },
  { "id": "...", "entity": "note", "status": "kept",    "sync_revision": 40 },
  { "id": "...", "entity": "label","status": "rejected","error": "name in use" }
] }
```

`status` ∈ `created | applied | kept | noop | rejected`. `kept` means the server
had a newer edit and the client should adopt the server version on its next pull.

## Attachments (blobs)

Metadata rides the delta feed (`id, url, mime, size, sha256`); the bytes move
over the existing routes:

- **Upload:** `POST /api/notes/<note_id>/attachments` (multipart, field `file`;
  optional field `id` to keep a client-minted attachment id). Re-uploading an id
  the note already has is an idempotent no-op. Server stores + hashes the bytes.
- **Download:** `GET /api/notes/<note_id>/attachments/<id>` (owner/shared scoped).
- The client uses `sha256` to skip blobs it already holds and to verify
  integrity after download. (Currently image mimes only; broadening to any file
  is tracked separately.)

## A sync cycle

1. **Push** local changes since the last sync (batched). Apply the per-item
   results (mark synced, adopt server version where `kept`).
2. **Pull** from the stored `since` cursor until `has_more` is false. Upsert
   notes/labels into the local store; delete rows whose `purged_at` is set;
   download any attachment blobs referenced by a new/changed `sha256`.
3. Persist the new `cursor`.

Initial sync is the same with `since=0`. Because everything is keyed by stable
ids and a monotonic cursor, the cycle is **idempotent and resumable** — a client
can crash mid-sync and simply resume from its last persisted cursor.