inkwell/docs/sync.md

# Inkwell sync protocol

The contract the local-first native clients (Tauri desktop, Android) implement
against. The server is the **sync hub**: each client keeps a full local store
(SQLite), works fully offline, and reconciles with the server when linked. The
web app does **not** use this API — it stays on the live REST API (`/api/notes`,
…) as an online-only stopgap.

All sync endpoints live under `/api/sync`. Everything is **owner-scoped** and
**deterministic** (no AI).

> **No Postgres CI lane.** Trigger/migration/sync behavior is verified by the
> operator on deploy, not in CI. Pure logic (LWW comparator, paging cursor,
> token hashing) is unit-tested.

## Protocol versioning — the compatibility handshake

Clients and servers update on their own schedules; a self-hosted server can sit on
an older release than the desktop app for months. So the wire protocol is
versioned **separately from either program's release version**, and each side
declares two numbers: what it speaks, and the oldest counterpart it accepts.

| | server (`src/inkwell/sync.py`) | client (`desktop/src-tauri/src/sync/compat.rs`) |
|---|---|---|
| speaks | `SYNC_PROTOCOL_VERSION` | `CLIENT_PROTOCOL_VERSION` |
| accepts down to | `MIN_CLIENT_PROTOCOL_VERSION` | `MIN_SERVER_PROTOCOL_VERSION` |

The server publishes its half on the **public, unauthenticated** `GET /api/config`
— a client must be able to ask "can I talk to you?" before it holds a device
token, or even has an account:

```json
{ "site_name": "...", "version": "0.1.0",
  "sync_protocol_version": 1,
  "min_client_protocol_version": 1,
  "sync_features": ["notes", "labels", "attachments", "tombstones", "revisions"],
  "trash_retention_days": 30 }
```

The client identifies itself on every request with
`X-Inkwell-Client: inkwell-desktop/<app version>` and
`X-Inkwell-Protocol: <n>`.

### `sync_features` — why versions alone aren't enough

A version number can only say "newer" or "older". `sync_features` names
capabilities, so a client tests for the one it needs instead of inferring it from
a number. That is what keeps an **additive** change from forcing a lockstep
upgrade: a newer client meeting an older server drops the missing feature and
syncs everything else.

### The policy

- **Any wire change** → bump `SYNC_PROTOCOL_VERSION`.
  - v2 (M13): `kind` and `title` left the wire; **floor raised**, because a v1
    client kept pushing both and read back notes carrying neither — and `title` was
    the note's NAME, so an old client showed nameless notes.
  - v3: attachments/tombstones/revisions.
  - v4 (M315): `color` left the note; **floor NOT raised**. Both directions degrade
    in silence and neither loses anything visible — an old client reading a v4 note
    falls back to the colour it derives locally, and one pushing `color` has the key
    ignored. The test is not "did a field leave" but "does either side end up
    showing something wrong".
  - v5 (#5168): attachments became a sync entity — an upload route keyed by the
    client's id, and `attachment`/`preview` deletes in push. Additive, so it is
    the `attachment_sync` feature and the floor stays.
  - v6 (#5175): shared notes — see "Shared notes" below. Opt-in per request
    (`changes?shares=1`), so it is the `shares` feature and the floor stays: a v5
    client never asks and gets exactly its own notes.
  - v7 (#5176): a recipient's own pin, archive and position on a shared note —
    see "Shared notes". Additive, so it is the `shared_state` feature and the
    floor stays: a v6 client pushes only text, and reads its own state as if the
    note were simply unpinned.
- **Additive change** (a new field, a new capability) → add a `sync_features`
  name. Do **not** raise a minimum. Old clients keep working.
- **Breaking change only** → raise `MIN_CLIENT_PROTOCOL_VERSION` (or the client's
  `MIN_SERVER_PROTOCOL_VERSION`). This is the switch that hard-blocks the other
  side, so it is the one to be stingy with.
- Never gate behavior on the *release* version (`version`) — it's for display.

### The three outcomes

The client evaluates the advertisement (`compat::evaluate`) and gets exactly one
of:

- **ok** — full parity; sync everything.
- **degraded** — safe to sync, but named capabilities are unavailable here; the UI
  says which.
- **incompatible** — do not sync. Carries `client_must_update` so the message can
  point at the side that can actually fix it, rather than just saying
  "incompatible".

A server that predates this handshake sends no protocol fields at all. That is
treated as **incompatible (update the server)** — deliberately not as a parse
error, which would look to the user like they mistyped the URL.

## Authentication — device bearer tokens

Native clients authenticate with a long-lived **device token**, not a session
cookie. Only the token's SHA-256 hash is stored server-side; the plaintext is
shown once at creation.

- **First link (no session):** `POST /api/auth/device-login` with
  `{email, password, name}` → `{token, device, user}`. Store `token`; send it as
  `Authorization: Bearer <token>` on every subsequent call.
- **From the web app (already signed in):** the user creates a token at
  `/account` ("Linked devices"); `POST /api/auth/devices` `{name}` → `{token,
  device}`. They paste it into the native app.
- **Manage:** `GET /api/auth/devices` (list), `DELETE /api/auth/devices/<id>`
  (revoke). A revoked token stops authenticating immediately.
- **Self-revoke:** `DELETE /api/auth/devices/self` retires the token presented in
  the `Authorization` header. This is what a native client calls when the user
  unlinks. It exists because a client can't use the id-keyed route: a token pasted
  from the web app arrives without a device id, and `/api/auth/me` describes the
  user, not the device row. A caller authenticated by session cookie gets `400` —
  it holds no device token, so there is nothing for it to mean.

  Unlinking is never blocked on this call. If the server is unreachable or too old
  to have the route, the client still unlinks locally and tells the user the token
  is still live and where to revoke it.

Every authenticated request (sync or otherwise) accepts the bearer token in
place of the session cookie.

## The revision cursor

Every syncable row carries a monotonic **`sync_revision`** (bigint), assigned by
a database trigger from a single shared sequence (`sync_revision_seq`) on every
insert/update. Because it's a shared sequence, one integer is a total-order
watermark across **all** of a user's notes and labels. It is not a timestamp —
it is immune to clock skew and is only ever compared with `>`.

The client persists the highest cursor it has fully consumed and passes it back
as `?since=`. `since=0` (or absent) is a **full initial sync**.

## Entities and what syncs

- **Note** — the primary sync unit. It travels with its **checklist items,
  label memberships, and attachment metadata inline** (same shape as the REST
  serialization). A change to any child bumps the parent note's `sync_revision`,
  so pulling the note re-syncs the whole thing.
- **Label** — the label *catalog* (name + color) syncs as its own entity so a
  rename/recolor/delete propagates independently of notes.
- **Derived, NOT synced:** `#tags` are parsed from the note body. Clients recompute
  them locally; the server recomputes them on push. They never travel over the wire.
  (`[[wiki-links]]` were derived the same way until they were removed — note 2897.)
- **Attachment blobs** sync by id over the existing upload/download routes (see
  Attachments below); only their metadata rides the delta feed.

## Tombstones (deletes)

Two levels, both propagate:

- **Trash** — `deleted_at` is a normal field, and it's **on the wire**: a trashed
  note still syncs with its content, the client shows it in its Trash, and the
  timestamp is what the client counts the retention window against. Restoring
  clears it.
- **Purge (permanent delete)** — becomes a **content-less tombstone**: `purged_at`
  is set, title/body/items/labels/attachments/previews/revisions are cleared or
  removed, and the row is kept. A client seeing `purged_at != null` deletes the row
  from its local store. `deleted_at` deliberately SURVIVES a purge, so ordinary
  server-side queries (`deleted_at IS NULL`) never see a tombstone as a live note.
  Tombstones themselves are retained indefinitely (cheap for a personal store);
  revisit if they ever grow large.

### Retention — trash expires

A trashed note is purged automatically once it is older than the server's
`trash_retention_days` setting (default **30**, `0` = keep forever), advertised on
`/api/config` so a client can show the countdown. A background sweep on the server
does the work; clients learn about it as ordinary tombstones and need no special
handling.

**A linked client must not run its own expiry.** The server owns the policy — one
clock, one window. A client that purged on its own schedule could destroy a note
the server was deliberately keeping and then push that delete upstream. An
*unlinked* client (offline-only, no server to defer to) expires its own trash on
its own default, which is the only case where nothing else can.

## Pull — `GET /api/sync/changes`

Query: `?since=<cursor>&limit=<n>` (limit default 500, max 1000).

Response:

```json
{
  "notes":  [ { "...full note...", "trashed": false, "deleted_at": null,
               "sync_revision": 42, "purged_at": null } ],
  "labels": [ { "id": "...", "name": "...", "color": "...",
               "sync_revision": 43, "purged_at": null, "created_at": "..." } ],
  "cursor": 43,
  "has_more": false
}
```

Returns **all** of the caller's notes + labels (any state — active, archived,
trash, purged) whose `sync_revision > since`, ascending by revision. Notes and
labels share the sequence, so paging merges the two streams: when either stream
fills a page, `cursor` advances only to the **smaller** of the two page
boundaries, so nothing between `cursor` and the next pull is skipped. Loop while
`has_more` is true, advancing `since = cursor` each time.

Note attachment metadata carries `{id, url, mime, size, sha256}`.

### Shared notes (`shares`, v6)

With `?shares=1` the feed carries every note the caller can **see**: their own, plus
those shared with them directly or through a group. Each note then also says how
it is held:

```json
{ "permission": "owner" | "edit" | "view",
  "shared": true,
  "shared_by": { "id": "...", "display_name": "..." } }
```

`shared_by` is null on the caller's own notes, and `shared` says whether they have
shared it with anyone. A note shared with the caller comes with `labels: []`:
labels are personal, and a recipient files nothing under the owner's tags.

Granting or changing a share moves the note to a new revision (without touching
`updated_at`, so last-write-wins is unaffected), which is how it passes a
recipient's cursor. Joining a group a note is shared with does the same for the
new member (#5177). Ending a share — or the owner purging the note, the person
leaving the group, or the admin deleting it — adds the note to the `revoked` list of
each person who can no longer see it:

```json
{ "notes": [ ... ], "labels": [ ... ], "revoked": ["<note id>", ...],
  "cursor": 51, "has_more": false }
```

A client deletes its copy of each revoked note. Revocations draw from the same
sequence and page on the same cursor as notes and labels (three streams now; the
smallest full boundary wins). Sharing with the person again deletes their
revocation, so a device that never saw it is never told to drop a note it has.

**Pin, archive and position are personal** (`shared_state`, v7). On a note shared
with the caller, the feed's `pinned`, `archived` and `position` are the caller's
own: unpinned and unarchived until they change them, and the owner's `position`
until they move it. They live in a per-person row with a revision of its own, so
the note's `sync_revision` in the feed is the later of the note's and that row's —
a recipient pinning a note reaches their devices and no one else's.

## Push — `POST /api/sync/push`

Body: `{ "changes": [ ... ] }` (max 1000 per batch). Each change:

```json
{ "entity": "note", "id": "<uuid>", "op": "upsert", "edited_at": "<iso8601>",
  "body": "...",
  "pinned": false, "archived": false, "trashed": false, "remind_at": null,
  "recurrence": null, "position": 0,
  "label_ids": ["<uuid>", ...], "created_at": "<iso8601, on create>" }
```

- **Client-generated ids.** Notes/labels are UUIDs; the client mints the id when
  it creates the row offline and sends it here. Create-if-absent, else update.
- **Whole-note semantics.** A note upsert carries the client's *full* current
  state (not a partial patch) — the server overwrites all scalar fields and sets
  manual label memberships from `label_ids` (tag-sourced labels are re-derived
  from the body). `#tags` are recomputed server-side. A checklist is `- [ ] ` lines
  inside `body` (M304), so there is no separate `items` array.
- **Fields a change may still carry, and the server reads past.** `title` and
  `kind` (removed in v2), `items` (M304) and `color` (v4, M315). The server reads
  its payload key by key and never validates the shape, which is exactly what lets
  an older client keep pushing a field this one has stopped storing — see the
  version policy above for why none of those needed a floor raise on their own.
- **`op: "delete"`** purges (tombstones) the row. Trashing is just an upsert with
  `trashed: true`.
- **Labels:** `{entity: "label", op: "upsert"|"delete", id, edited_at, name,
  color}`. A per-owner name clash on a *different* id is rejected (fix locally
  and retry).
- **Attachments and link previews** (`attachment_sync`): `{entity:
  "attachment"|"preview", op: "delete", id, edited_at}`. Delete is the only op —
  attachments are created by upload (below) and previews by the server. A removal
  is **not** weighed under last-write-wins: there is no rival version of a
  removed file, so it always applies. A row the caller can't see answers `noop`,
  the same as one that never existed. Removals are explicit rather than "the
  note's current attachment set" on purpose: push runs before pull, so a set
  would delete an attachment another device added that this one has not seen.
- **A note someone else owns** (`shares`): an upsert may carry two things, each
  under its own last-write-wins.
  - Its `body`, from a recipient at `edit`, compared on `edited_at` against the
    note. From a `view` recipient a body is `rejected` ("only its owner can change
    that").
  - Their own `pinned`, `archived` and `position` (`shared_state`), from anyone it
    is shared with, compared on **`state_at`** against what they last set. A
    client stamps `state_at` apart from the text's `edited_at` so that pinning a
    copy whose text is out of date never makes that text look newer than the
    owner's edit. Without `state_at` the three are ignored.

  Trashing, reminders, labels and deleting stay the owner's: those fields are
  ignored and a delete is `rejected`. The answer is `applied` if either part
  applied, else `kept`, with the revision as that caller's feed would show it. A
  note the caller can't see at all answers the generic "cannot apply".

### Conflict resolution — last-write-wins + history

On a clash the **most-recently-edited version wins**, by `edited_at`
(client wall-clock) compared against the server row's last-edit time
(`updated_at`). The client applies iff `client.edited_at >= server.updated_at`.
When a winning upsert overwrites an existing title/body, the server first
**snapshots the overwritten version into the note's version history**
(`note_revisions`) — so a "lost" edit is never truly lost; it's one Restore away.
A stale delete loses to a newer server edit.

Response — per item, so the client can mark its local rows synced:

```json
{ "results": [
  { "id": "...", "entity": "note", "status": "created", "sync_revision": 44 },
  { "id": "...", "entity": "note", "status": "kept",    "sync_revision": 40 },
  { "id": "...", "entity": "label","status": "rejected","error": "name in use" }
] }
```

`status` ∈ `created | applied | kept | noop | rejected`. `kept` means the server
had a newer edit and the client should adopt the server version on its next pull.

## Attachments (blobs)

Metadata rides the delta feed (`id, url, mime, size, sha256`); the bytes move
over the existing routes:

- **Sync upload** (`attachment_sync`): `PUT /api/sync/attachments/<id>?note_id=
  &filename=&sha256=`, body = the raw bytes, type in `Content-Type`. For a file
  attached offline, sent after the note itself has landed. `201` stores it under
  the client's id; `200 {"status": "exists"}` if the note already holds that id
  (a retry); `400` if the bytes don't hash to `sha256`; `409` if the id belongs to
  another note; `413` over `max_attachment_mb`; `404` for a note the caller
  doesn't own. A `4xx` other than `404` is permanent — the client records it on
  the attachment and stops retrying.
- **Web upload:** `POST /api/notes/<note_id>/attachments` (multipart, field
  `file`; optional field `id`, same idempotency).
- **Download:** `GET /api/notes/<note_id>/attachments/<id>` (owner/shared scoped).
- The client uses `sha256` to skip blobs it already holds and to verify
  integrity after download. Any file type syncs.
- Every insert or delete of an attachment or a link preview bumps its note's
  `sync_revision` (triggers from 0015 and 0031), so a change to either reaches
  other devices as the note.

## A sync cycle

1. **Push** local changes since the last sync (batched). Apply the per-item
   results (mark synced, adopt server version where `kept`). Then upload the bytes
   of attachments added offline whose notes have now landed.
2. **Pull** from the stored `since` cursor until `has_more` is false. Upsert
   notes/labels into the local store; delete rows whose `purged_at` is set;
   download any attachment blobs referenced by a new/changed `sha256`.
3. Persist the new `cursor`.

Initial sync is the same with `since=0`. Because everything is keyed by stable
ids and a monotonic cursor, the cycle is **idempotent and resumable** — a client
can crash mid-sync and simply resume from its last persisted cursor.