inkwell/docs/sync.md

Inkwell sync protocol

The contract the local-first native clients (Tauri desktop, Android) implement against. The server is the sync hub: each client keeps a full local store (SQLite), works fully offline, and reconciles with the server when linked. The web app does not use this API — it stays on the live REST API (/api/notes, …) as an online-only stopgap.

All sync endpoints live under /api/sync. Everything is owner-scoped and deterministic (no AI).

No Postgres CI lane. Trigger/migration/sync behavior is verified by the operator on deploy, not in CI. Pure logic (LWW comparator, paging cursor, token hashing) is unit-tested.

Protocol versioning — the compatibility handshake

Clients and servers update on their own schedules; a self-hosted server can sit on an older release than the desktop app for months. So the wire protocol is versioned separately from either program's release version, and each side declares two numbers: what it speaks, and the oldest counterpart it accepts.

	server (src/inkwell/sync.py)	client (desktop/src-tauri/src/sync/compat.rs)
speaks	SYNC_PROTOCOL_VERSION	CLIENT_PROTOCOL_VERSION
accepts down to	MIN_CLIENT_PROTOCOL_VERSION	MIN_SERVER_PROTOCOL_VERSION

The server publishes its half on the public, unauthenticated GET /api/config — a client must be able to ask "can I talk to you?" before it holds a device token, or even has an account:

{ "site_name": "...", "version": "0.1.0",
  "sync_protocol_version": 1,
  "min_client_protocol_version": 1,
  "sync_features": ["notes", "labels", "attachments", "tombstones", "revisions"],
  "trash_retention_days": 30 }

The client identifies itself on every request with X-Inkwell-Client: inkwell-desktop/<app version> and X-Inkwell-Protocol: <n>.

sync_features — why versions alone aren't enough

A version number can only say "newer" or "older". sync_features names capabilities, so a client tests for the one it needs instead of inferring it from a number. That is what keeps an additive change from forcing a lockstep upgrade: a newer client meeting an older server drops the missing feature and syncs everything else.

The policy

• Any wire change → bump SYNC_PROTOCOL_VERSION.
  • v2 (M13): kind and title left the wire; floor raised, because a v1 client kept pushing both and read back notes carrying neither — and title was the note's NAME, so an old client showed nameless notes.
  • v3: attachments/tombstones/revisions.
  • v4 (M315): color left the note; floor NOT raised. Both directions degrade in silence and neither loses anything visible — an old client reading a v4 note falls back to the colour it derives locally, and one pushing color has the key ignored. The test is not "did a field leave" but "does either side end up showing something wrong".
  • v5 (#5168): attachments became a sync entity — an upload route keyed by the client's id, and attachment/preview deletes in push. Additive, so it is the attachment_sync feature and the floor stays.
  • v6 (#5175): shared notes — see "Shared notes" below. Opt-in per request (changes?shares=1), so it is the shares feature and the floor stays: a v5 client never asks and gets exactly its own notes.
  • v7 (#5176): a recipient's own pin, archive and position on a shared note — see "Shared notes". Additive, so it is the shared_state feature and the floor stays: a v6 client pushes only text, and reads its own state as if the note were simply unpinned.
• Additive change (a new field, a new capability) → add a sync_features name. Do not raise a minimum. Old clients keep working.
• Breaking change only → raise MIN_CLIENT_PROTOCOL_VERSION (or the client's MIN_SERVER_PROTOCOL_VERSION). This is the switch that hard-blocks the other side, so it is the one to be stingy with.
• Never gate behavior on the release version (version) — it's for display.

The three outcomes

The client evaluates the advertisement (compat::evaluate) and gets exactly one of:

• ok — full parity; sync everything.
• degraded — safe to sync, but named capabilities are unavailable here; the UI says which.
• incompatible — do not sync. Carries client_must_update so the message can point at the side that can actually fix it, rather than just saying "incompatible".

A server that predates this handshake sends no protocol fields at all. That is treated as incompatible (update the server) — deliberately not as a parse error, which would look to the user like they mistyped the URL.

Authentication — device bearer tokens

Native clients authenticate with a long-lived device token, not a session cookie. Only the token's SHA-256 hash is stored server-side; the plaintext is shown once at creation.

• First link (no session): POST /api/auth/device-login with {email, password, name} → {token, device, user}. Store token; send it as Authorization: Bearer <token> on every subsequent call.

• From the web app (already signed in): the user creates a token at /account ("Linked devices"); POST /api/auth/devices {name} → {token, device}. They paste it into the native app.

• Manage: GET /api/auth/devices (list), DELETE /api/auth/devices/<id> (revoke). A revoked token stops authenticating immediately.

• Self-revoke: DELETE /api/auth/devices/self retires the token presented in the Authorization header. This is what a native client calls when the user unlinks. It exists because a client can't use the id-keyed route: a token pasted from the web app arrives without a device id, and /api/auth/me describes the user, not the device row. A caller authenticated by session cookie gets 400 — it holds no device token, so there is nothing for it to mean.

  Unlinking is never blocked on this call. If the server is unreachable or too old to have the route, the client still unlinks locally and tells the user the token is still live and where to revoke it.

Every authenticated request (sync or otherwise) accepts the bearer token in place of the session cookie.

The revision cursor

Every syncable row carries a monotonic sync_revision (bigint), assigned by a database trigger from a single shared sequence (sync_revision_seq) on every insert/update. Because it's a shared sequence, one integer is a total-order watermark across all of a user's notes and labels. It is not a timestamp — it is immune to clock skew and is only ever compared with >.

The client persists the highest cursor it has fully consumed and passes it back as ?since=. since=0 (or absent) is a full initial sync.

Entities and what syncs

• Note — the primary sync unit. It travels with its checklist items, label memberships, and attachment metadata inline (same shape as the REST serialization). A change to any child bumps the parent note's sync_revision, so pulling the note re-syncs the whole thing.
• Label — the label catalog (name + color) syncs as its own entity so a rename/recolor/delete propagates independently of notes.
• Derived, NOT synced: #tags are parsed from the note body. Clients recompute them locally; the server recomputes them on push. They never travel over the wire. ([[wiki-links]] were derived the same way until they were removed — note 2897.)
• Attachment blobs sync by id over the existing upload/download routes (see Attachments below); only their metadata rides the delta feed.

Tombstones (deletes)

Two levels, both propagate:

• Trash — deleted_at is a normal field, and it's on the wire: a trashed note still syncs with its content, the client shows it in its Trash, and the timestamp is what the client counts the retention window against. Restoring clears it.
• Purge (permanent delete) — becomes a content-less tombstone: purged_at is set, title/body/items/labels/attachments/previews/revisions are cleared or removed, and the row is kept. A client seeing purged_at != null deletes the row from its local store. deleted_at deliberately SURVIVES a purge, so ordinary server-side queries (deleted_at IS NULL) never see a tombstone as a live note. Tombstones themselves are retained indefinitely (cheap for a personal store); revisit if they ever grow large.

Retention — trash expires

A trashed note is purged automatically once it is older than the server's trash_retention_days setting (default 30, 0 = keep forever), advertised on /api/config so a client can show the countdown. A background sweep on the server does the work; clients learn about it as ordinary tombstones and need no special handling.

A linked client must not run its own expiry. The server owns the policy — one clock, one window. A client that purged on its own schedule could destroy a note the server was deliberately keeping and then push that delete upstream. An unlinked client (offline-only, no server to defer to) expires its own trash on its own default, which is the only case where nothing else can.

Pull — GET /api/sync/changes

Query: ?since=<cursor>&limit=<n> (limit default 500, max 1000).

Response:

{
  "notes":  [ { "...full note...", "trashed": false, "deleted_at": null,
               "sync_revision": 42, "purged_at": null } ],
  "labels": [ { "id": "...", "name": "...", "color": "...",
               "sync_revision": 43, "purged_at": null, "created_at": "..." } ],
  "cursor": 43,
  "has_more": false
}

Returns all of the caller's notes + labels (any state — active, archived, trash, purged) whose sync_revision > since, ascending by revision. Notes and labels share the sequence, so paging merges the two streams: when either stream fills a page, cursor advances only to the smaller of the two page boundaries, so nothing between cursor and the next pull is skipped. Loop while has_more is true, advancing since = cursor each time.

Note attachment metadata carries {id, url, mime, size, sha256}.

Shared notes (shares, v6)

With ?shares=1 the feed carries every note the caller can see: their own, plus those shared with them directly or through a group. Each note then also says how it is held:

{ "permission": "owner" | "edit" | "view",
  "shared": true,
  "shared_by": { "id": "...", "display_name": "..." } }

shared_by is null on the caller's own notes, and shared says whether they have shared it with anyone. A note shared with the caller comes with labels: []: labels are personal, and a recipient files nothing under the owner's tags.

Granting or changing a share moves the note to a new revision (without touching updated_at, so last-write-wins is unaffected), which is how it passes a recipient's cursor. Ending a share — or the owner purging the note — adds the note to the recipient's revoked list:

{ "notes": [ ... ], "labels": [ ... ], "revoked": ["<note id>", ...],
  "cursor": 51, "has_more": false }

A client deletes its copy of each revoked note. Revocations draw from the same sequence and page on the same cursor as notes and labels (three streams now; the smallest full boundary wins). Sharing with the person again deletes their revocation, so a device that never saw it is never told to drop a note it has.

Pin, archive and position are personal (shared_state, v7). On a note shared with the caller, the feed's pinned, archived and position are the caller's own: unpinned and unarchived until they change them, and the owner's position until they move it. They live in a per-person row with a revision of its own, so the note's sync_revision in the feed is the later of the note's and that row's — a recipient pinning a note reaches their devices and no one else's.

Push — POST /api/sync/push

Body: { "changes": [ ... ] } (max 1000 per batch). Each change:

{ "entity": "note", "id": "<uuid>", "op": "upsert", "edited_at": "<iso8601>",
  "body": "...",
  "pinned": false, "archived": false, "trashed": false, "remind_at": null,
  "recurrence": null, "position": 0,
  "label_ids": ["<uuid>", ...], "created_at": "<iso8601, on create>" }

• Client-generated ids. Notes/labels are UUIDs; the client mints the id when it creates the row offline and sends it here. Create-if-absent, else update.

• Whole-note semantics. A note upsert carries the client's full current state (not a partial patch) — the server overwrites all scalar fields and sets manual label memberships from label_ids (tag-sourced labels are re-derived from the body). #tags are recomputed server-side. A checklist is - [ ] lines inside body (M304), so there is no separate items array.

• Fields a change may still carry, and the server reads past. title and kind (removed in v2), items (M304) and color (v4, M315). The server reads its payload key by key and never validates the shape, which is exactly what lets an older client keep pushing a field this one has stopped storing — see the version policy above for why none of those needed a floor raise on their own.

• op: "delete" purges (tombstones) the row. Trashing is just an upsert with trashed: true.

• Labels: {entity: "label", op: "upsert"|"delete", id, edited_at, name, color}. A per-owner name clash on a different id is rejected (fix locally and retry).

• Attachments and link previews (attachment_sync): {entity: "attachment"|"preview", op: "delete", id, edited_at}. Delete is the only op — attachments are created by upload (below) and previews by the server. A removal is not weighed under last-write-wins: there is no rival version of a removed file, so it always applies. A row the caller can't see answers noop, the same as one that never existed. Removals are explicit rather than "the note's current attachment set" on purpose: push runs before pull, so a set would delete an attachment another device added that this one has not seen.

• A note someone else owns (shares): an upsert may carry two things, each under its own last-write-wins.

  • Its body, from a recipient at edit, compared on edited_at against the note. From a view recipient a body is rejected ("only its owner can change that").
  • Their own pinned, archived and position (shared_state), from anyone it is shared with, compared on state_at against what they last set. A client stamps state_at apart from the text's edited_at so that pinning a copy whose text is out of date never makes that text look newer than the owner's edit. Without state_at the three are ignored.

  Trashing, reminders, labels and deleting stay the owner's: those fields are ignored and a delete is rejected. The answer is applied if either part applied, else kept, with the revision as that caller's feed would show it. A note the caller can't see at all answers the generic "cannot apply".

Conflict resolution — last-write-wins + history

On a clash the most-recently-edited version wins, by edited_at (client wall-clock) compared against the server row's last-edit time (updated_at). The client applies iff client.edited_at >= server.updated_at. When a winning upsert overwrites an existing title/body, the server first snapshots the overwritten version into the note's version history (note_revisions) — so a "lost" edit is never truly lost; it's one Restore away. A stale delete loses to a newer server edit.

Response — per item, so the client can mark its local rows synced:

{ "results": [
  { "id": "...", "entity": "note", "status": "created", "sync_revision": 44 },
  { "id": "...", "entity": "note", "status": "kept",    "sync_revision": 40 },
  { "id": "...", "entity": "label","status": "rejected","error": "name in use" }
] }

status ∈ created | applied | kept | noop | rejected. kept means the server had a newer edit and the client should adopt the server version on its next pull.

Attachments (blobs)

Metadata rides the delta feed (id, url, mime, size, sha256); the bytes move over the existing routes:

• Sync upload (attachment_sync): PUT /api/sync/attachments/<id>?note_id= &filename=&sha256=, body = the raw bytes, type in Content-Type. For a file attached offline, sent after the note itself has landed. 201 stores it under the client's id; 200 {"status": "exists"} if the note already holds that id (a retry); 400 if the bytes don't hash to sha256; 409 if the id belongs to another note; 413 over max_attachment_mb; 404 for a note the caller doesn't own. A 4xx other than 404 is permanent — the client records it on the attachment and stops retrying.
• Web upload: POST /api/notes/<note_id>/attachments (multipart, field file; optional field id, same idempotency).
• Download: GET /api/notes/<note_id>/attachments/<id> (owner/shared scoped).
• The client uses sha256 to skip blobs it already holds and to verify integrity after download. Any file type syncs.
• Every insert or delete of an attachment or a link preview bumps its note's sync_revision (triggers from 0015 and 0031), so a change to either reaches other devices as the note.

A sync cycle

1. Push local changes since the last sync (batched). Apply the per-item results (mark synced, adopt server version where kept). Then upload the bytes of attachments added offline whose notes have now landed.
2. Pull from the stored since cursor until has_more is false. Upsert notes/labels into the local store; delete rows whose purged_at is set; download any attachment blobs referenced by a new/changed sha256.
3. Persist the new cursor.

Initial sync is the same with since=0. Because everything is keyed by stable ids and a monotonic cursor, the cycle is idempotent and resumable — a client can crash mid-sync and simply resume from its last persisted cursor.