server: the body is the checklist here too, and note_items is dropped
CI & Build / Build now, or wait for Android? (push) Successful in 2s
CI & Build / Python lint (push) Successful in 2s
CI & Build / TypeScript typecheck (push) Successful in 6s
CI & Build / Python tests (push) Failing after 8s
CI & Build / integration (push) Successful in 20s
CI & Build / Build & push image (push) Skipped

M304 steps 3 and the server half of 4. The client half landed in 668f7fa; these
belong in one deploy, and the protocol floor below is what enforces that.

notes/checklist.py is the Python half of a grammar that now exists three times —
here, core/src/local/derive.rs, and (next) frontend/src/notes/markdown.ts. That
triplication is the deliberate cost: the alternative is a round trip to the server
before a phone can draw a checkbox. Each copy names the other two, and each is
tested against the same table of cases, including the near-misses that must stay
prose: `-[ ] x`, `- []`, `- [ ]x`, a `[ ]` mid-sentence.

Routes: add/update/delete items stop touching rows and rewrite note.body, all
through one _rewrite_body that runs the same sequence the PATCH route runs for a
body change — because it IS a body change. Revisions, #tag reconciliation, the
name, and link unfurls therefore happen in one place rather than three routes each
remembering to.

The reorder route is gone (rule 22). Reordering a checklist is moving a line, and
no client ever called it — the only reference in the tree was a test asserting the
route existed.

The API still returns `items`, DERIVED from the body on the way out. That is not a
second source of truth and it cannot disagree with the body it came from; it keeps
the web client working across the rest of this milestone and saves any consumer
that only wants to draw checkboxes from carrying a parser.

Export drops its separate items block, in both formats. The body already ends with
those exact lines, so writing them again would double every checklist in an export
and then double it again on re-import. Import still ACCEPTS items, because a Keep
takeout has a list and not a blob; it folds them in before the Note is built, so
display_title and _reconcile_tags both see the finished text.

Protocol 3 on both sides now. A v2 client is refused rather than half-served —
which matters more than I first said: _apply_note_items returned early on an absent
`items` key, so an un-bumped v3 client against a v2 server would not have LOST the
rows, it would have kept them and then had the migration fold them a second time.
Duplicated lists rather than missing ones. The floor prevents both.

Migration 0027 folds every existing row into its note's body and drops the table.
It inlines its own copy of the fold on purpose — a migration has to keep producing
what it produced the day it ran — and a test pins that copy against the app's until
they are allowed to diverge. updated_at is deliberately untouched: a client holding
an unpushed edit keeps the newer timestamp, so last-write-wins keeps its work
instead of the migration silently winning.

The downgrade is honest rather than faithful. It recreates an empty note_items and
leaves the bodies alone, because once items are lines nothing distinguishes one this
migration wrote from one somebody typed, and a downgrade that guessed would eat
hand-written lists. Recreating the table is still necessary: 0015's downgrade drops
a trigger ON note_items, and IF EXISTS covers the trigger, not the table.
This commit is contained in:
2026-08-24 08:03:37 -04:00
parent 32dafca148
commit 761c3b5e82
13 changed files with 566 additions and 254 deletions
@@ -0,0 +1,120 @@
"""fold note_items into the note body and drop the table
Revision ID: 0027
Revises: 0026
Create Date: 2026-08-24
M304. A checklist item becomes a `- [ ] milk` line of `notes.body`, and `note_items`
goes. The reason is positional, not cosmetic: a row had a position in a table and no
position in the text, so a separate list could only ever render AFTER the prose. With
the items in the body, a list can sit between two paragraphs — which is the thing that
could not be built before and no amount of restyling would have delivered.
## This migration rewrites real content
Every note that has items gets its body appended to. The Google Keep import is genuine
content on this instance, not fixtures, so the rules here are strict:
* Rows are read BEFORE the table is dropped, in this one transaction.
* The existing body is never rewritten, only appended to.
* The layout — a blank line between prose and the list, nothing between consecutive
items — is byte-for-byte what `_note_markdown` has always exported and what
`derive::append_item` produces on every client. All three landing on the same text
is what lets the clients migrate their own SQLite stores independently and still
agree with the server, with no sync required to reconcile them.
## The fold is inlined on purpose
`notes/checklist.py` has this same function and this migration deliberately does not
import it. A migration has to keep producing what it produced the day it ran; if the
app's spacing rule ever changes, this file must not change with it.
## `updated_at` is left alone, and that is load-bearing
Raw SQL, so SQLAlchemy's `onupdate` never fires. Two reasons, and the second matters
more than the first. Every client folds the same rows the same way, so the new body is
news to nobody. And a client holding an UNPUSHED body edit still has the newer
`updated_at`, so when it pulls the migrated note last-write-wins keeps its edit instead
of the migration silently winning.
The `notes` row's own `sync_revision` trigger (migration 0015) does fire, so every
migrated note becomes pullable once. That is wanted: it is what makes a client whose
local fold somehow differed converge on the server's text.
## The downgrade is not a true inverse, and says so
It recreates an empty `note_items` and leaves the bodies alone. Nothing is lost —
every item is still there as text, which is where this migration put it — but the old
code would show those notes as prose with no checklist. A faithful inverse is not
possible: once the items are lines, nothing distinguishes a line this migration wrote
from one somebody typed, and a downgrade that guessed would eat hand-written task
lists. The real rollback is a database restore.
Recreating the table is not decoration, though. Migration 0015's downgrade runs
`DROP TRIGGER IF EXISTS trg_note_items_bump_note ON note_items`, and `IF EXISTS`
covers the trigger, not the table — against a missing table that statement errors. So
this is what keeps the migration chain runnable all the way back down.
"""
import re
import sqlalchemy as sa
from alembic import op
from sqlalchemy.dialects.postgresql import UUID
revision = "0027"
down_revision = "0026"
branch_labels = None
depends_on = None
_TASK_RE = re.compile(r"^\s*[-*] +\[[ xX]\](?: +.*)?$")
def _append_item(body: str, text: str, checked: bool) -> str:
mark = "x" if checked else " "
text = (text or "").strip()
line = f"- [{mark}] {text}" if text else f"- [{mark}]"
trimmed = (body or "").rstrip("\n")
if not trimmed.strip():
return line
follows_a_list = bool(_TASK_RE.match(trimmed.split("\n")[-1]))
return f"{trimmed}\n{line}" if follows_a_list else f"{trimmed}\n\n{line}"
def upgrade():
bind = op.get_bind()
rows = bind.execute(
sa.text("SELECT note_id, text, checked FROM note_items ORDER BY note_id, position, created_at")
).fetchall()
grouped: dict = {}
for note_id, text, checked in rows:
grouped.setdefault(note_id, []).append((text, bool(checked)))
for note_id, items in grouped.items():
body = bind.execute(sa.text("SELECT body FROM notes WHERE id = :id"), {"id": note_id}).scalar()
# An item whose note is already gone has nothing to fold into. The foreign key
# should make this impossible; skipping costs nothing and failing here would
# leave the database half-migrated.
if body is None:
continue
for text, checked in items:
body = _append_item(body, text, checked)
bind.execute(sa.text("UPDATE notes SET body = :body WHERE id = :id"), {"body": body, "id": note_id})
op.drop_table("note_items")
def downgrade():
# Column-for-column as migration 0006 created it, index name included: 0015's
# downgrade names both the table and its trigger, so a near-enough copy is not
# good enough.
op.create_table(
"note_items",
sa.Column("id", UUID(as_uuid=True), primary_key=True),
sa.Column("note_id", UUID(as_uuid=True), sa.ForeignKey("notes.id", ondelete="CASCADE"), nullable=False),
sa.Column("text", sa.Text(), nullable=False),
sa.Column("checked", sa.Boolean(), nullable=False, server_default=sa.false()),
sa.Column("position", sa.Integer(), nullable=False, server_default="0"),
sa.Column("created_at", sa.DateTime(timezone=True), nullable=False, server_default=sa.func.now()),
)
op.create_index("ix_note_items_note", "note_items", ["note_id"])
-1
View File
@@ -9,7 +9,6 @@ from . import ( # noqa: F401
label,
note,
note_attachment,
note_item,
note_link_preview,
note_revision,
saved_filter,
-1
View File
@@ -46,7 +46,6 @@ class Note(Base):
display_title: Mapped[str] = mapped_column(Text(), nullable=False, server_default="")
body: Mapped[str] = mapped_column(Text(), nullable=False, server_default="")
color: Mapped[str] = mapped_column(Text(), nullable=False, server_default="default")
# 'text' (freeform body) or 'list' (a checklist of note_items).
# Manual drag order (higher = earlier); 0 until the user reorders.
position: Mapped[int] = mapped_column(Integer(), nullable=False, server_default="0")
pinned: Mapped[bool] = mapped_column(Boolean(), nullable=False, server_default=func.false())
-29
View File
@@ -1,29 +0,0 @@
from __future__ import annotations
import uuid
from datetime import datetime
from sqlalchemy import Boolean, DateTime, ForeignKey, Integer, Text, func
from sqlalchemy.dialects.postgresql import UUID
from sqlalchemy.orm import Mapped, mapped_column
from . import Base
class NoteItem(Base):
"""A single checklist item on a note.
Any note can have them. There is no note "kind" gating this — a checklist is
something a note HAS, not something a note IS (M13 step 2).
"""
__tablename__ = "note_items"
id: Mapped[uuid.UUID] = mapped_column(UUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
note_id: Mapped[uuid.UUID] = mapped_column(
UUID(as_uuid=True), ForeignKey("notes.id", ondelete="CASCADE"), nullable=False
)
text: Mapped[str] = mapped_column(Text(), nullable=False)
checked: Mapped[bool] = mapped_column(Boolean(), nullable=False, server_default=func.false())
position: Mapped[int] = mapped_column(Integer(), nullable=False, server_default="0")
created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False, server_default=func.now())
+64 -73
View File
@@ -31,10 +31,16 @@ from ..labeling import reconcile_manual_labels, resolve_owned_label_ids
from ..models.label import Label, NoteLabel
from ..models.note import Note
from ..models.note_attachment import NoteAttachment
from ..models.note_item import NoteItem
from ..models.note_link_preview import NoteLinkPreview
from ..models.note_revision import NoteRevision
from ..revisions import should_snapshot
from .checklist import (
append_item,
parse_items,
remove_item,
set_item_checked,
set_item_text,
)
from ..responses import json_error, not_found, parse_uuid
from ..retention import purge_note
from ..settings import get_setting
@@ -70,7 +76,7 @@ from .tags import (
parse_tags,
)
from .recurrence import REMINDER_RECURRENCES, next_occurrence, normalize_recurrence
from .serialize import _items_for_notes, _labels_for_notes, _serialize_note, _serialize_notes
from .serialize import _labels_for_notes, _serialize_note, _serialize_notes
__all__ = [
"bp",
@@ -225,7 +231,6 @@ async def export_notes():
).all()
ids = [n.id for n in notes_list]
labels_map = await _labels_for_notes(db, ids)
items_map = await _items_for_notes(db, ids)
att_rows = (
(await db.scalars(select(NoteAttachment).where(NoteAttachment.note_id.in_(ids)))).all() if ids else []
)
@@ -247,7 +252,6 @@ async def export_notes():
with zipfile.ZipFile(buf, "w", zipfile.ZIP_DEFLATED) as zf:
for n in notes_list:
labels = labels_map.get(n.id, [])
items = items_map.get(n.id, [])
atts = att_by_note.get(n.id, [])
short = str(n.id)[:8]
payload["notes"].append(
@@ -263,13 +267,12 @@ async def export_notes():
"created_at": n.created_at.isoformat() if n.created_at else None,
"updated_at": n.updated_at.isoformat() if n.updated_at else None,
"labels": [lb["name"] for lb in labels],
"items": [{"text": it["text"], "checked": it["checked"]} for it in items],
"attachments": [
{"file": f"attachments/{short}/{os.path.basename(a.path)}", "mime": a.mime} for a in atts
],
}
)
zf.writestr(f"notes/{_slugify(n.display_title)}-{short}.md", _note_markdown(n, labels, items))
zf.writestr(f"notes/{_slugify(n.display_title)}-{short}.md", _note_markdown(n, labels))
for a in atts:
src = Config.media_root() / a.path
if src.is_file():
@@ -385,24 +388,6 @@ async def reorder_notes():
return jsonify({"ok": True})
async def _name_for(db, note: Note, item_texts: list[str] | None = None) -> str:
"""The note's display name, consulting its checklist only when the body is silent.
`item_texts` short-circuits the query for callers that already hold the items
(create, import). Everyone else pays one narrow SELECT, and only when the body
produced nothing — which is the uncommon case.
"""
name = derive_display_title(note.body)
if name:
return name
if item_texts is not None:
return derive_display_title("", item_texts[0] if item_texts else None)
first = await db.scalar(
select(NoteItem.text).where(NoteItem.note_id == note.id).order_by(NoteItem.position).limit(1)
)
return derive_display_title("", first)
@bp.post("")
@login_required
async def create_note():
@@ -419,17 +404,20 @@ async def create_note():
Note.owner_id == g.user_id, Note.deleted_at.is_(None)
)
)
# Items still arrive separately — a client holds a list, not a blob — but they
# are folded into the body, which is where a checklist lives now (M304).
for text in item_texts:
body = append_item(body, text)
note = Note(
owner_id=g.user_id,
display_title=derive_display_title(body, item_texts[0] if item_texts else None),
display_title=derive_display_title(body),
body=body,
color=normalize_color(data.get("color")),
position=int(max_pos) + 1,
)
db.add(note)
await db.flush() # assign note.id before writing items/links
for pos, text in enumerate(item_texts):
db.add(NoteItem(note_id=note.id, text=text, position=pos))
await db.flush() # assign note.id before writing links
# The FOLDED body: an item can carry a #tag too.
await _reconcile_tags(db, note)
await db.commit()
await db.refresh(note)
@@ -484,7 +472,7 @@ async def update_note(note_id: str):
if "recurrence" in data:
note.recurrence = normalize_recurrence(data["recurrence"])
if "body" in data:
note.display_title = await _name_for(db, note)
note.display_title = derive_display_title(note.body)
await _reconcile_tags(db, note)
# Version history: snapshot the PRE-edit body, once per editing session
# rather than once per write — see revisions.should_snapshot. Writing often
@@ -543,7 +531,7 @@ async def restore_revision(note_id: str, rev_id: str):
# the revision — with the same body ripple as a normal edit.
db.add(NoteRevision(note_id=note.id, body=note.body))
note.body = rev.body
note.display_title = await _name_for(db, note)
note.display_title = derive_display_title(note.body)
await _reconcile_tags(db, note)
await db.commit()
await db.refresh(note)
@@ -575,11 +563,35 @@ async def set_note_labels(note_id: str):
return jsonify(await _serialize_note(db, note))
async def _get_item(db, note: Note, item_id: str) -> NoteItem | None:
iid = parse_uuid(item_id)
if iid is None:
def _item_index(item_id: str) -> int | None:
"""An item's id is its ordinal (see serialize.items_of). Anything else is a stale
id from a client that has not reloaded, and the answer to those is 404."""
try:
index = int(item_id)
except (TypeError, ValueError):
return None
return await db.scalar(select(NoteItem).where(NoteItem.id == iid, NoteItem.note_id == note.id))
return index if index >= 0 else None
async def _rewrite_body(db, note: Note, body: str):
"""Every item mutation is a body edit, so all of them land here.
One place means one place that snapshots a revision, re-derives `#tags`, recomputes
the name and queues link unfurls — rather than three routes each remembering to.
Deliberately the same sequence the PATCH route runs for a body change, because it
IS a body change.
"""
old_body = note.body
if await should_snapshot(db, note.id, old_body, body):
db.add(NoteRevision(note_id=note.id, body=old_body))
note.body = body
note.display_title = derive_display_title(body)
await _reconcile_tags(db, note)
await db.commit()
await db.refresh(note)
if note.body != old_body:
schedule_unfurls(note.id, note.body)
return jsonify(await _serialize_note(db, note))
@bp.post("/<note_id>/items")
@@ -591,70 +603,49 @@ async def add_item(note_id: str):
note = await _get_owned(db, note_id)
if note is None:
return not_found()
max_pos = await db.scalar(
select(func.coalesce(func.max(NoteItem.position), -1)).where(NoteItem.note_id == note.id)
)
db.add(NoteItem(note_id=note.id, text=text, position=int(max_pos) + 1))
await db.commit()
return jsonify(await _serialize_note(db, note)), 201
response = await _rewrite_body(db, note, append_item(note.body, text))
return response, 201
@bp.patch("/<note_id>/items/<item_id>")
@login_required
async def update_item(note_id: str, item_id: str):
data = await request.get_json(silent=True) or {}
index = _item_index(item_id)
if index is None:
return not_found()
async with session_scope() as db:
note = await _get_owned(db, note_id)
if note is None:
return not_found()
item = await _get_item(db, note, item_id)
if item is None:
if index >= len(parse_items(note.body)):
return not_found()
body = note.body
if "text" in data and isinstance(data["text"], str):
item.text = data["text"]
body = set_item_text(body, index, data["text"])
if "checked" in data:
item.checked = bool(data["checked"])
await db.commit()
return jsonify(await _serialize_note(db, note))
body = set_item_checked(body, index, bool(data["checked"]))
return await _rewrite_body(db, note, body)
@bp.delete("/<note_id>/items/<item_id>")
@login_required
async def delete_item(note_id: str, item_id: str):
index = _item_index(item_id)
if index is None:
return not_found()
async with session_scope() as db:
note = await _get_owned(db, note_id)
if note is None:
return not_found()
item = await _get_item(db, note, item_id)
if item is None:
if index >= len(parse_items(note.body)):
return not_found()
await db.delete(item)
await db.commit()
return jsonify(await _serialize_note(db, note))
return await _rewrite_body(db, note, remove_item(note.body, index))
@bp.post("/<note_id>/items/reorder")
@login_required
async def reorder_items(note_id: str):
data = await request.get_json(silent=True) or {}
order = data.get("item_ids")
if not isinstance(order, list):
return json_error("item_ids must be a list", 400)
async with session_scope() as db:
note = await _get_owned(db, note_id)
if note is None:
return not_found()
existing = {
str(i.id): i for i in (await db.scalars(select(NoteItem).where(NoteItem.note_id == note.id))).all()
}
pos = 0
for iid in order:
item = existing.get(str(iid))
if item is not None:
item.position = pos
pos += 1
await db.commit()
return jsonify(await _serialize_note(db, note))
# The reorder route is gone with M304. Reordering a checklist is moving a line, which
# is something a text editor already does and no client ever called this for — the
# only reference to it in the tree was a test asserting the route existed.
@bp.post("/<note_id>/attachments")
+150
View File
@@ -0,0 +1,150 @@
"""Checklist items — a note's body IS its checklist (M304).
A `- [ ] milk` line is the item. There is no `note_items` table beside the body any
more, which is what lets a list sit BETWEEN two paragraphs: rows had a position in a
table and no position in the text, so a separate list could only ever render after
the prose no matter how it was styled.
The same shape as `tags.py`, one strength further along. Tags are derived from the
body too, but they MATERIALISE into `note_labels` rows because the board queries by
label. Items materialise into nothing, because nothing queries them — their only
readers are the card, the editor and `display_title`. So `parse_items` is the whole
storage layer for a checklist, and the rewriters below are how one is edited.
THE GRAMMAR IS SHARED. Three implementations exist and they have to agree, because a
difference between any two of them is a checklist that changes shape when it syncs:
core/src/local/derive.rs the native clients (desktop + Android)
src/thoughtsync/notes/checklist.py this file, the server
frontend/src/notes/markdown.ts the browser
optional indent, `-` or `*`, one-or-more spaces, `[ ]`/`[x]`/`[X]`,
then either end-of-line or one-or-more spaces and the text.
`*` is accepted because markdown.ts already takes it for a plain bullet, and a rule
that allowed `* item` but not `* [ ] item` would be one nobody could guess. `- [ ]`
with nothing after it IS an item with empty text — that is what pressing Enter on a
list leaves behind, and refusing to parse it would make a half-typed list stop being
a list. `- [X]` parses as checked and renders back lowercase, so one canonical form
survives a round trip.
"""
from __future__ import annotations
import re
from dataclasses import dataclass
# Anchored at both ends: a `[ ]` mid-sentence is prose, and `- [ ]x` (no space after
# the brackets) is a sentence that happens to start with brackets, not a marker.
_TASK_RE = re.compile(r"^(?P<indent>\s*)(?P<bullet>[-*]) +\[(?P<mark>[ xX])\](?: +(?P<text>.*))?$")
@dataclass(frozen=True)
class Item:
"""One checklist item. Its position in the parsed list is its identity — the same
thing `position` meant when these were rows, and all the wire ever carried."""
text: str
checked: bool
def parse_items(body: str | None) -> list[Item]:
"""Every checklist item in `body`, in the order they appear."""
out: list[Item] = []
for line in (body or "").split("\n"):
match = _TASK_RE.match(line)
if match:
out.append(Item(text=match.group("text") or "", checked=match.group("mark") in "xX"))
return out
def render_item(text: str, checked: bool, indent: str = "", bullet: str = "-") -> str:
"""One item as the line that stores it.
Always lowercase `x`, whatever was parsed: one canonical output is what makes a
round trip stable, so `- [X]` normalises the first time it is touched and never
again.
"""
mark = "x" if checked else " "
if not text:
return f"{indent}{bullet} [{mark}]"
return f"{indent}{bullet} [{mark}] {text}"
def strip_marker(line: str) -> str:
"""The text of a line with its task marker removed, or the line as it was.
For naming a note: a list-only note is named by its first item, and calling one
"- [ ] milk" would be showing someone the storage instead of the note.
"""
match = _TASK_RE.match(line)
return (match.group("text") or "") if match else line
def _rewrite(body: str, index: int, replace) -> str:
"""Rewrite the `index`-th task line with `replace`, or drop it when `replace`
returns None.
A body with fewer task lines than that is returned UNCHANGED rather than raising:
the index comes from a client that may be a moment behind the server, and a stale
request should do nothing rather than 500.
"""
lines = body.split("\n")
target = None
seen = 0
for n, line in enumerate(lines):
if _TASK_RE.match(line):
if seen == index:
target = n
break
seen += 1
if target is None:
return body
match = _TASK_RE.match(lines[target])
replacement = replace(match)
if replacement is None:
del lines[target]
else:
lines[target] = replacement
return "\n".join(lines)
def set_item_checked(body: str, index: int, checked: bool) -> str:
"""Tick or untick the `index`-th item, keeping its text, indent and bullet."""
return _rewrite(
body,
index,
lambda m: render_item(m.group("text") or "", checked, m.group("indent"), m.group("bullet")),
)
def set_item_text(body: str, index: int, text: str) -> str:
"""Replace the text of the `index`-th item, keeping its state and its bullet."""
return _rewrite(
body,
index,
lambda m: render_item(text.strip(), m.group("mark") in "xX", m.group("indent"), m.group("bullet")),
)
def remove_item(body: str, index: int) -> str:
"""Delete the `index`-th item, line and all."""
return _rewrite(body, index, lambda _m: None)
def append_item(body: str, text: str, checked: bool = False) -> str:
"""Add an item at the end of the body.
A blank line between prose and the list, nothing between consecutive items —
the layout `import_export._note_markdown` has always used when writing a checklist
out. That is not cosmetic: it is what the Alembic migration folds existing
`note_items` rows into AND what `derive::append_item` produces on every client, so
all three land on identical bodies. An export taken before the migration and one
taken after therefore differ in nothing.
"""
line = render_item(text.strip(), checked)
trimmed = body.rstrip("\n")
if not trimmed.strip():
return line
follows_a_list = bool(_TASK_RE.match(trimmed.split("\n")[-1]))
return f"{trimmed}\n{line}" if follows_a_list else f"{trimmed}\n\n{line}"
+22 -13
View File
@@ -3,6 +3,8 @@ board-filter narrowing, the owner-scoped fetch, and filename/slug sanitizers use
both the attachment routes and the importer."""
from __future__ import annotations
from .checklist import strip_marker
import os
import re
@@ -19,29 +21,36 @@ VALID_FILTERS = {"active", "archived", "trash"}
DISPLAY_TITLE_CAP = 200
def derive_display_title(body: str | None, first_item: str | None = None) -> str:
"""The note's display NAME: the first non-empty line of the body, else the first
checklist item's text (both trimmed and length-capped).
def derive_display_title(body: str | None) -> str:
"""The note's display NAME: the first line of the body that says anything.
There is no explicit title to prefer any more (M13 step 3) — a note is a body plus
optional items, and its name is simply the first thing written in it. Persisted as
notes.display_title so search results and export filenames have something to say.
There is no explicit title to prefer any more (M13 step 3) — a note is a body, and
its name is simply the first thing written in it. Persisted as notes.display_title
so search results and export filenames have something to say.
The item fallback is what step 2 bought: a note that is only a checklist would
otherwise have no name at all, which is exactly the hole that made removing the
title unsafe before checklists stopped being their own kind of thing.
The old `first_item` fallback is gone with M304: items ARE body lines now, so a
list-only note is named by its first item without anyone having to arrange it. What
replaced the fallback is stripping the task marker — calling that note "- [ ] milk"
would show someone the storage instead of the note — and skipping an EMPTY item, so
a half-typed list does not leave a note with no name.
Deterministic — a literal first line, never generated.
Mirrors `display_title` in core/src/local/store.rs. Deterministic — a literal first
line, never generated.
"""
for line in (body or "").splitlines():
stripped = line.strip()
stripped = strip_marker(line.strip()).strip()
if stripped:
return stripped[:DISPLAY_TITLE_CAP]
return (first_item or "").strip()[:DISPLAY_TITLE_CAP]
return ""
def is_empty_note(body: str | None, items: list | None = None) -> bool:
"""Nothing worth keeping: no body text and no checklist items."""
"""Nothing worth keeping: no body text and no checklist items.
`items` is still a separate argument because create still accepts them separately —
the importer holds a list, not a blob — and they are folded into the body only
after this check has decided the note is worth making at all.
"""
return not (body or "").strip() and not items
+15 -14
View File
@@ -20,7 +20,6 @@ from ..config import Config
from ..models.label import NoteLabel
from ..models.note import Note
from ..models.note_attachment import NoteAttachment
from ..models.note_item import NoteItem
from .helpers import (
ALLOWED_IMAGE_MIMES,
_attachment_ext,
@@ -28,11 +27,12 @@ from .helpers import (
derive_display_title,
is_empty_note,
)
from .checklist import append_item
from .tags import _find_or_create_label, _reconcile_tags
from .recurrence import normalize_recurrence
def _note_markdown(note: Note, labels: list, items: list) -> str:
def _note_markdown(note: Note, labels: list) -> str:
"""One note as a human-readable Markdown file with a small frontmatter block.
The authoritative machine format is notes.json; this is for reading/portability."""
fm = ["---"]
@@ -54,11 +54,9 @@ def _note_markdown(note: Note, labels: list, items: list) -> str:
# are written, body first, with a blank line between them when there is.
if note.body:
fm.append(note.body)
if items:
if note.body:
fm.append("")
for it in items:
fm.append(f"- [{'x' if it['checked'] else ' '}] {it['text']}")
# No separate items block any more. The body already ends with those exact lines
# (M304) — this function is where their layout was decided, and appending them a
# second time would double every checklist in an export.
return "\n".join(fm) + "\n"
@@ -294,9 +292,17 @@ async def _create_imported_note(
if is_empty_note(body, item_texts):
return False
# Items fold into the body, which is where a checklist lives now (M304). Done
# before the Note is built so display_title and _reconcile_tags both see the
# finished text — an imported item can carry a #tag like any other line.
for it in items:
text = (it.get("text") or "").strip()
if text:
body = append_item(body, text, bool(it.get("checked")))
note = Note(
owner_id=owner_id,
display_title=derive_display_title(body, item_texts[0] if item_texts else None),
display_title=derive_display_title(body),
body=body,
color=normalize_color(spec.get("color")),
pinned=bool(spec.get("pinned")),
@@ -316,12 +322,7 @@ async def _create_imported_note(
if spec.get("updated_at"):
note.updated_at = spec["updated_at"]
db.add(note)
await db.flush() # assign note.id before items/labels/attachments/links
for pos, it in enumerate(items):
text = (it.get("text") or "").strip()
if text:
db.add(NoteItem(note_id=note.id, text=text, checked=bool(it.get("checked")), position=pos))
await db.flush() # assign note.id before labels/attachments/links
# Explicit (picker-style) labels are manual — via_tag=False. Inline #tags in the
# body are handled by _reconcile_tags below, same as a normal create.
+19 -20
View File
@@ -8,8 +8,8 @@ from sqlalchemy import select
from ..models.label import Label, NoteLabel
from ..models.note import Note
from ..models.note_attachment import NoteAttachment
from ..models.note_item import NoteItem
from ..models.note_link_preview import NoteLinkPreview
from .checklist import parse_items
async def _labels_for_notes(db, note_ids: list) -> dict:
@@ -30,23 +30,23 @@ async def _labels_for_notes(db, note_ids: list) -> dict:
return result
def _serialize_item(item: NoteItem) -> dict:
return {"id": str(item.id), "text": item.text, "checked": item.checked, "position": item.position}
def items_of(body: str | None) -> list[dict]:
"""The note's checklist, read out of its body. No query, because there is no table.
Still emitted in the payload after M304, and that is not a second source of truth:
it is DERIVED on the way out, so it cannot disagree with the body it came from. It
saves every consumer that only wants to draw checkboxes from carrying a parser, and
the ones that do carry one (the native clients, the browser) are free to ignore it
and read the body.
async def _items_for_notes(db, note_ids: list) -> dict:
"""Map note_id -> [checklist items] in one query, ordered by position."""
result: dict = {}
if not note_ids:
return result
items = (
await db.scalars(
select(NoteItem).where(NoteItem.note_id.in_(note_ids)).order_by(NoteItem.position, NoteItem.created_at)
)
).all()
for item in items:
result.setdefault(item.note_id, []).append(_serialize_item(item))
return result
The id is the item's ORDINAL, which is what the rewriters in `checklist.py` take,
so a client holding one can act on it directly. It also shifts when an item is
removed — every mutation returns the reloaded note for exactly that reason.
"""
return [
{"id": str(i), "text": item.text, "checked": item.checked, "position": i}
for i, item in enumerate(parse_items(body))
]
def _attachment_url(note_id, att_id) -> str:
@@ -108,8 +108,7 @@ async def _serialize_note(db, note: Note) -> dict:
data = note.serialize()
labels = await _labels_for_notes(db, [note.id])
data["labels"] = labels.get(note.id, [])
items = await _items_for_notes(db, [note.id])
data["items"] = items.get(note.id, [])
data["items"] = items_of(note.body)
attachments = await _attachments_for_notes(db, [note.id])
data["attachments"] = attachments.get(note.id, [])
previews = await _previews_for_notes(db, [note.id])
@@ -120,14 +119,14 @@ async def _serialize_note(db, note: Note) -> dict:
async def _serialize_notes(db, notes: list) -> list:
ids = [n.id for n in notes]
labels_map = await _labels_for_notes(db, ids)
items_map = await _items_for_notes(db, ids)
attach_map = await _attachments_for_notes(db, ids)
preview_map = await _previews_for_notes(db, ids)
out = []
for n in notes:
data = n.serialize()
data["labels"] = labels_map.get(n.id, [])
data["items"] = items_map.get(n.id, [])
data["items"] = items_of(n.body)
data["attachments"] = attach_map.get(n.id, [])
data["previews"] = preview_map.get(n.id, [])
out.append(data)
-2
View File
@@ -32,7 +32,6 @@ from .db import session_scope
from .models.label import NoteLabel
from .models.note import Note
from .models.note_attachment import NoteAttachment
from .models.note_item import NoteItem
from .models.note_link_preview import NoteLinkPreview
from .models.note_revision import NoteRevision
from .settings import get_setting
@@ -85,7 +84,6 @@ async def purge_note(db, note: Note, edited_at: datetime | None = None) -> None:
# place would make the note reappear whole on the next sweep.
logger.warning("couldn't remove attachment file %s during purge", a.path, exc_info=True)
await db.execute(sa_delete(NoteAttachment).where(NoteAttachment.note_id == note.id))
await db.execute(sa_delete(NoteItem).where(NoteItem.note_id == note.id))
await db.execute(sa_delete(NoteLabel).where(NoteLabel.note_id == note.id))
await db.execute(sa_delete(NoteLinkPreview).where(NoteLinkPreview.note_id == note.id))
await db.execute(sa_delete(NoteRevision).where(NoteRevision.note_id == note.id))
+9 -48
View File
@@ -24,7 +24,6 @@ from .db import session_scope
from .labeling import reconcile_manual_labels, resolve_owned_label_ids
from .models.label import Label, NoteLabel
from .models.note import Note
from .models.note_item import NoteItem
from .models.note_revision import NoteRevision
from .revisions import should_snapshot
from .notes import (
@@ -63,8 +62,8 @@ MAX_PUSH = 1000 # per-batch change cap
#
# One bump for the pair: they landed in the same protocol generation, and nothing ever
# ran against a half-applied v2.
SYNC_PROTOCOL_VERSION = 2
MIN_CLIENT_PROTOCOL_VERSION = 2
SYNC_PROTOCOL_VERSION = 3
MIN_CLIENT_PROTOCOL_VERSION = 3
# Named capabilities beyond the base protocol. An ADDITIVE change earns a name
# here rather than a min-version bump, so a newer client meeting an older server
@@ -213,49 +212,12 @@ def _assign_note_fields(note: Note, ch: dict) -> None:
note.position = ch["position"]
def _first_item_text(ch: dict) -> str:
"""The first non-blank checklist item in a pushed change, or "".
Read straight from the payload rather than the database because the note's name is
computed BEFORE `_apply_note_items` has written anything — and a note whose body is
empty is named by its first item (M13 step 3).
"""
items = ch.get("items")
if not isinstance(items, list):
return ""
for it in items:
if isinstance(it, dict):
text = (it.get("text") or "").strip()
if text:
return text
return ""
async def _apply_note_items(db, note: Note, ch: dict) -> None:
"""Replace the note's checklist items with the client's (items sync inline).
Applies to ANY note. This used to delete every item when the note wasn't
`kind == "list"`, which was survivable only because nothing could produce a note
holding both a body and items. M13 makes that the normal shape — a checklist is
something a note HAS, not something a note IS — and against that shape the old
guard was a data-loss path: the first sync after adding a checklist to a note
would have wiped it.
Removed ahead of the UI that can create the state, deliberately, so there is no
window in which the two disagree.
"""
items = ch.get("items")
if not isinstance(items, list):
# Absent means "not telling us", not "empty". A client that omits the key
# leaves what the server has; only an explicit [] clears it.
return
await db.execute(sa_delete(NoteItem).where(NoteItem.note_id == note.id))
for pos, it in enumerate(items):
if not isinstance(it, dict):
continue
text = (it.get("text") or "").strip()
if text:
db.add(NoteItem(note_id=note.id, text=text, checked=bool(it.get("checked")), position=pos))
# `_first_item_text` and `_apply_note_items` lived here until M304. Both existed for
# one reason — a checklist was a table beside the body — and both are gone with it. A
# pushed change carries its items as `- [ ] ` lines inside `body`, so applying them is
# applying the body, and naming the note is reading its first line. A client that still
# sends an `items` array is a v2 client, and the version floor below turns it away
# before any of this runs.
async def _apply_note_manual_labels(db, note: Note, ch: dict) -> None:
@@ -315,7 +277,7 @@ async def _apply_note(db, ch: dict) -> dict:
old_body = note.body
_assign_note_fields(note, ch)
note.display_title = derive_display_title(note.body, _first_item_text(ch))
note.display_title = derive_display_title(note.body)
if edited_at is not None:
note.updated_at = edited_at
# Non-destructive LWW: snapshot the overwritten server body into history —
@@ -327,7 +289,6 @@ async def _apply_note(db, ch: dict) -> dict:
if not creating and await should_snapshot(db, note.id, old_body, note.body):
db.add(NoteRevision(note_id=note.id, body=old_body))
await db.flush() # assign note.id before items/labels/links
await _apply_note_items(db, note, ch)
await _reconcile_tags(db, note)
await _apply_note_manual_labels(db, note, ch)
await db.flush()
+41 -52
View File
@@ -26,21 +26,20 @@ from thoughtsync import ratelimit
from thoughtsync.app import create_app
from thoughtsync.db import dispose_engine, session_scope
from thoughtsync.models.note import Note
from thoughtsync.models.note_item import NoteItem
from thoughtsync.models.user import User
from thoughtsync.settings import get_setting, live, refresh_live, reset_live, set_settings
from thoughtsync.notes.checklist import parse_items, set_item_checked
from thoughtsync.notes.helpers import derive_display_title
from thoughtsync.models.note_link_preview import NoteLinkPreview
from thoughtsync.models.note_revision import NoteRevision
from thoughtsync.revisions import REVISION_WINDOW_MINUTES, should_snapshot
from thoughtsync.sync import _apply_note_items
from thoughtsync.unfurl_queue import _unfurl_new_urls, detect_urls
pytestmark = pytest.mark.integration
# Every table the tests touch, child-first so FKs never block the truncate.
# RESTART IDENTITY + CASCADE keeps this honest if a table gains children later.
_TABLES = "notes, note_items, note_revisions, note_labels, note_link_previews, labels, users"
_TABLES = "notes, note_revisions, note_labels, note_link_previews, labels, users"
@pytest_asyncio.fixture
@@ -160,65 +159,55 @@ async def test_the_search_vector_was_rebuilt_over_the_name(db, owner):
assert body_only == 1
async def test_a_note_keeps_both_its_body_and_its_items(db, owner):
"""The shape M13 step 2 made normal: a note HAS a checklist, it isn't one."""
note = Note(owner_id=owner.id, body="weekend shop", display_title="weekend shop")
db.add(note)
await db.flush()
db.add_all(
[
NoteItem(note_id=note.id, text="milk", position=0),
NoteItem(note_id=note.id, text="eggs", position=1),
]
)
await db.commit()
async def test_a_note_keeps_its_prose_on_both_sides_of_its_list(db, owner):
"""The shape M304 made expressible at all.
items = (
await db.scalars(select(NoteItem).where(NoteItem.note_id == note.id).order_by(NoteItem.position))
).all()
assert [i.text for i in items] == ["milk", "eggs"]
assert (await db.scalar(select(Note.body).where(Note.id == note.id))) == "weekend shop"
async def test_sync_no_longer_deletes_items_from_a_note_with_a_body(db, owner):
"""The data-loss path step 2 removed, pinned against a real database.
`_apply_note_items` used to delete every item when the note wasn't `kind = "list"`.
Nothing can produce that state any more, but this is the regression that would
have silently eaten a checklist, and it deserves a test that would catch its
return.
The old model could not hold this: a row had a position in a table and none in the
text, so a checklist could only ever render AFTER the body. Prose, list, prose is
the case that proves the storage changed, not just the styling.
"""
note = Note(owner_id=owner.id, body="packing", display_title="packing")
body = "weekend shop\n\n- [ ] milk\n- [x] eggs\n\nback before six"
note = Note(owner_id=owner.id, body=body, display_title=derive_display_title(body))
db.add(note)
await db.flush()
db.add(NoteItem(note_id=note.id, text="socks", position=0))
await db.commit()
# A change that says nothing about items must LEAVE them alone — absent means
# "not telling us", not "empty".
await _apply_note_items(db, note, {"body": "packing"})
await db.commit()
assert (await db.scalar(select(NoteItem.text).where(NoteItem.note_id == note.id))) == "socks"
# An explicit list replaces them.
await _apply_note_items(db, note, {"items": [{"text": "charger", "checked": True}]})
await db.commit()
rows = (await db.scalars(select(NoteItem).where(NoteItem.note_id == note.id))).all()
assert [(r.text, r.checked) for r in rows] == [("charger", True)]
stored = await db.scalar(select(Note.body).where(Note.id == note.id))
assert [(i.text, i.checked) for i in parse_items(stored)] == [("milk", False), ("eggs", True)]
assert stored.splitlines()[0] == "weekend shop"
assert stored.splitlines()[-1] == "back before six"
async def test_a_note_with_only_items_still_has_a_name(db, owner):
"""The hole that made removing the title unsafe until step 2 closed it."""
note = Note(owner_id=owner.id, body="", display_title="")
async def test_ticking_an_item_is_a_body_edit(db, owner):
"""What replaced `_apply_note_items`: there is no separate thing left to apply.
The regression that function guarded against — a sync silently eating a checklist
off a note that also had a body — cannot recur, because there is nothing to delete.
A pushed body either has the lines or it does not.
"""
body = "packing\n\n- [ ] socks"
note = Note(owner_id=owner.id, body=body, display_title="packing")
db.add(note)
await db.flush()
db.add(NoteItem(note_id=note.id, text="milk", position=0))
await db.commit()
first = await db.scalar(
select(NoteItem.text).where(NoteItem.note_id == note.id).order_by(NoteItem.position).limit(1)
)
note.display_title = derive_display_title(note.body, first)
note.body = set_item_checked(note.body, 0, True)
await db.commit()
stored = await db.scalar(select(Note.body).where(Note.id == note.id))
assert stored == "packing\n\n- [x] socks"
assert parse_items(stored)[0].checked
# The prose is untouched — a tick rewrites one line, not the note.
assert stored.splitlines()[0] == "packing"
async def test_a_note_with_only_a_list_still_has_a_name(db, owner):
"""The hole that made removing the title unsafe, still closed — by a different
mechanism. There is no item table to fall back to any more; the name comes from
the first line with its marker stripped, because calling the note "- [ ] milk"
would show someone the storage instead of the note.
"""
body = "- [ ] milk\n- [ ] eggs"
note = Note(owner_id=owner.id, body=body, display_title=derive_display_title(body))
db.add(note)
await db.commit()
assert (await db.scalar(select(Note.display_title).where(Note.id == note.id))) == "milk"
+126 -1
View File
@@ -5,6 +5,14 @@ import pytest
from thoughtsync.app import create_app
from thoughtsync.common import coerce_bool, parse_dt
from thoughtsync.models.note import NOTE_COLORS, Note
from thoughtsync.notes.checklist import (
append_item,
parse_items,
remove_item,
set_item_checked,
set_item_text,
strip_marker,
)
from thoughtsync.unfurl_queue import detect_urls
from thoughtsync.notes import (
_attachment_ext,
@@ -42,7 +50,7 @@ def test_all_note_routes_registered(app):
"reorder_notes", "create_note",
"get_note", "update_note", "list_revisions", "restore_revision",
"set_note_labels", "add_item", "update_item", "delete_item",
"reorder_items", "upload_attachment", "get_attachment",
"upload_attachment", "get_attachment",
"delete_attachment", "unfurl_link", "delete_preview", "trash_note",
"restore_note", "delete_note",
)
@@ -406,3 +414,120 @@ def test_detect_urls_ignores_non_http():
assert detect_urls("ftp://example.com and mailto:a@b.c and bare example.com") == []
assert detect_urls(None) == []
assert detect_urls("") == []
# --- checklist items: the body IS the checklist (M304) -----------------------
#
# The same table of cases as core/src/local/derive.rs. Deliberately duplicated
# rather than shared: the point of three implementations is that each is checked
# against the same grammar, and a test that only ran once would not catch the two
# drifting apart.
def test_parse_items_reads_a_list_out_of_prose():
body = "shopping\n\n- [ ] milk\n- [x] eggs"
assert [(i.text, i.checked) for i in parse_items(body)] == [("milk", False), ("eggs", True)]
def test_parse_items_between_paragraphs():
# The case a side table could not express, which is the whole reason for M304.
assert [i.text for i in parse_items("before\n- [ ] middle\nafter")] == ["middle"]
@pytest.mark.parametrize(
"body",
[
"-[ ] no space after the dash",
"- [] empty brackets",
"- [ ]no space after the brackets",
"- [y] not a mark",
"a [ ] mid sentence",
"[ ] no bullet at all",
],
)
def test_parse_items_rejects_near_misses(body):
assert parse_items(body) == []
def test_parse_items_accepts_star_bullets_and_indentation():
# `*` because markdown.ts already takes it for a plain bullet.
body = "* [ ] star\n - [x] indented"
assert [(i.text, i.checked) for i in parse_items(body)] == [("star", False), ("indented", True)]
def test_an_empty_item_is_still_an_item():
# What pressing Enter on a list leaves behind.
assert [i.text for i in parse_items("- [ ]")] == [""]
assert [i.text for i in parse_items("- [ ] ")] == [""]
def test_uppercase_x_parses_and_normalises_on_rewrite():
assert parse_items("- [X] done")[0].checked
assert set_item_checked("- [X] done", 0, True) == "- [x] done"
def test_rewriters_preserve_indent_bullet_and_neighbours():
assert set_item_checked(" * [ ] milk", 0, True) == " * [x] milk"
assert set_item_text("- [x] old", 0, "new") == "- [x] new"
assert remove_item("keep\n- [ ] drop\n- [ ] stay", 0) == "keep\n- [ ] stay"
# Addressed by ITEM, not by line.
assert set_item_checked("note\n- [ ] a\nprose\n- [ ] b", 1, True) == "note\n- [ ] a\nprose\n- [x] b"
def test_a_stale_index_does_nothing():
# The index comes from a client that may be a moment behind. A late request
# should be inert, not a 500.
body = "- [ ] only"
assert set_item_checked(body, 7, True) == body
assert remove_item(body, 7) == body
assert set_item_text(body, 7, "x") == body
def test_a_plain_body_is_returned_unchanged():
body = "just prose\nwith two lines"
assert set_item_checked(body, 0, True) == body
assert remove_item(body, 0) == body
def test_append_item_spacing():
# Prose, blank line, list — the layout _note_markdown has always exported, and
# what the migration folds existing rows into.
assert append_item("a note", "milk") == "a note\n\n- [ ] milk"
# Nothing between consecutive items.
assert append_item("a note\n\n- [ ] milk", "eggs") == "a note\n\n- [ ] milk\n- [ ] eggs"
# A list-only note starts at the first line.
assert append_item("", "milk") == "- [ ] milk"
assert append_item("\n\n", "milk") == "- [ ] milk"
# Carries state, which is what the migrations need of it.
assert append_item("", "done", True) == "- [x] done"
def test_strip_marker_and_display_title():
assert strip_marker("- [x] milk") == "milk"
assert strip_marker("just prose") == "just prose"
# A list-only note is named by its first item, without the marker.
assert derive_display_title("- [ ] milk\n- [ ] eggs") == "milk"
# An EMPTY item does not name the note "" — a half-typed list still has a name.
assert derive_display_title("- [ ]\n- [ ] eggs") == "eggs"
def test_the_migration_folds_exactly_like_the_app():
"""0027 inlines its own copy of append_item, deliberately — a migration has to keep
producing what it produced the day it ran, so it must not follow the app if the
app's spacing ever changes. This is what keeps the copy honest until then."""
import importlib.util
from pathlib import Path
path = Path(__file__).resolve().parents[1] / "alembic" / "versions" / "0027_checklist_items_into_body.py"
spec = importlib.util.spec_from_file_location("_m0027", path)
module = importlib.util.module_from_spec(spec)
spec.loader.exec_module(module)
for body, text, checked in [
("a note", "milk", False),
("a note\n\n- [ ] milk", "eggs", True),
("", "milk", False),
("\n\n", "milk", False),
("prose\n", " padded ", True),
]:
assert module._append_item(body, text, checked) == append_item(body, text, checked)