Server comments describe the code to a stranger (#6221)
CI & Build / Python lint (push) Successful in 2s
CI & Build / Build now, or wait for Android? (push) Successful in 2s
Android / Build, or is the channel already serving this? (push) Successful in 3s
Android / Core and FFI clippy and tests (push) Skipped
Android / Kotlin + Rust (APK) (push) Skipped
Desktop (Tauri) / Build, or is the channel already serving this? (push) Successful in 2s
CI & Build / Web typecheck and unit tests (push) Successful in 9s
CI & Build / Python tests (push) Successful in 13s
Android / Build the server image (push) Skipped
Desktop (Tauri) / Web tests, clippy, Rust tests and rustfmt (push) Skipped
Desktop (Tauri) / Tauri desktop (Linux) (push) Skipped
Desktop (Tauri) / Windows installer (cross-compiled) (push) Skipped
Desktop (Tauri) / Update manifest (push) Skipped
CI & Build / integration (push) Successful in 1m30s
CI & Build / Build & push image (push) Successful in 32s

Rule 32 "Comments describe the product to a stranger" sweep over
src/, alembic/ and tests/. Comments and docstrings now say what the code
does and the reason as a present-tense constraint, briefly.

Removed:
- issue, task, note, rule, milestone and family-idea references
- "used to" / before-and-after narration
- quotes of decisions and incident stories from the deploying instance
- paragraph-length arguments, where a line or two carries the fact

Kept every hazard as a present-tense fact, e.g. timing oracles, the
proxy hop direction, and migrations that must stay frozen.

Also:
- Three test names lose "still"/"used to"/"pre_m304" phrasing.
- pyproject's description drops "FabledSword family".

The history these comments carried stays in Scribe and in git log.
A comment-stripping diff (Scribe snippet #6226 "codeonly.py — prove a
diff changed only comments") shows no code changed apart from the
renamed tests and the description string.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-10-21 20:07:47 -04:00
co-authored by Claude Opus 5.5
parent 5556bb2136
commit a51a563fe7
108 changed files with 779 additions and 1610 deletions
+2 -2
View File
@@ -4,8 +4,8 @@ Revision ID: 0001
Revises: Revises:
Create Date: 2026-07-19 Create Date: 2026-07-19
The M0 identity + sharing-ACL spine. No notes yet (those arrive in M1); the Identity and the sharing ACL. The shares table is polymorphic, so any shareable
shares table is polymorphic so notes reuse it without a schema retrofit. entity uses it.
""" """
from alembic import op from alembic import op
import sqlalchemy as sa import sqlalchemy as sa
+1 -1
View File
@@ -4,7 +4,7 @@ Revision ID: 0002
Revises: 0001 Revises: 0001
Create Date: 2026-07-19 Create Date: 2026-07-19
The M1 capture core: the notes table the masonry board renders. The notes table the board renders.
""" """
from alembic import op from alembic import op
import sqlalchemy as sa import sqlalchemy as sa
+1 -1
View File
@@ -5,7 +5,7 @@ Revises: 0002
Create Date: 2026-07-19 Create Date: 2026-07-19
Adds the admin role bit and the DB-backed settings table. No rows are seeded — Adds the admin role bit and the DB-backed settings table. No rows are seeded —
defaults live in the code registry (thoughtsync.settings), so an empty table defaults live in the code registry (inkwell.settings), so an empty table
means every setting is at its default. means every setting is at its default.
""" """
from alembic import op from alembic import op
+2 -4
View File
@@ -16,10 +16,8 @@ depends_on = None
def upgrade() -> None: def upgrade() -> None:
# The note's display NAME: explicit title if set, else the first body line. # The note's display NAME: explicit title if set, else the first body line.
op.add_column("notes", sa.Column("display_title", sa.Text(), nullable=False, server_default="")) op.add_column("notes", sa.Column("display_title", sa.Text(), nullable=False, server_default=""))
# Best-effort backfill (kept intentionally simple + unambiguous — CI has no # Best-effort backfill, capped at 200 chars. The app recomputes the exact value
# Postgres lane, so this only runs on a real deploy). The app recomputes the # (derive_display_title, which also skips leading blank lines) on the next save.
# precise value via derive_display_title on the next save (which additionally
# skips leading blank lines). Capped at 200 chars.
op.execute( op.execute(
r""" r"""
UPDATE notes SET display_title = LEFT(btrim( UPDATE notes SET display_title = LEFT(btrim(
+1 -4
View File
@@ -1,4 +1,4 @@
"""sync revision + tombstone infrastructure (M8 sync hub, step 1) """sync revision + tombstone infrastructure
Revision ID: 0015 Revision ID: 0015
Revises: 0014 Revises: 0014
@@ -11,9 +11,6 @@ forget to bump it. Child-row changes (items / attachments / label membership) bu
their parent note's revision, since a note syncs as a whole. `purged_at` is the their parent note's revision, since a note syncs as a whole. `purged_at` is the
hard-delete tombstone (distinct from `deleted_at` = recoverable trash) so an hard-delete tombstone (distinct from `deleted_at` = recoverable trash) so an
offline client learns a row is gone instead of resurrecting it. offline client learns a row is gone instead of resurrecting it.
No API behavior changes here — this is pure infrastructure. Triggers/behavior are
operator-verified on deploy (no Postgres CI lane).
""" """
from alembic import op from alembic import op
import sqlalchemy as sa import sqlalchemy as sa
+1 -1
View File
@@ -1,4 +1,4 @@
"""device_tokens (M8 sync hub, step 2) """device_tokens
Revision ID: 0016 Revision ID: 0016
Revises: 0015 Revises: 0015
+1 -1
View File
@@ -1,4 +1,4 @@
"""labels.updated_at (M8 sync hub, step 4 — LWW field for the label catalog) """labels.updated_at, the last-write-wins field for the label catalog
Revision ID: 0017 Revision ID: 0017
Revises: 0016 Revises: 0016
+1 -1
View File
@@ -1,4 +1,4 @@
"""note_attachments.sha256 (M8 sync hub, step 5 — attachment blob sync) """note_attachments.sha256, for attachment blob sync
Revision ID: 0018 Revision ID: 0018
Revises: 0017 Revises: 0017
+1 -1
View File
@@ -1,4 +1,4 @@
"""note_attachments.filename (any-file attachments — M6 1900) """note_attachments.filename, for attachments of any file type
Revision ID: 0019 Revision ID: 0019
Revises: 0018 Revises: 0018
+1 -1
View File
@@ -1,4 +1,4 @@
"""note_link_previews (URL unfurl — M6 1901) """note_link_previews, cached URL unfurls
Revision ID: 0020 Revision ID: 0020
Revises: 0019 Revises: 0019
+1 -1
View File
@@ -1,4 +1,4 @@
"""saved_filters (richer search + saved views — M6 1902) """saved_filters, saved search views
Revision ID: 0021 Revision ID: 0021
Revises: 0020 Revises: 0020
+1 -1
View File
@@ -1,4 +1,4 @@
"""notes.recurrence (recurring reminders — M6 1908) """notes.recurrence, for recurring reminders
Revision ID: 0022 Revision ID: 0022
Revises: 0021 Revises: 0021
+3 -6
View File
@@ -1,14 +1,11 @@
"""note_links.target_id — resolve [[links]] to a note, not to a string (M13 step 1) """note_links.target_id — resolve [[links]] to a note, not to a string
Revision ID: 0023 Revision ID: 0023
Revises: 0022 Revises: 0022
Create Date: 2026-08-22 Create Date: 2026-08-22
A wiki-link stored only as normalized TEXT means a note's name IS the edge: rename A wiki-link stored only as normalized text makes a note's name the edge, so renaming
the note and every inbound link stops matching. The old answer was to rewrite the the note breaks every inbound link. Resolving to an id survives a rename.
`[[Old Name]]` text inside every note that linked to it — workable while an explicit
title existed to hold still, untenable once a note's name is just its first body
line (M13).
`target_norm` stays: it is what an UNRESOLVED link carries, since linking to a note `target_norm` stays: it is what an UNRESOLVED link carries, since linking to a note
that doesn't exist yet is a supported way to create one. that doesn't exist yet is a supported way to create one.
+5 -13
View File
@@ -1,22 +1,14 @@
"""drop note_links — [[wiki-links]] are removed (note 2897) """drop note_links — [[wiki-links]] are removed
Revision ID: 0024 Revision ID: 0024
Revises: 0023 Revises: 0023
Create Date: 2026-08-22 Create Date: 2026-08-22
ThoughtSync is an intermediary surface for capture and recall; a linking system is 0023 stays in the chain: removing an applied revision would strand a database's
organization, which is not what it is for. Backlinks, the graph and the name index alembic_version pointer.
went with it.
0023 (which added `note_links.target_id`) is deliberately left in the chain rather note_links was derived from note bodies, so nothing typed is lost; the `[[text]]` is
than deleted. It shipped in an image and may already be applied, and removing an still in every body.
applied revision would strand a database's alembic_version pointer. So the column is
dropped here along with the table it lived on, and the history stays honest about the
fact that it existed for a day.
No down-migration data concern: note_links was always DERIVED from note bodies. The
`[[text]]` is still sitting in every body it was written in; nothing a person typed is
lost by this.
""" """
from alembic import op from alembic import op
import sqlalchemy as sa import sqlalchemy as sa
+5 -14
View File
@@ -1,19 +1,11 @@
"""drop notes.kind — a checklist is something a note HAS (M13 step 2) """drop notes.kind — a checklist is something a note has
Revision ID: 0025 Revision ID: 0025
Revises: 0024 Revises: 0024
Create Date: 2026-08-22 Create Date: 2026-08-22
`kind` was never a type: a plain TEXT column with no enum and no CHECK, compared A note is a body plus optional checkable items. Items are rows in their own table,
against a hardcoded ("text", "list") tuple in six places. `note_items` was always an so a `kind = 'list'` note keeps every one.
ordinary child table keyed by note_id, serialization always emitted `items` whatever
the kind, and the Android editor already toggled between the two losslessly. The
storage has modelled "a body plus optional checkable items" the whole time; only the
gates forbade it.
Nothing is lost. Items were already rows in their own table, and a note that was
`kind = 'list'` keeps every one of them — it just stops being a different sort of
thing from the note next to it.
""" """
from alembic import op from alembic import op
import sqlalchemy as sa import sqlalchemy as sa
@@ -29,7 +21,6 @@ def upgrade() -> None:
def downgrade() -> None: def downgrade() -> None:
# server_default so existing rows get a value; every note comes back as 'text', # server_default so existing rows get a value. Every note comes back as 'text':
# which is right — a restored note with items would previously have hidden its # nothing records which were lists.
# body, and there is no record of which ones were once lists.
op.add_column("notes", sa.Column("kind", sa.Text(), nullable=False, server_default="text")) op.add_column("notes", sa.Column("kind", sa.Text(), nullable=False, server_default="text"))
+6 -25
View File
@@ -4,33 +4,14 @@ Revision ID: 0026
Revises: 0025 Revises: 0025
Create Date: 2026-08-22 Create Date: 2026-08-22
M13 step 3. A note is a body plus optional checkable items; its NAME is the first A note's name is its first non-empty body line, carried by `display_title`.
non-empty line of that body, falling back to its first checklist item. There is no
separate field to type into, and `display_title` (already persisted, already what
search results and export filenames read) carries the name.
## The search vector has to be rebuilt, not just left alone `notes.search_vector` is a stored generated column over `title` (0005, weight A), and
Postgres won't drop a column a generated one depends on. So it is recreated over
`display_title`, keeping the name ranked above the body. That recomputes every row
and rebuilds the GIN index, which takes a while on a large table.
`notes.search_vector` is a STORED GENERATED column whose expression names `title` `display_title` was already derived from the title, so every note keeps its name.
(migration 0005, weight A) — Postgres will refuse to drop a column another generated
column depends on, and even if it didn't, the weighting would be wrong. So it is
dropped and recreated over `display_title` instead, which keeps the original
intent: the note's NAME ranks above the rest of its body.
Rebuilding a stored generated column re-computes every row, and the GIN index is
rebuilt with it. On a personal instance that is milliseconds; it is worth knowing
before running this against something large.
## What happens to existing titles
Nothing preserves them, deliberately: `display_title` was already derived from the
title when one was set, so every note keeps the NAME it had. What is lost is the
distinction between "this note has an explicit title" and "this note's first line is
its name" — which is the distinction being removed.
Imports are the exception and are handled in code, not here: a Keep note's title, or
one in an export taken before this, is folded in as the note's first body line rather
than dropped (see `_create_imported_note`).
""" """
from alembic import op from alembic import op
import sqlalchemy as sa import sqlalchemy as sa
@@ -4,64 +4,31 @@ Revision ID: 0027
Revises: 0026 Revises: 0026
Create Date: 2026-08-24 Create Date: 2026-08-24
M304. A checklist item becomes a `- [ ] milk` line of `notes.body`, and `note_items` A checklist item becomes a `- [ ] milk` line of `notes.body`, so a list can sit
goes. The reason is positional, not cosmetic: a row had a position in a table and no between two paragraphs, and `note_items` goes.
position in the text, so a separate list could only ever render AFTER the prose. With
the items in the body, a list can sit between two paragraphs — which is the thing that
could not be built before and no amount of restyling would have delivered.
## This migration rewrites note bodies ## This migration rewrites note bodies
Every note that has items gets its body appended to. The rules below are strict * Rows are read before the table is dropped, in this one transaction.
because rewriting somebody's text deserves it — not, as an earlier draft of this * The existing body is only appended to.
docstring claimed, because this instance holds imported Google Keep notes. It does * The layout (a blank line between prose and the list, nothing between items) is
not; note 2916's headline is that nothing here is anyone's work but the operator's what the export writes and what `derive::append_item` produces, so the clients
test data. What 2916 actually says about imports is conditional — text arriving from migrate their own stores independently and still agree with the server.
another app WOULD be real, and any import path has to treat it that way — and the
importer this migration shares a format with is one nobody here has run.
Careful was still the right call. It cost little, and the same care is what the rule The fold is inlined rather than imported from `notes/checklist.py`: a migration must
demands the day someone does import something: keep producing what it produced the day it ran.
* Rows are read BEFORE the table is dropped, in this one transaction. ## `updated_at` is left alone
* The existing body is never rewritten, only appended to.
* The layout — a blank line between prose and the list, nothing between consecutive
items — is byte-for-byte what `_note_markdown` has always exported and what
`derive::append_item` produces on every client. All three landing on the same text
is what lets the clients migrate their own SQLite stores independently and still
agree with the server, with no sync required to reconcile them.
## The fold is inlined on purpose Raw SQL, so `onupdate` never fires, and a client holding an unpushed edit keeps the
newer `updated_at` and wins last-write-wins. The `sync_revision` trigger does fire,
so every migrated note is pulled once and clients converge.
`notes/checklist.py` has this same function and this migration deliberately does not ## The downgrade is not a true inverse
import it. A migration has to keep producing what it produced the day it ran; if the
app's spacing rule ever changes, this file must not change with it.
## `updated_at` is left alone, and that is load-bearing It recreates an empty `note_items` and leaves the bodies alone: nothing tells a line
this wrote from one somebody typed. The real rollback is a database restore. The
Raw SQL, so SQLAlchemy's `onupdate` never fires. Two reasons, and the second matters table must exist because 0015's downgrade drops a trigger on it.
more than the first. Every client folds the same rows the same way, so the new body is
news to nobody. And a client holding an UNPUSHED body edit still has the newer
`updated_at`, so when it pulls the migrated note last-write-wins keeps its edit instead
of the migration silently winning.
The `notes` row's own `sync_revision` trigger (migration 0015) does fire, so every
migrated note becomes pullable once. That is wanted: it is what makes a client whose
local fold somehow differed converge on the server's text.
## The downgrade is not a true inverse, and says so
It recreates an empty `note_items` and leaves the bodies alone. Nothing is lost —
every item is still there as text, which is where this migration put it — but the old
code would show those notes as prose with no checklist. A faithful inverse is not
possible: once the items are lines, nothing distinguishes a line this migration wrote
from one somebody typed, and a downgrade that guessed would eat hand-written task
lists. The real rollback is a database restore.
Recreating the table is not decoration, though. Migration 0015's downgrade runs
`DROP TRIGGER IF EXISTS trg_note_items_bump_note ON note_items`, and `IF EXISTS`
covers the trigger, not the table — against a missing table that statement errors. So
this is what keeps the migration chain runnable all the way back down.
""" """
import re import re
@@ -100,9 +67,8 @@ def upgrade():
for note_id, items in grouped.items(): for note_id, items in grouped.items():
body = bind.execute(sa.text("SELECT body FROM notes WHERE id = :id"), {"id": note_id}).scalar() body = bind.execute(sa.text("SELECT body FROM notes WHERE id = :id"), {"id": note_id}).scalar()
# An item whose note is already gone has nothing to fold into. The foreign key # An item whose note is gone has nothing to fold into. The foreign key should
# should make this impossible; skipping costs nothing and failing here would # prevent it; skipping avoids a half-migrated database.
# leave the database half-migrated.
if body is None: if body is None:
continue continue
for text, checked in items: for text, checked in items:
@@ -113,9 +79,8 @@ def upgrade():
def downgrade(): def downgrade():
# Column-for-column as migration 0006 created it, index name included: 0015's # Exactly as 0006 created it, index name included: 0015's downgrade names both
# downgrade names both the table and its trigger, so a near-enough copy is not # the table and its trigger.
# good enough.
op.create_table( op.create_table(
"note_items", "note_items",
sa.Column("id", UUID(as_uuid=True), primary_key=True), sa.Column("id", UUID(as_uuid=True), primary_key=True),
+19 -71
View File
@@ -4,73 +4,27 @@ Revision ID: 0028
Revises: 0027 Revises: 0027
Create Date: 2026-08-26 Create Date: 2026-08-26
M311. A `#tag` was being shown twice — once as the text you typed and once as a chip — A tag standing on its own line shows as a chip, so its text is removed, by the same
and with the chip moved to the top of the card the text is redundant. This removes it, rule as `notes/tags.py:split_body_tags`:
but only from notes where the tag was standing on its own.
## This migration rewrites note bodies * A line containing nothing but tags and whitespace is removed.
* Every other line is left exactly as written, so `remember to call #mom` stays.
* A line inside a ``` fence is never touched.
* A note that is nothing but tags keeps its text, rather than becoming a blank card.
The rule is deliberately narrow, and the same one `notes/tags.py:split_body_tags` The label rows flip to `via_tag = false` in the same transaction: the text backing
applies from here on: them is gone, and `_lift_and_reconcile_tags` would otherwise detach them on the next
save.
* A line containing nothing but tags and whitespace is REMOVED. The transform and `_display_title` are inlined: a migration must keep producing what
* Every other line is left exactly as written. it produced the day it ran. The title is recomputed only where the body moved.
So `#todo` on its own line goes, and `remember to call #mom tomorrow` does not. The Raw SQL, so `onupdate` never fires and a client's unpushed edit still wins
looser reading — also stripping a trailing tag off a prose line — was rejected because last-write-wins. The `sync_revision` trigger fires, so clients pull the server's text.
the text does not say which kind it is: `buy milk #grocery` is filing, `remember to
call #mom` is the sentence's object, and lifting the second leaves "remember to call".
Rewriting somebody's words to save a duplicate chip is a bad trade, and a migration is
the worst possible place to make it.
Two guards, both of which cost a note nothing: The downgrade does nothing. Restoring the lines would be guessing, and flipping the
rows back to `via_tag = true` would detach every label on the next save. The real
* A line inside a ``` fence is never touched. A `#tag` there is a shell comment in a rollback is a database restore.
snippet somebody pasted, and deleting it would eat a line of their example.
* A note that is NOTHING but tags keeps its text. Lifting would leave a blank card,
which is worse than the duplication this fixes.
## The label rows have to graduate in the same transaction
A `via_tag` row means "this label is backed by text still in the body". Once the text
is gone that is false, and leaving it true is not cosmetic: `_lift_and_reconcile_tags`
detaches any `via_tag` row it cannot find a `#tag` for, so the note would lose the tag
on its very next save. The flip to `via_tag = false` is what makes the label the record
instead — and what makes the chip's × appear in both editors, which is now the only way
to remove a tag whose text no longer exists.
## The transform is inlined, like 0027's
`split_body_tags` is deliberately NOT imported. A migration has to keep producing what
it produced the day it ran; if the app's rule is ever loosened, this file must not
loosen with it and start eating prose it previously left alone.
`_display_title` is inlined for the same reason, and is only recomputed for a note whose
body actually moved — a note named after a `#todo` line needs a new name, and reading it
from the app would couple this migration to a rule that has already changed once (M13).
## `updated_at` is left alone, and that is load-bearing
Raw SQL, so SQLAlchemy's `onupdate` never fires. A client holding an UNPUSHED body edit
keeps the newer `updated_at`, so when it pulls the migrated note last-write-wins keeps
its edit instead of the migration silently winning.
The `sync_revision` trigger (migration 0015) does fire, so every rewritten note becomes
pullable once and clients converge on the server's text. That is wanted here: unlike
0027, the clients do NOT yet apply this rule locally, so the server's copy is the only
correct one until they do.
## The downgrade is not a true inverse, and says so
It cannot be. Nothing distinguishes a `#todo` line this migration deleted from one that
was never there, and putting one back would be guessing at where in the note it went.
Nothing is lost, though, which is why that is acceptable: the tag still exists as a
label on the note, and the chip still shows it. What a downgrade cannot restore is the
DUPLICATE — which is the thing this migration set out to remove. Rolling the rows back
to `via_tag = true` would be actively harmful: the text that flag claims to be backed by
is gone, so the next save would detach the label and lose the tag for real. So the
downgrade leaves both alone. The real rollback is a database restore.
""" """
import re import re
@@ -152,17 +106,11 @@ def upgrade():
sa.text("UPDATE notes SET body = :body, display_title = :title WHERE id = :id"), sa.text("UPDATE notes SET body = :body, display_title = :title WHERE id = :id"),
{"body": lifted, "title": _display_title(lifted), "id": note_id}, {"body": lifted, "title": _display_title(lifted), "id": note_id},
) )
# Even when the body did not move, a tag can be standalone only in the sense # The flip is driven by the tag list, not by whether the text changed: an
# that its line was already removed by an earlier pass — so the flip is driven # earlier pass may already have removed the line.
# by the tag list, not by whether the text changed.
if standalone: if standalone:
bind.execute(flip, {"nid": note_id, "names": [n.lower() for n in standalone]}) bind.execute(flip, {"nid": note_id, "names": [n.lower() for n in standalone]})
def downgrade(): def downgrade():
"""Deliberately empty — see the module docstring. """Deliberately empty; see the module docstring."""
Restoring the deleted lines would be guessing, and flipping the rows back to
`via_tag = true` would be worse than doing nothing: the text that flag claims backs
them is gone, so the next save would detach the label and lose the tag for real.
"""
+9 -42
View File
@@ -4,50 +4,18 @@ Revision ID: 0029
Revises: 0028 Revises: 0028
Create Date: 2026-08-28 Create Date: 2026-08-28
M315 step 3. A note's colour was set by a picker and read by three card renderers. Cards are one neutral surface per theme; the only colour on a board is a tag's.
Steps 1 and 2 stopped every one of those reads: the card is one neutral per theme and `labels.color` is untouched. A colour a note was given is dropped with the column.
the only coloured thing on a board is a tag. This drops the column that nothing has
been reading since, and the picker goes with it.
`labels.color` is untouched. That is the colour that survived, and the one the whole `saved_filters.params` may carry a `"color"` facet, so stored views are swept too. In
milestone was about keeping. Python rather than `params::jsonb - 'color'`, because Postgres has no try-cast and one
malformed blob would abort the migration; a row that doesn't parse is left as it is.
## What is lost, and why that is the change rather than a cost of it The search vector never named `color`, so it needs no rebuild.
Any colour a note was explicitly given. There is nowhere to preserve it TO — the field
it would be preserved in is the one being dropped — and nothing renders it, so a
preserved value would be a column kept warm for a feature that was deliberately
removed. A note that had a colour now takes its identity from its tags, which is what
the operator asked for: "strip color from the cards ... and keep the color for tags
just on the tag."
The palette itself is not lost. `NOTE_COLORS` moved from `models/note.py` to
`colors.py` in the same change — labels still name a colour, and leaving the vocabulary
defined on the model that lost one would be an invitation to put the column back.
## The saved-filter sweep is not optional
`saved_filters.params` is opaque JSON mirroring the `GET /api/notes` facet query, and
a stored view could carry `"color": "teal"`. With the facet gone that key would sit
there forever, and `clean_params` only guards what is written FROM here on. A view that
silently filters on a field the app no longer has is worse than one that visibly lost a
criterion, so the stored rows are swept too.
Done in Python rather than as `params::jsonb - 'color'`, deliberately. Postgres has no
try-cast: one malformed blob would abort the whole migration, and these rows are
somebody's saved views. `json.loads` in a try/except lets a corrupt row keep whatever it
holds and lets every other row be fixed.
## Search is not affected
`notes.search_vector` is a stored generated column over `display_title` and `body`
(rebuilt in 0026). It never named `color`, so unlike the title drop there is nothing
here to tear down and recreate.
## Downgrade ## Downgrade
Restores the column, empty, at its old default. The values are not recoverable — see Restores the column, empty, at its old default. The values are not recoverable.
above. It is the schema that comes back, not the data.
""" """
import json import json
@@ -71,9 +39,8 @@ def upgrade() -> None:
try: try:
parsed = json.loads(params) parsed = json.loads(params)
except (ValueError, TypeError): except (ValueError, TypeError):
# A blob that does not parse cannot be edited safely. Leaving it is # A blob that doesn't parse can't be edited safely; the app couldn't read
# correct: it was already unreadable by the app, and this migration is not # it either.
# the place to decide what it should have said.
continue continue
if not isinstance(parsed, dict) or "color" not in parsed: if not isinstance(parsed, dict) or "color" not in parsed:
continue continue
+3 -9
View File
@@ -4,15 +4,9 @@ Revision ID: 0030
Revises: 0029 Revises: 0029
Create Date: 2026-10-06 Create Date: 2026-10-06
The product is renamed ThoughtSync → Inkwell (Scribe note 5071), and the `site_name` The app was renamed from ThoughtSync to Inkwell. The Settings page saves every key,
registry default moves with it. On its own that only reaches servers whose settings so most servers have a stored `site_name` of the old default without anyone choosing
table has no `site_name` row — and most servers have one, because the Settings page it. The row is rewritten only when it holds exactly that; a typed name is left alone.
saves EVERY key on Save, not just the one that changed. An admin who opened Settings
to flip registration off has persisted `"ThoughtSync"` without ever choosing it.
So the row is rewritten, but only when it holds exactly the old default. A name the
admin actually typed is theirs and is left alone; the old default is the one value we
can be sure nobody chose.
The match is on the stored JSON text (`settings.value` is `json.dumps(value)`), so The match is on the stored JSON text (`settings.value` is `json.dumps(value)`), so
`'"ThoughtSync"'` is the whole of what it looks for. `'"ThoughtSync"'` is the whole of what it looks for.
@@ -4,16 +4,9 @@ Revision ID: 0031
Revises: 0030 Revises: 0030
Create Date: 2026-10-07 Create Date: 2026-10-07
0015 made every child table bump its parent note's `sync_revision`, so a note syncs A note syncs as a whole, so every child table bumps its note's `sync_revision`. With
as a whole. `note_link_previews` arrived later (0020) and was never added, and two this, a preview fetched after a save, or dismissed on the web, reaches every device.
things have been missing on every linked device since: The trigger is the function 0015 installs on the other child tables.
- A preview fetched in the background after a save (`unfurl_queue.py`) never reached
a device that had already pulled the note. The note's revision was assigned when
the TEXT was saved, before the preview existed, and nothing moved it afterwards.
- A preview dismissed on the web stayed on every other device, for the same reason.
The trigger is the same function 0015 installs on the other child tables.
## Downgrade ## Downgrade
+2 -3
View File
@@ -4,9 +4,8 @@ Revision ID: 0032
Revises: 0031 Revises: 0031
Create Date: 2026-10-07 Create Date: 2026-10-07
Until now the only way to add a second person was to re-open registration to the An invite lets one person register while registration stays closed. Only the token's
whole internet while they signed up (#2939 §1). An invite lets one person register SHA-256 hash is stored.
while registration stays closed. Only the token's SHA-256 hash is stored.
## Downgrade ## Downgrade
+5 -6
View File
@@ -4,20 +4,19 @@ Revision ID: 0033
Revises: 0032 Revises: 0032
Create Date: 2026-10-07 Create Date: 2026-10-07
A forgotten password used to need a hand on the database (#2939 §2), and the app has An admin makes a reset link for the account and hands it over. Only the token's
no mail path to send a reset by. An admin makes a reset link for the account and SHA-256 hash is stored.
hands it over (#5173). Only the token's SHA-256 hash is stored.
`users.session_epoch` is what lets a reset sign the account out everywhere. Sessions `users.session_epoch` is what lets a reset sign the account out everywhere. Sessions
are signed cookies held by the browser, so the server can't delete them; each one are signed cookies held by the browser, so the server can't delete them; each one
carries the epoch it was signed in under, and a reset moves the account's epoch on. carries the epoch it was signed in under, and a reset moves the account's epoch on.
It starts at 0, the value a cookie from before this migration is read as, so nobody It starts at 0, the value a cookie without one is read as, so the upgrade signs
is signed out by the upgrade itself. nobody out.
## Downgrade ## Downgrade
Drops the table and the column. Outstanding reset links stop working; sessions keep Drops the table and the column. Outstanding reset links stop working; sessions keep
working, since nothing checks an epoch any more. working, since nothing checks an epoch.
""" """
import sqlalchemy as sa import sqlalchemy as sa
from alembic import op from alembic import op
+2 -3
View File
@@ -4,11 +4,10 @@ Revision ID: 0034
Revises: 0033 Revises: 0033
Create Date: 2026-10-07 Create Date: 2026-10-07
Nothing wrote a share until #5174, so the table never needed these. Now the Share For the Share dialog:
dialog does:
- A unique index on (resource_type, resource_id, shared_with_user_id) where a user is - A unique index on (resource_type, resource_id, shared_with_user_id) where a user is
the target, so sharing a note with someone twice updates the one grant rather than the target, so sharing a note with someone twice updates the one grant rather than
stacking two (the same for a group, ahead of step 15). stacking two (the same for a group).
- A check that `permission` is `view` or `edit`. - A check that `permission` is `view` or `edit`.
- An index on `shared_with_user_id` for "Shared with me". - An index on `shared_with_user_id` for "Shared with me".
+1 -1
View File
@@ -5,7 +5,7 @@ Revises: 0034
Create Date: 2026-10-07 Create Date: 2026-10-07
The change feed carries every note a person can see, including notes shared with The change feed carries every note a person can see, including notes shared with
them (#5175). When a share ends, the note stops being visible, so it simply stops them. When a share ends, the note stops being visible, so it simply stops
appearing in the feed. A device that already holds a copy would keep it forever. A appearing in the feed. A device that already holds a copy would keep it forever. A
revocation is that missing signal: one row per (note, person) who lost the note, revocation is that missing signal: one row per (note, person) who lost the note,
stamped from the same `sync_revision_seq` the notes and labels draw from, so it sits stamped from the same `sync_revision_seq` the notes and labels draw from, so it sits
+4 -6
View File
@@ -4,12 +4,10 @@ Revision ID: 0036
Revises: 0035 Revises: 0035
Create Date: 2026-10-07 Create Date: 2026-10-07
Pin, archive and board order are personal organization, and until now they were Pin, archive and board order are personal. The owner's are columns on `notes`; this
columns on `notes`, so on a shared note they could only ever mean the owner's table holds everyone else's, one row per (note, person it is shared with) who has
(#5176). This table holds them for everyone else: one row per (note, person it is pinned, archived or moved it. A recipient with no row sees the note unpinned,
shared with) who has pinned, archived or moved it. The owner keeps the note's own unarchived, in the owner's order.
columns; a recipient with no row sees the note unpinned, unarchived, in the owner's
order.
`sync_revision` is stamped by the same `ts_set_sync_revision()` trigger the notes `sync_revision` is stamped by the same `ts_set_sync_revision()` trigger the notes
and labels use (0015), so a recipient's change reaches their own devices on the one and labels use (0015), so a recipient's change reaches their own devices on the one
@@ -4,12 +4,8 @@ Revision ID: 0037
Revises: 0036 Revises: 0036
Create Date: 2026-10-07 Create Date: 2026-10-07
Both came with the foundation (0001) for features that were never built (#5178). Nothing set `email_verified` or read `avatar_path`, and a field whose value means
Nothing ever set `email_verified`, so it was false for every account on the server nothing would be trusted by a later feature.
while the offline desktop reported it as true; nothing read or wrote `avatar_path`.
A field whose value means nothing is worse than no field: a later feature would
trust it. If address verification is ever built, it adds its own column, with a
migration that says what the existing accounts are.
## Downgrade ## Downgrade
+1 -3
View File
@@ -4,9 +4,7 @@ Revision ID: 0038
Revises: 0037 Revises: 0037
Create Date: 2026-10-08 Create Date: 2026-10-08
Saved views were removed in #5180 (note 2897: recall first, organizing second). They Saved views are removed; tags in the drawer give one-click recall.
lived only on the web; the desktop kept its own set that never synced, and Android
had none. Tags in the drawer already give one-click recall.
## Downgrade ## Downgrade
+2 -3
View File
@@ -4,9 +4,8 @@ Revision ID: 0039
Revises: 0038 Revises: 0038
Create Date: 2026-10-08 Create Date: 2026-10-08
Credential events were only written to the app log, which is not queryable and is Keeps credential events in the database for `audit_retention_days`, for admins to
gone at the container's log rotation (#2939 §5). This keeps them in the database for read in Settings → Activity; the app log isn't queryable and rotates away.
`audit_retention_days`, for admins to read in Settings → Activity.
## Downgrade ## Downgrade
+3 -5
View File
@@ -4,11 +4,9 @@ Revision ID: 0040
Revises: 0039 Revises: 0039
Create Date: 2026-10-08 Create Date: 2026-10-08
Deleting a tag now leaves a tombstone row (`purged_at` set) so the change feed can Deleting a tag leaves a tombstone row (`purged_at` set) so the change feed tells
tell linked devices about it (#5382). Before, the web deleted the row outright and linked devices. A tombstone must not hold its name, so `(owner_id, name)` is unique
devices never heard. A tombstone must not hold its name: creating `#grocery` again only where `purged_at IS NULL`.
after deleting it has to make a live tag, on the web and from a device alike. So
`(owner_id, name)` stays unique only where `purged_at IS NULL`.
## Downgrade ## Downgrade
+1 -1
View File
@@ -5,7 +5,7 @@ build-backend = "setuptools.build_meta"
[project] [project]
name = "inkwell" name = "inkwell"
version = "0.2.0" version = "0.2.0"
description = "Self-hosted personal thought-capture web app (FabledSword family)" description = "A self-hosted web app for capturing and finding thoughts"
requires-python = ">=3.12" requires-python = ">=3.12"
dependencies = [ dependencies = [
"quart>=0.19", "quart>=0.19",
+3 -14
View File
@@ -1,16 +1,5 @@
"""Inkwell — self-hosted personal thought-capture web app (FabledSword family).""" """Inkwell — a self-hosted web app for capturing and finding thoughts."""
# PACKAGING METADATA, and nothing else. Not the version any running server reports. # Packaging metadata only. A running server reports APP_VERSION from the image, or
# # "unknown"; this string is never shown and bumping it changes nothing.
# A built image carries APP_VERSION in the environment, derived from the server's
# own shipped file set (packaging/version.sh); `app.py` reads that and reports an
# explicit "unknown" when it is absent, so this string never reaches a user and
# bumping it changes nothing anybody sees.
#
# It exists because a Python package needs a version and "unknown" is not a legal
# one here. It used to double as app.py's fallback, which meant a server run from a
# checkout confidently reported `0.2.0` — a real-looking version naming no build
# that exists. Note 3127 §5 is why that matters more than it reads: with version
# tags gone, a build's self-report is the only answer to "which build is this?",
# and there is nothing left to catch it lying.
__version__ = "0.2.0" __version__ = "0.2.0"
+1 -1
View File
@@ -1,5 +1,5 @@
"""Admin routes over the instance's accounts: list them, make a password reset link """Admin routes over the instance's accounts: list them, make a password reset link
for one (#5173), and read what has happened to them (the audit log, #2939 §5). for one, and read what has happened to them (the audit log).
What a reset link is, and how it is used, is in `password_resets.py`. What a reset link is, and how it is used, is in `password_resets.py`.
""" """
+5 -6
View File
@@ -7,9 +7,9 @@ from sqlalchemy import ColumnElement, exists, or_, select
from .models.group import GroupMember from .models.group import GroupMember
from .models.share import Share from .models.share import Share
# What a share grants (#5174). `edit` is the note's body and checklist; everything # What a share grants. `edit` covers the note's body and checklist; everything else
# else about a note stays its owner's. Ordered weakest first, so the strongest of a # stays the owner's. Ordered weakest first, so a person's strongest grant has the
# person's grants is the one with the highest index. # highest index.
VIEW = "view" VIEW = "view"
EDIT = "edit" EDIT = "edit"
PERMISSIONS = (VIEW, EDIT) PERMISSIONS = (VIEW, EDIT)
@@ -26,9 +26,8 @@ def visible_to_user(
"""Boolean SQL predicate: is this resource visible to ``user_id``? """Boolean SQL predicate: is this resource visible to ``user_id``?
A resource is visible if the user OWNS it, OR it is shared DIRECTLY with the A resource is visible if the user OWNS it, OR it is shared DIRECTLY with the
user, OR it is shared with a GROUP the user belongs to. Scope every read (and user, OR it is shared with a GROUP the user belongs to. Scope every read and
mutation) of shareable user data through this — never assume a single operator mutation of shareable user data through this; the server is multi-user.
(family rule 47).
With ``permission``, only shares at that level count: ``permission=EDIT`` is With ``permission``, only shares at that level count: ``permission=EDIT`` is
"may this user change it", which the owner always may. "may this user change it", which the owner always may.
+32 -69
View File
@@ -41,13 +41,9 @@ from .settings_api import bp as settings_bp
from .shares_api import bp as shares_bp from .shares_api import bp as shares_bp
from .sync import bp as sync_bp, protocol_advertisement from .sync import bp as sync_bp, protocol_advertisement
# Without this, `logger.info` from this package goes nowhere: hypercorn configures its # Hypercorn leaves the root logger at WARNING, which would hide this package's INFO
# own access/error loggers and leaves the root logger at WARNING, so the credential # credential events from `docker compose logs`. A host that already configured
# events in auth.py would be invisible in `docker compose logs` — which is exactly # logging keeps its own setup.
# where they are read live. (Settings → Activity keeps most of them, `audit.py`.)
#
# `force=False` (the default) so a host that has already configured logging keeps its
# own setup; LOG_LEVEL lets an operator turn it up without a code change.
logging.basicConfig( logging.basicConfig(
level=os.environ.get("INKWELL_LOG_LEVEL", "INFO").upper(), level=os.environ.get("INKWELL_LOG_LEVEL", "INFO").upper(),
format="%(asctime)s %(levelname)s %(name)s %(message)s", format="%(asctime)s %(levelname)s %(name)s %(message)s",
@@ -65,10 +61,8 @@ class _AutoSecureSessionInterface(SecureCookieSessionInterface):
"""Mark the session cookie `Secure` whenever the request arrived over HTTPS — """Mark the session cookie `Secure` whenever the request arrived over HTTPS —
directly, or via a TLS-terminating reverse proxy that sets X-Forwarded-Proto. directly, or via a TLS-terminating reverse proxy that sets X-Forwarded-Proto.
Auto-detecting per request (rather than a fixed SESSION_COOKIE_SECURE flag) Per request rather than a fixed flag, because a forced Secure cookie is never sent
hardens the cookie on HTTPS deployments without breaking a plain-HTTP install on back over plain HTTP, which would break sign-in on a LAN install.
a trusted LAN, where a hard-forced Secure flag would stop the browser from ever
sending the cookie back — i.e. silently break login. No configuration required.
""" """
def get_cookie_secure(self, app: Quart) -> bool: def get_cookie_secure(self, app: Quart) -> bool:
@@ -83,19 +77,8 @@ def create_app() -> Quart:
# Ephemeral/env secret so the app (and DB-free unit tests) construct without a # Ephemeral/env secret so the app (and DB-free unit tests) construct without a
# database. before_serving swaps in the real, DB-persisted key before serving. # database. before_serving swaps in the real, DB-persisted key before serving.
app.secret_key = Config.secret_key_env() or secrets.token_urlsafe(48) app.secret_key = Config.secret_key_env() or secrets.token_urlsafe(48)
# The RUNNING build, or an explicit "unknown" — never the packaging fallback. # The running build, or "unknown". Never `__version__`: that names no build, and
# # this string is the only answer to "which build is this?".
# This read `os.environ.get("APP_VERSION", __version__)`, so a server started
# from a checkout reported `0.2.0`: a real-looking version that names no build
# anybody could get. `__init__.py` already claimed the honest answer was
# "APP_VERSION being missing, which app.py already handles" — it did not, and a
# comment asserting a behaviour two files away from the code is how that stayed
# true-sounding for months.
#
# It matters more than it used to. Note 3127 §5 removed version tags, so this
# string is the only answer to "which build is this?" and nothing exists to
# contradict it when it is wrong. `__version__` stays where it belongs, as
# packaging metadata, which is the one place "unknown" is not a legal value.
app.config["APP_VERSION"] = os.environ.get("APP_VERSION") or "unknown" app.config["APP_VERSION"] = os.environ.get("APP_VERSION") or "unknown"
app.config["SESSION_COOKIE_HTTPONLY"] = True app.config["SESSION_COOKIE_HTTPONLY"] = True
app.config["SESSION_COOKIE_SAMESITE"] = "Lax" app.config["SESSION_COOKIE_SAMESITE"] = "Lax"
@@ -130,14 +113,11 @@ def create_app() -> Quart:
apply_session_ttl(app, await get_setting(db, "session_ttl_days")) apply_session_ttl(app, await get_setting(db, "session_ttl_days"))
except (ValueError, TypeError, KeyError): except (ValueError, TypeError, KeyError):
pass pass
# The security settings the throttle and the proxy trust read on hot # Security settings read on hot paths, cached rather than queried per
# paths. Cached rather than queried per request; until this runs they # request. Until this runs they hold their registry defaults.
# hold their registry defaults, which is the correct behaviour for a
# server that has not finished starting.
await refresh_live(db) await refresh_live(db)
# Credentials stored before sealing existed are sealed now (#5927). A data # Seal any credentials still stored in the clear. A data folder that can't
# folder that can't be written leaves them as they are; it never stops the # be written leaves them as they are; it never stops the server starting.
# server starting.
try: try:
if count := await seal_stored_secrets(db): if count := await seal_stored_secrets(db):
await db.commit() await db.commit()
@@ -145,11 +125,9 @@ def create_app() -> Quart:
except SealError as exc: except SealError as exc:
await db.rollback() await db.rollback()
logger.warning("stored credentials left unsealed: %s", exc) logger.warning("stored credentials left unsealed: %s", exc)
# Expire old trash in the background (retention.py). One task per process is # Expire old trash in the background (retention.py). One task per process,
# correct because the image serves with a single hypercorn worker (Dockerfile); # because the image runs one hypercorn worker; with several, each would sweep,
# if that ever gains `--workers`, this needs a lock so N workers don't each # which is wasteful but harmless.
# sweep. Duplicate sweeps would be harmless but wasteful — a purged row is
# skipped by `purged_at IS NULL` — so this is about load, not correctness.
app.config["TRASH_SWEEPER"] = asyncio.create_task(run_sweeper()) app.config["TRASH_SWEEPER"] = asyncio.create_task(run_sweeper())
# The daily backup (backup_job.py), on the same single-worker footing. It also # The daily backup (backup_job.py), on the same single-worker footing. It also
# takes a Postgres advisory lock, so a second process could not double it. # takes a Postgres advisory lock, so a second process could not double it.
@@ -172,23 +150,16 @@ def create_app() -> Quart:
None of these change how the app behaves for a legitimate caller; they narrow None of these change how the app behaves for a legitimate caller; they narrow
what a browser will do if something else goes wrong. what a browser will do if something else goes wrong.
The CSP is the substantive one. `script-src 'self'` means that even if some The CSP allows only the app's own scripts; it has no inline or third-party
future path did manage to reflect user text into the page, the browser would ones. The exceptions:
refuse to run it — the app has no inline scripts and no third-party scripts, - `style-src 'unsafe-inline'`: Vue writes inline styles (v-show, FLIP
so nothing legitimate is given up. The exceptions are honest ones: transitions). Inline style is not an execution primitive.
- `style-src 'unsafe-inline'` — Vue writes inline styles itself (`v-show` - `img-src https: http:`: link previews show the linked page's image, from
toggling display, TransitionGroup's FLIP setting transforms). Inline any host. http too, for LAN installs; an https page blocks those anyway.
STYLE is not an execution primitive the way inline script is. - `blob:`/`data:`: attachment previews and the desktop's blob URIs.
- `img-src https: http:` — link previews render the remote og:image of
whatever was linked, which is an arbitrary host by definition. Both
schemes, because a LAN install is served over http and would otherwise
lose every preview image; on an https instance the browser blocks the
http ones as mixed content anyway, so naming it concedes nothing.
- `blob:`/`data:` — attachment previews and the desktop's blob URI scheme.
`frame-ancestors 'none'` replaces the older X-Frame-Options and is what stops `frame-ancestors 'none'` prevents clickjacking; `form-action 'self'` keeps
the app being framed for clickjacking; `form-action 'self'` stops a form from forms from posting to another origin.
being pointed at another origin.
""" """
response.headers.setdefault( response.headers.setdefault(
"Content-Security-Policy", "Content-Security-Policy",
@@ -204,9 +175,7 @@ def create_app() -> Quart:
"media-src 'self' blob:; " "media-src 'self' blob:; "
"connect-src 'self'", "connect-src 'self'",
) )
# Content-type sniffing turns a file we said was text into whatever the bytes # Stop the browser sniffing a response into a different content type.
# look like. The attachment route already sets this; every other response
# deserves it too.
response.headers.setdefault("X-Content-Type-Options", "nosniff") response.headers.setdefault("X-Content-Type-Options", "nosniff")
# Note titles and label names end up in the URL of a search or a label lens, # Note titles and label names end up in the URL of a search or a label lens,
# and a full Referer would hand them to any site a link preview points at. # and a full Referer would hand them to any site a link preview points at.
@@ -216,14 +185,9 @@ def create_app() -> Quart:
response.headers.setdefault( response.headers.setdefault(
"Permissions-Policy", "camera=(), microphone=(), geolocation=(), interest-cohort=()" "Permissions-Policy", "camera=(), microphone=(), geolocation=(), interest-cohort=()"
) )
# HSTS only where the request already arrived over TLS — the same detection # HSTS only on requests that arrived over TLS: on a plain-HTTP install it would
# the session cookie uses. Sending it on a plain-HTTP LAN install would tell # lock the browser out. This host only, no includeSubDomains or preload. The
# the browser to refuse the only scheme that install serves. # browser keeps the policy for a year, even after the header stops.
#
# Scoped to this host: no `includeSubDomains` and no `preload`, both of which
# commit domains this app does not own. A browser still remembers the policy
# for up to a year after the header stops being sent, which is the point of
# it — worth knowing before putting a hostname behind TLS temporarily.
if is_https(): if is_https():
response.headers.setdefault("Strict-Transport-Security", "max-age=31536000") response.headers.setdefault("Strict-Transport-Security", "max-age=31536000")
return response return response
@@ -238,13 +202,12 @@ def create_app() -> Quart:
async with session_scope() as db: async with session_scope() as db:
data = await get_public_config(db) data = await get_public_config(db)
data["version"] = app.config["APP_VERSION"] data["version"] = app.config["APP_VERSION"]
# The sync-protocol handshake (M10.6). A native client reads this BEFORE # The sync-protocol handshake. A native client reads this before linking, with
# linking — while it still has no token and possibly no account — to decide # no token yet, to decide whether it can talk to this server and which
# whether it can talk to this server, and which optional features to offer. # optional features to offer.
data.update(protocol_advertisement()) data.update(protocol_advertisement())
# Which CLIENTS this server can hand out, if any, under `clients`. Absent # The clients this server can hand out. Absent when it has none, so the web UI
# rather than null when it has none, so the web UI hides a download instead # hides the download.
# of offering a button that 404s.
data.update(client_advertisement()) data.update(client_advertisement())
return jsonify(data) return jsonify(data)
+1 -1
View File
@@ -1,4 +1,4 @@
"""The audit log: what happened to accounts, kept where an admin can read it (#2939 §5). """The audit log: what happened to accounts, kept where an admin can read it.
Sign-ins, failed sign-ins, password changes and resets, devices linked and Sign-ins, failed sign-ins, password changes and resets, devices linked and
unlinked, invites made and revoked. Each is still written to the app log as it unlinked, invites made and revoked. Each is still written to the app log as it
+40 -73
View File
@@ -32,30 +32,20 @@ from .storage import storage_limit, storage_used
bp = Blueprint("auth", __name__, url_prefix="/api/auth") bp = Blueprint("auth", __name__, url_prefix="/api/auth")
# Every credential event goes to the app log, which is the live view of whether # Every credential event goes to the app log (`docker compose logs`); most also go to
# anyone is knocking: `docker compose logs`. Most also go to the audit log # the audit log (Settings → Activity). The attempted email is logged so a failure can
# (`audit.py`), which keeps them for admins to read in Settings → Activity. # be acted on; passwords never are.
#
# The attempted email is included deliberately. It is the operator's own server, and
# "somebody failed a login" without saying against WHICH account tells you nothing you
# can act on. Passwords, obviously, never appear.
logger = logging.getLogger(__name__) logger = logging.getLogger(__name__)
SESSION_KEY = "user_id" SESSION_KEY = "user_id"
# The account's `session_epoch` when this session signed in. A password reset moves # The account's `session_epoch` when this session signed in. A password reset moves
# the account's epoch on, which is how it ends sessions it can't reach: they are # the epoch on, which ends sessions held as signed cookies in other browsers. A
# signed cookies in other people's browsers. A cookie from before epochs existed has # cookie without the key reads as 0, every account's starting epoch.
# no key and reads as 0, the epoch every account started at.
EPOCH_KEY = "epoch" EPOCH_KEY = "epoch"
# The first account can only be created this long after the server starts (family # How long after start the first account may be created. Otherwise a fresh server on
# idea #5105, practice 8). Without it, a fresh server on a public address belongs to # a public address belongs to whoever reaches /register first. A constant, since
# whoever reaches /register first, and a new DNS name is found within minutes. # there is no admin yet to change a setting; restarting the server reopens it.
#
# A constant, not a Setting: there is no admin yet to change one. Anyone who misses
# the window restarts the server, which opens it again; once an account exists the
# window no longer matters. Measured from `create_app` (STARTED_AT), so each start
# counts afresh.
SETUP_WINDOW_S = 30 * 60 SETUP_WINDOW_S = 30 * 60
SETUP_CLOSED = ( SETUP_CLOSED = (
"this server's setup window has closed. Restart the server, then create the " "this server's setup window has closed. Restart the server, then create the "
@@ -136,9 +126,8 @@ def login_required(fn):
"""Guard: 401 unless authenticated. Accepts a web session cookie OR a device """Guard: 401 unless authenticated. Accepts a web session cookie OR a device
bearer token (native clients). Sets g.user_id for the view. bearer token (native clients). Sets g.user_id for the view.
The session path reads one column by primary key. It used to read nothing, which The session path reads the account's epoch, so a password reset or a deleted
left no way to end a session short of its expiry: not for a reset password, and account ends the session before its expiry."""
not for a deleted account (#5173)."""
@functools.wraps(fn) @functools.wraps(fn)
async def wrapper(*args, **kwargs): async def wrapper(*args, **kwargs):
@@ -182,10 +171,8 @@ def require_admin(fn):
def _throttled(retry_after: int): def _throttled(retry_after: int):
"""The 429 every throttled credential route returns. """The 429 every throttled credential route returns.
Deliberately says nothing about WHICH limit was hit or how many attempts are Says nothing about which limit was hit or what is left, which would reveal
left — that would tell someone probing whether the email they guessed exists. whether a guessed email exists. Only `Retry-After`.
`Retry-After` is standard and is the one thing a legitimate client (or person)
genuinely needs.
""" """
logger.warning("throttled credential attempt from=%s retry_after=%ss", client_address(), retry_after) logger.warning("throttled credential attempt from=%s retry_after=%ss", client_address(), retry_after)
return too_many("too many attempts — try again shortly", retry_after) return too_many("too many attempts — try again shortly", retry_after)
@@ -194,10 +181,8 @@ def _throttled(retry_after: int):
def _sign_in_block(email: str) -> int | None: def _sign_in_block(email: str) -> int | None:
"""Seconds to wait before this sign-in may be attempted, or None to proceed. """Seconds to wait before this sign-in may be attempted, or None to proceed.
Checked BEFORE the password is verified, so a throttled attempt costs no bcrypt Checked before the password is hashed, so a throttled attempt costs no bcrypt:
— which is the other half of what this protects: hashing is deliberately slow, unlimited hashing would let an anonymous caller exhaust the CPU.
and an unauthenticated caller who can trigger it without limit has a CPU
exhaustion primitive, not just a guessing one.
""" """
address = client_address() address = client_address()
waits = [ waits = [
@@ -237,9 +222,8 @@ async def register():
if not display_name: if not display_name:
display_name = email.split("@", 1)[0] display_name = email.split("@", 1)[0]
# Counted by attempt rather than by failure: a rejected registration still cost a # Counted per attempt, not per failure: with signups open, what is bounded is how
# round trip and a uniqueness check, and on an instance with signups open the # fast accounts can appear.
# thing worth bounding is how fast accounts can appear at all.
wait = register_by_address.retry_after(client_address()) wait = register_by_address.retry_after(client_address())
if wait is not None: if wait is not None:
return _throttled(wait) return _throttled(wait)
@@ -252,11 +236,10 @@ async def register():
logger.warning("registration refused (setup window closed) email=%s from=%s", email, client_address()) logger.warning("registration refused (setup window closed) email=%s from=%s", email, client_address())
await audit.record(audit.REGISTRATION_REFUSED, email=email, detail="setup window closed") await audit.record(audit.REGISTRATION_REFUSED, email=email, detail="setup window closed")
return json_error(SETUP_CLOSED, 403) return json_error(SETUP_CLOSED, 403)
# The first account bootstraps the admin and is allowed, inside the setup # The first account becomes the admin and is allowed inside the setup window
# window above, even when registration is otherwise closed. After that, an invite lets one person in # even with registration closed. After that, an invite lets one person in while
# while it is closed (#5172). One that was offered is redeemed even while # it is closed. An offered invite is redeemed even when registration is open, so
# registration is open, so the list still says who used it, and one that # the list says who used it, and an invalid one is refused, not ignored.
# doesn't hold is refused rather than ignored.
invite_id = None invite_id = None
if not is_first and invite: if not is_first and invite:
invite_id = await redeem(db, invite, email) invite_id = await redeem(db, invite, email)
@@ -281,17 +264,9 @@ async def register():
) )
db.add(user) db.add(user)
if is_first: if is_first:
# Registration CLOSES the moment the instance has an owner. # Registration closes as soon as the admin account exists, so there is no
# # window between "my account exists" and remembering to close it. More
# Not "defaults closed" — that would still need the first person to get in # people join by invite (Settings → Invites).
# somehow. Closed as a CONSEQUENCE of the admin account existing, which is
# the only formulation with no open window in it. Leaving the setting on
# meant the gap between "my account exists" and "I remembered to turn it
# off in Settings" was wide open, and on a public host that gap is the
# entire exposure — it starts the moment DNS resolves.
#
# An admin who wants a second person sends them an invite (Settings →
# Invites), which works with registration closed.
await set_settings(db, {"allow_registration": False}) await set_settings(db, {"allow_registration": False})
if invite_id is not None: if invite_id is not None:
await db.flush() await db.flush()
@@ -312,9 +287,8 @@ async def _check_credentials(db, email: str, password: str, route: str) -> User
"""The account these credentials open, or None. Counts the failure against the """The account these credentials open, or None. Counts the failure against the
throttle, logs it under `route`, and clears the account's history on success. throttle, logs it under `route`, and clears the account's history on success.
A missing account still pays for a hash. Without that, "no such account" returns A missing account still pays for a hash, so response time doesn't reveal which
in microseconds while a wrong password takes bcrypt's deliberate ~100ms, and the emails have accounts.
difference is a reliable oracle for which emails have accounts here.
""" """
user = await db.scalar(select(User).where(User.email == email)) user = await db.scalar(select(User).where(User.email == email))
if user is None or not user.password_hash: if user is None or not user.password_hash:
@@ -378,7 +352,7 @@ async def me():
@login_required @login_required
async def storage(): async def storage():
"""How much attachment storage this account uses, and its limit (None when it """How much attachment storage this account uses, and its limit (None when it
has none). For the Account page (#2939 §4).""" has none). For the Account page."""
async with session_scope() as db: async with session_scope() as db:
return jsonify( return jsonify(
{ {
@@ -394,7 +368,7 @@ FORGOT_SENT = "If an account uses that address, a reset link is on its way. It w
@bp.post("/forgot-password") @bp.post("/forgot-password")
async def forgot_password(): async def forgot_password():
"""Email a reset link to whoever owns this address (#5266). """Email a reset link to whoever owns this address.
Says the same thing, at the same speed, whether or not the address has an Says the same thing, at the same speed, whether or not the address has an
account: the link is made and sent off the request. Throttled per visitor address account: the link is made and sent off the request. Throttled per visitor address
@@ -428,7 +402,7 @@ async def forgot_password():
@bp.post("/reset-password") @bp.post("/reset-password")
async def reset_password(): async def reset_password():
"""Set a new password with a reset link an admin made (#5173), and sign the """Set a new password with a reset link, and sign the
account out everywhere else: every other session, and every linked device. account out everywhere else: every other session, and every linked device.
The browser that used the link is signed in under the new password, since holding The browser that used the link is signed in under the new password, since holding
@@ -497,8 +471,8 @@ def _stay_signed_in(user: User) -> None:
@login_required @login_required
async def change_password(): async def change_password():
"""Change the account's password, and sign it out everywhere else: every other """Change the account's password, and sign it out everywhere else: every other
web session, and every linked device (family idea #5105, practice 4). The web session, and every linked device. The browser that changed it stays signed
browser that changed it stays signed in. in.
The current password is required, so a session left open on someone else's The current password is required, so a session left open on someone else's
screen can't take the account over. A wrong one counts against the sign-in screen can't take the account over. A wrong one counts against the sign-in
@@ -536,8 +510,8 @@ async def change_password():
@login_required @login_required
async def sign_out_elsewhere(): async def sign_out_elsewhere():
"""Sign the account out of every other browser and unlink every device, keeping """Sign the account out of every other browser and unlink every device, keeping
this one signed in (family idea #5105, practice 4). For a session or a device this one signed in. For a session or device the person doesn't recognise, or a
the person doesn't recognise, or one on a machine they no longer have.""" machine they no longer have."""
async with session_scope() as db: async with session_scope() as db:
user = await db.get(User, g.user_id) user = await db.get(User, g.user_id)
if user is None: if user is None:
@@ -551,7 +525,7 @@ async def sign_out_elsewhere():
return jsonify({"ok": True, "devices_unlinked": unlinked}) return jsonify({"ok": True, "devices_unlinked": unlinked})
# --- Device (bearer) tokens for native clients — M8 sync hub --- # --- Device (bearer) tokens for native clients ---
async def _email_of(db) -> str | None: async def _email_of(db) -> str | None:
@@ -592,9 +566,8 @@ async def device_login():
if not email or not password: if not email or not password:
return json_error("email and password are required", 400) return json_error("email and password are required", 400)
# Same budget as the web sign-in, and the SAME counters — this route hands out a # The same counters as the web sign-in: this route hands out a long-lived token,
# long-lived bearer token, so leaving it unthrottled would just move the guessing # so an unthrottled one would just move the guessing here.
# here from /login.
wait = _sign_in_block(email) wait = _sign_in_block(email)
if wait is not None: if wait is not None:
return _throttled(wait) return _throttled(wait)
@@ -643,21 +616,15 @@ async def list_devices():
async def revoke_own_device(): async def revoke_own_device():
"""Revoke the device token presented on THIS request. """Revoke the device token presented on THIS request.
What makes "unlink" on a native client actually stop its access. The client How "unlink" on a native client ends its access. A client doesn't reliably know
can't use the id-keyed route below, because it doesn't reliably know its own its own device id (a token pasted from the web app arrives without one), so the
device id: a token pasted from the web app arrives without one, and `/me` row is found by the token it presents.
describes the user, not the device row. Identifying the row by the presented
token needs nothing the caller doesn't already hold, so it works for both ways
a client can be linked.
Declared above the `<device_id>` rule for reading order only — Werkzeug ranks a Werkzeug ranks this static rule ahead of `<device_id>` whatever the order.
static rule ahead of a converter regardless of registration order.
""" """
token = _bearer_token() token = _bearer_token()
if token is None: if token is None:
# A session-cookie caller holds no device token, so "revoke the one I'm # A session caller holds no device token to revoke; the web app revokes by id.
# using" is meaningless rather than merely unauthorized. The web app
# revokes by id.
return json_error("no device token was presented", 400) return json_error("no device token was presented", 400)
# Owner-scoped like every other device route. The hash already pins a single # Owner-scoped like every other device route. The hash already pins a single
# row; the guarantee shouldn't rest on one column. # row; the guarantee shouldn't rest on one column.
+1 -1
View File
@@ -1,4 +1,4 @@
"""Admin routes for Settings → Backups (#5920): where backups stand, Back up now, and """Admin routes for Settings → Backups: where backups stand, Back up now, and
Test destination. What a backup holds and when it runs is in `backup_job`.""" Test destination. What a backup holds and when it runs is in `backup_job`."""
from __future__ import annotations from __future__ import annotations
+4 -5
View File
@@ -1,4 +1,4 @@
"""What goes into a backup, and putting it back (#5918, #5921). """What goes into a backup, and putting it back.
Every table the models define is exported, so a table added later is backed up Every table the models define is exported, so a table added later is backed up
without anyone remembering to add it here. The few that are left out are named in without anyone remembering to add it here. The few that are left out are named in
@@ -74,9 +74,8 @@ def _decoder(column):
async def export_contents(db) -> Contents: async def export_contents(db) -> Contents:
"""Every exported table's rows in primary-key order, and the attachment files. """Every exported table's rows in primary-key order, and the attachment files.
Primary-key order is load-bearing: Postgres returns rows in heap order, and an Primary-key order keeps an unchanged table byte-identical between runs: heap order
UPDATE moves a row to the end of the heap, so without ORDER BY an unchanged table shifts whenever a row is updated, and then nothing deduplicates."""
would come out shuffled after any edit and nothing would deduplicate."""
tables: dict[str, list[dict]] = {} tables: dict[str, list[dict]] = {}
for table in exported_tables(): for table in exported_tables():
query = select(table).order_by(*table.primary_key.columns) query = select(table).order_by(*table.primary_key.columns)
@@ -143,7 +142,7 @@ async def restore_tables(db, members: Iterable[Member]) -> dict[str, int]:
for member in members: for member in members:
if member.kind == "table": if member.kind == "table":
if member.name not in by_name: if member.name not in by_name:
# A table this version no longer has. Nothing to put it in. # A table this version doesn't have. Nothing to put it in.
logger.warning("restore: skipping table %s, which this version doesn't have", member.name) logger.warning("restore: skipping table %s, which this version doesn't have", member.name)
member.fileobj.read() member.fileobj.read()
continue continue
+3 -4
View File
@@ -1,10 +1,9 @@
"""Where the daily backup goes, and getting a backup file there (#5919). """Where the daily backup goes, and getting a backup file there.
Two kinds of destination, both set in Settings → Backups: Two kinds of destination, both set in Settings → Backups:
- **folder**: a directory inside the container, usually a mounted share. A backup - **folder**: a directory inside the container, usually a mounted share, from which
service that watches the share (Fabled Archives' folder mode, say) or a person a backup service or a person takes the files.
takes the files from there.
- **s3**: any S3-compatible bucket, addressed by path. Inkwell only puts and lists, - **s3**: any S3-compatible bucket, addressed by path. Inkwell only puts and lists,
so a key that may only add objects is enough. so a key that may only add objects is enough.
+1 -4
View File
@@ -1,4 +1,4 @@
"""The full backup as one file: an uncompressed tar per run (#5918). """The full backup as one file: an uncompressed tar per run.
A scheduled backup lands somewhere that keeps a history of them: a folder a backup A scheduled backup lands somewhere that keeps a history of them: a folder a backup
service watches, or an S3 bucket. From there it is usually copied off-site by a service watches, or an S3 bucket. From there it is usually copied off-site by a
@@ -25,9 +25,6 @@ that whatever did not change between two runs is the same bytes both times.
can thin the history on its own schedule. can thin the history on its own schedule.
- **Written atomically** through a dot-named `.part` file, which a folder watcher - **Written atomically** through a dot-named `.part` file, which a folder watcher
ignores until the rename. ignores until the rename.
The shape is Scribe's (snippet #5875), with attachment files added: in a notes app
the files are as much the user's data as the rows.
""" """
from __future__ import annotations from __future__ import annotations
+3 -3
View File
@@ -1,12 +1,12 @@
"""The daily backup: when it runs, running it, and what happened (#5920). """The daily backup: when it runs, running it, and what happened.
- **When.** A tick every few minutes asks whether a backup is due: switched on, and - **When.** A tick every few minutes asks whether a backup is due: switched on, and
no good backup since the most recent scheduled hour. A server that was down at no good backup since the most recent scheduled hour. A server that was down at
that hour backs up as soon as it is back. Switching backups on runs the first one that hour backs up as soon as it is back. Switching backups on runs the first one
within a tick, which is also the quickest proof the destination works. within a tick, which is also the quickest proof the destination works.
- **Failure is "not yet".** A failed run is tried again after an hour, then two, - **Failure is "not yet".** A failed run is tried again after an hour, then two,
then four, then waits for the next day's hour, rather than at once or never then four, then waits for the next day's hour, rather than at once or never. The
(rule 193). The destination may simply be restarting. destination may simply be restarting.
- **One at a time.** An in-process lock, plus a Postgres advisory lock in case a - **One at a time.** An in-process lock, plus a Postgres advisory lock in case a
second server process is ever started against the same database. Back up now second server process is ever started against the same database. Back up now
while one is running is answered 409. while one is running is answered 409.
+8 -10
View File
@@ -1,14 +1,12 @@
"""A small S3 client for sending backups: AWS Signature Version 4, path-style """A small S3 client for sending backups: AWS Signature Version 4, path-style
addressing (endpoint/bucket/key), and the two calls a backup needs. addressing (endpoint/bucket/key), and the two calls a backup needs.
Ported from FabledSteward's and FabledScribe's backup clients (Scribe snippet #5679), Uses the standard library's `http.client` on a worker thread, since two calls don't
which are tested against the same AWS examples as tests/test_backup_s3.py. Those use justify an HTTP client dependency. The signing is checked against AWS's published
httpx. This one uses the standard library's `http.client`, on a worker thread, examples (tests/test_backup_s3.py).
because Inkwell has no HTTP client dependency and two calls don't justify adding one.
Inkwell puts and lists; it never deletes. The bucket keeps the history and thins it It puts and lists, never deletes, so an add-only key is enough; a stolen one can't
by its own policy, so a key that may only add objects is enough. It is also the safer wipe the history the bucket keeps.
key to give a service: a stolen one cannot wipe the history it writes into.
""" """
from __future__ import annotations from __future__ import annotations
@@ -128,15 +126,15 @@ def _says(status: int, reason: str, body: bytes) -> str:
} }
said = hints.get(code) or message or reason said = hints.get(code) or message or reason
if status == 404 and not code: if status == 404 and not code:
# Seen live: an address without its port lands on port 80 and gets a plain # An address without its port lands on port 80 and gets a plain 404 from
# 404 from whatever answers there (Scribe #5871). # whatever answers there.
said = "nothing answered as S3 at that address; check the port" said = "nothing answered as S3 at that address; check the port"
return f"HTTP {status}{f' {code}' if code else ''}: {said}." return f"HTTP {status}{f' {code}' if code else ''}: {said}."
class _Throttled: class _Throttled:
"""A file wrapper that reads no faster than `limit` bytes a second, so an upload """A file wrapper that reads no faster than `limit` bytes a second, so an upload
leaves room on the connection for everything else (rule 193).""" leaves room on the connection for everything else."""
def __init__(self, f: IO[bytes], limit: int): def __init__(self, f: IO[bytes], limit: int):
self._f = f self._f = f
+63 -162
View File
@@ -1,73 +1,33 @@
"""The server hands out the clients it is in step with. """The server hands out the clients it is in step with.
## Why the server, and not a release page A self-hoster gets the apps from the server that already holds their notes, with no
account anywhere else. Client and server negotiate a sync protocol before linking,
The Fabled-Git instance is private (issue 2091), so `install.sh` already cannot so a server never hands out a client it can't talk to.
fetch for anyone but the operator — and a self-hoster should not need an account
on someone else's forge to get the app for their own notes. The server they
already trust with the notes is the obvious place to get the client from.
It also keeps the two in step by construction. Client and server already
negotiate a sync protocol version before linking, so a server that also serves
the client cannot hand out a phone it is unable to talk to.
That argument was never Android-specific, which is why this module now serves a
TABLE of platforms rather than the one it was written for.
## Where the files come from ## Where the files come from
Two places, checked in that order, per platform: Per platform, the first of:
1. `DATA_DIR/client/` — the mounted volume that already holds attachments. An 1. `DATA_DIR/client/`, on the mounted volume: a build dropped there wins.
operator who wants a SPECIFIC build drops it there and it wins. 2. The copy baked into the image, inside the package rather than under DATA_DIR,
2. the copy baked into the image at build time — CI fetches the newest published which a volume mount would hide. CI bakes the newest build of each client into
build of each client into every image, so `:dev` and `:latest` both carry a every image.
full set and `docker compose pull` delivers new ones with nothing copied by
hand.
The precedence is the point: the image is the default, and a person who wants to Precedence is per platform: a drop-in `.deb` doesn't shadow the baked APK.
override it should not have to fight it. The baked copy sits inside the package
rather than under DATA_DIR because DATA_DIR is a volume mount, and anything the
image wrote there would be hidden the moment one is attached.
**Precedence is decided per platform, not for the set.** A drop-in `.deb` does
not shadow the baked APK. The alternative — first directory holding anything
wins — would mean replacing one client silently retracts the other four.
## What a platform needs on disk ## What a platform needs on disk
Two files, and both must be present: - the artifact, at a fixed name (`inkwell.deb`). A version in the filename would
mean globbing, and a glob can pick the older of two builds.
- the artifact, at a FIXED name (`inkwell.deb`, not - its sidecar, `{version_name, version_code, size, sha256}`, written by CI, because
`Inkwell_2026.08.30.0307_amd64.deb`) a version can't be read reliably from an APK, a `.deb` or an AppImage.
- its sidecar, `{version_name, version_code, size, sha256}` - for the AppImage, `inkwell.AppImage.sig`: the signature the desktop updater
verifies. Without it the AppImage counts as absent.
**Fixed names, and the version only in the sidecar.** A version-stamped filename
would force this module to glob, and a glob over a directory an operator can drop
files into is how you serve the older of two builds — `write-manifest.sh` carries
a comment about exactly that, from the time it advertised a new version while
pointing at an old binary.
The sidecar exists because a version is not reliably readable from the artifact:
an APK keeps it in a binary AXML manifest Python cannot parse without the Android
build tools, and a `.deb` or an AppImage would each need a different unpacker. CI
writes the sidecar where the real value is already known.
The AppImage needs a THIRD file, `inkwell.AppImage.sig`. That is the minisign
signature the desktop updater verifies before replacing the running binary, and a
bundle that cannot be verified cannot be offered as an update — so a missing
signature makes the AppImage absent rather than merely unsigned.
## Absence is normal ## Absence is normal
A server with no client for a platform advertises none, and the web UI hides that A platform with no build, or whose sidecar size doesn't match the file, is simply
download rather than offering a button that 404s. Same for a mismatched pair: if not advertised, and the web UI hides that download.
the sidecar's recorded size does not match the file on disk, the two did not
arrive together, and the server says it has nothing rather than serving one build
while describing another.
This is the ONLY state available before a platform's first build has ever
published, so it is an ordinary answer and never an error.
""" """
from __future__ import annotations from __future__ import annotations
@@ -90,29 +50,17 @@ class Platform:
"""One installable client, and where its files sit.""" """One installable client, and where its files sit."""
id: str id: str
# What a person calls it. Named for the DISTRO rather than the package format # What a person calls it: the system ("Debian / Ubuntu"), not the package format.
# ("Debian / Ubuntu", not ".deb") — someone knows which system they run and
# does not necessarily know which packaging it uses.
label: str label: str
artifact: str artifact: str
sidecar: str sidecar: str
mimetype: str mimetype: str
# An updater-verifiable bundle: `<artifact>.sig` must be present too, and its # An updater-verifiable bundle: `<artifact>.sig` must be present, and is published
# contents are published with the metadata. Only the AppImage, because it is # with the metadata. Only the AppImage can replace itself in place.
# the only bundle that can replace itself in place — a package-manager install
# cannot, by design (see the desktop's update.rs).
signed: bool = False signed: bool = False
# Whether this platform's ordering key is an INTEGER. # Whether this platform's ordering key (`version_code`) is an integer. Android's
# # is, and must stay a JSON number: core/src/sync/client.rs reads it as `i64`. The
# `version_code` is "whatever this platform's comparator reads", and that is not # desktop's is Tauri's semver key, `1.0.<minutes>`.
# one type. Android's is an int because Android's own install gate compares one,
# and it must stay a JSON number — `ClientRelease` in core/src/sync/client.rs
# declares it `i64` and a string would fail to deserialize on every phone in the
# field. The desktop's is Tauri's semver key, `1.0.<minutes>`, which is the value
# its updater compares and is not an integer at all.
#
# Coercing everything to int was inherited from the days when Android was the
# only platform, and would have rejected every desktop sidecar written.
code_is_int: bool = True code_is_int: bool = True
@property @property
@@ -124,17 +72,14 @@ class Platform:
return f"/api/client/{self.id}/download" return f"/api/client/{self.id}/download"
# ONE definition of what this server can hand out. Every route, the /api/config # What this server can hand out. Every route, /api/config and the tests read this
# advertisement and the tests all read this table; adding a platform is adding a # table; adding a platform is adding a row.
# row.
PLATFORMS: tuple[Platform, ...] = ( PLATFORMS: tuple[Platform, ...] = (
Platform( Platform(
id="android", id="android",
label="Android", label="Android",
# UNCHANGED, and it must stay unchanged: the Android lane publishes these # Fixed: installed phones poll `/api/client/android`, and the Android lane
# exact names, CI bakes them in under them, and clients in the field poll # publishes these exact names.
# `/api/client/android`. Renaming them to match the pattern below would
# buy tidiness and strand every installed phone.
artifact="inkwell.apk", artifact="inkwell.apk",
sidecar="inkwell-android.json", sidecar="inkwell-android.json",
mimetype="application/vnd.android.package-archive", mimetype="application/vnd.android.package-archive",
@@ -160,9 +105,7 @@ PLATFORMS: tuple[Platform, ...] = (
label="Other Linux (AppImage)", label="Other Linux (AppImage)",
artifact="inkwell.AppImage", artifact="inkwell.AppImage",
sidecar="inkwell-linux-appimage.json", sidecar="inkwell-linux-appimage.json",
# No registered type for an AppImage, and guessing one buys nothing: it is # AppImage has no registered type; it is served as an attachment either way.
# served as an attachment either way, and octet-stream is the answer that
# cannot be wrong.
mimetype="application/octet-stream", mimetype="application/octet-stream",
signed=True, signed=True,
code_is_int=False, code_is_int=False,
@@ -189,10 +132,8 @@ bp = Blueprint("client_dist", __name__)
def _read(root: Path, platform: Platform) -> dict | None: def _read(root: Path, platform: Platform) -> dict | None:
"""One platform's build in one directory, or None. """One platform's build in one directory, or None.
Never raises. A missing directory, an unreadable sidecar, malformed JSON, a Never raises. A missing directory, an unreadable or malformed sidecar, a sidecar
sidecar that describes a different file and — for a signed bundle — a missing describing a different file, or a missing signature all mean "no client here".
signature are all the same answer to the only question being asked, "is there
a client here I can honestly offer?", and that answer is no.
""" """
try: try:
size = (root / platform.artifact).stat().st_size size = (root / platform.artifact).stat().st_size
@@ -211,9 +152,8 @@ def _read(root: Path, platform: Platform) -> dict | None:
# not a version, and it would render as a blank on the download card. # not a version, and it would render as a blank on the download card.
return None return None
# The pair has to describe one build. A sidecar left behind by a previous # The pair must describe one build, or a stale sidecar would advertise a version
# release would otherwise advertise a version this server cannot serve, and the # the file isn't.
# client would download something other than what it was promised.
if recorded != size: if recorded != size:
return None return None
@@ -225,22 +165,17 @@ def _read(root: Path, platform: Platform) -> dict | None:
# sorts like one, which is not how "is this newer" works. # sorts like one, which is not how "is this newer" works.
"version_code": code, "version_code": code,
"size": size, "size": size,
# Computed by CI over the same bytes it uploaded, so a client can tell a # Lets a client spot a truncated download before installing it. Not a trust
# truncated download from a complete one BEFORE handing it to an installer. # anchor; the signature is.
# Not a trust anchor — the signature is that.
"sha256": digest, "sha256": digest,
# A PATH, never an absolute URL: the client joins it to the base it is # A path, never an absolute URL: the client joins it to the server it is
# already linked to, so a compromised or misconfigured server cannot # linked to, so a misconfigured server can't redirect the download.
# redirect the download somewhere else. `core/src/sync/client.rs` relies on
# this and says so.
"url": platform.download_path, "url": platform.download_path,
} }
if platform.signed: if platform.signed:
# The signature travels WITH the metadata rather than behind its own route. # The signature travels with the metadata, so one response can't pair a
# It is ~100 bytes, it is public wherever these bundles are published, and # version with another build's signature.
# the updater needs the version and the signature in the same breath — one
# request that cannot return a signature belonging to a different build.
try: try:
sig = (root / platform.signature).read_text(encoding="utf-8").strip() sig = (root / platform.signature).read_text(encoding="utf-8").strip()
except OSError: except OSError:
@@ -255,14 +190,11 @@ def _read(root: Path, platform: Platform) -> dict | None:
def _resolve(platform: Platform) -> tuple[Path, dict] | None: def _resolve(platform: Platform) -> tuple[Path, dict] | None:
"""Which directory this server serves a platform from, and what is in it. """Which directory this server serves a platform from, and what is in it.
The operator's drop-in beats the baked copy — someone who deliberately put a A drop-in on the volume beats the baked copy, older or not. A broken or
build on the volume wants that build, older or not. A directory holding a half-copied pair there doesn't shadow the image; the search moves on.
broken or half-copied pair does NOT shadow the image: it simply is not a
client, so the search moves on.
""" """
# Both wrapped in Path(), not just the first. The asymmetry was arbitrary and it # Both must be Paths: `_read` swallows the TypeError of str / str and would report
# fails quietly: `_read` catches the TypeError a str `/` str raises and reports # a good directory as empty.
# "no client here", so a directory that is perfectly fine reads as empty.
for root in (Path(Config.client_root()), Path(BAKED_ROOT)): for root in (Path(Config.client_root()), Path(BAKED_ROOT)):
release = _read(root, platform) release = _read(root, platform)
if release is not None: if release is not None:
@@ -282,9 +214,7 @@ def release(platform_id: str) -> dict | None:
def releases() -> dict[str, dict]: def releases() -> dict[str, dict]:
"""Every client this server can hand out, keyed by platform id. """Every client this server can hand out, keyed by platform id.
Platforms it holds nothing for are ABSENT rather than present-and-null, so a A platform with no build is absent, not null.
caller can test for the key instead of distinguishing "no build" from "a
server that never had this platform".
""" """
found = {p.id: _resolve(p) for p in PLATFORMS} found = {p.id: _resolve(p) for p in PLATFORMS}
return {pid: r[1] for pid, r in found.items() if r is not None} return {pid: r[1] for pid, r in found.items() if r is not None}
@@ -295,9 +225,7 @@ def advertisement() -> dict:
whole table, which the web UI renders the downloads section from. Phones ask whole table, which the web UI renders the downloads section from. Phones ask
`/api/client/android` for their own update instead. `/api/client/android` for their own update instead.
ABSENT rather than empty when the server holds none, so a client testing for the Absent, not empty, when the server holds none.
key gets one unambiguous answer instead of having to distinguish "no client" from
"an old server that never had this field".
""" """
found = releases() found = releases()
return {"clients": found} if found else {} return {"clients": found} if found else {}
@@ -305,11 +233,7 @@ def advertisement() -> dict:
@bp.get("/api/client") @bp.get("/api/client")
async def client_index(): async def client_index():
"""Everything this server holds, in one request. """Everything this server holds, in one request, for the downloads UI."""
The downloads UI needs all five to decide what to lead with, and five requests
to answer one question is five chances to render half a page.
"""
return jsonify({"clients": releases()}) return jsonify({"clients": releases()})
@@ -317,12 +241,8 @@ async def client_index():
async def client_metadata(platform_id: str): async def client_metadata(platform_id: str):
"""Version and digest without the payload. What an updater polls. """Version and digest without the payload. What an updater polls.
Public, because a client has to be able to ask "is there something newer?" Public, so a client can ask "is there something newer?" before it has a token.
cheaply — before it has a token, in the case of a first pairing. An unknown platform and one with no build both 404.
An unknown platform and a platform with no build both 404. They are the same
answer to the caller ("not here"), and distinguishing them would only tell an
unauthenticated stranger which platforms this build of the server knows about.
""" """
found = release(platform_id) found = release(platform_id)
if found is None: if found is None:
@@ -334,25 +254,17 @@ async def client_metadata(platform_id: str):
async def client_update_manifest(platform_id: str): async def client_update_manifest(platform_id: str):
"""The same build, in the shape the desktop's in-app updater reads. """The same build, in the shape the desktop's in-app updater reads.
Tauri's updater takes a manifest, not this module's metadata: `version` is the A translation of `release()` into Tauri's single-build manifest: `version` is
ordering key it compares (the desktop's `1.0.<minutes>`, i.e. `version_code`), the desktop's `version_code`, with `url` and `signature` at the top level.
and `url` + `signature` at the top level are its single-build form. So this is a
translation of `release()`, never a second description of the build.
Only for a SIGNED platform. The updater verifies the signature against the key Signed platforms only. The updater checks the signature against the key built
baked into the app before it replaces anything, so a server mirrors builds here into the app, so a server can mirror builds but not sign them. No AppImage: 404,
and cannot sign them: the trust anchor does not move with the feed. A platform which the desktop shows as "no update to offer", not "up to date".
with no signature has no manifest, and a server holding no AppImage 404s, which
the desktop reports as "this server has no update to offer", never as "up to
date".
`url` must be absolute for the updater, which is the one place this module `url` is absolute, built from the address this request arrived on rather than
hands out more than a path. It is built from the address THIS request arrived Settings, because the updater sends the same Authorization header there.
on, not from Settings, because the updater sends the same Authorization header
to it that it sent here: the token goes back to the host that was asked, and
nowhere else.
Public, like the metadata above. The bytes `url` names are not. Public; the bytes `url` names are not.
""" """
platform = BY_ID.get(platform_id) platform = BY_ID.get(platform_id)
if platform is None or not platform.signed: if platform is None or not platform.signed:
@@ -376,13 +288,9 @@ async def client_update_manifest(platform_id: str):
async def client_download(platform_id: str): async def client_download(platform_id: str):
"""The bytes themselves. """The bytes themselves.
Authenticated — by session cookie from a browser, or by device bearer token Needs a session or a device token. Throttled per account (Settings → Security),
from a client updating itself; `login_required` accepts either. The metadata because each answer is tens of megabytes and a looping client could fill the
above is public; the bytes are not for anyone who can reach the port. uplink.
Throttled per account (Settings → Security), because each answer is tens of
megabytes: one account or a leaked device token looping here could fill the
server's uplink (family idea #5103, practice 6).
""" """
platform = BY_ID.get(platform_id) platform = BY_ID.get(platform_id)
if platform is None: if platform is None:
@@ -406,23 +314,16 @@ async def client_download(platform_id: str):
async def send_artifact(root: Path, platform: Platform, found: dict) -> Response: async def send_artifact(root: Path, platform: Platform, found: dict) -> Response:
"""The artifact, with its sha256 as the ETag and Range honoured against it. """The artifact, with its sha256 as the ETag and Range honoured against it.
Quart's own ETag is the file's mtime, size and path. That changes when a Quart's ETag is mtime, size and path, which changes when a container restarts on
container restarts on the same bytes, so a phone resuming a download across a the same bytes, so a resumed download would be told its partial copy is stale.
redeploy would be told its partial copy is stale. It also says nothing about Quart's ETag and conditional handling are off, and the conditional pass runs here
content. The sidecar's sha256 is the identity the client already checks, so after the sha256 ETag is set.
`If-Range` is answered against the bytes themselves (family idea #5103,
practice 6).
So Quart's ETag and conditional handling are both turned off, and the
conditional pass runs here after the right ETag is set. Run before it, Range
and If-None-Match would be judged against the wrong one.
""" """
response = await send_from_directory( response = await send_from_directory(
root, platform.artifact, mimetype=platform.mimetype, add_etags=False, conditional=False root, platform.artifact, mimetype=platform.mimetype, add_etags=False, conditional=False
) )
response.set_etag(found["sha256"]) response.set_etag(found["sha256"])
# Only behind a login, so no shared cache may keep a copy (family idea #5105, # Behind a login, so no shared cache may keep a copy. Quart marks files `public`.
# practice 11). Quart marks every file it sends `public`.
response.cache_control.public = False response.cache_control.public = False
response.cache_control.private = True response.cache_control.private = True
# Without this some browsers try to render it, and Android's download handler # Without this some browsers try to render it, and Android's download handler
+4 -13
View File
@@ -1,15 +1,8 @@
from __future__ import annotations from __future__ import annotations
# The colour palette, and the one place that clamps to it. # The label colour palette, and the one place that clamps to it. Keys, not tints: the
# # colours live in each client (frontend/src/notes/colors.ts, NoteTint.kt), so they can
# It lived on `models/note.py` until M315, when a note stopped having a colour. A # be retuned without a migration.
# palette defined on the model that lost one would be a standing invitation to put the
# column back; here it reads as what it now is — a LABEL's vocabulary, shared with the
# import path that still names a colour.
#
# Keys, not tints. The actual colours live in each client (frontend/src/notes/colors.ts
# and NoteTint.kt), so they can be retuned without a schema migration — which M315 spent
# two steps doing.
NOTE_COLORS = { NOTE_COLORS = {
"default", "default",
"red", "red",
@@ -29,8 +22,6 @@ __all__ = ["NOTE_COLORS", "normalize_color"]
def normalize_color(color: object) -> str: def normalize_color(color: object) -> str:
"""Return `color` if it's a known palette key, else the default. """Return `color` if it's a known palette key, else the default.
`default` is no longer something anybody can CHOOSE — nothing offers a colour Nothing offers `default` as a choice; it is where unrecognised input lands.
picker since M315 — but it is still where unrecognised input has to land, so this
fallback is unreachable by choice rather than dead.
""" """
return color if color in NOTE_COLORS else "default" return color if color in NOTE_COLORS else "default"
+3 -7
View File
@@ -3,9 +3,7 @@ from __future__ import annotations
import asyncio import asyncio
from datetime import datetime, timedelta from datetime import datetime, timedelta
# Small, dependency-free value coercions shared across the blueprints. Kept in one # Small value coercions shared across the blueprints.
# place so the "parse an ISO date" / "is this flag truthy" logic has a single
# definition instead of a near-identical copy per module.
def iso(dt: datetime | None) -> str | None: def iso(dt: datetime | None) -> str | None:
@@ -65,10 +63,8 @@ def detach(running: set[asyncio.Task], coro) -> asyncio.Task:
def expired_before(now: datetime, retention_days: int) -> datetime | None: def expired_before(now: datetime, retention_days: int) -> datetime | None:
"""The cutoff: anything older than this has expired. `None` = retention is off. """The cutoff: anything older than this has expired. `None` = retention is off.
Kept separate from the query so the window arithmetic — including the two ways 0 and negative both mean "never". Separate from the query so it is testable
to say "never" (0 and negative, the latter reachable by typing a stray minus in without a database. Trash and the audit log both expire through it.
Settings) — is testable without a database. Trash and the audit log both expire
through it.
""" """
if retention_days <= 0: if retention_days <= 0:
return None return None
+10 -19
View File
@@ -7,19 +7,14 @@ from pathlib import Path
class Config: class Config:
"""Bootstrap configuration. """Bootstrap configuration.
For a basic install, ``INKWELL_DATABASE_URL`` is the ONLY required env var — ``INKWELL_DATABASE_URL`` is the only required env var; everything else is in
every other tunable lives in the DB-backed Settings UI (rule 25). The only other the Settings UI, stored in the database. Optional:
env var is an optional "break-glass" item:
- ``INKWELL_SECRET_KEY`` — optional override for the cookie-signing secret. - ``INKWELL_SECRET_KEY`` overrides the cookie-signing secret. Unset, a key is
If unset, a key is generated and persisted in the DB (see generated and stored in the database, so sessions survive restarts.
``inkwell.settings.load_or_create_secret_key``), so sessions survive
restarts with no volume required.
Uploaded media lives under ``DATA_DIR`` — a fixed, authoritative path Uploads live under ``DATA_DIR`` (``/var/inkwell``), which is fixed. Mount a
(``/var/inkwell``), intentionally NOT configurable (a mutable data path only volume there to keep them across container recreation.
invites breakage). Mount a volume there if you want uploads to persist across
container recreation; a text-notes-only install never writes to it.
""" """
DATA_DIR = "/var/inkwell" DATA_DIR = "/var/inkwell"
@@ -39,15 +34,11 @@ class Config:
@classmethod @classmethod
def client_root(cls) -> Path: def client_root(cls) -> Path:
"""Where an operator DROPS IN clients for this server to hand out. """Where client builds can be dropped in to override the image's own.
One directory for every platform; `client_dist.py` picks files out of it by One directory for every platform; `client_dist.py` picks files by name. On
name. Under DATA_DIR because it is a mounted volume: a build placed here the data volume, so a build placed here survives container recreation.
survives container recreation, and it beats the copy baked into the image, Usually empty.
which is the whole point of the directory existing.
Empty is the ordinary case — the image ships its own set and most operators
never touch this.
""" """
return Path(cls.DATA_DIR) / "client" return Path(cls.DATA_DIR) / "client"
+2 -4
View File
@@ -24,10 +24,8 @@ async def _probe(url: str) -> None:
async def wait_for_db(attempts: int = ATTEMPTS, delay: float = DELAY_SECONDS) -> bool: async def wait_for_db(attempts: int = ATTEMPTS, delay: float = DELAY_SECONDS) -> bool:
"""Poll the database until it accepts a connection, up to `attempts` tries. """Poll the database until it accepts a connection, up to `attempts` tries.
Startup runs `alembic upgrade head` as the container's first DB touch, so a Runs before `alembic upgrade head`, so a database that is slow to come up delays
database that is a moment slow to accept connections would otherwise crash the the start instead of failing it.
container. This gives it a short window to come up, verifying once per `delay`
seconds, instead of failing on the first missed connection.
""" """
url = Config.DATABASE_URL url = Config.DATABASE_URL
for attempt in range(1, attempts + 1): for attempt in range(1, attempts + 1):
+1 -1
View File
@@ -1,4 +1,4 @@
"""Admin routes over groups: named sets of people a note can be shared with (#5177). """Admin routes over groups: named sets of people a note can be shared with.
A group is the instance's, not anyone's: the admin makes it and says who is in it, A group is the instance's, not anyone's: the admin makes it and says who is in it,
and every member of the instance can share a note with it. Sharing with a group and every member of the instance can share a note with it. Sharing with a group
+14 -29
View File
@@ -2,28 +2,16 @@
curl -fsSL https://notes.example.com/install.sh | sh curl -fsSL https://notes.example.com/install.sh | sh
## Why the server templates it `curl | sh` gives a script no way to learn where it came from, so the server writes
its own address into the copy it hands out.
`curl | sh` gives a script no way to learn where it came from: no argv, no referrer, Arbitrary text in a piped script is the whole attack, so exactly one variable is
nothing. So the server writes its own address into the copy it hands out, and the substituted, `INKWELL_SERVER_DEFAULT`, with a value that passed [SAFE_ADDRESS]
script installs from there without being told. Anything else means a self-hoster (nothing a shell reads as more than characters). The script checks the shape again.
editing a URL by hand, which is the friction `client_dist.py` exists to remove. An address that can't pass is refused.
## Which way it fails The script is `desktop/packaging/install.sh`; the image carries a copy beside this
module, and a source checkout reads the original.
A server that can put arbitrary text into a script a person pipes to `sh` is the
whole attack. So the substitution is ONE variable, `INKWELL_SERVER_DEFAULT`, given a value
that has already passed [SAFE_ADDRESS] — no quote, no space, no `$`, nothing a shell
reads as anything but characters — and the script checks the shape again before it
uses it. If the address cannot pass, the route refuses rather than serving a script
that would install from somewhere else.
## Where the script comes from
ONE script. The forge serves `desktop/packaging/install.sh` raw, and that is how the
operator installs. The image carries a copy beside this module (the Dockerfile copies
it in), and a source checkout reads the original. Same bytes either way; this module
changes exactly one line of them.
""" """
from __future__ import annotations from __future__ import annotations
@@ -47,11 +35,10 @@ SOURCE_SCRIPT = Path(__file__).resolve().parents[2] / "desktop" / "packaging" /
# `render` refuses a script that does not, rather than serving one with no address in. # `render` refuses a script that does not, rather than serving one with no address in.
PLACEHOLDER = 'INKWELL_SERVER_DEFAULT=""' PLACEHOLDER = 'INKWELL_SERVER_DEFAULT=""'
# What a server address may look like before it goes into a shell script: a scheme, # A server address safe to put in a shell script: scheme, host name or IPv4, optional
# a host name or IPv4 address, an optional port and an optional path. Deliberately # port and path. Narrower than a URL on purpose; every character is inert inside
# narrower than a URL. Every character it admits is inert inside single quotes, and # single quotes. An address it refuses (IPv6, `%` in a path) is fixed by setting
# an address it turns away (an IPv6 literal, a path with `%` in it) is answered by # Settings → Public address.
# setting Settings → Public address, not by widening this.
SAFE_ADDRESS = re.compile(r"https?://[A-Za-z0-9.-]+(:[0-9]{1,5})?(/[A-Za-z0-9._~/-]*)?") SAFE_ADDRESS = re.compile(r"https?://[A-Za-z0-9.-]+(:[0-9]{1,5})?(/[A-Za-z0-9._~/-]*)?")
@@ -68,10 +55,8 @@ def script_text() -> str | None:
def server_address(public_url: str, scheme: str, host: str) -> str | None: def server_address(public_url: str, scheme: str, host: str) -> str | None:
"""Where the installer should fetch from, or None if no safe answer exists. """Where the installer should fetch from, or None if no safe answer exists.
Settings → Public address when it is set: it is the address the operator says Settings → Public address when set, since behind a proxy the request's Host may
people use, and behind a reverse proxy the request's own Host may be an internal be internal. Otherwise the address this request arrived on.
one. Otherwise the address this request arrived on, which is the one the person
running `curl` just reached.
""" """
address = (public_url or f"{scheme}://{host}").rstrip("/") address = (public_url or f"{scheme}://{host}").rstrip("/")
return address if SAFE_ADDRESS.fullmatch(address) else None return address if SAFE_ADDRESS.fullmatch(address) else None
+3 -7
View File
@@ -1,16 +1,12 @@
"""Invites: an admin lets one person register while registration is closed (#5172). """Invites: an admin lets one person register while registration is closed.
Before these, adding a second person meant turning registration back on for the
whole internet while they signed up, with no record of who was let in (#2939 §1).
An invite is a link carrying a random token. It works once, until it expires, and An invite is a link carrying a random token. It works once, until it expires, and
optionally only for one email address. Only the token's hash is kept, so the link is optionally only for one email address. Only the token's hash is kept, so the link is
shown once, when it is made; an admin who loses it revokes it and makes another. shown once, when it is made; an admin who loses it revokes it and makes another.
This module is the invite itself: its status, its wire shape, and redemption, which This module is the invite itself: its status, wire shape, and redemption, which
`auth.register` calls inside the transaction that creates the account. The admin `auth.register` calls inside the transaction that creates the account. The admin
routes are in `invites_api.py`, the same split as `settings` and `settings_api`, and routes are in `invites_api.py`, because they need `auth` and `auth` needs this.
for the same reason: the routes need `auth`, and `auth` needs this.
""" """
from __future__ import annotations from __future__ import annotations
+1 -1
View File
@@ -1,4 +1,4 @@
"""Admin routes for invites: make one, list them, revoke one (#5172). """Admin routes for invites: make one, list them, revoke one.
What an invite is, and how it is redeemed, is in `invites.py`. What an invite is, and how it is redeemed, is in `invites.py`.
""" """
+4 -6
View File
@@ -1,10 +1,8 @@
"""Shared note↔label membership reconciliation. The "set a note's MANUAL (picker) """Note↔label membership. Sets a note's manual (picker) labels for both the labels
labels, leave the tag-sourced ones alone" logic was duplicated line-for-line between API and sync push, leaving via_tag=True rows, which track the body's #tags, to
the labels-picker API (notes.set_note_labels) and sync push (sync._apply_note_manual_labels). _lift_and_reconcile_tags.
Single home so both stay in lockstep. via_tag=True rows track the body #tags and are
governed by _lift_and_reconcile_tags — this function never touches them.
Also the one definition of a LIVE tag and of deleting one (#5382).""" Also the one definition of a live tag and of deleting one."""
from __future__ import annotations from __future__ import annotations
from datetime import datetime, timezone from datetime import datetime, timezone
+15 -27
View File
@@ -33,13 +33,11 @@ async def _label_note_count(db, label_id) -> int:
async def _merge_into(db, source: Label, target: Label) -> None: async def _merge_into(db, source: Label, target: Label) -> None:
"""Move every note tagged with `source` onto `target`, then tombstone `source`. """Move every note tagged with `source` onto `target`, then tombstone `source`.
Shared by the explicit `/merge` route and by a rename that lands on a name Used by `/merge` and by a rename onto a name another tag holds, which are the
another tag already holds — those are the same operation, and having one body same operation.
is what stops them drifting into two answers for one question.
Note-body `#tags` are NOT rewritten, so a note whose body still literally Body `#tags` are not rewritten, so a note still containing the source `#tag`
contains the source `#tag` will re-mint that tag on its next edit. Retiring a re-creates it on its next edit; retiring a tag means editing it out of the text.
tag means editing it out of the text; a known, documented nuance.
""" """
# Notes already carrying the target: a note can't hold the same label twice # Notes already carrying the target: a note can't hold the same label twice
# (composite PK), so the source attachment there is just dropped as a dup. # (composite PK), so the source attachment there is just dropped as a dup.
@@ -81,9 +79,8 @@ async def list_labels():
) )
).all() ).all()
) if labels else {} ) if labels else {}
# When each was last used: the newest edit among the notes carrying it, trash # When each was last used: the newest edit among its notes, trash aside. The
# aside. The rail shows the few most recent (#5931), so a tag in daily use # rail shows the most recent few, so a tag in daily use stays in reach.
# stays in reach however many others there are.
last_used = dict( last_used = dict(
( (
await db.execute( await db.execute(
@@ -114,10 +111,9 @@ async def create_label():
if not name: if not name:
return json_error("tag name is required", 400) return json_error("tag name is required", 400)
async with session_scope() as db: async with session_scope() as db:
# Idempotent: creating an existing tag just returns it. Case-INSENSITIVE, # Creating an existing tag returns it. Case-insensitive, like the clients:
# like the clients' `find_or_create_label` — a case-sensitive match here was # their `labels` index is unique on `lower(name)`, so they can't hold both
# the path that MINTED the "Groceries" beside "groceries" pair that no synced # "Groceries" and "groceries".
# client can hold, since their `labels` index is unique on `lower(name)`.
existing = await db.scalar( existing = await db.scalar(
select(Label).where(named(g.user_id, name)) select(Label).where(named(g.user_id, name))
) )
@@ -146,11 +142,8 @@ async def update_label(label_id: str):
if label is None: if label is None:
return not_found() return not_found()
if has_name: if has_name:
# Case-INSENSITIVE, matching the clients' `find_or_create_label` and the # Case-insensitive, like the clients' unique `lower(name)` index. Postgres
# local store's `lower(name)` unique index. The Postgres constraint here # would accept both spellings, and a phone's pull would then fail.
# is on the raw name, so the database would happily hold "Groceries"
# beside "groceries" — but no synced client can store both, so letting
# one be made is letting a pull fail later on a phone.
clash = await db.scalar( clash = await db.scalar(
select(Label).where( select(Label).where(
named(g.user_id, name), named(g.user_id, name),
@@ -158,12 +151,9 @@ async def update_label(label_id: str):
) )
) )
if clash is not None: if clash is not None:
# Renaming onto an existing tag MERGES the two rather than failing: # Renaming onto an existing tag merges the two. The older row survives
# typing an existing tag's name onto this one says they are the same # and takes the new spelling, so A→B and B→A agree; ties go to the
# thing. The OLDER row survives and takes the new spelling — age is # incumbent. Mirrors `store::rename_label`.
# the one property that does not depend on which of the two the
# caller happened to be renaming, so A→B and B→A agree. Ties go to
# the incumbent. Mirrors `store::rename_label` exactly.
if clash.created_at <= label.created_at: if clash.created_at <= label.created_at:
survivor, doomed = clash, label survivor, doomed = clash, label
else: else:
@@ -195,9 +185,7 @@ async def delete_label(label_id: str):
async def merge_label(label_id: str): async def merge_label(label_id: str):
"""Merge `label_id` (source) INTO the label given by body {"into": <id>}: move """Merge `label_id` (source) INTO the label given by body {"into": <id>}: move
every note tagged with the source onto the target, then tombstone the source. Both every note tagged with the source onto the target, then tombstone the source. Both
must be owned by the caller. Note-body `#tags` are NOT rewritten, so a note whose must be owned by the caller. Body `#tags` are not rewritten (see `merge_into`)."""
body still literally contains the source #tag will re-mint that label on its next
edit — retire a tag by editing it out of the text (a known, documented nuance)."""
data = await request.get_json(silent=True) or {} data = await request.get_json(silent=True) or {}
into = data.get("into") into = data.get("into")
async with session_scope() as db: async with session_scope() as db:
+7 -10
View File
@@ -1,18 +1,15 @@
"""What the server's emails say, and the one layout they all share (#5917). """What the server's emails say, and the one layout they all share.
Each email is a function returning an `Email`: a subject, a plain-text part and an Each email is a function returning an `Email`: a subject, a plain-text part and an
HTML part. `mailer.send` sends both as one multipart/alternative message, so a client HTML part. `mailer.send` sends both as one multipart/alternative message, so a client
that shows HTML gets the branded version and one that doesn't still gets every word that shows HTML gets the branded version and one that doesn't still gets every word
and the link. and the link.
The HTML is deliberately email-shaped rather than web-shaped. It uses tables and The HTML uses tables and inline styles, because many mail clients drop `<style>`
inline styles, because many mail clients drop `<style>` blocks and ignore flexbox. blocks and ignore flexbox. It loads nothing remote. The logo travels inside the
It fetches nothing: the app sends with no outbound network beyond the mail server message as an inline part (`cid:`), which clients show without asking; SVG and
(rule 164), and a remote image is a tracking pixel anyway. The one image, the `data:` images are dropped by too many clients. Everything taken from settings or a
Inkwell logo, travels inside the message as an inline part (`cid:`), which mail person (site name, address, link) is escaped.
clients show without asking, unlike a remote image; SVG and `data:` images are
dropped by too many clients to rely on. Everything taken from settings or from a person (the site name, an address,
a link) is escaped before it goes into the HTML.
Links come from Settings → General → Public address, never from the request: a Links come from Settings → General → Public address, never from the request: a
forged Host header would otherwise mail a stranger a link to the forger's site. forged Host header would otherwise mail a stranger a link to the forger's site.
@@ -64,7 +61,7 @@ def _render(
site: str, subject: str, heading: str, lines: list[str], button: _Button | None, footer: str site: str, subject: str, heading: str, lines: list[str], button: _Button | None, footer: str
) -> Email: ) -> Email:
"""One email in the shared layout: the site's name, a heading, the message, a """One email in the shared layout: the site's name, a heading, the message, a
button for the link, and a footer saying who sent it and why (preference 194). button for the link, and a footer saying who sent it and why.
`lines` are paragraphs of plain text. The button goes after the first one, and the `lines` are paragraphs of plain text. The button goes after the first one, and the
text part spells its link out in full, since that is the only way a text-only text part spells its link out in full, since that is the only way a text-only
+4 -5
View File
@@ -1,10 +1,9 @@
"""Sending email over the SMTP server set in Settings → Email (#5266). """Sending email over the SMTP server set in Settings → Email.
Used for password reset links, invites, and the admin's test message. What each one Used for password reset links, invites, and the admin's test message. What each one
says is in `mail_templates`; this module only delivers. Email is an optional says is in `mail_templates`; this module only delivers. Email is optional: with no
feature (rule 164): with no server set, `mail_settings` answers None and everything server set, `mail_settings` answers None and everything that would send says it is
that would send says it is not available. A send that fails is logged and never not available. A failed send is logged and takes nothing else down.
takes anything else down with it.
stdlib `smtplib`, run on a worker thread, so the event loop never waits on a mail stdlib `smtplib`, run on a worker thread, so the event loop never waits on a mail
server and nothing new is installed. server and nothing new is installed.
+1 -1
View File
@@ -12,7 +12,7 @@ from . import Base
class AuditEvent(Base): class AuditEvent(Base):
"""One thing that happened to an account: a sign-in, a failed one, a password """One thing that happened to an account: a sign-in, a failed one, a password
changed, a device linked (#2939 §5). Written by `audit.record`, read by admins in changed, a device linked. Written by `audit.record`, read by admins in
Settings → Activity, and deleted after `audit_retention_days`. Settings → Activity, and deleted after `audit_retention_days`.
""" """
+1 -1
View File
@@ -12,7 +12,7 @@ from . import Base
class Invite(Base): class Invite(Base):
"""A single-use, expiring link an admin hands to one person so they can register """A single-use, expiring link an admin hands to one person so they can register
while registration is closed (#5172). while registration is closed.
Only the token's SHA-256 hash is stored, as for device tokens; the link is shown Only the token's SHA-256 hash is stored, as for device tokens; the link is shown
once, when the invite is made. Rows are never deleted by the app: a used, expired once, when the invite is made. Rows are never deleted by the app: a used, expired
+4 -4
View File
@@ -12,8 +12,8 @@ from . import Base
class Label(Base): class Label(Base):
__tablename__ = "labels" __tablename__ = "labels"
# A name is unique among LIVE tags only: a tombstone keeps its row for the change # A name is unique among live tags only: a tombstone keeps its row for the change
# feed but gives its name back, so `#grocery` can be made again (#5382, 0040). # feed but gives its name back, so `#grocery` can be made again.
__table_args__ = ( __table_args__ = (
Index( Index(
"uq_labels_owner_name_live", "uq_labels_owner_name_live",
@@ -36,8 +36,8 @@ class Label(Base):
updated_at: Mapped[datetime] = mapped_column( updated_at: Mapped[datetime] = mapped_column(
DateTime(timezone=True), nullable=False, server_default=func.now(), onupdate=func.now() DateTime(timezone=True), nullable=False, server_default=func.now(), onupdate=func.now()
) )
# Sync (M8): monotonic per-row revision (from sync_revision_seq via DB trigger) so a # Monotonic per-row revision (sync_revision_seq, set by a DB trigger), so a rename,
# label rename/recolor/merge/delete propagates to native clients independently of notes. # recolour, merge or delete reaches native clients independently of notes.
sync_revision: Mapped[int | None] = mapped_column(BigInteger(), nullable=True) sync_revision: Mapped[int | None] = mapped_column(BigInteger(), nullable=True)
# Tombstone: set means deleted. The row stays so the change feed can carry the # Tombstone: set means deleted. The row stays so the change feed can carry the
# delete to devices; every read of the catalog skips it (`labeling.live`). # delete to devices; every read of the catalog skips it (`labeling.live`).
+6 -8
View File
@@ -22,11 +22,9 @@ class Note(Base):
owner_id: Mapped[uuid.UUID] = mapped_column( owner_id: Mapped[uuid.UUID] = mapped_column(
UUID(as_uuid=True), ForeignKey("users.id", ondelete="CASCADE"), nullable=False UUID(as_uuid=True), ForeignKey("users.id", ondelete="CASCADE"), nullable=False
) )
# The note's NAME: its first non-empty body line, else its first checklist item # The note's name: its first non-empty body line (notes.derive_display_title).
# (see notes.derive_display_title). There is no title field to prefer — a note is # Stored for search results and export filenames, and so full-text search can
# a body plus optional items, and this is simply the first thing written in it. # weight it above the rest of the body.
# Persisted so search results and export filenames have something to say, and so
# the full-text vector can weight it above the rest of the body.
display_title: Mapped[str] = mapped_column(Text(), nullable=False, server_default="") display_title: Mapped[str] = mapped_column(Text(), nullable=False, server_default="")
body: Mapped[str] = mapped_column(Text(), nullable=False, server_default="") body: Mapped[str] = mapped_column(Text(), nullable=False, server_default="")
# Manual drag order (higher = earlier); 0 until the user reorders. # Manual drag order (higher = earlier); 0 until the user reorders.
@@ -44,9 +42,9 @@ class Note(Base):
updated_at: Mapped[datetime] = mapped_column( updated_at: Mapped[datetime] = mapped_column(
DateTime(timezone=True), nullable=False, server_default=func.now(), onupdate=func.now() DateTime(timezone=True), nullable=False, server_default=func.now(), onupdate=func.now()
) )
# Sync (M8): a monotonic per-row revision drawn from sync_revision_seq and assigned # Monotonic per-row revision from sync_revision_seq, set by a DB trigger on every
# by a DB trigger on every insert/update — the delta cursor native clients pull # write: the cursor native clients pull against. Nullable in the ORM because the
# against. Nullable in the ORM because the trigger populates it server-side. # trigger fills it.
sync_revision: Mapped[int | None] = mapped_column(BigInteger(), nullable=True) sync_revision: Mapped[int | None] = mapped_column(BigInteger(), nullable=True)
# Hard-delete tombstone: non-null => permanently deleted (content cleared), kept so # Hard-delete tombstone: non-null => permanently deleted (content cleared), kept so
# offline clients learn the row is gone. Distinct from deleted_at (= recoverable trash). # offline clients learn the row is gone. Distinct from deleted_at (= recoverable trash).
+1 -2
View File
@@ -25,7 +25,6 @@ class NoteAttachment(Base):
filename: Mapped[str | None] = mapped_column(Text(), nullable=True) filename: Mapped[str | None] = mapped_column(Text(), nullable=True)
mime: Mapped[str] = mapped_column(Text(), nullable=False) mime: Mapped[str] = mapped_column(Text(), nullable=False)
size: Mapped[int] = mapped_column(BigInteger(), nullable=False) size: Mapped[int] = mapped_column(BigInteger(), nullable=False)
# Content hash (sha256 hex) for client-side dedupe + integrity over sync. Nullable # sha256 hex, for client-side dedupe and integrity over sync. Null on older rows.
# for rows created before this column (no backfill).
sha256: Mapped[str | None] = mapped_column(Text(), nullable=True) sha256: Mapped[str | None] = mapped_column(Text(), nullable=True)
created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False, server_default=func.now()) created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False, server_default=func.now())
+1 -1
View File
@@ -11,7 +11,7 @@ from . import Base
class NoteUserState(Base): class NoteUserState(Base):
"""A recipient's own pin, archive and place for a note shared with them (#5176). """A recipient's own pin, archive and place for a note shared with them.
The owner's live on the note itself. `position` null means "where the owner put The owner's live on the note itself. `position` null means "where the owner put
it", so a recipient who only pins a note doesn't freeze its order. `updated_at` it", so a recipient who only pins a note doesn't freeze its order. `updated_at`
+2 -2
View File
@@ -11,8 +11,8 @@ from . import Base
class PasswordReset(Base): class PasswordReset(Base):
"""A one-time link an admin makes so a person who forgot their password can set """A one-time link for setting a new password, made by an admin or emailed by
a new one (#5173). There is no mail path; the admin hands the link over. "forgot password".
Only the token's SHA-256 hash is stored, as for invites and device tokens. Making Only the token's SHA-256 hash is stored, as for invites and device tokens. Making
a new link for an account deletes its unused ones, so only the newest works. a new link for an account deletes its unused ones, so only the newest works.
+1 -1
View File
@@ -11,7 +11,7 @@ from . import Base
class Setting(Base): class Setting(Base):
"""A single persisted key/value setting. Values are JSON-encoded text; the """A single persisted key/value setting. Values are JSON-encoded text; the
typed defaults + metadata live in the code registry (inkwell.settings), so typed defaults + metadata live in the code registry (inkwell.settings), so
an empty table means "all defaults" (rule 26).""" an empty table means "all defaults"."""
__tablename__ = "settings" __tablename__ = "settings"
+3 -3
View File
@@ -15,9 +15,9 @@ class Share(Base):
A resource's OWNER is tracked on the resource row itself (its owner_id column); A resource's OWNER is tracked on the resource row itself (its owner_id column);
this table records grants BEYOND the owner — to a single user or to a whole this table records grants BEYOND the owner — to a single user or to a whole
group. It is polymorphic (resource_type + resource_id) so every future group. Polymorphic (resource_type + resource_id), so any shareable entity uses
shareable entity (notes first, in M1) reuses one table and one ACL predicate. one table and one ACL predicate. Exactly one of shared_with_user_id /
Exactly one of shared_with_user_id / shared_with_group_id is set. shared_with_group_id is set.
""" """
__tablename__ = "shares" __tablename__ = "shares"
+1 -1
View File
@@ -11,7 +11,7 @@ from . import Base
class ShareRevocation(Base): class ShareRevocation(Base):
"""A note that someone lost access to, for their devices to delete (#5175). """A note that someone lost access to, for their devices to delete.
The change feed only carries notes a person can see, so a share ending would The change feed only carries notes a person can see, so a share ending would
otherwise leave a stale copy on every device that had synced it. The revision otherwise leave a stale copy on every device that had synced it. The revision
+1 -1
View File
@@ -18,7 +18,7 @@ class User(Base):
email: Mapped[str] = mapped_column(CITEXT(), nullable=False, unique=True) email: Mapped[str] = mapped_column(CITEXT(), nullable=False, unique=True)
# First registered user becomes admin (see auth.register); admin gates Settings. # First registered user becomes admin (see auth.register); admin gates Settings.
is_admin: Mapped[bool] = mapped_column(Boolean(), nullable=False, server_default=func.false()) is_admin: Mapped[bool] = mapped_column(Boolean(), nullable=False, server_default=func.false())
# Nullable: leaves room for external-identity-only accounts later (rule 26). # Nullable, for accounts that sign in only through an external identity.
password_hash: Mapped[str | None] = mapped_column(Text(), nullable=True) password_hash: Mapped[str | None] = mapped_column(Text(), nullable=True)
display_name: Mapped[str] = mapped_column(Text(), nullable=False) display_name: Mapped[str] = mapped_column(Text(), nullable=False)
# Moved on by a password reset to sign the account out everywhere: every session # Moved on by a password reset to sign the account out everywhere: every session
+1 -1
View File
@@ -1,4 +1,4 @@
"""How a note sits on one person's board: pinned, archived, and where (#5176). """How a note sits on one person's board: pinned, archived, and where.
For the owner that is the note's own columns. For someone it is shared with it is For the owner that is the note's own columns. For someone it is shared with it is
their `NoteUserState` row, so the owner pinning a shared shopping list doesn't their `NoteUserState` row, so the owner pinning a shared shopping list doesn't
+20 -38
View File
@@ -72,11 +72,9 @@ async def list_notes():
label_params = request.args.getlist("label") label_params = request.args.getlist("label")
has_attachment = coerce_bool(request.args.get("has_attachment")) has_attachment = coerce_bool(request.args.get("has_attachment"))
query_text = (request.args.get("q") or "").strip() query_text = (request.args.get("q") or "").strip()
# Optional creation-date range — the "browse by when" / Timeline lens. Both bounds # Optional creation-date range, half-open [created_after, created_before), so a
# are ISO-8601 instants forming a HALF-OPEN interval [created_after, created_before), # client can pass local day boundaries. `sort=created` is newest first; the
# so a client can pass local day-boundaries (start-of-day .. start-of-next-day) # default is the board's pinned/position/updated order.
# without off-by-one. `sort=created` orders newest-captured first for a chronological
# timeline; the default keeps the board's pinned/position/updated order.
after_param = request.args.get("created_after") after_param = request.args.get("created_after")
before_param = request.args.get("created_before") before_param = request.args.get("created_before")
sort = request.args.get("sort") sort = request.args.get("sort")
@@ -85,7 +83,7 @@ async def list_notes():
return json_error("invalid shared", 400) return json_error("invalid shared", 400)
async with session_scope() as db: async with session_scope() as db:
# Through the viewer's own pin, archive and place: on a note shared with them # Through the viewer's own pin, archive and place: on a note shared with them
# those are theirs, not the owner's (#5176). # those are theirs, not the owner's.
stmt = join_state(select(Note), g.user_id).where(note_visible()) stmt = join_state(select(Note), g.user_id).where(note_visible())
stmt = apply_filter(stmt, filter_name, archived_for(g.user_id)) stmt = apply_filter(stmt, filter_name, archived_for(g.user_id))
# Trash is the owner's: a note its owner trashed leaves a recipient's board # Trash is the owner's: a note its owner trashed leaves a recipient's board
@@ -112,10 +110,8 @@ async def list_notes():
return json_error("invalid created_before", 400) return json_error("invalid created_before", 400)
stmt = stmt.where(Note.created_at < before_dt) stmt = stmt.where(Note.created_at < before_dt)
if query_text: if query_text:
# Full-text match over the note's name + body (generated tsvector, # Ranked full-text match over the note's name and body (a generated
# migrations 0005/0026), ranked. This is the ONLY text search now: the # tsvector). The only text search, so it combines with every other filter.
# separate facet-less `/search` route was removed because landing on it
# was the one place you could not also narrow by tag (note 2930).
tsquery = func.websearch_to_tsquery("english", query_text) tsquery = func.websearch_to_tsquery("english", query_text)
search_col = literal_column("notes.search_vector") search_col = literal_column("notes.search_vector")
stmt = stmt.where(search_col.op("@@")(tsquery)).order_by( stmt = stmt.where(search_col.op("@@")(tsquery)).order_by(
@@ -168,8 +164,7 @@ async def complete_reminder(note_id: str):
return jsonify(await _serialize_note(db, note, g.user_id)) return jsonify(await _serialize_note(db, note, g.user_id))
# The longest snooze: 30 days. The core clamps to the same range # The longest snooze: 30 days, the same as the core's `store::SNOOZE_MAX_MINUTES`.
# (`store::SNOOZE_MAX_MINUTES`), so a device and the web agree (#5386).
SNOOZE_MAX_MINUTES = 60 * 24 * 30 SNOOZE_MAX_MINUTES = 60 * 24 * 30
@@ -283,9 +278,8 @@ async def import_notes():
if len(zf.infolist()) > IMPORT_MAX_ENTRIES: if len(zf.infolist()) > IMPORT_MAX_ENTRIES:
return json_error("that archive has too many files to import", 413) return json_error("that archive has too many files to import", 413)
# Bound total decompressed bytes so a zip bomb can't exhaust memory/disk. Any entry # Bound decompressed bytes so a zip bomb can't exhaust memory or disk. Going over
# (or the cumulative total) blowing the budget aborts the whole import — nothing is # aborts the whole import; the session rolls back, so nothing is committed.
# committed, since the session rolls back when the exception exits its block.
budget = _ImportBudget() budget = _ImportBudget()
try: try:
specs, source = _read_import_specs(zf, budget) specs, source = _read_import_specs(zf, budget)
@@ -321,10 +315,7 @@ async def import_notes():
async def list_titles(): async def list_titles():
"""Owner's non-trashed notes, keyed by their display NAME. """Owner's non-trashed notes, keyed by their display NAME.
Survived the removal of [[wiki-links]] (note 2897) because it was serving two What the command palette lists, so someone can jump to a note by name.
different things, and only one of them was linking. This is what the command
palette lists so someone can jump to a note by name — which is recall, the thing
this app is actually for. The `[[` autocomplete that also read it is gone.
""" """
async with session_scope() as db: async with session_scope() as db:
rows = ( rows = (
@@ -367,7 +358,7 @@ async def reorder_notes():
note = visible.get(nid) note = visible.get(nid)
if note is None: if note is None:
continue continue
# A note shared with the caller moves on their board only (#5176). # A note shared with the caller moves on their board only.
if note.owner_id == g.user_id: if note.owner_id == g.user_id:
note.position = total - index note.position = total - index
else: else:
@@ -386,8 +377,8 @@ async def create_note():
if is_empty_note(body, item_texts): if is_empty_note(body, item_texts):
return json_error("note is empty", 400) return json_error("note is empty", 400)
async with session_scope() as db: async with session_scope() as db:
# Items still arrive separately — a client holds a list, not a blob — but they # Items may arrive separately, from a client holding a list, and are folded
# are folded into the body, which is where a checklist lives now (M304). # into the body, where a checklist lives.
for text in item_texts: for text in item_texts:
body = append_item(body, text) body = append_item(body, text)
note = Note(owner_id=g.user_id, body="", display_title="", position=await top_position(db, g.user_id) + 1) note = Note(owner_id=g.user_id, body="", display_title="", position=await top_position(db, g.user_id) + 1)
@@ -418,9 +409,8 @@ async def update_note(note_id: str):
owned = note.owner_id == g.user_id owned = note.owner_id == g.user_id
if not owned: if not owned:
# Someone the note is shared with may pin and archive it on their own # Someone the note is shared with may pin and archive it on their own
# board (#5176), and change its text if it was shared at `edit`; nothing # board, and change its text if shared at `edit`; nothing else. Text from
# else. They can already read it, so saying so is no oracle. Text from a # a view share 404s.
# view share 404s, as it did before they could do anything at all.
if set(data) - {"body", *OWN_FIELDS}: if set(data) - {"body", *OWN_FIELDS}:
return json_error("only the owner can change that", 403) return json_error("only the owner can change that", 403)
if "body" in data and await _get_editable(db, note_id) is None: if "body" in data and await _get_editable(db, note_id) is None:
@@ -576,11 +566,6 @@ async def update_item(note_id: str, item_id: str):
return await _rewrite_body(db, note, body) return await _rewrite_body(db, note, body)
# The reorder route is gone with M304. Reordering a checklist is moving a line, which
# is something a text editor already does and no client ever called this for — the
# only reference to it in the tree was a test asserting the route existed.
@bp.post("/<note_id>/attachments") @bp.post("/<note_id>/attachments")
@login_required @login_required
async def upload_attachment(note_id: str): async def upload_attachment(note_id: str):
@@ -621,7 +606,7 @@ async def get_attachment(note_id: str, att_id: str):
if aid is None: if aid is None:
return not_found() return not_found()
async with session_scope() as db: async with session_scope() as db:
# Owner OR shared may view (rule 47). A purged note's files are gone with it. # Owner or a share may view. A purged note's files are gone with it.
note = await _get_visible(db, note_id) note = await _get_visible(db, note_id)
if note is None: if note is None:
return not_found() return not_found()
@@ -638,9 +623,8 @@ async def get_attachment(note_id: str, att_id: str):
response = await send_file(str(file_path), mimetype=mime) response = await send_file(str(file_path), mimetype=mime)
response.headers["X-Content-Type-Options"] = "nosniff" response.headers["X-Content-Type-Options"] = "nosniff"
response.headers["Cache-Control"] = "private, max-age=86400" response.headers["Cache-Control"] = "private, max-age=86400"
# Only trusted raster image types render inline; everything else — notably # Only trusted raster image types render inline. Everything else downloads,
# image/svg+xml, which can carry script — downloads, so a shared note's attachment # notably SVG, which can carry script into a viewer's session.
# can't execute script in a viewer's session (rule 47 = shares).
disposition = "inline" if mime in ALLOWED_IMAGE_MIMES else "attachment" disposition = "inline" if mime in ALLOWED_IMAGE_MIMES else "attachment"
response.headers["Content-Disposition"] = f'{disposition}; filename="{_header_filename(filename)}"' response.headers["Content-Disposition"] = f'{disposition}; filename="{_header_filename(filename)}"'
return response return response
@@ -722,10 +706,8 @@ async def delete_note(note_id: str):
return not_found() return not_found()
if note.deleted_at is None: if note.deleted_at is None:
return json_error("note must be trashed before permanent delete", 409) return json_error("note must be trashed before permanent delete", 409)
# A tombstone, not a dropped row. Deleting the row outright would leave the # A tombstone, not a dropped row, so a device that was offline learns of the
# server with no record the note ever existed, so a linked device that was # delete instead of keeping its copy and pushing it back.
# offline at the time would keep its copy forever — and push it back the
# next time it was edited. The delete has to be something clients can LEARN.
await purge_note(db, note) await purge_note(db, note)
await db.commit() await db.commit()
return jsonify({"ok": True}) return jsonify({"ok": True})
+2 -3
View File
@@ -2,7 +2,6 @@ from __future__ import annotations
from quart import Blueprint from quart import Blueprint
# The notes blueprint lives in its own tiny module so every route module in the # In its own module so route modules can `from ._bp import bp` without an import cycle
# package can `from ._bp import bp` without importing the package __init__ (which # through the package __init__.
# imports the route modules) — i.e. no import cycle.
bp = Blueprint("notes", __name__, url_prefix="/api/notes") bp = Blueprint("notes", __name__, url_prefix="/api/notes")
+6 -10
View File
@@ -1,16 +1,12 @@
"""Writing a note's text — the one path every surface goes through. """Writing a note's text — the one path every surface goes through.
A body edit is never only the body. It may keep a revision of the old text, it A body edit may keep a revision of the old text, renames the note, and lifts and
renames the note, it lifts and reconciles `#tags`, and once committed it queues link reconciles `#tags`. PATCH, the item routes, restoring a revision and sync push all
previews. That sequence used to be written out at each place a body could change — come through here; the caller says how the old text is kept.
PATCH, the item routes, restoring a revision and sync push — and the copies drifted:
restoring a revision never queued previews, so a restored link stayed a bare URL
(#5164). Now a caller states how the old text should be kept and this does the rest.
Previews are the one part that cannot happen here. They are queued AFTER the commit Link previews are queued by the caller after its commit (`unfurl_queue.schedule`),
(`unfurl_queue.schedule`), because the fetch reads the note in a session of its own because the fetch reads the note in its own session. So this returns whether the
and a note it cannot see yet gets no preview. So this returns whether the text text changed.
changed, and the caller queues them once its transaction is committed.
""" """
from __future__ import annotations from __future__ import annotations
+12 -25
View File
@@ -1,18 +1,11 @@
"""Checklist items — a note's body IS its checklist (M304). """Checklist items: a note's body is its checklist.
A `- [ ] milk` line is the item. There is no `note_items` table beside the body any A `- [ ] milk` line is the item, so a list can sit between two paragraphs. Unlike
more, which is what lets a list sit BETWEEN two paragraphs: rows had a position in a tags, items are stored nowhere else: nothing queries them, so `parse_items` is the
table and no position in the text, so a separate list could only ever render after whole storage layer and the rewriters below are how one is edited.
the prose no matter how it was styled.
The same shape as `tags.py`, one strength further along. Tags are derived from the The grammar has three implementations that must agree, or a checklist changes shape
body too, but they MATERIALISE into `note_labels` rows because the board queries by when it syncs:
label. Items materialise into nothing, because nothing queries them — their only
readers are the card, the editor and `display_title`. So `parse_items` is the whole
storage layer for a checklist, and the rewriters below are how one is edited.
THE GRAMMAR IS SHARED. Three implementations exist and they have to agree, because a
difference between any two of them is a checklist that changes shape when it syncs:
core/src/local/derive.rs the native clients (desktop + Android) core/src/local/derive.rs the native clients (desktop + Android)
src/inkwell/notes/checklist.py this file, the server src/inkwell/notes/checklist.py this file, the server
@@ -21,12 +14,9 @@ difference between any two of them is a checklist that changes shape when it syn
optional indent, `-` or `*`, one-or-more spaces, `[ ]`/`[x]`/`[X]`, optional indent, `-` or `*`, one-or-more spaces, `[ ]`/`[x]`/`[X]`,
then either end-of-line or one-or-more spaces and the text. then either end-of-line or one-or-more spaces and the text.
`*` is accepted because markdown.ts already takes it for a plain bullet, and a rule `*` is accepted because it is a plain bullet too. `- [ ]` alone is an item with empty
that allowed `* item` but not `* [ ] item` would be one nobody could guess. `- [ ]` text, which is what Enter on a list leaves behind. `- [X]` parses as checked and
with nothing after it IS an item with empty text — that is what pressing Enter on a renders back lowercase, so one canonical form survives a round trip.
list leaves behind, and refusing to parse it would make a half-typed list stop being
a list. `- [X]` parses as checked and renders back lowercase, so one canonical form
survives a round trip.
""" """
from __future__ import annotations from __future__ import annotations
@@ -101,12 +91,9 @@ def set_item_checked(body: str, index: int, checked: bool) -> str:
def append_item(body: str, text: str, checked: bool = False) -> str: def append_item(body: str, text: str, checked: bool = False) -> str:
"""Add an item at the end of the body. """Add an item at the end of the body.
A blank line between prose and the list, nothing between consecutive items — A blank line between prose and the list, nothing between consecutive items: the
the layout `import_export._note_markdown` has always used when writing a checklist same layout as the export and the core's `derive::append_item`, so every surface
out. That is not cosmetic: it is what the Alembic migration folds existing lands on identical bodies.
`note_items` rows into AND what `derive::append_item` produces on every client, so
all three land on identical bodies. An export taken before the migration and one
taken after therefore differ in nothing.
""" """
line = render_item(text.strip(), checked) line = render_item(text.strip(), checked)
trimmed = body.rstrip("\n") trimmed = body.rstrip("\n")
+15 -25
View File
@@ -33,21 +33,13 @@ DISPLAY_TITLE_CAP = 200
def derive_display_title(body: str | None) -> str: def derive_display_title(body: str | None) -> str:
"""The note's display NAME: the first line of the body that says anything. """The note's display NAME: the first line of the body that says anything.
There is no explicit title to prefer any more (M13 step 3) — a note is a body, and Stored as notes.display_title for search results and export filenames. A task
its name is simply the first thing written in it. Persisted as notes.display_title line is named without its marker ("milk", not "- [ ] milk"), and an empty item
so search results and export filenames have something to say. is skipped.
The old `first_item` fallback is gone with M304: items ARE body lines now, so a Lines split on "\n" alone, like every other reader of the grammar (`splitlines`
list-only note is named by its first item without anyone having to arrange it. What would also break on "\r" and U+2028). Mirrors `derive::display_title` in the
replaced the fallback is stripping the task marker — calling that note "- [ ] milk" core; both run the `display_titles` cases in core/testdata/grammar.json.
would show someone the storage instead of the note — and skipping an EMPTY item, so
a half-typed list does not leave a note with no name.
Lines split on "\n" alone, as every other reading of the grammar does;
`splitlines` also broke on a lone "\r" or a U+2028, so a phone and the server named
the same note differently. Mirrors `derive::display_title` in the core, and both
run the `display_titles` cases in core/testdata/grammar.json. Deterministic — a
literal first line, never generated.
""" """
for line in (body or "").split("\n"): for line in (body or "").split("\n"):
stripped = strip_marker(line.strip()).strip() stripped = strip_marker(line.strip()).strip()
@@ -59,9 +51,8 @@ def derive_display_title(body: str | None) -> str:
def is_empty_note(body: str | None, items: list | None = None) -> bool: def is_empty_note(body: str | None, items: list | None = None) -> bool:
"""Nothing worth keeping: no body text and no checklist items. """Nothing worth keeping: no body text and no checklist items.
`items` is still a separate argument because create still accepts them separately — `items` is separate because create accepts them separately (the importer holds a
the importer holds a list, not a blob — and they are folded into the body only list); they are folded into the body after this check.
after this check has decided the note is worth making at all.
""" """
return not (body or "").strip() and not items return not (body or "").strip() and not items
@@ -77,7 +68,7 @@ def apply_filter(stmt, filter_name: str, archived):
"""Narrow a notes query to one board view. """Narrow a notes query to one board view.
`archived` is the viewer's archive flag (`note_state.archived_for`): a note `archived` is the viewer's archive flag (`note_state.archived_for`): a note
someone shared with you is archived on your board when YOU archived it (#5176). someone shared with you is archived on your board when you archived it.
Every branch excludes purge tombstones — content-less rows kept only so the sync Every branch excludes purge tombstones — content-less rows kept only so the sync
feed can tell offline clients a note is gone (see `retention.purge_note`). The feed can tell offline clients a note is gone (see `retention.purge_note`). The
@@ -121,12 +112,11 @@ async def _fetch_note(db, note_id: str, gate) -> Note | None:
async def _get_owned(db, note_id: str) -> Note | None: async def _get_owned(db, note_id: str) -> Note | None:
"""Fetch a note the current user OWNS, for every change only an owner may make: """Fetch a note the current user OWNS, for every change only an owner may make:
trash, delete, labels, reminders, attachments, previews, history and sharing trash, delete, labels, reminders, attachments, previews, history and sharing.
(#5174). Pin and archive are everyone's own (#5176). Pin and archive are everyone's own.
Someone the note is shared with gets None here, and so a 404, even when they can Someone the note is shared with gets None, and so a 404, even when they can read
read the note: a 403 would confirm the note exists to a caller who only has its it: a 403 would confirm the note exists to a caller who only has its id.
id (#1984).
A purged note reads as absent: the REST API must treat it as gone, so opening, A purged note reads as absent: the REST API must treat it as gone, so opening,
editing or restoring one 404s. The sync push path looks rows up directly rather editing or restoring one 404s. The sync push path looks rows up directly rather
@@ -138,14 +128,14 @@ async def _get_owned(db, note_id: str) -> Note | None:
async def _get_visible(db, note_id: str) -> Note | None: async def _get_visible(db, note_id: str) -> Note | None:
"""Fetch a note the current user can see: theirs, or shared with them at any """Fetch a note the current user can see: theirs, or shared with them at any
level. What a recipient may do to it from here is their own pin, archive and level. What a recipient may do to it from here is their own pin, archive and
place (#5176); `_get_editable` and `_get_owned` gate the rest.""" place; `_get_editable` and `_get_owned` gate the rest."""
return await _fetch_note(db, note_id, note_visible()) return await _fetch_note(db, note_id, note_visible())
async def _get_editable(db, note_id: str) -> Note | None: async def _get_editable(db, note_id: str) -> Note | None:
"""Fetch a note the current user may change the TEXT of: its body and its """Fetch a note the current user may change the TEXT of: its body and its
checklist. The owner, or someone it is shared with at `edit`; a view share gets checklist. The owner, or someone it is shared with at `edit`; a view share gets
None, and so a 404, like a stranger (#5174).""" None, and so a 404, like a stranger."""
return await _fetch_note(db, note_id, note_visible(EDIT)) return await _fetch_note(db, note_id, note_visible(EDIT))
+10 -25
View File
@@ -42,22 +42,16 @@ def _note_markdown(note: Note, labels: list) -> str:
fm.append(f"updated: {iso(note.updated_at) or ''}") fm.append(f"updated: {iso(note.updated_at) or ''}")
fm.append("---") fm.append("---")
fm.append("") fm.append("")
# Body and checklist are no longer alternatives — a note can carry both, so both
# are written, body first, with a blank line between them when there is.
if note.body: if note.body:
fm.append(note.body) fm.append(note.body)
# No separate items block any more. The body already ends with those exact lines # The checklist is part of the body, so it is not written a second time.
# (M304) — this function is where their layout was decided, and appending them a
# second time would double every checklist in an export.
return "\n".join(fm) + "\n" return "\n".join(fm) + "\n"
# --- Import: Inkwell's own export (round-trip) OR a Google Keep Takeout zip --- # --- Import: Inkwell's own export (round-trip) OR a Google Keep Takeout zip ---
# The `app` value an export's notes.json carries. "thoughtsync" is what every export # The `app` values an Inkwell export's notes.json may carry. Exports from when the app
# written before the rename to Inkwell says (Scribe note 5071), and those zips are # was called ThoughtSync say "thoughtsync", and still import.
# people's backups, so they keep importing. That is the one reason the old name is
# still read here; nothing writes it any more.
NATIVE_APP_MARKERS = ("inkwell", "thoughtsync") NATIVE_APP_MARKERS = ("inkwell", "thoughtsync")
# Reverse of ALLOWED_IMAGE_MIMES, for inferring an attachment's mime from its # Reverse of ALLOWED_IMAGE_MIMES, for inferring an attachment's mime from its
@@ -106,9 +100,7 @@ def _keep_spec(kn: dict, keep_dir: str) -> dict:
"""Normalize one Google Keep note (Takeout <note>.json) into the common import """Normalize one Google Keep note (Takeout <note>.json) into the common import
spec. `keep_dir` is the note JSON's folder, used to resolve attachment paths.""" spec. `keep_dir` is the note JSON's folder, used to resolve attachment paths."""
list_content = kn.get("listContent") if isinstance(kn.get("listContent"), list) else [] list_content = kn.get("listContent") if isinstance(kn.get("listContent"), list) else []
# Keep's own notes are one or the other, but its text was being DISCARDED whenever # A Keep note may carry both text and list content; both are kept.
# a note also had list content, because the target model could only hold one.
# It can hold both now, so both are kept.
body = kn.get("textContent") or "" body = kn.get("textContent") or ""
# Keep stores link annotations (e.g. shared URLs) separately from the text — # Keep stores link annotations (e.g. shared URLs) separately from the text —
# fold any URLs into the body so the content survives the move. # fold any URLs into the body so the content survives the move.
@@ -247,28 +239,21 @@ async def _create_imported_note(
name derivation + tag reconciliation as create_note. Returns False (nothing name derivation + tag reconciliation as create_note. Returns False (nothing
written) when the spec is empty.""" written) when the spec is empty."""
body = spec.get("body") or "" body = spec.get("body") or ""
# An imported title becomes the note's FIRST BODY LINE. # An imported title (Keep has them) becomes the first body line, which is where a
# # note's name lives. Skipped when the body already opens with it, so re-importing
# Inkwell has no title field any more (M13 step 3), but the things people # an export doesn't stack duplicates.
# import from do — Keep notes carry one, and so does any export taken before this.
# Dropping it would silently lose text someone wrote; folding it into the body puts
# it exactly where a name now lives, so the note comes in named the way it was.
# Skipped when the body already opens with that line, so re-importing an export
# this code produced doesn't stack duplicates.
title = (spec.get("title") or "").strip() title = (spec.get("title") or "").strip()
if title and body.lstrip().split("\n", 1)[0].strip() != title: if title and body.lstrip().split("\n", 1)[0].strip() != title:
body = f"{title}\n{body}" if body.strip() else title body = f"{title}\n{body}" if body.strip() else title
items = spec.get("items") or [] items = spec.get("items") or []
item_texts = [t for t in ((it.get("text") or "").strip() for it in items) if t] item_texts = [t for t in ((it.get("text") or "").strip() for it in items) if t]
# A note that is only a photo is still a note — Keep has plenty of them, and they # A note that is only a photo is still a note, as in the core's importer.
# used to be skipped as empty. The core's importer keeps them too (portable.rs).
if is_empty_note(body, item_texts) and not spec.get("attachments"): if is_empty_note(body, item_texts) and not spec.get("attachments"):
return False return False
# Items fold into the body, which is where a checklist lives now (M304). Done # Items fold into the body before the Note is built, so display_title and
# before the Note is built so display_title and _lift_and_reconcile_tags both see the # _lift_and_reconcile_tags see the finished text; an item can carry a #tag.
# finished text — an imported item can carry a #tag like any other line.
for it in items: for it in items:
text = (it.get("text") or "").strip() text = (it.get("text") or "").strip()
if text: if text:
+6 -10
View File
@@ -36,15 +36,11 @@ async def _labels_for_notes(db, note_ids: list) -> dict:
def items_of(body: str | None) -> list[dict]: def items_of(body: str | None) -> list[dict]:
"""The note's checklist, read out of its body. No query, because there is no table. """The note's checklist, read out of its body. No query, because there is no table.
Still emitted in the payload after M304, and that is not a second source of truth: Derived on the way out, so it can't disagree with the body; it spares a consumer
it is DERIVED on the way out, so it cannot disagree with the body it came from. It that only draws checkboxes from carrying a parser.
saves every consumer that only wants to draw checkboxes from carrying a parser, and
the ones that do carry one (the native clients, the browser) are free to ignore it
and read the body.
The id is the item's ORDINAL, which is what the rewriters in `checklist.py` take, The id is the item's ordinal, which the rewriters in `checklist.py` take. It
so a client holding one can act on it directly. It also shifts when an item is shifts when an item is removed, so every mutation returns the reloaded note.
removed — every mutation returns the reloaded note for exactly that reason.
""" """
return [ return [
{"id": str(i), "text": item.text, "checked": item.checked, "position": i} {"id": str(i), "text": item.text, "checked": item.checked, "position": i}
@@ -108,7 +104,7 @@ async def _previews_for_notes(db, note_ids: list) -> dict:
async def _sharing_for_notes(db, notes: list, viewer) -> dict: async def _sharing_for_notes(db, notes: list, viewer) -> dict:
"""note_id -> how `viewer` holds the note (#5174): `permission` (owner, edit or """note_id -> how `viewer` holds the note: `permission` (owner, edit or
view), `shared` (whether it is shared at all, as far as the viewer may know) and view), `shared` (whether it is shared at all, as far as the viewer may know) and
`shared_by` (the owner, for a note someone shared with them).""" `shared_by` (the owner, for a note someone shared with them)."""
foreign = [n for n in notes if n.owner_id != viewer] foreign = [n for n in notes if n.owner_id != viewer]
@@ -142,7 +138,7 @@ async def _serialize_notes(db, notes: list, viewer=None) -> list:
"""The API shape of each note. With `viewer`, each also says how the viewer holds """The API shape of each note. With `viewer`, each also says how the viewer holds
it, and a note someone shared with them comes without the owner's labels: labels it, and a note someone shared with them comes without the owner's labels: labels
are personal, and a recipient's board files nothing under someone else's tags. are personal, and a recipient's board files nothing under someone else's tags.
For the same reason its pin, archive and position are the viewer's own (#5176). For the same reason its pin, archive and position are the viewer's own.
Sync passes no viewer to a client that asked only for its own notes.""" Sync passes no viewer to a client that asked only for its own notes."""
ids = [n.id for n in notes] ids = [n.id for n in notes]
labels_map = await _labels_for_notes(db, ids) labels_map = await _labels_for_notes(db, ids)
+21 -57
View File
@@ -1,16 +1,6 @@
"""#tags — parsing note bodies, LIFTING the standalone ones out of the text, and """#tags: parsing note bodies, lifting the standalone ones out of the text (they show
keeping the still-in-text ones in sync with the note_labels rows. Manual (picker) as chips), and keeping the in-text ones in sync with the note_labels rows. Manual
labels are NOT touched here (see the labeling module). (picker) labels are not touched here (see the labeling module).
A tag used to be shown twice: once as the `#todo` you typed and once as a chip. The
chip moved to the top of the card (M311) and the text now goes, but only when the tag
was standing on its own — see `split_body_tags` for the rule and why it is the
conservative one.
Was `links.py`, and also owned `[[wiki-links]]` until they were removed (note 2897):
this app is an intermediary surface for capture and recall, and a linking system is
organization, which is not what it is for. A file called links.py holding no links
would have been exactly the kind of drift that removal was meant to end.
""" """
from __future__ import annotations from __future__ import annotations
@@ -26,22 +16,15 @@ from .helpers import derive_display_title
# A #tag: `#` at the start of a line or after whitespace, then a LETTER, then # A #tag: `#` at the start of a line or after whitespace, then a LETTER, then
# letters, digits, `_` and `-`. The same rule as `line_tags` in # letters, digits, `_` and `-`. The same rule as `line_tags` in
# core/src/local/derive.rs and the web's markdown.ts, and all three run the cases in # core/src/local/derive.rs and the web's markdown.ts, and all three run the cases in
# core/testdata/grammar.json. # core/testdata/grammar.json; a difference would change a note's labels on sync. So
# # `(#todo)`, `/#section` in a URL and `#1st` are not tags.
# The three used to differ. The core took a tag after any punctuation, so `(#todo)`
# was a label on the phone and plain text here, and the `/#section` of a pasted URL
# became a label too. This side let a tag start with a digit or `_` (`#1st`). So a
# note's labels changed every time it synced. The shared rule is the strict one:
# nothing is a tag now that was not already one everywhere (#5166).
# #
# `[^\W\d_]` is "a word character that is not a digit or `_`", which is a letter. # `[^\W\d_]` is "a word character that is not a digit or `_`", which is a letter.
_TAG_RE = re.compile(r"(?<!\S)#([^\W\d_][\w-]*)") _TAG_RE = re.compile(r"(?<!\S)#([^\W\d_][\w-]*)")
# A fence opens or closes a code block. A `#tag` inside one is CODE — the shell # A fence opens or closes a code block. A `#tag` inside one is code (a pasted shell
# comment in a snippet someone pasted — and lifting it would delete a line of their # comment), so it is never lifted out of the text, though it still becomes a label.
# example. It still becomes a label, because it always has and that is a separate
# question from whether the text may be touched.
_FENCE_RE = re.compile(r"^\s*(?:```|~~~)") _FENCE_RE = re.compile(r"^\s*(?:```|~~~)")
@@ -72,17 +55,12 @@ def split_body_tags(body: str | None) -> tuple[list[str], list[str], str]:
THE RULE: a line containing nothing but tags and whitespace is removed. Anything THE RULE: a line containing nothing but tags and whitespace is removed. Anything
else is left exactly as written. else is left exactly as written.
That is deliberately the conservative reading of "standalone". The looser one — A trailing tag on a prose line stays: `buy milk #grocery` is filing, but
also stripping a trailing tag off a prose line — was rejected because a trailing `remember to call #mom` is the sentence's object, and the text can't tell them
tag is ambiguous and the text does not say which it is: `buy milk #grocery` is apart.
filing, `remember to call #mom` is the sentence's object, and lifting the second
leaves "remember to call". Mangling a sentence to save a duplicate chip is a bad
trade. A tag sharing a line with words keeps its words.
`standalone` tags become ORDINARY labels — the text no longer backs them, so `standalone` tags become ordinary labels, removed with the chip's ×. `inline`
nothing can derive them any more, and the way to remove one becomes the chip's × tags stay derived (`via_tag`: backed by text still present).
rather than deleting the text. `inline` tags stay derived exactly as before. That
is the whole meaning of `via_tag` after this change: backed by text still present.
""" """
if not body: if not body:
return [], [], body or "" return [], [], body or ""
@@ -108,8 +86,7 @@ def split_body_tags(body: str | None) -> tuple[list[str], list[str], str]:
standalone.extend(names) standalone.extend(names)
lifted = re.sub(r"\n{3,}", "\n\n", "\n".join(kept)).strip("\n") lifted = re.sub(r"\n{3,}", "\n\n", "\n".join(kept)).strip("\n")
if body.strip() and not lifted.strip(): if body.strip() and not lifted.strip():
# The note was NOTHING but tags. Lifting would leave a blank card, which is a # The note was nothing but tags. Lifting would leave a blank card, so its tags
# worse outcome than a duplicated chip — so leave it alone and let its tags
# stay derived. # stay derived.
return [], _dedupe(standalone + inline), body return [], _dedupe(standalone + inline), body
# A tag that ALSO appears in prose stays derived: the prose copy still backs it, # A tag that ALSO appears in prose stays derived: the prose copy still backs it,
@@ -138,24 +115,12 @@ async def _lift_and_reconcile_tags(db, note: Note) -> None:
"""Attach the note's tag labels, LIFT its standalone tags out of the body, and """Attach the note's tag labels, LIFT its standalone tags out of the body, and
re-derive display_title if the body moved. re-derive display_title if the body moved.
NAMED FOR THE MUTATION. It used to be `_reconcile_tags` and only touched rows; it It rewrites `note.body`, so the title is re-derived here rather than at each
now rewrites `note.body`, and a caller that does not expect that will compute a call site, where it would be computed from text about to be deleted.
display_title from text this function is about to delete.
Which is why the lift and the re-derivation both live HERE rather than at the standalone lifted out, attached as an ordinary label (via_tag=False), removed
seven call sites that would each have to remember. Spreading a derived-value with the chip's ×.
update across every place a body can be written is precisely the failure #2965 inline left in place, attached via_tag=True, detached when its text goes.
named about label minting: "easy to miss, and it is the common one".
The two kinds of tag are handled differently, and that difference IS what
`via_tag` means from here on — backed by text still in the body:
standalone lifted out, attached as an ORDINARY label (via_tag=False). Nothing
derives it any more because nothing is left to derive it from, and
the way to remove it becomes the chip's × — which both editors
already offer for exactly this class of label.
inline left in place, attached via_tag=True, and still detached when its
text goes. Unchanged from before.
""" """
standalone, inline, lifted = split_body_tags(note.body) standalone, inline, lifted = split_body_tags(note.body)
standalone_ids: set = set() standalone_ids: set = set()
@@ -171,10 +136,9 @@ async def _lift_and_reconcile_tags(db, note: Note) -> None:
if not r.via_tag: if not r.via_tag:
attached.add(r.label_id) # manual already: a #tag of the same name changes nothing attached.add(r.label_id) # manual already: a #tag of the same name changes nothing
elif r.label_id in standalone_ids: elif r.label_id in standalone_ids:
# It GRADUATED. The text backing it is about to be deleted, so the row has # The text backing it is about to go, so the row becomes the record. Must
# to become the record instead. This must run BEFORE the detach below, or # run before the detach below, or the row would be dropped for no longer
# the same row would be dropped for no longer being in the body — which is # being in the body.
# the bug that makes a naive lift delete every tag it touches.
r.via_tag = False r.via_tag = False
attached.add(r.label_id) attached.add(r.label_id)
elif r.label_id in inline_ids: elif r.label_id in inline_ids:
+5 -5
View File
@@ -1,17 +1,17 @@
"""Password resets: an admin makes a one-hour link for an account (#5173). """Password resets: an admin makes a one-hour link for an account.
Before this, a forgotten password needed a hand on the database (#2939 §2). This reuses what invites established: a random token, only its hash A random token with only its hash kept, as for invites; the link is shown once to the
kept, the link shown once to the admin who hands it over. admin who hands it over.
Using the link sets a new password and signs the account out everywhere. Its web Using the link sets a new password and signs the account out everywhere. Its web
sessions end because the account's `session_epoch` moves on (see `auth`), and its sessions end because the account's `session_epoch` moves on (see `auth`), and its
device tokens are deleted, so each linked app has to sign in again. device tokens are deleted, so each linked app has to sign in again.
A person can also ask for one themselves when the server can send email (#5266): A person can also ask for one themselves when the server can send email:
`mail_reset_link` makes the same link and mails it to the account's address. `mail_reset_link` makes the same link and mails it to the account's address.
This module is the reset itself; `auth.reset_password` redeems it and the admin route This module is the reset itself; `auth.reset_password` redeems it and the admin route
is in `accounts_api.py`, the same split as invites and for the same reason. is in `accounts_api.py`.
""" """
from __future__ import annotations from __future__ import annotations
+10 -24
View File
@@ -1,27 +1,15 @@
"""Reading what the proxies in front of this app say about a request. """Reading what the proxies in front of this app say about a request.
Two headers carry information the app cannot see for itself — who the client is `X-Forwarded-For` (who the client is) and `X-Forwarded-Proto` (whether they arrived
(`X-Forwarded-For`) and whether they arrived over TLS (`X-Forwarded-Proto`) — and both over TLS) are trusted by one rule, kept here.
are trusted by the same rule, so the rule lives in one place. Writing it twice is
precisely how issue 2183 happened: two places holding one decision, and only one of
them updated.
## The rule A forwarding header grows left to right: each hop appends what it saw, so a caller's
forged entries sit to the left of what our proxies wrote. So we count in from the
right by the **Trusted proxy hops** setting (default 1), and a forged prefix is never
selected.
A forwarding header grows LEFT to RIGHT. Each hop appends what IT saw, so the Too high a count is the dangerous direction: it believes entries no proxy of ours
rightmost entries are the ones our own infrastructure wrote, and anything a caller wrote. When the header is shorter than configured, the socket address is used.
sent arrives to the LEFT of those.
That inverts the intuitive reading. The leftmost entry is nominally "the original
client" — and is exactly the one a caller can forge, by sending the header themselves.
So we count in from the right by the number of proxies we actually run
(the **Trusted proxy hops** setting, default 1), and a forged prefix can never be
selected no matter how much of it there is.
Too HIGH a hop count is the dangerous direction: it starts believing entries no proxy
of ours wrote. Too low just means several callers share a bucket. So when the header
is shorter than configured — fewer proxies than expected — we fall back to the socket
address rather than reaching further left.
""" """
from __future__ import annotations from __future__ import annotations
@@ -61,10 +49,8 @@ def client_address() -> str:
def is_https() -> bool: def is_https() -> bool:
"""Whether this request reached us over TLS — directly, or via a trusted proxy. """Whether this request reached us over TLS — directly, or via a trusted proxy.
Shared by the session cookie's `Secure` flag and by HSTS, because they are the same For the session cookie's `Secure` flag and HSTS. Read with the same hop count as
question. Read with the same hop count as the address: a caller who sets the address, so a forged `X-Forwarded-Proto: https` is never what gets read.
`X-Forwarded-Proto: https` on a plain-HTTP request puts it to the left of whatever
our proxy appended, so it is not what gets read.
""" """
if not has_request_context(): if not has_request_context():
return False return False
+22 -48
View File
@@ -4,37 +4,22 @@ The credential endpoints are rate-limited: login, register, and the native
device-link exchange. Everything else already needs a session or a device token to device-link exchange. Everything else already needs a session or a device token to
reach, so an attacker has to get through one of these three first. reach, so an attacker has to get through one of these three first.
The one exception is the app downloads (`client_dist`): signed in, but each one is App downloads (`client_dist`) are throttled too: each is tens of megabytes, so a
tens of megabytes, so one account or a leaked device token looping on it could fill looping client could fill the server's uplink.
the server's uplink (family idea #5103, practice 6).
## Why in-process is enough here, and where that stops being true State is per-process, which matches the image's single hypercorn worker. With several
workers the effective limit would multiply; the fix would be a shared store.
State lives in module-level dicts, so it is per-process. That is correct for how Every attempt counts against both the account tried and the caller's address, and
this image actually serves — one hypercorn worker (see the Dockerfile, and the either can refuse it:
same assumption the trash sweeper documents in app.py). If that ever gains
``--workers N``, each worker would keep its own counters and the effective limit
would multiply by N; the fix then is a shared store (the Postgres connection is
already there), not a bigger number here.
## Two keys, on purpose - **The account** can't be forged, and stops credential stuffing against one email
from any number of addresses.
- **The address** bounds one source spraying many accounts. Read via `proxy.py`, so
it is only as good as the Trusted proxy hops setting.
Every attempt is counted against BOTH the account being tried and the address it Sign-in counts failures, so a correct password never moves someone closer to a
came from, and either one can refuse it: lockout; registration counts attempts.
- **The account** is the key that matters, and the key that cannot be forged. It
is what stops credential stuffing against one known email, no matter how many
addresses the attempts arrive from.
- **The address** bounds the damage from one source spraying many accounts. It is
read from ``X-Forwarded-For``, counting in from the RIGHT by
the **Trusted proxy hops** setting so that only entries our own proxies wrote are
believed — a forged header lands to the left of those and is never selected. It is
still the weaker of the two keys, because it depends on that setting matching the
deployment; the account key depends on nothing.
Counting is by failure for the sign-in routes and by attempt for registration: a
correct password should never move someone closer to being locked out, but every
registration is a row in the users table whether it succeeds or not.
""" """
from __future__ import annotations from __future__ import annotations
@@ -45,26 +30,18 @@ from collections.abc import Callable
from .settings import live from .settings import live
# Never let the bookkeeping become the denial of service: an attacker rotating an # A cap on tracked keys, so rotating addresses can't grow the limiter without bound.
# address could otherwise mint an unbounded number of buckets. Well above any real # Far above any real caller count; past it the oldest buckets are dropped.
# deployment's distinct-caller count, so a legitimate instance never reaches it; when
# it is reached the oldest buckets are dropped, which at worst forgives some attempts.
#
# Not a setting: it protects the limiter from itself rather than the app from a
# caller, and there is no operator judgment to apply to it.
MAX_BUCKETS = 10_000 MAX_BUCKETS = 10_000
class SlidingWindow: class SlidingWindow:
"""Counts events per key over a trailing window. """Counts events per key over a trailing window.
Sliding rather than a fixed window because a fixed one lets twice the limit Sliding, because a fixed window lets twice the limit through across a boundary.
through across a boundary — 10 at 14:59 and 10 at 15:00 — which for a login
limiter is the difference between the number meaning something and not.
The limit and window are SUPPLIERS, not values, so an admin saving a new number in The limit and window are suppliers, read per call from the settings cache, so a
Settings takes effect on the next attempt instead of the next deploy. They are read change in Settings applies to the next attempt.
per call, which is a dict lookup — the settings cache never touches the database.
""" """
def __init__(self, limit: Callable[[], int], window_s: Callable[[], float]) -> None: def __init__(self, limit: Callable[[], int], window_s: Callable[[], float]) -> None:
@@ -108,9 +85,8 @@ class SlidingWindow:
self._prune(key, now).append(now) self._prune(key, now).append(now)
def forget(self, key: str) -> None: def forget(self, key: str) -> None:
"""Drop a key's history. Used after a successful sign-in, so someone who """Drop a key's history. Used after a successful sign-in, so earlier failures
fumbled a password twice and then got it right starts clean rather than don't carry over."""
carrying those two for the next quarter of an hour."""
self._hits.pop(key, None) self._hits.pop(key, None)
def clear(self) -> None: def clear(self) -> None:
@@ -131,16 +107,14 @@ register_by_address = SlidingWindow(
lambda: live("register_limit_per_address"), _minutes("register_window_minutes") lambda: live("register_limit_per_address"), _minutes("register_window_minutes")
) )
# Reset emails, counted per address typed in whether or not it has an account, so # Reset emails, counted per address typed in whether or not it has an account, so
# the cap can't be used to learn which ones do (#5266). # the cap can't be used to learn which ones do.
reset_mail_by_account = SlidingWindow( reset_mail_by_account = SlidingWindow(
lambda: live("reset_emails_per_account"), _minutes("signin_window_minutes") lambda: live("reset_emails_per_account"), _minutes("signin_window_minutes")
) )
# App downloads, per account. Counted per REQUEST, so resuming a broken download # App downloads, per account, per hour. Counted per request, so each piece of a
# counts each piece; the setting's default leaves room for that. The window is fixed # resumed download counts; the default leaves room for that.
# at an hour because the setting's label says "per hour", and one number is easier
# to judge than two.
downloads_by_account = SlidingWindow(lambda: live("client_downloads_per_hour"), lambda: 3600.0) downloads_by_account = SlidingWindow(lambda: live("client_downloads_per_hour"), lambda: 3600.0)
+2 -3
View File
@@ -4,9 +4,8 @@ import uuid
from quart import jsonify from quart import jsonify
# The app's single JSON error shape + the two guards that pair with it, so every # The app's JSON error shape and the two guards that pair with it, so every blueprint
# blueprint returns errors and parses path ids the same way instead of hand-building # returns errors and parses path ids the same way.
# `jsonify({"error": ...}), code` and try/except uuid blocks at ~35 call sites.
def json_error(message: str, status: int): def json_error(message: str, status: int):
+1 -1
View File
@@ -1,4 +1,4 @@
"""Bring a fresh Inkwell back from a backup file (#5921). """Bring a fresh Inkwell back from a backup file.
docker compose exec app python -m inkwell.restore /backups/inkwell-20261016T020000Z.tar docker compose exec app python -m inkwell.restore /backups/inkwell-20261016T020000Z.tar
+19 -38
View File
@@ -2,21 +2,14 @@
Two things live here, deliberately together: Two things live here, deliberately together:
**`purge_note`** — the single definition of destroying a note. Three callers reach **`purge_note`** is the one definition of destroying a note, used by Delete forever
permanent deletion by different routes (the user's Delete forever in the web UI, in the web UI, a client's `op=delete` over sync, and the sweeper.
a client's `op=delete` over sync, and the sweeper below), and if each had its own
idea of what to tear down they would drift — one would forget the files, another
the revision history, and "permanently deleted" would quietly mean three different
things depending on how you got there.
**The sweeper** — trash that nobody empties is not free: a trashed note keeps its **The sweeper** expires trash, which still holds attachment bytes on disk. The window
attachment BYTES on disk for as long as it sits there. So trash expires. The window is the `trash_retention_days` setting (default 30, `0` = forever), re-read each pass.
is the `trash_retention_days` setting (default 30, `0` = keep forever), re-read on
every pass so a change in admin Settings takes effect without a restart.
A purged note is not a deleted ROW — it's a content-less tombstone. That's what lets A purged note is a content-less tombstone, not a deleted row, so an offline client
an offline client that reappears next month learn the note is gone instead of that comes back learns the note is gone instead of pushing it back.
faithfully resurrecting it on the next push.
""" """
from __future__ import annotations from __future__ import annotations
@@ -43,17 +36,14 @@ from .storage import unlink_media
logger = logging.getLogger(__name__) logger = logging.getLogger(__name__)
# How often the sweeper wakes. Retention is measured in days, so anything under # How often the sweeper wakes. Retention is in days, so more often only adds load.
# "a few times a day" buys nothing but load — a note trashed at 09:00 expiring at
# 14:00 rather than 09:00 thirty days later is not a difference anyone can feel.
SWEEP_INTERVAL_SECONDS = 6 * 60 * 60 SWEEP_INTERVAL_SECONDS = 6 * 60 * 60
# Let the app finish booting (migrations, first requests) before the first sweep. # Let the app finish booting (migrations, first requests) before the first sweep.
SWEEP_STARTUP_DELAY_SECONDS = 60 SWEEP_STARTUP_DELAY_SECONDS = 60
# Rows purged per transaction. A long-neglected install could have thousands of # Rows purged per transaction, so a large first sweep isn't one long transaction
# expired notes on the first sweep; committing in batches keeps that from becoming # holding locks while it deletes files.
# one enormous transaction holding locks while it deletes files.
SWEEP_BATCH = 200 SWEEP_BATCH = 200
@@ -61,11 +51,9 @@ async def purge_note(db, note: Note, edited_at: datetime | None = None) -> None:
"""Turn a note into a content-less tombstone: delete its children (and the """Turn a note into a content-less tombstone: delete its children (and the
attachment files on disk), clear its content, stamp `purged_at`. attachment files on disk), clear its content, stamp `purged_at`.
The row survives on purpose — offline clients read it off the delta feed and The row survives so offline clients learn of it from the delta feed. Everything
learn the note is gone. Everything that carries the note's CONTENT goes, and carrying the note's content goes, revisions included, since each holds a full
that includes history: a revision row holds the full body, so leaving revisions body.
behind would mean the text of a "permanently deleted" note is still on the
server, recoverable by anyone who can read the table.
""" """
atts = (await db.scalars(select(NoteAttachment).where(NoteAttachment.note_id == note.id))).all() atts = (await db.scalars(select(NoteAttachment).where(NoteAttachment.note_id == note.id))).all()
for a in atts: for a in atts:
@@ -80,12 +68,8 @@ async def purge_note(db, note: Note, edited_at: datetime | None = None) -> None:
await db.execute(sa_delete(Share).where(Share.resource_type == "note", Share.resource_id == note.id)) await db.execute(sa_delete(Share).where(Share.resource_type == "note", Share.resource_id == note.id))
note.body = "" note.body = ""
note.display_title = "" note.display_title = ""
# `deleted_at` deliberately SURVIVES. It's still true — that is when the note was # `deleted_at` stays, so every "not trashed" query (`deleted_at IS NULL`) excludes
# deleted — and keeping it means every ordinary query, present and future, that # tombstones for free. Only the Trash view has to name `purged_at`.
# says "not trashed" (`deleted_at IS NULL`) excludes tombstones for free. Clearing
# it would leave a content-less row looking like a perfectly normal active note,
# and it would surface on the board as a blank card. Only the Trash view, which
# asks for `deleted_at IS NOT NULL`, has to name `purged_at` explicitly.
note.remind_at = None note.remind_at = None
note.purged_at = datetime.now(timezone.utc) note.purged_at = datetime.now(timezone.utc)
if edited_at is not None: if edited_at is not None:
@@ -95,9 +79,8 @@ async def purge_note(db, note: Note, edited_at: datetime | None = None) -> None:
async def sweep_expired_trash(db, retention_days: int, *, now: datetime | None = None) -> int: async def sweep_expired_trash(db, retention_days: int, *, now: datetime | None = None) -> int:
"""Purge every note whose trash has expired. Returns how many were purged. """Purge every note whose trash has expired. Returns how many were purged.
Runs across ALL owners — it's a server-wide policy, not a per-user action, and Across all owners: a server-wide policy, not a user's read, so the ACL doesn't
the sweeper has no session to scope it by (rule 47 is about honoring the ACL on apply.
user-initiated reads, not about exempting rows from server maintenance).
""" """
cutoff = expired_before(now or datetime.now(timezone.utc), retention_days) cutoff = expired_before(now or datetime.now(timezone.utc), retention_days)
if cutoff is None: if cutoff is None:
@@ -110,9 +93,8 @@ async def sweep_expired_trash(db, retention_days: int, *, now: datetime | None =
.where( .where(
Note.deleted_at.is_not(None), Note.deleted_at.is_not(None),
Note.deleted_at < cutoff, Note.deleted_at < cutoff,
# Already a tombstone. Without this the purge would re-run on # Already a tombstone. Purging again would bump sync_revision
# every sweep forever, bumping sync_revision each time and # every sweep and send clients endless changes.
# handing clients an endless stream of "news" about one note.
Note.purged_at.is_(None), Note.purged_at.is_(None),
) )
.order_by(Note.deleted_at) .order_by(Note.deleted_at)
@@ -140,8 +122,7 @@ async def sweep_once() -> int:
async def run_sweeper() -> None: async def run_sweeper() -> None:
"""The background loop. Started in `before_serving`, cancelled on shutdown. """The background loop. Started in `before_serving`, cancelled on shutdown.
A sweep failure (DB blip, unreadable media directory) must never take the loop A failed sweep never ends the loop; the next pass tries the same rows again.
down with it — the next pass simply finds the same expired rows and tries again.
""" """
await asyncio.sleep(SWEEP_STARTUP_DELAY_SECONDS) await asyncio.sleep(SWEEP_STARTUP_DELAY_SECONDS)
while True: while True:
+8 -22
View File
@@ -1,25 +1,13 @@
"""When a body change is worth keeping a version of. """When a body change is worth keeping a version of.
A note's body used to snapshot into `note_revisions` on EVERY write, which made a A body change is snapshotted only if it is the first of an editing session, so a
write expensive — and the clients compensated by writing as rarely as they could client may autosave as often as it likes. A snapshot stores the body as it was
get away with, saving only when an editor closed. That is durability paying for before the edit, so each sitting yields one revision: the note as you found it.
version history: a crash mid-session lost everything typed, so that the revision
list would stay tidy. The safety property is worth more than the feature it was
subsidising.
The rule here breaks that trade. A body change earns a snapshot only if it is the It is a time rule rather than a protocol flag, and `sync.py` applies pushed bodies
first one of an editing session, so a client may write as often as it likes. through the same check.
Session granularity falls out of the window rather than being declared. A snapshot Restoring a revision always snapshots, so a restore is itself undoable.
stores the body as it was BEFORE the edit, so the first write of a sitting captures
the note as you found it and every write after it inside the window adds nothing —
one revision per sitting, with no "commit" flag for a client to send and no wire
surface to carry it. That last part is why this is a time rule and not a protocol
one: `sync.py` applies pushed bodies through the same check, so a client autosaving
every second cannot make the server snapshot every second either.
Deliberately NOT applied to restoring a revision. That is a considered act rather
than a keystroke, and it snapshots unconditionally so restoring is itself undoable.
""" """
from __future__ import annotations from __future__ import annotations
@@ -30,10 +18,8 @@ from sqlalchemy import select
from .models.note_revision import NoteRevision from .models.note_revision import NoteRevision
# How long one editing session is assumed to last. A constant rather than a setting: # How long one editing session is assumed to last: longer than a sitting, shorter
# it is not a preference anyone holds, and the value only has to be longer than a # than the gap between two.
# sitting and shorter than the gap between two of them. Promote it to the settings
# registry (rule 25) if that ever stops being true.
REVISION_WINDOW_MINUTES = 10 REVISION_WINDOW_MINUTES = 10
+7 -10
View File
@@ -1,14 +1,11 @@
"""Credentials at rest: the SMTP password and the backup bucket's secret key (#5927). """Credentials at rest: the SMTP password and the backup bucket's secret key.
The family's secret.Box (#5185): a stored credential is encrypted under a key the A stored credential is encrypted under a key the database does not hold, so a copy
database does not hold, so a copy of the database (a dump, a copied volume) cannot of the database (a dump, a copied volume) can't open anything else.
read the keys to anything else.
- **Where the key lives.** A random key in a file under DATA_DIR, made on first use. - **Where the key lives.** A random key in a file under DATA_DIR, made on first use.
Not the signing secret: by default that is kept in the settings table (rule 25, Not the signing secret, which by default sits in the settings table beside what it
no env var needed), so a key derived from it would sit beside what it seals. The would seal. Backups take only attachment files from DATA_DIR, never this file.
data volume and the database volume are separate, and backups take only the
attachment files from DATA_DIR, never this file.
- **Per purpose, per row.** The cipher key is HKDF-derived from the file's key with - **Per purpose, per row.** The cipher key is HKDF-derived from the file's key with
a purpose label, and each value is bound to its setting's name, so a sealed value a purpose label, and each value is bound to its setting's name, so a sealed value
copied onto another setting does not open there. copied onto another setting does not open there.
@@ -112,8 +109,8 @@ def seal(plain: str, owner: str) -> str:
def unseal(stored: str, owner: str) -> str | None: def unseal(stored: str, owner: str) -> str | None:
"""The plain value, or None when it can't be opened (key file lost or replaced, """The plain value, or None when it can't be opened (key file lost or replaced,
or the value was copied from another setting). A value stored before sealing or the value was copied from another setting). A value stored in the clear comes
existed comes back as it is; startup seals it (seal_stored_secrets).""" back as it is; startup seals it (seal_stored_secrets)."""
if not is_sealed(stored): if not is_sealed(stored):
return stored return stored
try: try:
+3 -4
View File
@@ -14,10 +14,9 @@ def hash_password(password: str) -> str:
return bcrypt.hashpw(password.encode("utf-8")[:_MAX_BCRYPT_BYTES], bcrypt.gensalt()).decode("utf-8") return bcrypt.hashpw(password.encode("utf-8")[:_MAX_BCRYPT_BYTES], bcrypt.gensalt()).decode("utf-8")
# A real bcrypt hash of a value nobody can present, used only to spend the time a # A real bcrypt hash of a value nobody can present, to spend the time a verification
# verification would have. Computed once at import — generating it per call would # would. Computed once at import: per call, "no such user" would be slower than a
# cost a gensalt+hash on top of the checkpw and make the "no such user" path SLOWER # real check, the same timing oracle reversed.
# than the real one, which is the same oracle pointing the other way.
_DUMMY_HASH = bcrypt.hashpw(secrets.token_bytes(32), bcrypt.gensalt()) _DUMMY_HASH = bcrypt.hashpw(secrets.token_bytes(32), bcrypt.gensalt())
+3 -5
View File
@@ -1,8 +1,6 @@
"""Shared JSON serializers — the single home for turning ORM rows into the dict """Shared JSON serializers for labels and people, so a field name or format is
shapes the API returns, so a field name or format is defined once. Each entity has a defined once. Callers add what their surface needs (a usage count in the labels API,
BASE serializer with the fields common to every context; callers layer on the sync_revision/purged_at in sync deltas)."""
extras their surface needs (usage count in the labels API, sync_revision/purged_at
in sync deltas). Labels and people so far."""
from __future__ import annotations from __future__ import annotations
from .common import iso from .common import iso
+21 -32
View File
@@ -33,11 +33,9 @@ class SettingDef:
label: str label: str
description: str description: str
group: str group: str
# Ints only. Enforced server-side in validate_updates and passed to the UI so the # Ints only. Enforced in validate_updates and passed to the UI's number input.
# number input carries them too. These exist because several of the security # Some typos are dangerous: a hop count of 50 trusts anything a caller sends, and
# values have ranges where a typo is not merely wrong but dangerous — a proxy hop # a sign-in limit of 0 locks every account out.
# count of 50 would trust anything a caller sent, and a sign-in limit of 0 would
# lock every account out permanently.
minimum: int | None = None minimum: int | None = None
maximum: int | None = None maximum: int | None = None
# Strings only: the values allowed, shown as a choice rather than a text field. # Strings only: the values allowed, shown as a choice rather than a text field.
@@ -46,7 +44,7 @@ class SettingDef:
# value is shown, which reads fine for "Inkwell" and badly for "starttls". # value is shown, which reads fine for "Inkwell" and badly for "starttls".
choice_labels: tuple[str, ...] | None = None choice_labels: tuple[str, ...] | None = None
# A credential. The admin API never sends its value back, only whether one is # A credential. The admin API never sends its value back, only whether one is
# set, and saving it empty leaves it as it was (#5266). # set, and saving it empty leaves it as it was.
secret: bool = False secret: bool = False
# An http(s) address, stored without a trailing slash. # An http(s) address, stored without a trailing slash.
url: bool = False url: bool = False
@@ -128,10 +126,10 @@ REGISTRY: list[SettingDef] = [
"The server contacts the linked site; private/internal addresses are always blocked.", "The server contacts the linked site; private/internal addresses are always blocked.",
"Links", "Links",
), ),
# --- Email (#5266) --------------------------------------------------------------- # --- Email ---------------------------------------------------------------------
# #
# Only for password reset links so far. Off until a server, a sender and the # For reset links and invites. Off until a server, a sender and the public address
# public address are all set (`mailer.mail_settings`). # are all set (`mailer.mail_settings`).
SettingDef("smtp_host", "string", "", "SMTP server", "Leave empty to turn email off.", "Email"), SettingDef("smtp_host", "string", "", "SMTP server", "Leave empty to turn email off.", "Email"),
SettingDef( SettingDef(
"smtp_port", "smtp_port",
@@ -158,7 +156,7 @@ REGISTRY: list[SettingDef] = [
"smtp_password", "string", "", "SMTP password", "Kept on the server and never shown again.", "Email", secret=True "smtp_password", "string", "", "SMTP password", "Kept on the server and never shown again.", "Email", secret=True
), ),
SettingDef("smtp_from", "string", "", "From address", "Who emails come from. Defaults to the username.", "Email"), SettingDef("smtp_from", "string", "", "From address", "Who emails come from. Defaults to the username.", "Email"),
# --- Backups (#5918) ------------------------------------------------------------ # --- Backups --------------------------------------------------------------------
# #
# A dated backup file each day, to a folder or an S3-compatible bucket # A dated backup file each day, to a folder or an S3-compatible bucket
# (backup_job.py). Off until switched on, because a folder has to be mounted or a # (backup_job.py). Off until switched on, because a folder has to be mounted or a
@@ -249,9 +247,8 @@ REGISTRY: list[SettingDef] = [
), ),
# --- Security ----------------------------------------------------------------- # --- Security -----------------------------------------------------------------
# #
# Read on paths too hot for a database round trip (the credential throttle checks # Cached (see `live()` below): the credential throttle reads them before opening
# them BEFORE opening a connection, which is the point of checking a throttle # a database connection.
# before doing expensive work), so they are cached — see `live()` below.
SettingDef( SettingDef(
"trusted_proxy_hops", "trusted_proxy_hops",
"int", "int",
@@ -370,8 +367,7 @@ def _coerce(defn: SettingDef, raw: Any) -> Any:
n = int(raw) n = int(raw)
except (ValueError, TypeError): except (ValueError, TypeError):
return defn.default return defn.default
# A value saved before its bounds existed is still in the row; read it as the # A stored value outside its bounds is read as the nearest bound.
# nearest bound rather than let it act (#5384).
if defn.minimum is not None: if defn.minimum is not None:
n = max(n, defn.minimum) n = max(n, defn.minimum)
if defn.maximum is not None: if defn.maximum is not None:
@@ -437,11 +433,10 @@ async def get_public_config(db) -> dict:
"site_name": await get_setting(db, "site_name"), "site_name": await get_setting(db, "site_name"),
"allow_registration": await get_setting(db, "allow_registration"), "allow_registration": await get_setting(db, "allow_registration"),
"enable_url_unfurl": await get_setting(db, "enable_url_unfurl"), "enable_url_unfurl": await get_setting(db, "enable_url_unfurl"),
# Server policy, not user data: clients need it to say how long a note has # Clients show how long a note has left in Trash, and a native client reads it
# left in Trash. A native client also reads it BEFORE linking, which is why # before linking, so it is on the unauthenticated config.
# it belongs on the unauthenticated config rather than behind login.
"trash_retention_days": await get_setting(db, "trash_retention_days"), "trash_retention_days": await get_setting(db, "trash_retention_days"),
# Whether the sign-in screen offers "Forgot password?" (#5266). # Whether the sign-in screen offers "Forgot password?".
"password_reset_by_email": await mail_configured(db), "password_reset_by_email": await mail_configured(db),
"min_password_length": MIN_PASSWORD_LEN, "min_password_length": MIN_PASSWORD_LEN,
} }
@@ -482,16 +477,11 @@ async def get_admin_settings(db) -> list[dict]:
return result return result
# Settings the app must be able to read WITHOUT awaiting a database. # Settings read without awaiting a database: the credential throttle needs them
# before opening a connection, so a refused attempt costs nothing.
# #
# The credential throttle consults these before opening a connection — deliberately, # Seeded from the registry defaults (unit tests run with no Postgres), then refreshed
# because a refused attempt is supposed to cost nothing, and the proxy hop count is # from the database at boot and whenever an admin saves.
# needed to know who is even asking. A per-request query would undo both.
#
# Seeded from the registry defaults so the app works before (and without) a database —
# unit tests construct it with no Postgres at all — then refreshed from the DB at boot
# and again whenever an admin saves. Same live-update contract `session_ttl_days`
# already has in settings_api.py.
_LIVE_KEYS = ( _LIVE_KEYS = (
# Read by GET /install.sh, which writes it into the installer it serves # Read by GET /install.sh, which writes it into the installer it serves
# (installer.py) and must not need a database to answer. # (installer.py) and must not need a database to answer.
@@ -516,7 +506,7 @@ def live(key: str) -> Any:
def apply_session_ttl(app, days) -> None: def apply_session_ttl(app, days) -> None:
"""Sessions signed from now on last `days` days. At boot and on every admin save, """Sessions signed from now on last `days` days. At boot and on every admin save,
so a changed lifetime applies without a restart (rule 25).""" so a changed lifetime applies without a restart."""
app.config["PERMANENT_SESSION_LIFETIME"] = timedelta(days=int(days)) app.config["PERMANENT_SESSION_LIFETIME"] = timedelta(days=int(days))
@@ -545,9 +535,8 @@ def validate_updates(updates: dict) -> tuple[dict, str | None]:
n = int(val) n = int(val)
except (ValueError, TypeError): except (ValueError, TypeError):
return {}, f"{defn.label} must be a whole number" return {}, f"{defn.label} must be a whole number"
# Rejected rather than clamped: silently accepting a number and storing a # Rejected rather than clamped, so nobody believes a protection is set to
# different one is how somebody ends up believing a protection is set to # a number it isn't.
# something it is not.
if defn.minimum is not None and n < defn.minimum: if defn.minimum is not None and n < defn.minimum:
return {}, f"{defn.label} must be at least {defn.minimum}" return {}, f"{defn.label} must be at least {defn.minimum}"
if defn.maximum is not None and n > defn.maximum: if defn.maximum is not None and n > defn.maximum:
+4 -5
View File
@@ -46,13 +46,12 @@ async def update_settings():
# Nothing is saved: a credential is never stored unsealed by a save. # Nothing is saved: a credential is never stored unsealed by a save.
return json_error(f"Couldn't encrypt the password or key to store it. {exc}", 500) return json_error(f"Couldn't encrypt the password or key to store it. {exc}", 500)
await db.commit() await db.commit()
# Re-read the cached security values so a saved limit or hop count applies to # Re-read the cached security values so a saved limit applies to the next
# the very next request. Unconditional: cheap, and a conditional here would be # request.
# one more place that has to know which keys are hot.
await refresh_live(db) await refresh_live(db)
result = await get_admin_settings(db) result = await get_admin_settings(db)
# Apply the live-tunable knob without a restart (rule 25). # Apply the session lifetime without a restart.
if "session_ttl_days" in clean: if "session_ttl_days" in clean:
apply_session_ttl(current_app, clean["session_ttl_days"]) apply_session_ttl(current_app, clean["session_ttl_days"])
@@ -63,7 +62,7 @@ async def update_settings():
@require_admin @require_admin
async def test_email(): async def test_email():
"""Send a test message to the signed-in admin with the SAVED email settings, so """Send a test message to the signed-in admin with the SAVED email settings, so
"Send test email" checks what reset links will actually use (#5266). An admin is "Send test email" checks what reset links will actually use. An admin is
shown the server's own error, which is what fixing the settings takes.""" shown the server's own error, which is what fixing the settings takes."""
async with session_scope() as db: async with session_scope() as db:
cfg = await mail_settings(db) cfg = await mail_settings(db)
+2 -2
View File
@@ -1,4 +1,4 @@
"""What sharing a note does to the change feed (#5175). """What sharing a note does to the change feed.
The feed carries every note a person can see, so a share changing has to show up in The feed carries every note a person can see, so a share changing has to show up in
it: granting or changing a share moves the note past each device's cursor, and ending it: granting or changing a share moves the note past each device's cursor, and ending
@@ -72,7 +72,7 @@ async def granted(db, note_id: uuid.UUID, user_id: uuid.UUID) -> None:
await bump_note(db, note_id) await bump_note(db, note_id)
# --- Group membership (#5177) ------------------------------------------------ # --- Group membership --------------------------------------------------------
# #
# A note shared with a group reaches whoever is in it NOW, so joining or leaving # A note shared with a group reaches whoever is in it NOW, so joining or leaving
# changes what that one person can see without any share changing. Their devices # changes what that one person can see without any share changing. Their devices
+5 -6
View File
@@ -1,9 +1,8 @@
"""Sharing a note with another member of this instance (#5174). """Sharing a note with another member of this instance.
`acl.visible_to_user` has gated every read since M0, but nothing wrote a share. This The note's owner lists, grants and revokes access, and anyone signed in can read the
is the writing half: the note's owner lists, grants and revokes access, and anyone member directory the Share dialog picks people from. A share goes to one person or
signed in can read the member directory the Share dialog picks people from. A share to a group the admin made. Reads are gated by `acl.visible_to_user`.
goes to one person or to a group the admin made (#5177).
What a share grants is in `acl` (view, or edit: the body and checklist) and is What a share grants is in `acl` (view, or edit: the body and checklist) and is
enforced by the note routes (`notes.helpers._get_owned` / `_get_editable`). enforced by the note routes (`notes.helpers._get_owned` / `_get_editable`).
@@ -50,7 +49,7 @@ async def _member_counts(db, group_ids: list) -> dict:
@login_required @login_required
async def directory(): async def directory():
"""Who on this instance a note can be shared with: everyone but you, and every """Who on this instance a note can be shared with: everyone but you, and every
group the admin has made (#5177). group the admin has made.
Signed-in only. Each person is a name and an email and nothing more, which is Signed-in only. Each person is a name and an email and nothing more, which is
what choosing someone takes and all a fellow member needs to know; a group is its what choosing someone takes and all a fellow member needs to know; a group is its
+4 -6
View File
@@ -1,4 +1,4 @@
"""How much attachment storage an account may use (#2939 §4). """How much attachment storage an account may use.
`max_attachment_mb` caps one file; this caps an account's total, so one account `max_attachment_mb` caps one file; this caps an account's total, so one account
can't fill the volume. The limit is Settings → General → `storage_quota_gb`; can't fill the volume. The limit is Settings → General → `storage_quota_gb`;
@@ -7,11 +7,9 @@ can't fill the volume. The limit is Settings → General → `storage_quota_gb`;
What counts is every attachment on every note the account owns, trashed notes What counts is every attachment on every note the account owns, trashed notes
included: their files stay on disk until the trash is emptied. included: their files stay on disk until the trash is emptied.
An upload that would go over the limit is answered 507 Insufficient Storage, not An upload over the limit is answered 507, not 413: native clients never retry a 4xx
413. The native clients treat a 4xx as permanent and never retry the file (`client::upload_attachment` in the core), and freeing space should let the file
(`client::upload_attachment` in the core), and a 5xx as worth another try. Over the through on the next sync.
limit is neither: freeing space should let the file through on the next sync, with
nothing for the person to redo.
The one way an attachment's file leaves the disk is here too (`unlink_media`). The one way an attachment's file leaves the disk is here too (`unlink_media`).
""" """
+21 -42
View File
@@ -1,9 +1,9 @@
"""Delta-sync API for the local-first native clients (M8 sync hub). """Delta-sync API for the local-first native clients.
Pull: `GET /api/sync/changes?since=<cursor>` returns every note + label the caller Pull: `GET /api/sync/changes?since=<cursor>` returns every note + label the caller
owns whose sync_revision advanced past the cursor, newest-revision last, paginated. owns whose sync_revision advanced past the cursor, newest-revision last, paginated.
With `shares=1` it also carries the notes shared with the caller, and `revoked`: the With `shares=1` it also carries the notes shared with the caller, and `revoked`: the
notes that stopped being shared with them (#5175). notes that stopped being shared with them.
Notes and labels both draw from ONE shared sequence (sync_revision_seq), so the Notes and labels both draw from ONE shared sequence (sync_revision_seq), so the
cursor is a single monotonic watermark across both entity types. cursor is a single monotonic watermark across both entity types.
@@ -56,18 +56,14 @@ MAX_PUSH = 1000 # per-batch change cap
# SYNC_PROTOCOL_VERSION what this server speaks. # SYNC_PROTOCOL_VERSION what this server speaks.
# MIN_CLIENT_PROTOCOL_VERSION the oldest client protocol it still accepts. # MIN_CLIENT_PROTOCOL_VERSION the oldest client protocol it still accepts.
# #
# Bump SYNC_PROTOCOL_VERSION for ANY wire change. Raise the floor only for a # Bump SYNC_PROTOCOL_VERSION for any wire change. Raise the floor only for a breaking
# BREAKING one: it hard-blocks older clients, so additive changes leave it alone. # one, since it blocks older clients. docs/sync.md ("The policy") lists each version.
# What each version changed, and why the floor moved or didn't, is in docs/sync.md
# ("The policy").
SYNC_PROTOCOL_VERSION = 7 SYNC_PROTOCOL_VERSION = 7
MIN_CLIENT_PROTOCOL_VERSION = 3 MIN_CLIENT_PROTOCOL_VERSION = 3
# Named capabilities beyond the base protocol. An ADDITIVE change earns a name # Named capabilities beyond the base protocol. An additive change gets a name here
# here rather than a min-version bump, so a newer client meeting an older server # rather than a floor bump, so a newer client meeting an older server loses a feature
# can degrade to "some features unavailable" instead of refusing to sync. Clients # instead of refusing to sync. Clients test for the name, never a version number.
# test for the name, never infer a capability from a version number — that's what
# keeps feature gating independent of release lockstep.
SYNC_FEATURES: tuple[str, ...] = ( SYNC_FEATURES: tuple[str, ...] = (
"notes", # note delta sync (pull + push) "notes", # note delta sync (pull + push)
"labels", # the label catalog as its own entity "labels", # the label catalog as its own entity
@@ -83,8 +79,8 @@ SYNC_FEATURES: tuple[str, ...] = (
def protocol_advertisement() -> dict: def protocol_advertisement() -> dict:
"""What the server publishes about the sync protocol, merged into `/api/config`. """What the server publishes about the sync protocol, merged into `/api/config`.
DB-free and unauthenticated on purpose: a client has to be able to ask "can I DB-free and unauthenticated: a client asks before it holds a device token or an
talk to you at all?" before it holds a device token — or even has an account. account.
""" """
return { return {
"sync_protocol_version": SYNC_PROTOCOL_VERSION, "sync_protocol_version": SYNC_PROTOCOL_VERSION,
@@ -131,9 +127,8 @@ def _page_cursor(streams: list[list[int]], since: int, limit: int) -> tuple[int,
async def changes(): async def changes():
since = _parse_since(request.args.get("since")) since = _parse_since(request.args.get("since"))
limit = _clamp_limit(request.args.get("limit")) limit = _clamp_limit(request.args.get("limit"))
# `shares=1` is a client that understands shared notes (protocol 6, `shares`). # `shares=1` is a client that understands shared notes (protocol 6). Without it a
# Anything older gets only its own notes, as before, rather than someone else's # client gets only its own notes, since it would treat others' as its own.
# notes it would treat as its own: pinning them, labelling them, pushing them.
with_shares = request.args.get("shares") == "1" with_shares = request.args.get("shares") == "1"
if with_shares: if with_shares:
visible = note_visible() visible = note_visible()
@@ -143,7 +138,7 @@ async def changes():
# Every note the caller can see, in any state (active/archived/trash/purged), # Every note the caller can see, in any state (active/archived/trash/purged),
# since a client mirrors everything; ordered by the shared revision. On a note # since a client mirrors everything; ordered by the shared revision. On a note
# shared with the caller, pinning it moves their own state row and not the # shared with the caller, pinning it moves their own state row and not the
# note, so its revision here is whichever of the two moved last (#5176). # note, so its revision here is whichever of the two moved last.
revision = revision_for() revision = revision_for()
note_rows = ( note_rows = (
await db.execute( await db.execute(
@@ -238,14 +233,6 @@ def _assign_note_fields(note: Note, ch: dict) -> None:
note.position = ch["position"] note.position = ch["position"]
# `_first_item_text` and `_apply_note_items` lived here until M304. Both existed for
# one reason — a checklist was a table beside the body — and both are gone with it. A
# pushed change carries its items as `- [ ] ` lines inside `body`, so applying them is
# applying the body, and naming the note is reading its first line. A client that still
# sends an `items` array is a v2 client, and the version floor below turns it away
# before any of this runs.
async def _apply_note_manual_labels(db, note: Note, ch: dict) -> None: async def _apply_note_manual_labels(db, note: Note, ch: dict) -> None:
"""Set the note's MANUAL (picker) label memberships from client label_ids, leaving """Set the note's MANUAL (picker) label memberships from client label_ids, leaving
tag-sourced (via_tag) rows to _lift_and_reconcile_tags. Only labels the caller owns count.""" tag-sourced (via_tag) rows to _lift_and_reconcile_tags. Only labels the caller owns count."""
@@ -307,12 +294,9 @@ async def _apply_note(db, ch: dict, previews: list[tuple[uuid.UUID, str]]) -> di
note.purged_at = None # client re-created/edited → clear the tombstone note.purged_at = None # client re-created/edited → clear the tombstone
_assign_note_fields(note, ch) _assign_note_fields(note, ch)
# Non-destructive LWW: the overwritten server body snapshots into history, subject # The overwritten server body snapshots into history under the same session window
# to the same session window as a direct edit (revisions.should_snapshot). This # as a direct edit (revisions.should_snapshot), so a client autosaving every second
# path is why the window is a time rule rather than a flag on the wire: a client # doesn't make the server snapshot every second.
# autosaving every second pushes a body change every second, and without the
# check the SERVER would snapshot each one no matter how restrained the client's
# own store was being.
body = ch["body"] if isinstance(ch.get("body"), str) else "" body = ch["body"] if isinstance(ch.get("body"), str) else ""
if await write_body(db, note, body, snapshot="never" if creating else "session"): if await write_body(db, note, body, snapshot="never" if creating else "session"):
# A note pushed from a linked client gets the same link previews as one typed # A note pushed from a linked client gets the same link previews as one typed
@@ -327,7 +311,7 @@ async def _apply_note(db, ch: dict, previews: list[tuple[uuid.UUID, str]]) -> di
async def _apply_shared_note(db, note: Note, ch: dict, edited_at, previews: list[tuple[uuid.UUID, str]]) -> dict: async def _apply_shared_note(db, note: Note, ch: dict, edited_at, previews: list[tuple[uuid.UUID, str]]) -> dict:
"""Apply a pushed change to a note someone else owns (#5175, #5176). """Apply a pushed change to a note someone else owns.
Two separate things can arrive, each under its own last-write-wins: Two separate things can arrive, each under its own last-write-wins:
@@ -436,10 +420,8 @@ _CHILD_ENTITIES = {"attachment": NoteAttachment, "preview": NoteLinkPreview}
async def _apply_child_delete(db, ch: dict, removed_files: list[str]) -> dict: async def _apply_child_delete(db, ch: dict, removed_files: list[str]) -> dict:
"""Delete one attachment or link preview the client removed. """Delete one attachment or link preview the client removed.
Not subject to last-write-wins. LWW settles which of two versions of the SAME Not subject to last-write-wins: nothing can re-create a removed id, so there is no
thing survives; a removal has no rival version — nothing can re-create an id once rival version, and a removal that lost to a newer note edit would come back.
it is gone — so it always applies. That is what makes a deletion stick: a removal
that lost to a newer edit of the note would come straight back on the next pull.
A row this caller cannot see answers `noop`, exactly like one that never existed, A row this caller cannot see answers `noop`, exactly like one that never existed,
so the reply cannot be used to learn whether someone else's id is real. so the reply cannot be used to learn whether someone else's id is real.
@@ -494,10 +476,8 @@ async def push():
else: else:
results.append({"id": ch.get("id"), "status": "rejected", "error": "unknown entity"}) results.append({"id": ch.get("id"), "status": "rejected", "error": "unknown entity"})
await db.commit() await db.commit()
# After the batch commits, never inside it: the fetch reads each note in a # After the batch commits: the fetch reads each note in its own session.
# session of its own, and these were scheduled mid-batch until #5164, where a # Scheduled, not awaited, so a push never waits on somebody else's website.
# fast fetch could look for a note that was not committed yet. Scheduled rather
# than awaited — a push must not wait on somebody else's website.
for note_id, text in previews: for note_id, text in previews:
schedule_unfurls(note_id, text) schedule_unfurls(note_id, text)
# Files go once their rows have committed, so a failed batch never leaves a row # Files go once their rows have committed, so a failed batch never leaves a row
@@ -513,8 +493,7 @@ async def put_attachment(att_id: str):
"""Upload a file a client attached while offline, under the id it chose there. """Upload a file a client attached while offline, under the id it chose there.
The body is the file's raw bytes; `note_id`, `filename` and `sha256` ride in the The body is the file's raw bytes; `note_id`, `filename` and `sha256` ride in the
query and the type in Content-Type. Raw rather than multipart so a native client query and the type in Content-Type, so a native client needs no form encoder.
needs no form encoder for what is one blob and three strings.
Idempotent: re-sending an id the note already holds answers 200 and changes Idempotent: re-sending an id the note already holds answers 200 and changes
nothing, which is what lets a client retry an upload whose reply it never got. nothing, which is what lets a client retry an upload whose reply it never got.
+4 -5
View File
@@ -1,8 +1,7 @@
"""SSRF-hardened link unfurling — fetch a URL server-side and extract an OG/meta """SSRF-hardened link unfurling: fetch a URL server-side and extract an OG/meta
preview. Dependency-free (stdlib only), matching the project's hand-rolled ethos. preview, with the standard library only.
The server fetching arbitrary user-supplied URLs is a classic SSRF surface, so the Fetching user-supplied URLs is an SSRF surface, so the defences are layered:
defenses are deliberate and layered:
- http/https only (no file://, gopher://, …). - http/https only (no file://, gopher://, …).
- Resolve the host and require EVERY resolved address to be public — reject - Resolve the host and require EVERY resolved address to be public — reject
private / loopback / link-local / reserved / multicast / unspecified ranges. private / loopback / link-local / reserved / multicast / unspecified ranges.
@@ -11,7 +10,7 @@ defenses are deliberate and layered:
can't slip through. can't slip through.
- At most 3 redirects, each hop re-validated the same way. - At most 3 redirects, each hop re-validated the same way.
- 5s timeout, 512 KB body cap, text/html only. - 5s timeout, 512 KB body cap, text/html only.
No AI — this just parses OpenGraph/Twitter/`<title>` meta tags. It parses OpenGraph, Twitter and `<title>` meta tags.
""" """
from __future__ import annotations from __future__ import annotations
+8 -22
View File
@@ -1,25 +1,13 @@
"""Unfurling a note's URLs in the background, after the note is already saved. """Unfurling a note's URLs in the background, after the note is already saved.
## Why this is not done inline A save never waits on a fetch from a host nobody controls; the preview catches up.
Capture speed is the product. Unfurling is a 5-second-timeout network call to a host It runs on the server, which sees every note from every surface. A linked desktop or
nobody controls, and a note must persist the instant someone stops typing — so the Android client gets the preview on its next pull; an unlinked one has none until it
save returns first and the preview catches up. A person who pastes a link and closes links.
the composer has already done the thing they came to do.
## Why it is on the server rather than in each client A preview that can't be fetched fails quietly: the note just has no card, and the
link is still in the body.
The server sees every note that reaches it, from all three surfaces, so detection and
fetching live in one place instead of three. A linked desktop or Android client pushes
its note and picks the preview up on the next pull; an unlinked one has no server to
ask and simply has no preview until it links, which is the honest consequence of being
offline rather than a gap to paper over.
## What it deliberately does not do
Fail loudly. A preview that could not be fetched is not an error the person needs —
the note is fine, it just has no card. The link is still in the body, still clickable,
still searchable.
""" """
from __future__ import annotations from __future__ import annotations
@@ -39,10 +27,8 @@ from .unfurl import UnfurlError, unfurl
logger = logging.getLogger(__name__) logger = logging.getLogger(__name__)
# Matches the frontend's detector (NoteEditor.vue) so both surfaces agree on what # Matches the frontend's detector (NoteEditor.vue). Trailing sentence punctuation is
# counts as a link. Trailing sentence punctuation is stripped below rather than in the # stripped below rather than in the pattern, since a URL may end in those characters.
# pattern — a URL can legitimately end in most of these characters, just not when the
# sentence does.
_URL_RE = re.compile(r"(https?://[^\s<>\"'\])]+)") _URL_RE = re.compile(r"(https?://[^\s<>\"'\])]+)")
# Per note, per save. A body pasted full of links should not turn into a burst of # Per note, per save. A body pasted full of links should not turn into a burst of
+2 -3
View File
@@ -6,9 +6,8 @@ from inkwell.models.user import User
def test_visible_to_user_builds_owner_or_shared_predicate(): def test_visible_to_user_builds_owner_or_shared_predicate():
"""DB-free smoke test: the predicate compiles and references both the direct """DB-free smoke test: the predicate compiles and references both the direct
(shares) and group (group_members) grant paths. Functional row-visibility is (shares) and group (group_members) grant paths. Row visibility against a real
verified against a live database in M1, once notes are a real shareable database is in test_integration.py.
resource (family practice: DB-backed tests run against the dev image, not CI).
""" """
uid = uuid.uuid4() uid = uuid.uuid4()
# User stands in as a shareable resource purely to exercise the SQL builder. # User stands in as a shareable resource purely to exercise the SQL builder.
+2 -5
View File
@@ -31,11 +31,8 @@ async def test_unknown_api_route_404s(app):
# --- the version a running server reports ------------------------------------ # --- the version a running server reports ------------------------------------
# #
# Note 3127 §5 removed version tags, so this string is the only answer to "which # The only answer to "which build is this?", so the fallback matters most: it must
# build is this?" and nothing exists to contradict it when it is wrong. That makes # be "unknown", never a plausible-looking `__version__`.
# the FALLBACK the interesting case rather than the happy path: it used to be
# `__version__`, so a server run from a checkout reported `0.2.0` — a real-looking
# version naming no build anybody could obtain.
async def reported_version() -> str: async def reported_version() -> str:
+2 -3
View File
@@ -1,8 +1,7 @@
"""The channel -> release tag mapping — `packaging/channel-tag.sh`. """The channel -> release tag mapping — `packaging/channel-tag.sh`.
A channel's release tag must never be spelled like a branch. The dev channel's tag A channel's release tag must never be spelled like a branch: a `dev` tag shadows
was `dev`, which shadowed the `dev` branch: once a clone had fetched tags, the `dev` branch, and `git push origin dev` then fails as an ambiguous refspec.
`git push origin dev` failed as an ambiguous refspec (Scribe #2184).
These run the script itself rather than reading it, so a mapping that is correct These run the script itself rather than reading it, so a mapping that is correct
on the page but broken in execution still fails here. on the page but broken in execution still fails here.
+16 -34
View File
@@ -14,28 +14,17 @@ from inkwell.client_dist import (
) )
from inkwell.config import Config from inkwell.config import Config
# DB-free, like the rest of this suite — the test lane runs no Postgres. That is # DB-free, so the advertisement is asserted through `advertisement()` rather than
# why the advertisement is asserted through `advertisement()` rather than through # `/api/config`, whose other half needs a database. The routes below are exercised:
# `/api/config`: the route is a one-line merge of this dict into a payload whose # none opens a session.
# other half needs a database, and testing it here tests the part that can be wrong.
#
# The routes below ARE exercised, because none opens a session: the metadata route
# only stats files, and the download's 401 is returned before any token lookup.
PAYLOAD = b"not really a client, but the server only ever stats it" PAYLOAD = b"not really a client, but the server only ever stats it"
# Every test that is about the MECHANISM rather than about one platform runs # Tests about the mechanism rather than one platform run against every platform.
# against all of them. The bugs this module can have — a sidecar describing a
# different build, a half-finished copy shadowing a good one — are not
# platform-specific, and a suite that only ever exercised Android is how the other
# four would ship untested.
ALL_IDS = [p.id for p in PLATFORMS] ALL_IDS = [p.id for p in PLATFORMS]
# `version_code` is "whatever this platform's comparator reads", and that is not one # `version_code` is an integer for Android and Tauri's semver key for the desktop,
# type. Android's install gate compares an integer; the desktop's updater compares # so the tests write both shapes.
# Tauri's semver key. The tests carry both shapes for the same reason the module
# does — a suite that only ever wrote integers would pass while every desktop
# sidecar CI writes was being rejected.
ANDROID_CODE = 3503708 ANDROID_CODE = 3503708
DESKTOP_CODE = "1.0.3503707" DESKTOP_CODE = "1.0.3503707"
@@ -58,10 +47,8 @@ def coded(platform_id: str, value: int):
def _empty_baked_client(tmp_path, monkeypatch): def _empty_baked_client(tmp_path, monkeypatch):
"""Point the baked-in copy at an empty directory. """Point the baked-in copy at an empty directory.
In a source checkout `src/inkwell/client/` does not exist, so these tests A built image has real clients in `src/inkwell/client/`, so the tests must not
would pass anyway — but only by accident of where they are run. A built image depend on which tree they run in.
has real clients there, and a test that silently depends on which tree it is in
is one that will eventually lie.
""" """
monkeypatch.setattr(client_dist, "BAKED_ROOT", tmp_path / "baked") monkeypatch.setattr(client_dist, "BAKED_ROOT", tmp_path / "baked")
yield yield
@@ -121,8 +108,7 @@ def test_the_android_names_are_the_ones_the_image_build_writes():
with nothing checking them against each other, so a rename on one side reads as with nothing checking them against each other, so a rename on one side reads as
"no Android client" on the other rather than as an error. "no Android client" on the other rather than as an error.
Phones never ask for these names; they poll `/api/client/android`. The names Phones never ask for these names; they poll `/api/client/android`.
changed once, with the rename to Inkwell (Scribe note 5071).
""" """
assert BY_ID["android"].artifact == "inkwell.apk" assert BY_ID["android"].artifact == "inkwell.apk"
assert BY_ID["android"].sidecar == "inkwell-android.json" assert BY_ID["android"].sidecar == "inkwell-android.json"
@@ -139,8 +125,7 @@ def test_an_absent_client_is_no_client(platform_id):
def test_a_server_holding_nothing_advertises_no_keys_at_all(): def test_a_server_holding_nothing_advertises_no_keys_at_all():
"""The KEYS are missing, not null. """The KEYS are missing, not null.
A client testing for one then gets an unambiguous answer rather than having to A client testing for one gets an unambiguous answer.
tell "this server has no client" apart from "this server predates the feature".
""" """
assert releases() == {} assert releases() == {}
assert advertisement() == {} assert advertisement() == {}
@@ -234,7 +219,7 @@ def test_a_present_client_reports_what_a_comparator_reads(platform_id):
assert not found["url"].startswith("http") assert not found["url"].startswith("http")
def test_the_android_payload_still_carries_every_field_it_used_to(): def test_the_android_payload_carries_every_field_phones_read():
"""Phones in the field parse this. Fields may be ADDED — `ClientRelease` in """Phones in the field parse this. Fields may be ADDED — `ClientRelease` in
core/src/sync/client.rs is a plain serde struct and ignores what it does not core/src/sync/client.rs is a plain serde struct and ignores what it does not
know — but none of these may move or change meaning.""" know — but none of these may move or change meaning."""
@@ -303,15 +288,13 @@ def test_a_dropped_in_build_beats_the_one_the_image_shipped():
"""Someone who deliberately put a build on the volume wants that build.""" """Someone who deliberately put a build on the volume wants that build."""
place("android", root=client_dist.BAKED_ROOT, version_code=300) place("android", root=client_dist.BAKED_ROOT, version_code=300)
place("android", version_code=99) place("android", version_code=99)
# Lower version and all — precedence is about intent, not about newness. An # Even at a lower version: dropping in an older client is deliberate.
# operator pinning an older client is doing it on purpose.
assert release("android")["version_code"] == 99 assert release("android")["version_code"] == 99
def test_precedence_is_decided_per_platform_not_for_the_whole_set(): def test_precedence_is_decided_per_platform_not_for_the_whole_set():
"""THE trap this table introduces. Dropping in one client must not retract the """Dropping in one client must not retract the other four, as "first directory
other four — "first directory holding anything wins" would mean overriding the holding anything wins" would."""
APK silently takes the desktop downloads offline."""
for platform_id in ALL_IDS: for platform_id in ALL_IDS:
place(platform_id, root=client_dist.BAKED_ROOT, version_code=300) place(platform_id, root=client_dist.BAKED_ROOT, version_code=300)
place("android", version_code=99) place("android", version_code=99)
@@ -326,8 +309,7 @@ def test_precedence_is_decided_per_platform_not_for_the_whole_set():
def test_a_broken_drop_in_does_not_shadow_the_baked_copy(): def test_a_broken_drop_in_does_not_shadow_the_baked_copy():
"""A half-finished copy onto the volume must not take the app offline. """A half-finished copy onto the volume must not take the app offline.
This is the failure the copy-order advice in docs/android-distribution.md is The failure the copy-order advice in docs/android-distribution.md is about.
about, and the server should ride it out rather than go dark.
""" """
place("android", root=client_dist.BAKED_ROOT, version_code=300) place("android", root=client_dist.BAKED_ROOT, version_code=300)
place("android", size=999_999) # sidecar describing a different build place("android", size=999_999) # sidecar describing a different build
@@ -424,7 +406,7 @@ async def test_a_server_without_the_appimage_404s_its_updater_manifest(app):
assert resp.status_code == 404 assert resp.status_code == 404
# --- serving the bytes (#5118, family idea #5103 practice 6) ----------------- # --- serving the bytes ------------------------------------------------------
# #
# `send_artifact` is called directly in a request context: the route in front of it # `send_artifact` is called directly in a request context: the route in front of it
# needs a signed-in account, and this suite has no database to sign one in with. # needs a signed-in account, and this suite has no database to sign one in with.
+5 -7
View File
@@ -34,18 +34,16 @@ async def test_revoke_self_requires_auth(app):
def test_revoke_self_routes_to_the_self_revoke_view(app): def test_revoke_self_routes_to_the_self_revoke_view(app):
# The static rule must win over `/devices/<device_id>`: if "self" fell through to # The static rule must win over `/devices/<device_id>`, or "self" would be read as
# the id-keyed route it would be read as a malformed UUID and answer 404. Checked # a malformed UUID and 404. Checked on the URL map, because every route behind
# on the URL map, because every route behind login_required now reads the # login_required reads the database. The view's 400 is in test_integration.py.
# account's session epoch (#5173). The view's 400 for a caller with no bearer
# token is in test_integration.py.
endpoint, _ = app.url_map.bind("localhost").match("/api/auth/devices/self", method="DELETE") endpoint, _ = app.url_map.bind("localhost").match("/api/auth/devices/self", method="DELETE")
assert endpoint == "auth.revoke_own_device" assert endpoint == "auth.revoke_own_device"
async def test_device_login_validates_input(app): async def test_device_login_validates_input(app):
# Missing credentials → 400 BEFORE any DB access, so it's checkable in the # Missing credentials → 400 before any DB access, so it's checkable without a
# DB-free unit lane (invalid-cred and success paths are operator-verified). # database. The other paths are in test_integration.py.
client = app.test_client() client = app.test_client()
resp = await client.post("/api/auth/device-login", json={}) resp = await client.post("/api/auth/device-login", json={})
assert resp.status_code == 400 assert resp.status_code == 400
+1 -1
View File
@@ -76,7 +76,7 @@ def test_a_script_without_exactly_one_placeholder_is_not_served(script):
def test_the_public_address_setting_beats_the_request_host(): def test_the_public_address_setting_beats_the_request_host():
"""Behind a reverse proxy the Host this request arrived with may be an internal """Behind a reverse proxy the Host this request arrived with may be an internal
one; Settings → Public address is what the operator says people use.""" one; Settings → Public address is the one people use."""
assert server_address("https://notes.example.com", "http", "inkwell:5000") == "https://notes.example.com" assert server_address("https://notes.example.com", "http", "inkwell:5000") == "https://notes.example.com"

Some files were not shown because too many files have changed in this diff Show More