Server comments describe the code to a stranger (#6221)
CI & Build / Python lint (push) Successful in 2s
CI & Build / Build now, or wait for Android? (push) Successful in 2s
Android / Build, or is the channel already serving this? (push) Successful in 3s
Android / Core and FFI clippy and tests (push) Skipped
Android / Kotlin + Rust (APK) (push) Skipped
Desktop (Tauri) / Build, or is the channel already serving this? (push) Successful in 2s
CI & Build / Web typecheck and unit tests (push) Successful in 9s
CI & Build / Python tests (push) Successful in 13s
Android / Build the server image (push) Skipped
Desktop (Tauri) / Web tests, clippy, Rust tests and rustfmt (push) Skipped
Desktop (Tauri) / Tauri desktop (Linux) (push) Skipped
Desktop (Tauri) / Windows installer (cross-compiled) (push) Skipped
Desktop (Tauri) / Update manifest (push) Skipped
CI & Build / integration (push) Successful in 1m30s
CI & Build / Build & push image (push) Successful in 32s

Rule 32 "Comments describe the product to a stranger" sweep over
src/, alembic/ and tests/. Comments and docstrings now say what the code
does and the reason as a present-tense constraint, briefly.

Removed:
- issue, task, note, rule, milestone and family-idea references
- "used to" / before-and-after narration
- quotes of decisions and incident stories from the deploying instance
- paragraph-length arguments, where a line or two carries the fact

Kept every hazard as a present-tense fact, e.g. timing oracles, the
proxy hop direction, and migrations that must stay frozen.

Also:
- Three test names lose "still"/"used to"/"pre_m304" phrasing.
- pyproject's description drops "FabledSword family".

The history these comments carried stays in Scribe and in git log.
A comment-stripping diff (Scribe snippet #6226 "codeonly.py — prove a
diff changed only comments") shows no code changed apart from the
renamed tests and the description string.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-10-21 20:07:47 -04:00
co-authored by Claude Opus 5.5
parent 5556bb2136
commit a51a563fe7
108 changed files with 779 additions and 1610 deletions
+2 -2
View File
@@ -4,8 +4,8 @@ Revision ID: 0001
Revises:
Create Date: 2026-07-19
The M0 identity + sharing-ACL spine. No notes yet (those arrive in M1); the
shares table is polymorphic so notes reuse it without a schema retrofit.
Identity and the sharing ACL. The shares table is polymorphic, so any shareable
entity uses it.
"""
from alembic import op
import sqlalchemy as sa
+1 -1
View File
@@ -4,7 +4,7 @@ Revision ID: 0002
Revises: 0001
Create Date: 2026-07-19
The M1 capture core: the notes table the masonry board renders.
The notes table the board renders.
"""
from alembic import op
import sqlalchemy as sa
+1 -1
View File
@@ -5,7 +5,7 @@ Revises: 0002
Create Date: 2026-07-19
Adds the admin role bit and the DB-backed settings table. No rows are seeded —
defaults live in the code registry (thoughtsync.settings), so an empty table
defaults live in the code registry (inkwell.settings), so an empty table
means every setting is at its default.
"""
from alembic import op
+2 -4
View File
@@ -16,10 +16,8 @@ depends_on = None
def upgrade() -> None:
# The note's display NAME: explicit title if set, else the first body line.
op.add_column("notes", sa.Column("display_title", sa.Text(), nullable=False, server_default=""))
# Best-effort backfill (kept intentionally simple + unambiguous — CI has no
# Postgres lane, so this only runs on a real deploy). The app recomputes the
# precise value via derive_display_title on the next save (which additionally
# skips leading blank lines). Capped at 200 chars.
# Best-effort backfill, capped at 200 chars. The app recomputes the exact value
# (derive_display_title, which also skips leading blank lines) on the next save.
op.execute(
r"""
UPDATE notes SET display_title = LEFT(btrim(
+1 -4
View File
@@ -1,4 +1,4 @@
"""sync revision + tombstone infrastructure (M8 sync hub, step 1)
"""sync revision + tombstone infrastructure
Revision ID: 0015
Revises: 0014
@@ -11,9 +11,6 @@ forget to bump it. Child-row changes (items / attachments / label membership) bu
their parent note's revision, since a note syncs as a whole. `purged_at` is the
hard-delete tombstone (distinct from `deleted_at` = recoverable trash) so an
offline client learns a row is gone instead of resurrecting it.
No API behavior changes here — this is pure infrastructure. Triggers/behavior are
operator-verified on deploy (no Postgres CI lane).
"""
from alembic import op
import sqlalchemy as sa
+1 -1
View File
@@ -1,4 +1,4 @@
"""device_tokens (M8 sync hub, step 2)
"""device_tokens
Revision ID: 0016
Revises: 0015
+1 -1
View File
@@ -1,4 +1,4 @@
"""labels.updated_at (M8 sync hub, step 4 — LWW field for the label catalog)
"""labels.updated_at, the last-write-wins field for the label catalog
Revision ID: 0017
Revises: 0016
+1 -1
View File
@@ -1,4 +1,4 @@
"""note_attachments.sha256 (M8 sync hub, step 5 — attachment blob sync)
"""note_attachments.sha256, for attachment blob sync
Revision ID: 0018
Revises: 0017
+1 -1
View File
@@ -1,4 +1,4 @@
"""note_attachments.filename (any-file attachments — M6 1900)
"""note_attachments.filename, for attachments of any file type
Revision ID: 0019
Revises: 0018
+1 -1
View File
@@ -1,4 +1,4 @@
"""note_link_previews (URL unfurl — M6 1901)
"""note_link_previews, cached URL unfurls
Revision ID: 0020
Revises: 0019
+1 -1
View File
@@ -1,4 +1,4 @@
"""saved_filters (richer search + saved views — M6 1902)
"""saved_filters, saved search views
Revision ID: 0021
Revises: 0020
+1 -1
View File
@@ -1,4 +1,4 @@
"""notes.recurrence (recurring reminders — M6 1908)
"""notes.recurrence, for recurring reminders
Revision ID: 0022
Revises: 0021
+3 -6
View File
@@ -1,14 +1,11 @@
"""note_links.target_id — resolve [[links]] to a note, not to a string (M13 step 1)
"""note_links.target_id — resolve [[links]] to a note, not to a string
Revision ID: 0023
Revises: 0022
Create Date: 2026-08-22
A wiki-link stored only as normalized TEXT means a note's name IS the edge: rename
the note and every inbound link stops matching. The old answer was to rewrite the
`[[Old Name]]` text inside every note that linked to it — workable while an explicit
title existed to hold still, untenable once a note's name is just its first body
line (M13).
A wiki-link stored only as normalized text makes a note's name the edge, so renaming
the note breaks every inbound link. Resolving to an id survives a rename.
`target_norm` stays: it is what an UNRESOLVED link carries, since linking to a note
that doesn't exist yet is a supported way to create one.
+5 -13
View File
@@ -1,22 +1,14 @@
"""drop note_links — [[wiki-links]] are removed (note 2897)
"""drop note_links — [[wiki-links]] are removed
Revision ID: 0024
Revises: 0023
Create Date: 2026-08-22
ThoughtSync is an intermediary surface for capture and recall; a linking system is
organization, which is not what it is for. Backlinks, the graph and the name index
went with it.
0023 stays in the chain: removing an applied revision would strand a database's
alembic_version pointer.
0023 (which added `note_links.target_id`) is deliberately left in the chain rather
than deleted. It shipped in an image and may already be applied, and removing an
applied revision would strand a database's alembic_version pointer. So the column is
dropped here along with the table it lived on, and the history stays honest about the
fact that it existed for a day.
No down-migration data concern: note_links was always DERIVED from note bodies. The
`[[text]]` is still sitting in every body it was written in; nothing a person typed is
lost by this.
note_links was derived from note bodies, so nothing typed is lost; the `[[text]]` is
still in every body.
"""
from alembic import op
import sqlalchemy as sa
+5 -14
View File
@@ -1,19 +1,11 @@
"""drop notes.kind — a checklist is something a note HAS (M13 step 2)
"""drop notes.kind — a checklist is something a note has
Revision ID: 0025
Revises: 0024
Create Date: 2026-08-22
`kind` was never a type: a plain TEXT column with no enum and no CHECK, compared
against a hardcoded ("text", "list") tuple in six places. `note_items` was always an
ordinary child table keyed by note_id, serialization always emitted `items` whatever
the kind, and the Android editor already toggled between the two losslessly. The
storage has modelled "a body plus optional checkable items" the whole time; only the
gates forbade it.
Nothing is lost. Items were already rows in their own table, and a note that was
`kind = 'list'` keeps every one of them — it just stops being a different sort of
thing from the note next to it.
A note is a body plus optional checkable items. Items are rows in their own table,
so a `kind = 'list'` note keeps every one.
"""
from alembic import op
import sqlalchemy as sa
@@ -29,7 +21,6 @@ def upgrade() -> None:
def downgrade() -> None:
# server_default so existing rows get a value; every note comes back as 'text',
# which is right — a restored note with items would previously have hidden its
# body, and there is no record of which ones were once lists.
# server_default so existing rows get a value. Every note comes back as 'text':
# nothing records which were lists.
op.add_column("notes", sa.Column("kind", sa.Text(), nullable=False, server_default="text"))
+6 -25
View File
@@ -4,33 +4,14 @@ Revision ID: 0026
Revises: 0025
Create Date: 2026-08-22
M13 step 3. A note is a body plus optional checkable items; its NAME is the first
non-empty line of that body, falling back to its first checklist item. There is no
separate field to type into, and `display_title` (already persisted, already what
search results and export filenames read) carries the name.
A note's name is its first non-empty body line, carried by `display_title`.
## The search vector has to be rebuilt, not just left alone
`notes.search_vector` is a stored generated column over `title` (0005, weight A), and
Postgres won't drop a column a generated one depends on. So it is recreated over
`display_title`, keeping the name ranked above the body. That recomputes every row
and rebuilds the GIN index, which takes a while on a large table.
`notes.search_vector` is a STORED GENERATED column whose expression names `title`
(migration 0005, weight A) — Postgres will refuse to drop a column another generated
column depends on, and even if it didn't, the weighting would be wrong. So it is
dropped and recreated over `display_title` instead, which keeps the original
intent: the note's NAME ranks above the rest of its body.
Rebuilding a stored generated column re-computes every row, and the GIN index is
rebuilt with it. On a personal instance that is milliseconds; it is worth knowing
before running this against something large.
## What happens to existing titles
Nothing preserves them, deliberately: `display_title` was already derived from the
title when one was set, so every note keeps the NAME it had. What is lost is the
distinction between "this note has an explicit title" and "this note's first line is
its name" — which is the distinction being removed.
Imports are the exception and are handled in code, not here: a Keep note's title, or
one in an export taken before this, is folded in as the note's first body line rather
than dropped (see `_create_imported_note`).
`display_title` was already derived from the title, so every note keeps its name.
"""
from alembic import op
import sqlalchemy as sa
@@ -4,64 +4,31 @@ Revision ID: 0027
Revises: 0026
Create Date: 2026-08-24
M304. A checklist item becomes a `- [ ] milk` line of `notes.body`, and `note_items`
goes. The reason is positional, not cosmetic: a row had a position in a table and no
position in the text, so a separate list could only ever render AFTER the prose. With
the items in the body, a list can sit between two paragraphs — which is the thing that
could not be built before and no amount of restyling would have delivered.
A checklist item becomes a `- [ ] milk` line of `notes.body`, so a list can sit
between two paragraphs, and `note_items` goes.
## This migration rewrites note bodies
Every note that has items gets its body appended to. The rules below are strict
because rewriting somebody's text deserves it — not, as an earlier draft of this
docstring claimed, because this instance holds imported Google Keep notes. It does
not; note 2916's headline is that nothing here is anyone's work but the operator's
test data. What 2916 actually says about imports is conditional — text arriving from
another app WOULD be real, and any import path has to treat it that way — and the
importer this migration shares a format with is one nobody here has run.
* Rows are read before the table is dropped, in this one transaction.
* The existing body is only appended to.
* The layout (a blank line between prose and the list, nothing between items) is
what the export writes and what `derive::append_item` produces, so the clients
migrate their own stores independently and still agree with the server.
Careful was still the right call. It cost little, and the same care is what the rule
demands the day someone does import something:
The fold is inlined rather than imported from `notes/checklist.py`: a migration must
keep producing what it produced the day it ran.
* Rows are read BEFORE the table is dropped, in this one transaction.
* The existing body is never rewritten, only appended to.
* The layout — a blank line between prose and the list, nothing between consecutive
items — is byte-for-byte what `_note_markdown` has always exported and what
`derive::append_item` produces on every client. All three landing on the same text
is what lets the clients migrate their own SQLite stores independently and still
agree with the server, with no sync required to reconcile them.
## `updated_at` is left alone
## The fold is inlined on purpose
Raw SQL, so `onupdate` never fires, and a client holding an unpushed edit keeps the
newer `updated_at` and wins last-write-wins. The `sync_revision` trigger does fire,
so every migrated note is pulled once and clients converge.
`notes/checklist.py` has this same function and this migration deliberately does not
import it. A migration has to keep producing what it produced the day it ran; if the
app's spacing rule ever changes, this file must not change with it.
## The downgrade is not a true inverse
## `updated_at` is left alone, and that is load-bearing
Raw SQL, so SQLAlchemy's `onupdate` never fires. Two reasons, and the second matters
more than the first. Every client folds the same rows the same way, so the new body is
news to nobody. And a client holding an UNPUSHED body edit still has the newer
`updated_at`, so when it pulls the migrated note last-write-wins keeps its edit instead
of the migration silently winning.
The `notes` row's own `sync_revision` trigger (migration 0015) does fire, so every
migrated note becomes pullable once. That is wanted: it is what makes a client whose
local fold somehow differed converge on the server's text.
## The downgrade is not a true inverse, and says so
It recreates an empty `note_items` and leaves the bodies alone. Nothing is lost —
every item is still there as text, which is where this migration put it — but the old
code would show those notes as prose with no checklist. A faithful inverse is not
possible: once the items are lines, nothing distinguishes a line this migration wrote
from one somebody typed, and a downgrade that guessed would eat hand-written task
lists. The real rollback is a database restore.
Recreating the table is not decoration, though. Migration 0015's downgrade runs
`DROP TRIGGER IF EXISTS trg_note_items_bump_note ON note_items`, and `IF EXISTS`
covers the trigger, not the table — against a missing table that statement errors. So
this is what keeps the migration chain runnable all the way back down.
It recreates an empty `note_items` and leaves the bodies alone: nothing tells a line
this wrote from one somebody typed. The real rollback is a database restore. The
table must exist because 0015's downgrade drops a trigger on it.
"""
import re
@@ -100,9 +67,8 @@ def upgrade():
for note_id, items in grouped.items():
body = bind.execute(sa.text("SELECT body FROM notes WHERE id = :id"), {"id": note_id}).scalar()
# An item whose note is already gone has nothing to fold into. The foreign key
# should make this impossible; skipping costs nothing and failing here would
# leave the database half-migrated.
# An item whose note is gone has nothing to fold into. The foreign key should
# prevent it; skipping avoids a half-migrated database.
if body is None:
continue
for text, checked in items:
@@ -113,9 +79,8 @@ def upgrade():
def downgrade():
# Column-for-column as migration 0006 created it, index name included: 0015's
# downgrade names both the table and its trigger, so a near-enough copy is not
# good enough.
# Exactly as 0006 created it, index name included: 0015's downgrade names both
# the table and its trigger.
op.create_table(
"note_items",
sa.Column("id", UUID(as_uuid=True), primary_key=True),
+19 -71
View File
@@ -4,73 +4,27 @@ Revision ID: 0028
Revises: 0027
Create Date: 2026-08-26
M311. A `#tag` was being shown twice — once as the text you typed and once as a chip —
and with the chip moved to the top of the card the text is redundant. This removes it,
but only from notes where the tag was standing on its own.
A tag standing on its own line shows as a chip, so its text is removed, by the same
rule as `notes/tags.py:split_body_tags`:
## This migration rewrites note bodies
* A line containing nothing but tags and whitespace is removed.
* Every other line is left exactly as written, so `remember to call #mom` stays.
* A line inside a ``` fence is never touched.
* A note that is nothing but tags keeps its text, rather than becoming a blank card.
The rule is deliberately narrow, and the same one `notes/tags.py:split_body_tags`
applies from here on:
The label rows flip to `via_tag = false` in the same transaction: the text backing
them is gone, and `_lift_and_reconcile_tags` would otherwise detach them on the next
save.
* A line containing nothing but tags and whitespace is REMOVED.
* Every other line is left exactly as written.
The transform and `_display_title` are inlined: a migration must keep producing what
it produced the day it ran. The title is recomputed only where the body moved.
So `#todo` on its own line goes, and `remember to call #mom tomorrow` does not. The
looser reading — also stripping a trailing tag off a prose line — was rejected because
the text does not say which kind it is: `buy milk #grocery` is filing, `remember to
call #mom` is the sentence's object, and lifting the second leaves "remember to call".
Rewriting somebody's words to save a duplicate chip is a bad trade, and a migration is
the worst possible place to make it.
Raw SQL, so `onupdate` never fires and a client's unpushed edit still wins
last-write-wins. The `sync_revision` trigger fires, so clients pull the server's text.
Two guards, both of which cost a note nothing:
* A line inside a ``` fence is never touched. A `#tag` there is a shell comment in a
snippet somebody pasted, and deleting it would eat a line of their example.
* A note that is NOTHING but tags keeps its text. Lifting would leave a blank card,
which is worse than the duplication this fixes.
## The label rows have to graduate in the same transaction
A `via_tag` row means "this label is backed by text still in the body". Once the text
is gone that is false, and leaving it true is not cosmetic: `_lift_and_reconcile_tags`
detaches any `via_tag` row it cannot find a `#tag` for, so the note would lose the tag
on its very next save. The flip to `via_tag = false` is what makes the label the record
instead — and what makes the chip's × appear in both editors, which is now the only way
to remove a tag whose text no longer exists.
## The transform is inlined, like 0027's
`split_body_tags` is deliberately NOT imported. A migration has to keep producing what
it produced the day it ran; if the app's rule is ever loosened, this file must not
loosen with it and start eating prose it previously left alone.
`_display_title` is inlined for the same reason, and is only recomputed for a note whose
body actually moved — a note named after a `#todo` line needs a new name, and reading it
from the app would couple this migration to a rule that has already changed once (M13).
## `updated_at` is left alone, and that is load-bearing
Raw SQL, so SQLAlchemy's `onupdate` never fires. A client holding an UNPUSHED body edit
keeps the newer `updated_at`, so when it pulls the migrated note last-write-wins keeps
its edit instead of the migration silently winning.
The `sync_revision` trigger (migration 0015) does fire, so every rewritten note becomes
pullable once and clients converge on the server's text. That is wanted here: unlike
0027, the clients do NOT yet apply this rule locally, so the server's copy is the only
correct one until they do.
## The downgrade is not a true inverse, and says so
It cannot be. Nothing distinguishes a `#todo` line this migration deleted from one that
was never there, and putting one back would be guessing at where in the note it went.
Nothing is lost, though, which is why that is acceptable: the tag still exists as a
label on the note, and the chip still shows it. What a downgrade cannot restore is the
DUPLICATE — which is the thing this migration set out to remove. Rolling the rows back
to `via_tag = true` would be actively harmful: the text that flag claims to be backed by
is gone, so the next save would detach the label and lose the tag for real. So the
downgrade leaves both alone. The real rollback is a database restore.
The downgrade does nothing. Restoring the lines would be guessing, and flipping the
rows back to `via_tag = true` would detach every label on the next save. The real
rollback is a database restore.
"""
import re
@@ -152,17 +106,11 @@ def upgrade():
sa.text("UPDATE notes SET body = :body, display_title = :title WHERE id = :id"),
{"body": lifted, "title": _display_title(lifted), "id": note_id},
)
# Even when the body did not move, a tag can be standalone only in the sense
# that its line was already removed by an earlier pass — so the flip is driven
# by the tag list, not by whether the text changed.
# The flip is driven by the tag list, not by whether the text changed: an
# earlier pass may already have removed the line.
if standalone:
bind.execute(flip, {"nid": note_id, "names": [n.lower() for n in standalone]})
def downgrade():
"""Deliberately empty — see the module docstring.
Restoring the deleted lines would be guessing, and flipping the rows back to
`via_tag = true` would be worse than doing nothing: the text that flag claims backs
them is gone, so the next save would detach the label and lose the tag for real.
"""
"""Deliberately empty; see the module docstring."""
+9 -42
View File
@@ -4,50 +4,18 @@ Revision ID: 0029
Revises: 0028
Create Date: 2026-08-28
M315 step 3. A note's colour was set by a picker and read by three card renderers.
Steps 1 and 2 stopped every one of those reads: the card is one neutral per theme and
the only coloured thing on a board is a tag. This drops the column that nothing has
been reading since, and the picker goes with it.
Cards are one neutral surface per theme; the only colour on a board is a tag's.
`labels.color` is untouched. A colour a note was given is dropped with the column.
`labels.color` is untouched. That is the colour that survived, and the one the whole
milestone was about keeping.
`saved_filters.params` may carry a `"color"` facet, so stored views are swept too. In
Python rather than `params::jsonb - 'color'`, because Postgres has no try-cast and one
malformed blob would abort the migration; a row that doesn't parse is left as it is.
## What is lost, and why that is the change rather than a cost of it
Any colour a note was explicitly given. There is nowhere to preserve it TO — the field
it would be preserved in is the one being dropped — and nothing renders it, so a
preserved value would be a column kept warm for a feature that was deliberately
removed. A note that had a colour now takes its identity from its tags, which is what
the operator asked for: "strip color from the cards ... and keep the color for tags
just on the tag."
The palette itself is not lost. `NOTE_COLORS` moved from `models/note.py` to
`colors.py` in the same change — labels still name a colour, and leaving the vocabulary
defined on the model that lost one would be an invitation to put the column back.
## The saved-filter sweep is not optional
`saved_filters.params` is opaque JSON mirroring the `GET /api/notes` facet query, and
a stored view could carry `"color": "teal"`. With the facet gone that key would sit
there forever, and `clean_params` only guards what is written FROM here on. A view that
silently filters on a field the app no longer has is worse than one that visibly lost a
criterion, so the stored rows are swept too.
Done in Python rather than as `params::jsonb - 'color'`, deliberately. Postgres has no
try-cast: one malformed blob would abort the whole migration, and these rows are
somebody's saved views. `json.loads` in a try/except lets a corrupt row keep whatever it
holds and lets every other row be fixed.
## Search is not affected
`notes.search_vector` is a stored generated column over `display_title` and `body`
(rebuilt in 0026). It never named `color`, so unlike the title drop there is nothing
here to tear down and recreate.
The search vector never named `color`, so it needs no rebuild.
## Downgrade
Restores the column, empty, at its old default. The values are not recoverable — see
above. It is the schema that comes back, not the data.
Restores the column, empty, at its old default. The values are not recoverable.
"""
import json
@@ -71,9 +39,8 @@ def upgrade() -> None:
try:
parsed = json.loads(params)
except (ValueError, TypeError):
# A blob that does not parse cannot be edited safely. Leaving it is
# correct: it was already unreadable by the app, and this migration is not
# the place to decide what it should have said.
# A blob that doesn't parse can't be edited safely; the app couldn't read
# it either.
continue
if not isinstance(parsed, dict) or "color" not in parsed:
continue
+3 -9
View File
@@ -4,15 +4,9 @@ Revision ID: 0030
Revises: 0029
Create Date: 2026-10-06
The product is renamed ThoughtSync → Inkwell (Scribe note 5071), and the `site_name`
registry default moves with it. On its own that only reaches servers whose settings
table has no `site_name` row — and most servers have one, because the Settings page
saves EVERY key on Save, not just the one that changed. An admin who opened Settings
to flip registration off has persisted `"ThoughtSync"` without ever choosing it.
So the row is rewritten, but only when it holds exactly the old default. A name the
admin actually typed is theirs and is left alone; the old default is the one value we
can be sure nobody chose.
The app was renamed from ThoughtSync to Inkwell. The Settings page saves every key,
so most servers have a stored `site_name` of the old default without anyone choosing
it. The row is rewritten only when it holds exactly that; a typed name is left alone.
The match is on the stored JSON text (`settings.value` is `json.dumps(value)`), so
`'"ThoughtSync"'` is the whole of what it looks for.
@@ -4,16 +4,9 @@ Revision ID: 0031
Revises: 0030
Create Date: 2026-10-07
0015 made every child table bump its parent note's `sync_revision`, so a note syncs
as a whole. `note_link_previews` arrived later (0020) and was never added, and two
things have been missing on every linked device since:
- A preview fetched in the background after a save (`unfurl_queue.py`) never reached
a device that had already pulled the note. The note's revision was assigned when
the TEXT was saved, before the preview existed, and nothing moved it afterwards.
- A preview dismissed on the web stayed on every other device, for the same reason.
The trigger is the same function 0015 installs on the other child tables.
A note syncs as a whole, so every child table bumps its note's `sync_revision`. With
this, a preview fetched after a save, or dismissed on the web, reaches every device.
The trigger is the function 0015 installs on the other child tables.
## Downgrade
+2 -3
View File
@@ -4,9 +4,8 @@ Revision ID: 0032
Revises: 0031
Create Date: 2026-10-07
Until now the only way to add a second person was to re-open registration to the
whole internet while they signed up (#2939 §1). An invite lets one person register
while registration stays closed. Only the token's SHA-256 hash is stored.
An invite lets one person register while registration stays closed. Only the token's
SHA-256 hash is stored.
## Downgrade
+5 -6
View File
@@ -4,20 +4,19 @@ Revision ID: 0033
Revises: 0032
Create Date: 2026-10-07
A forgotten password used to need a hand on the database (#2939 §2), and the app has
no mail path to send a reset by. An admin makes a reset link for the account and
hands it over (#5173). Only the token's SHA-256 hash is stored.
An admin makes a reset link for the account and hands it over. Only the token's
SHA-256 hash is stored.
`users.session_epoch` is what lets a reset sign the account out everywhere. Sessions
are signed cookies held by the browser, so the server can't delete them; each one
carries the epoch it was signed in under, and a reset moves the account's epoch on.
It starts at 0, the value a cookie from before this migration is read as, so nobody
is signed out by the upgrade itself.
It starts at 0, the value a cookie without one is read as, so the upgrade signs
nobody out.
## Downgrade
Drops the table and the column. Outstanding reset links stop working; sessions keep
working, since nothing checks an epoch any more.
working, since nothing checks an epoch.
"""
import sqlalchemy as sa
from alembic import op
+2 -3
View File
@@ -4,11 +4,10 @@ Revision ID: 0034
Revises: 0033
Create Date: 2026-10-07
Nothing wrote a share until #5174, so the table never needed these. Now the Share
dialog does:
For the Share dialog:
- A unique index on (resource_type, resource_id, shared_with_user_id) where a user is
the target, so sharing a note with someone twice updates the one grant rather than
stacking two (the same for a group, ahead of step 15).
stacking two (the same for a group).
- A check that `permission` is `view` or `edit`.
- An index on `shared_with_user_id` for "Shared with me".
+1 -1
View File
@@ -5,7 +5,7 @@ Revises: 0034
Create Date: 2026-10-07
The change feed carries every note a person can see, including notes shared with
them (#5175). When a share ends, the note stops being visible, so it simply stops
them. When a share ends, the note stops being visible, so it simply stops
appearing in the feed. A device that already holds a copy would keep it forever. A
revocation is that missing signal: one row per (note, person) who lost the note,
stamped from the same `sync_revision_seq` the notes and labels draw from, so it sits
+4 -6
View File
@@ -4,12 +4,10 @@ Revision ID: 0036
Revises: 0035
Create Date: 2026-10-07
Pin, archive and board order are personal organization, and until now they were
columns on `notes`, so on a shared note they could only ever mean the owner's
(#5176). This table holds them for everyone else: one row per (note, person it is
shared with) who has pinned, archived or moved it. The owner keeps the note's own
columns; a recipient with no row sees the note unpinned, unarchived, in the owner's
order.
Pin, archive and board order are personal. The owner's are columns on `notes`; this
table holds everyone else's, one row per (note, person it is shared with) who has
pinned, archived or moved it. A recipient with no row sees the note unpinned,
unarchived, in the owner's order.
`sync_revision` is stamped by the same `ts_set_sync_revision()` trigger the notes
and labels use (0015), so a recipient's change reaches their own devices on the one
@@ -4,12 +4,8 @@ Revision ID: 0037
Revises: 0036
Create Date: 2026-10-07
Both came with the foundation (0001) for features that were never built (#5178).
Nothing ever set `email_verified`, so it was false for every account on the server
while the offline desktop reported it as true; nothing read or wrote `avatar_path`.
A field whose value means nothing is worse than no field: a later feature would
trust it. If address verification is ever built, it adds its own column, with a
migration that says what the existing accounts are.
Nothing set `email_verified` or read `avatar_path`, and a field whose value means
nothing would be trusted by a later feature.
## Downgrade
+1 -3
View File
@@ -4,9 +4,7 @@ Revision ID: 0038
Revises: 0037
Create Date: 2026-10-08
Saved views were removed in #5180 (note 2897: recall first, organizing second). They
lived only on the web; the desktop kept its own set that never synced, and Android
had none. Tags in the drawer already give one-click recall.
Saved views are removed; tags in the drawer give one-click recall.
## Downgrade
+2 -3
View File
@@ -4,9 +4,8 @@ Revision ID: 0039
Revises: 0038
Create Date: 2026-10-08
Credential events were only written to the app log, which is not queryable and is
gone at the container's log rotation (#2939 §5). This keeps them in the database for
`audit_retention_days`, for admins to read in Settings → Activity.
Keeps credential events in the database for `audit_retention_days`, for admins to
read in Settings → Activity; the app log isn't queryable and rotates away.
## Downgrade
+3 -5
View File
@@ -4,11 +4,9 @@ Revision ID: 0040
Revises: 0039
Create Date: 2026-10-08
Deleting a tag now leaves a tombstone row (`purged_at` set) so the change feed can
tell linked devices about it (#5382). Before, the web deleted the row outright and
devices never heard. A tombstone must not hold its name: creating `#grocery` again
after deleting it has to make a live tag, on the web and from a device alike. So
`(owner_id, name)` stays unique only where `purged_at IS NULL`.
Deleting a tag leaves a tombstone row (`purged_at` set) so the change feed tells
linked devices. A tombstone must not hold its name, so `(owner_id, name)` is unique
only where `purged_at IS NULL`.
## Downgrade
+1 -1
View File
@@ -5,7 +5,7 @@ build-backend = "setuptools.build_meta"
[project]
name = "inkwell"
version = "0.2.0"
description = "Self-hosted personal thought-capture web app (FabledSword family)"
description = "A self-hosted web app for capturing and finding thoughts"
requires-python = ">=3.12"
dependencies = [
"quart>=0.19",
+3 -14
View File
@@ -1,16 +1,5 @@
"""Inkwell — self-hosted personal thought-capture web app (FabledSword family)."""
"""Inkwell — a self-hosted web app for capturing and finding thoughts."""
# PACKAGING METADATA, and nothing else. Not the version any running server reports.
#
# A built image carries APP_VERSION in the environment, derived from the server's
# own shipped file set (packaging/version.sh); `app.py` reads that and reports an
# explicit "unknown" when it is absent, so this string never reaches a user and
# bumping it changes nothing anybody sees.
#
# It exists because a Python package needs a version and "unknown" is not a legal
# one here. It used to double as app.py's fallback, which meant a server run from a
# checkout confidently reported `0.2.0` — a real-looking version naming no build
# that exists. Note 3127 §5 is why that matters more than it reads: with version
# tags gone, a build's self-report is the only answer to "which build is this?",
# and there is nothing left to catch it lying.
# Packaging metadata only. A running server reports APP_VERSION from the image, or
# "unknown"; this string is never shown and bumping it changes nothing.
__version__ = "0.2.0"
+1 -1
View File
@@ -1,5 +1,5 @@
"""Admin routes over the instance's accounts: list them, make a password reset link
for one (#5173), and read what has happened to them (the audit log, #2939 §5).
for one, and read what has happened to them (the audit log).
What a reset link is, and how it is used, is in `password_resets.py`.
"""
+5 -6
View File
@@ -7,9 +7,9 @@ from sqlalchemy import ColumnElement, exists, or_, select
from .models.group import GroupMember
from .models.share import Share
# What a share grants (#5174). `edit` is the note's body and checklist; everything
# else about a note stays its owner's. Ordered weakest first, so the strongest of a
# person's grants is the one with the highest index.
# What a share grants. `edit` covers the note's body and checklist; everything else
# stays the owner's. Ordered weakest first, so a person's strongest grant has the
# highest index.
VIEW = "view"
EDIT = "edit"
PERMISSIONS = (VIEW, EDIT)
@@ -26,9 +26,8 @@ def visible_to_user(
"""Boolean SQL predicate: is this resource visible to ``user_id``?
A resource is visible if the user OWNS it, OR it is shared DIRECTLY with the
user, OR it is shared with a GROUP the user belongs to. Scope every read (and
mutation) of shareable user data through this — never assume a single operator
(family rule 47).
user, OR it is shared with a GROUP the user belongs to. Scope every read and
mutation of shareable user data through this; the server is multi-user.
With ``permission``, only shares at that level count: ``permission=EDIT`` is
"may this user change it", which the owner always may.
+32 -69
View File
@@ -41,13 +41,9 @@ from .settings_api import bp as settings_bp
from .shares_api import bp as shares_bp
from .sync import bp as sync_bp, protocol_advertisement
# Without this, `logger.info` from this package goes nowhere: hypercorn configures its
# own access/error loggers and leaves the root logger at WARNING, so the credential
# events in auth.py would be invisible in `docker compose logs` — which is exactly
# where they are read live. (Settings → Activity keeps most of them, `audit.py`.)
#
# `force=False` (the default) so a host that has already configured logging keeps its
# own setup; LOG_LEVEL lets an operator turn it up without a code change.
# Hypercorn leaves the root logger at WARNING, which would hide this package's INFO
# credential events from `docker compose logs`. A host that already configured
# logging keeps its own setup.
logging.basicConfig(
level=os.environ.get("INKWELL_LOG_LEVEL", "INFO").upper(),
format="%(asctime)s %(levelname)s %(name)s %(message)s",
@@ -65,10 +61,8 @@ class _AutoSecureSessionInterface(SecureCookieSessionInterface):
"""Mark the session cookie `Secure` whenever the request arrived over HTTPS —
directly, or via a TLS-terminating reverse proxy that sets X-Forwarded-Proto.
Auto-detecting per request (rather than a fixed SESSION_COOKIE_SECURE flag)
hardens the cookie on HTTPS deployments without breaking a plain-HTTP install on
a trusted LAN, where a hard-forced Secure flag would stop the browser from ever
sending the cookie back — i.e. silently break login. No configuration required.
Per request rather than a fixed flag, because a forced Secure cookie is never sent
back over plain HTTP, which would break sign-in on a LAN install.
"""
def get_cookie_secure(self, app: Quart) -> bool:
@@ -83,19 +77,8 @@ def create_app() -> Quart:
# Ephemeral/env secret so the app (and DB-free unit tests) construct without a
# database. before_serving swaps in the real, DB-persisted key before serving.
app.secret_key = Config.secret_key_env() or secrets.token_urlsafe(48)
# The RUNNING build, or an explicit "unknown" — never the packaging fallback.
#
# This read `os.environ.get("APP_VERSION", __version__)`, so a server started
# from a checkout reported `0.2.0`: a real-looking version that names no build
# anybody could get. `__init__.py` already claimed the honest answer was
# "APP_VERSION being missing, which app.py already handles" — it did not, and a
# comment asserting a behaviour two files away from the code is how that stayed
# true-sounding for months.
#
# It matters more than it used to. Note 3127 §5 removed version tags, so this
# string is the only answer to "which build is this?" and nothing exists to
# contradict it when it is wrong. `__version__` stays where it belongs, as
# packaging metadata, which is the one place "unknown" is not a legal value.
# The running build, or "unknown". Never `__version__`: that names no build, and
# this string is the only answer to "which build is this?".
app.config["APP_VERSION"] = os.environ.get("APP_VERSION") or "unknown"
app.config["SESSION_COOKIE_HTTPONLY"] = True
app.config["SESSION_COOKIE_SAMESITE"] = "Lax"
@@ -130,14 +113,11 @@ def create_app() -> Quart:
apply_session_ttl(app, await get_setting(db, "session_ttl_days"))
except (ValueError, TypeError, KeyError):
pass
# The security settings the throttle and the proxy trust read on hot
# paths. Cached rather than queried per request; until this runs they
# hold their registry defaults, which is the correct behaviour for a
# server that has not finished starting.
# Security settings read on hot paths, cached rather than queried per
# request. Until this runs they hold their registry defaults.
await refresh_live(db)
# Credentials stored before sealing existed are sealed now (#5927). A data
# folder that can't be written leaves them as they are; it never stops the
# server starting.
# Seal any credentials still stored in the clear. A data folder that can't
# be written leaves them as they are; it never stops the server starting.
try:
if count := await seal_stored_secrets(db):
await db.commit()
@@ -145,11 +125,9 @@ def create_app() -> Quart:
except SealError as exc:
await db.rollback()
logger.warning("stored credentials left unsealed: %s", exc)
# Expire old trash in the background (retention.py). One task per process is
# correct because the image serves with a single hypercorn worker (Dockerfile);
# if that ever gains `--workers`, this needs a lock so N workers don't each
# sweep. Duplicate sweeps would be harmless but wasteful — a purged row is
# skipped by `purged_at IS NULL` — so this is about load, not correctness.
# Expire old trash in the background (retention.py). One task per process,
# because the image runs one hypercorn worker; with several, each would sweep,
# which is wasteful but harmless.
app.config["TRASH_SWEEPER"] = asyncio.create_task(run_sweeper())
# The daily backup (backup_job.py), on the same single-worker footing. It also
# takes a Postgres advisory lock, so a second process could not double it.
@@ -172,23 +150,16 @@ def create_app() -> Quart:
None of these change how the app behaves for a legitimate caller; they narrow
what a browser will do if something else goes wrong.
The CSP is the substantive one. `script-src 'self'` means that even if some
future path did manage to reflect user text into the page, the browser would
refuse to run it — the app has no inline scripts and no third-party scripts,
so nothing legitimate is given up. The exceptions are honest ones:
- `style-src 'unsafe-inline'` — Vue writes inline styles itself (`v-show`
toggling display, TransitionGroup's FLIP setting transforms). Inline
STYLE is not an execution primitive the way inline script is.
- `img-src https: http:` — link previews render the remote og:image of
whatever was linked, which is an arbitrary host by definition. Both
schemes, because a LAN install is served over http and would otherwise
lose every preview image; on an https instance the browser blocks the
http ones as mixed content anyway, so naming it concedes nothing.
- `blob:`/`data:` — attachment previews and the desktop's blob URI scheme.
The CSP allows only the app's own scripts; it has no inline or third-party
ones. The exceptions:
- `style-src 'unsafe-inline'`: Vue writes inline styles (v-show, FLIP
transitions). Inline style is not an execution primitive.
- `img-src https: http:`: link previews show the linked page's image, from
any host. http too, for LAN installs; an https page blocks those anyway.
- `blob:`/`data:`: attachment previews and the desktop's blob URIs.
`frame-ancestors 'none'` replaces the older X-Frame-Options and is what stops
the app being framed for clickjacking; `form-action 'self'` stops a form from
being pointed at another origin.
`frame-ancestors 'none'` prevents clickjacking; `form-action 'self'` keeps
forms from posting to another origin.
"""
response.headers.setdefault(
"Content-Security-Policy",
@@ -204,9 +175,7 @@ def create_app() -> Quart:
"media-src 'self' blob:; "
"connect-src 'self'",
)
# Content-type sniffing turns a file we said was text into whatever the bytes
# look like. The attachment route already sets this; every other response
# deserves it too.
# Stop the browser sniffing a response into a different content type.
response.headers.setdefault("X-Content-Type-Options", "nosniff")
# Note titles and label names end up in the URL of a search or a label lens,
# and a full Referer would hand them to any site a link preview points at.
@@ -216,14 +185,9 @@ def create_app() -> Quart:
response.headers.setdefault(
"Permissions-Policy", "camera=(), microphone=(), geolocation=(), interest-cohort=()"
)
# HSTS only where the request already arrived over TLS — the same detection
# the session cookie uses. Sending it on a plain-HTTP LAN install would tell
# the browser to refuse the only scheme that install serves.
#
# Scoped to this host: no `includeSubDomains` and no `preload`, both of which
# commit domains this app does not own. A browser still remembers the policy
# for up to a year after the header stops being sent, which is the point of
# it — worth knowing before putting a hostname behind TLS temporarily.
# HSTS only on requests that arrived over TLS: on a plain-HTTP install it would
# lock the browser out. This host only, no includeSubDomains or preload. The
# browser keeps the policy for a year, even after the header stops.
if is_https():
response.headers.setdefault("Strict-Transport-Security", "max-age=31536000")
return response
@@ -238,13 +202,12 @@ def create_app() -> Quart:
async with session_scope() as db:
data = await get_public_config(db)
data["version"] = app.config["APP_VERSION"]
# The sync-protocol handshake (M10.6). A native client reads this BEFORE
# linking — while it still has no token and possibly no account — to decide
# whether it can talk to this server, and which optional features to offer.
# The sync-protocol handshake. A native client reads this before linking, with
# no token yet, to decide whether it can talk to this server and which
# optional features to offer.
data.update(protocol_advertisement())
# Which CLIENTS this server can hand out, if any, under `clients`. Absent
# rather than null when it has none, so the web UI hides a download instead
# of offering a button that 404s.
# The clients this server can hand out. Absent when it has none, so the web UI
# hides the download.
data.update(client_advertisement())
return jsonify(data)
+1 -1
View File
@@ -1,4 +1,4 @@
"""The audit log: what happened to accounts, kept where an admin can read it (#2939 §5).
"""The audit log: what happened to accounts, kept where an admin can read it.
Sign-ins, failed sign-ins, password changes and resets, devices linked and
unlinked, invites made and revoked. Each is still written to the app log as it
+40 -73
View File
@@ -32,30 +32,20 @@ from .storage import storage_limit, storage_used
bp = Blueprint("auth", __name__, url_prefix="/api/auth")
# Every credential event goes to the app log, which is the live view of whether
# anyone is knocking: `docker compose logs`. Most also go to the audit log
# (`audit.py`), which keeps them for admins to read in Settings → Activity.
#
# The attempted email is included deliberately. It is the operator's own server, and
# "somebody failed a login" without saying against WHICH account tells you nothing you
# can act on. Passwords, obviously, never appear.
# Every credential event goes to the app log (`docker compose logs`); most also go to
# the audit log (Settings → Activity). The attempted email is logged so a failure can
# be acted on; passwords never are.
logger = logging.getLogger(__name__)
SESSION_KEY = "user_id"
# The account's `session_epoch` when this session signed in. A password reset moves
# the account's epoch on, which is how it ends sessions it can't reach: they are
# signed cookies in other people's browsers. A cookie from before epochs existed has
# no key and reads as 0, the epoch every account started at.
# the epoch on, which ends sessions held as signed cookies in other browsers. A
# cookie without the key reads as 0, every account's starting epoch.
EPOCH_KEY = "epoch"
# The first account can only be created this long after the server starts (family
# idea #5105, practice 8). Without it, a fresh server on a public address belongs to
# whoever reaches /register first, and a new DNS name is found within minutes.
#
# A constant, not a Setting: there is no admin yet to change one. Anyone who misses
# the window restarts the server, which opens it again; once an account exists the
# window no longer matters. Measured from `create_app` (STARTED_AT), so each start
# counts afresh.
# How long after start the first account may be created. Otherwise a fresh server on
# a public address belongs to whoever reaches /register first. A constant, since
# there is no admin yet to change a setting; restarting the server reopens it.
SETUP_WINDOW_S = 30 * 60
SETUP_CLOSED = (
"this server's setup window has closed. Restart the server, then create the "
@@ -136,9 +126,8 @@ def login_required(fn):
"""Guard: 401 unless authenticated. Accepts a web session cookie OR a device
bearer token (native clients). Sets g.user_id for the view.
The session path reads one column by primary key. It used to read nothing, which
left no way to end a session short of its expiry: not for a reset password, and
not for a deleted account (#5173)."""
The session path reads the account's epoch, so a password reset or a deleted
account ends the session before its expiry."""
@functools.wraps(fn)
async def wrapper(*args, **kwargs):
@@ -182,10 +171,8 @@ def require_admin(fn):
def _throttled(retry_after: int):
"""The 429 every throttled credential route returns.
Deliberately says nothing about WHICH limit was hit or how many attempts are
left — that would tell someone probing whether the email they guessed exists.
`Retry-After` is standard and is the one thing a legitimate client (or person)
genuinely needs.
Says nothing about which limit was hit or what is left, which would reveal
whether a guessed email exists. Only `Retry-After`.
"""
logger.warning("throttled credential attempt from=%s retry_after=%ss", client_address(), retry_after)
return too_many("too many attempts — try again shortly", retry_after)
@@ -194,10 +181,8 @@ def _throttled(retry_after: int):
def _sign_in_block(email: str) -> int | None:
"""Seconds to wait before this sign-in may be attempted, or None to proceed.
Checked BEFORE the password is verified, so a throttled attempt costs no bcrypt
— which is the other half of what this protects: hashing is deliberately slow,
and an unauthenticated caller who can trigger it without limit has a CPU
exhaustion primitive, not just a guessing one.
Checked before the password is hashed, so a throttled attempt costs no bcrypt:
unlimited hashing would let an anonymous caller exhaust the CPU.
"""
address = client_address()
waits = [
@@ -237,9 +222,8 @@ async def register():
if not display_name:
display_name = email.split("@", 1)[0]
# Counted by attempt rather than by failure: a rejected registration still cost a
# round trip and a uniqueness check, and on an instance with signups open the
# thing worth bounding is how fast accounts can appear at all.
# Counted per attempt, not per failure: with signups open, what is bounded is how
# fast accounts can appear.
wait = register_by_address.retry_after(client_address())
if wait is not None:
return _throttled(wait)
@@ -252,11 +236,10 @@ async def register():
logger.warning("registration refused (setup window closed) email=%s from=%s", email, client_address())
await audit.record(audit.REGISTRATION_REFUSED, email=email, detail="setup window closed")
return json_error(SETUP_CLOSED, 403)
# The first account bootstraps the admin and is allowed, inside the setup
# window above, even when registration is otherwise closed. After that, an invite lets one person in
# while it is closed (#5172). One that was offered is redeemed even while
# registration is open, so the list still says who used it, and one that
# doesn't hold is refused rather than ignored.
# The first account becomes the admin and is allowed inside the setup window
# even with registration closed. After that, an invite lets one person in while
# it is closed. An offered invite is redeemed even when registration is open, so
# the list says who used it, and an invalid one is refused, not ignored.
invite_id = None
if not is_first and invite:
invite_id = await redeem(db, invite, email)
@@ -281,17 +264,9 @@ async def register():
)
db.add(user)
if is_first:
# Registration CLOSES the moment the instance has an owner.
#
# Not "defaults closed" — that would still need the first person to get in
# somehow. Closed as a CONSEQUENCE of the admin account existing, which is
# the only formulation with no open window in it. Leaving the setting on
# meant the gap between "my account exists" and "I remembered to turn it
# off in Settings" was wide open, and on a public host that gap is the
# entire exposure — it starts the moment DNS resolves.
#
# An admin who wants a second person sends them an invite (Settings →
# Invites), which works with registration closed.
# Registration closes as soon as the admin account exists, so there is no
# window between "my account exists" and remembering to close it. More
# people join by invite (Settings → Invites).
await set_settings(db, {"allow_registration": False})
if invite_id is not None:
await db.flush()
@@ -312,9 +287,8 @@ async def _check_credentials(db, email: str, password: str, route: str) -> User
"""The account these credentials open, or None. Counts the failure against the
throttle, logs it under `route`, and clears the account's history on success.
A missing account still pays for a hash. Without that, "no such account" returns
in microseconds while a wrong password takes bcrypt's deliberate ~100ms, and the
difference is a reliable oracle for which emails have accounts here.
A missing account still pays for a hash, so response time doesn't reveal which
emails have accounts.
"""
user = await db.scalar(select(User).where(User.email == email))
if user is None or not user.password_hash:
@@ -378,7 +352,7 @@ async def me():
@login_required
async def storage():
"""How much attachment storage this account uses, and its limit (None when it
has none). For the Account page (#2939 §4)."""
has none). For the Account page."""
async with session_scope() as db:
return jsonify(
{
@@ -394,7 +368,7 @@ FORGOT_SENT = "If an account uses that address, a reset link is on its way. It w
@bp.post("/forgot-password")
async def forgot_password():
"""Email a reset link to whoever owns this address (#5266).
"""Email a reset link to whoever owns this address.
Says the same thing, at the same speed, whether or not the address has an
account: the link is made and sent off the request. Throttled per visitor address
@@ -428,7 +402,7 @@ async def forgot_password():
@bp.post("/reset-password")
async def reset_password():
"""Set a new password with a reset link an admin made (#5173), and sign the
"""Set a new password with a reset link, and sign the
account out everywhere else: every other session, and every linked device.
The browser that used the link is signed in under the new password, since holding
@@ -497,8 +471,8 @@ def _stay_signed_in(user: User) -> None:
@login_required
async def change_password():
"""Change the account's password, and sign it out everywhere else: every other
web session, and every linked device (family idea #5105, practice 4). The
browser that changed it stays signed in.
web session, and every linked device. The browser that changed it stays signed
in.
The current password is required, so a session left open on someone else's
screen can't take the account over. A wrong one counts against the sign-in
@@ -536,8 +510,8 @@ async def change_password():
@login_required
async def sign_out_elsewhere():
"""Sign the account out of every other browser and unlink every device, keeping
this one signed in (family idea #5105, practice 4). For a session or a device
the person doesn't recognise, or one on a machine they no longer have."""
this one signed in. For a session or device the person doesn't recognise, or a
machine they no longer have."""
async with session_scope() as db:
user = await db.get(User, g.user_id)
if user is None:
@@ -551,7 +525,7 @@ async def sign_out_elsewhere():
return jsonify({"ok": True, "devices_unlinked": unlinked})
# --- Device (bearer) tokens for native clients — M8 sync hub ---
# --- Device (bearer) tokens for native clients ---
async def _email_of(db) -> str | None:
@@ -592,9 +566,8 @@ async def device_login():
if not email or not password:
return json_error("email and password are required", 400)
# Same budget as the web sign-in, and the SAME counters — this route hands out a
# long-lived bearer token, so leaving it unthrottled would just move the guessing
# here from /login.
# The same counters as the web sign-in: this route hands out a long-lived token,
# so an unthrottled one would just move the guessing here.
wait = _sign_in_block(email)
if wait is not None:
return _throttled(wait)
@@ -643,21 +616,15 @@ async def list_devices():
async def revoke_own_device():
"""Revoke the device token presented on THIS request.
What makes "unlink" on a native client actually stop its access. The client
can't use the id-keyed route below, because it doesn't reliably know its own
device id: a token pasted from the web app arrives without one, and `/me`
describes the user, not the device row. Identifying the row by the presented
token needs nothing the caller doesn't already hold, so it works for both ways
a client can be linked.
How "unlink" on a native client ends its access. A client doesn't reliably know
its own device id (a token pasted from the web app arrives without one), so the
row is found by the token it presents.
Declared above the `<device_id>` rule for reading order only — Werkzeug ranks a
static rule ahead of a converter regardless of registration order.
Werkzeug ranks this static rule ahead of `<device_id>` whatever the order.
"""
token = _bearer_token()
if token is None:
# A session-cookie caller holds no device token, so "revoke the one I'm
# using" is meaningless rather than merely unauthorized. The web app
# revokes by id.
# A session caller holds no device token to revoke; the web app revokes by id.
return json_error("no device token was presented", 400)
# Owner-scoped like every other device route. The hash already pins a single
# row; the guarantee shouldn't rest on one column.
+1 -1
View File
@@ -1,4 +1,4 @@
"""Admin routes for Settings → Backups (#5920): where backups stand, Back up now, and
"""Admin routes for Settings → Backups: where backups stand, Back up now, and
Test destination. What a backup holds and when it runs is in `backup_job`."""
from __future__ import annotations
+4 -5
View File
@@ -1,4 +1,4 @@
"""What goes into a backup, and putting it back (#5918, #5921).
"""What goes into a backup, and putting it back.
Every table the models define is exported, so a table added later is backed up
without anyone remembering to add it here. The few that are left out are named in
@@ -74,9 +74,8 @@ def _decoder(column):
async def export_contents(db) -> Contents:
"""Every exported table's rows in primary-key order, and the attachment files.
Primary-key order is load-bearing: Postgres returns rows in heap order, and an
UPDATE moves a row to the end of the heap, so without ORDER BY an unchanged table
would come out shuffled after any edit and nothing would deduplicate."""
Primary-key order keeps an unchanged table byte-identical between runs: heap order
shifts whenever a row is updated, and then nothing deduplicates."""
tables: dict[str, list[dict]] = {}
for table in exported_tables():
query = select(table).order_by(*table.primary_key.columns)
@@ -143,7 +142,7 @@ async def restore_tables(db, members: Iterable[Member]) -> dict[str, int]:
for member in members:
if member.kind == "table":
if member.name not in by_name:
# A table this version no longer has. Nothing to put it in.
# A table this version doesn't have. Nothing to put it in.
logger.warning("restore: skipping table %s, which this version doesn't have", member.name)
member.fileobj.read()
continue
+3 -4
View File
@@ -1,10 +1,9 @@
"""Where the daily backup goes, and getting a backup file there (#5919).
"""Where the daily backup goes, and getting a backup file there.
Two kinds of destination, both set in Settings → Backups:
- **folder**: a directory inside the container, usually a mounted share. A backup
service that watches the share (Fabled Archives' folder mode, say) or a person
takes the files from there.
- **folder**: a directory inside the container, usually a mounted share, from which
a backup service or a person takes the files.
- **s3**: any S3-compatible bucket, addressed by path. Inkwell only puts and lists,
so a key that may only add objects is enough.
+1 -4
View File
@@ -1,4 +1,4 @@
"""The full backup as one file: an uncompressed tar per run (#5918).
"""The full backup as one file: an uncompressed tar per run.
A scheduled backup lands somewhere that keeps a history of them: a folder a backup
service watches, or an S3 bucket. From there it is usually copied off-site by a
@@ -25,9 +25,6 @@ that whatever did not change between two runs is the same bytes both times.
can thin the history on its own schedule.
- **Written atomically** through a dot-named `.part` file, which a folder watcher
ignores until the rename.
The shape is Scribe's (snippet #5875), with attachment files added: in a notes app
the files are as much the user's data as the rows.
"""
from __future__ import annotations
+3 -3
View File
@@ -1,12 +1,12 @@
"""The daily backup: when it runs, running it, and what happened (#5920).
"""The daily backup: when it runs, running it, and what happened.
- **When.** A tick every few minutes asks whether a backup is due: switched on, and
no good backup since the most recent scheduled hour. A server that was down at
that hour backs up as soon as it is back. Switching backups on runs the first one
within a tick, which is also the quickest proof the destination works.
- **Failure is "not yet".** A failed run is tried again after an hour, then two,
then four, then waits for the next day's hour, rather than at once or never
(rule 193). The destination may simply be restarting.
then four, then waits for the next day's hour, rather than at once or never. The
destination may simply be restarting.
- **One at a time.** An in-process lock, plus a Postgres advisory lock in case a
second server process is ever started against the same database. Back up now
while one is running is answered 409.
+8 -10
View File
@@ -1,14 +1,12 @@
"""A small S3 client for sending backups: AWS Signature Version 4, path-style
addressing (endpoint/bucket/key), and the two calls a backup needs.
Ported from FabledSteward's and FabledScribe's backup clients (Scribe snippet #5679),
which are tested against the same AWS examples as tests/test_backup_s3.py. Those use
httpx. This one uses the standard library's `http.client`, on a worker thread,
because Inkwell has no HTTP client dependency and two calls don't justify adding one.
Uses the standard library's `http.client` on a worker thread, since two calls don't
justify an HTTP client dependency. The signing is checked against AWS's published
examples (tests/test_backup_s3.py).
Inkwell puts and lists; it never deletes. The bucket keeps the history and thins it
by its own policy, so a key that may only add objects is enough. It is also the safer
key to give a service: a stolen one cannot wipe the history it writes into.
It puts and lists, never deletes, so an add-only key is enough; a stolen one can't
wipe the history the bucket keeps.
"""
from __future__ import annotations
@@ -128,15 +126,15 @@ def _says(status: int, reason: str, body: bytes) -> str:
}
said = hints.get(code) or message or reason
if status == 404 and not code:
# Seen live: an address without its port lands on port 80 and gets a plain
# 404 from whatever answers there (Scribe #5871).
# An address without its port lands on port 80 and gets a plain 404 from
# whatever answers there.
said = "nothing answered as S3 at that address; check the port"
return f"HTTP {status}{f' {code}' if code else ''}: {said}."
class _Throttled:
"""A file wrapper that reads no faster than `limit` bytes a second, so an upload
leaves room on the connection for everything else (rule 193)."""
leaves room on the connection for everything else."""
def __init__(self, f: IO[bytes], limit: int):
self._f = f
+63 -162
View File
@@ -1,73 +1,33 @@
"""The server hands out the clients it is in step with.
## Why the server, and not a release page
The Fabled-Git instance is private (issue 2091), so `install.sh` already cannot
fetch for anyone but the operator — and a self-hoster should not need an account
on someone else's forge to get the app for their own notes. The server they
already trust with the notes is the obvious place to get the client from.
It also keeps the two in step by construction. Client and server already
negotiate a sync protocol version before linking, so a server that also serves
the client cannot hand out a phone it is unable to talk to.
That argument was never Android-specific, which is why this module now serves a
TABLE of platforms rather than the one it was written for.
A self-hoster gets the apps from the server that already holds their notes, with no
account anywhere else. Client and server negotiate a sync protocol before linking,
so a server never hands out a client it can't talk to.
## Where the files come from
Two places, checked in that order, per platform:
Per platform, the first of:
1. `DATA_DIR/client/` — the mounted volume that already holds attachments. An
operator who wants a SPECIFIC build drops it there and it wins.
2. the copy baked into the image at build time — CI fetches the newest published
build of each client into every image, so `:dev` and `:latest` both carry a
full set and `docker compose pull` delivers new ones with nothing copied by
hand.
1. `DATA_DIR/client/`, on the mounted volume: a build dropped there wins.
2. The copy baked into the image, inside the package rather than under DATA_DIR,
which a volume mount would hide. CI bakes the newest build of each client into
every image.
The precedence is the point: the image is the default, and a person who wants to
override it should not have to fight it. The baked copy sits inside the package
rather than under DATA_DIR because DATA_DIR is a volume mount, and anything the
image wrote there would be hidden the moment one is attached.
**Precedence is decided per platform, not for the set.** A drop-in `.deb` does
not shadow the baked APK. The alternative — first directory holding anything
wins — would mean replacing one client silently retracts the other four.
Precedence is per platform: a drop-in `.deb` doesn't shadow the baked APK.
## What a platform needs on disk
Two files, and both must be present:
- the artifact, at a FIXED name (`inkwell.deb`, not
`Inkwell_2026.08.30.0307_amd64.deb`)
- its sidecar, `{version_name, version_code, size, sha256}`
**Fixed names, and the version only in the sidecar.** A version-stamped filename
would force this module to glob, and a glob over a directory an operator can drop
files into is how you serve the older of two builds — `write-manifest.sh` carries
a comment about exactly that, from the time it advertised a new version while
pointing at an old binary.
The sidecar exists because a version is not reliably readable from the artifact:
an APK keeps it in a binary AXML manifest Python cannot parse without the Android
build tools, and a `.deb` or an AppImage would each need a different unpacker. CI
writes the sidecar where the real value is already known.
The AppImage needs a THIRD file, `inkwell.AppImage.sig`. That is the minisign
signature the desktop updater verifies before replacing the running binary, and a
bundle that cannot be verified cannot be offered as an update — so a missing
signature makes the AppImage absent rather than merely unsigned.
- the artifact, at a fixed name (`inkwell.deb`). A version in the filename would
mean globbing, and a glob can pick the older of two builds.
- its sidecar, `{version_name, version_code, size, sha256}`, written by CI, because
a version can't be read reliably from an APK, a `.deb` or an AppImage.
- for the AppImage, `inkwell.AppImage.sig`: the signature the desktop updater
verifies. Without it the AppImage counts as absent.
## Absence is normal
A server with no client for a platform advertises none, and the web UI hides that
download rather than offering a button that 404s. Same for a mismatched pair: if
the sidecar's recorded size does not match the file on disk, the two did not
arrive together, and the server says it has nothing rather than serving one build
while describing another.
This is the ONLY state available before a platform's first build has ever
published, so it is an ordinary answer and never an error.
A platform with no build, or whose sidecar size doesn't match the file, is simply
not advertised, and the web UI hides that download.
"""
from __future__ import annotations
@@ -90,29 +50,17 @@ class Platform:
"""One installable client, and where its files sit."""
id: str
# What a person calls it. Named for the DISTRO rather than the package format
# ("Debian / Ubuntu", not ".deb") — someone knows which system they run and
# does not necessarily know which packaging it uses.
# What a person calls it: the system ("Debian / Ubuntu"), not the package format.
label: str
artifact: str
sidecar: str
mimetype: str
# An updater-verifiable bundle: `<artifact>.sig` must be present too, and its
# contents are published with the metadata. Only the AppImage, because it is
# the only bundle that can replace itself in place — a package-manager install
# cannot, by design (see the desktop's update.rs).
# An updater-verifiable bundle: `<artifact>.sig` must be present, and is published
# with the metadata. Only the AppImage can replace itself in place.
signed: bool = False
# Whether this platform's ordering key is an INTEGER.
#
# `version_code` is "whatever this platform's comparator reads", and that is not
# one type. Android's is an int because Android's own install gate compares one,
# and it must stay a JSON number — `ClientRelease` in core/src/sync/client.rs
# declares it `i64` and a string would fail to deserialize on every phone in the
# field. The desktop's is Tauri's semver key, `1.0.<minutes>`, which is the value
# its updater compares and is not an integer at all.
#
# Coercing everything to int was inherited from the days when Android was the
# only platform, and would have rejected every desktop sidecar written.
# Whether this platform's ordering key (`version_code`) is an integer. Android's
# is, and must stay a JSON number: core/src/sync/client.rs reads it as `i64`. The
# desktop's is Tauri's semver key, `1.0.<minutes>`.
code_is_int: bool = True
@property
@@ -124,17 +72,14 @@ class Platform:
return f"/api/client/{self.id}/download"
# ONE definition of what this server can hand out. Every route, the /api/config
# advertisement and the tests all read this table; adding a platform is adding a
# row.
# What this server can hand out. Every route, /api/config and the tests read this
# table; adding a platform is adding a row.
PLATFORMS: tuple[Platform, ...] = (
Platform(
id="android",
label="Android",
# UNCHANGED, and it must stay unchanged: the Android lane publishes these
# exact names, CI bakes them in under them, and clients in the field poll
# `/api/client/android`. Renaming them to match the pattern below would
# buy tidiness and strand every installed phone.
# Fixed: installed phones poll `/api/client/android`, and the Android lane
# publishes these exact names.
artifact="inkwell.apk",
sidecar="inkwell-android.json",
mimetype="application/vnd.android.package-archive",
@@ -160,9 +105,7 @@ PLATFORMS: tuple[Platform, ...] = (
label="Other Linux (AppImage)",
artifact="inkwell.AppImage",
sidecar="inkwell-linux-appimage.json",
# No registered type for an AppImage, and guessing one buys nothing: it is
# served as an attachment either way, and octet-stream is the answer that
# cannot be wrong.
# AppImage has no registered type; it is served as an attachment either way.
mimetype="application/octet-stream",
signed=True,
code_is_int=False,
@@ -189,10 +132,8 @@ bp = Blueprint("client_dist", __name__)
def _read(root: Path, platform: Platform) -> dict | None:
"""One platform's build in one directory, or None.
Never raises. A missing directory, an unreadable sidecar, malformed JSON, a
sidecar that describes a different file and — for a signed bundle — a missing
signature are all the same answer to the only question being asked, "is there
a client here I can honestly offer?", and that answer is no.
Never raises. A missing directory, an unreadable or malformed sidecar, a sidecar
describing a different file, or a missing signature all mean "no client here".
"""
try:
size = (root / platform.artifact).stat().st_size
@@ -211,9 +152,8 @@ def _read(root: Path, platform: Platform) -> dict | None:
# not a version, and it would render as a blank on the download card.
return None
# The pair has to describe one build. A sidecar left behind by a previous
# release would otherwise advertise a version this server cannot serve, and the
# client would download something other than what it was promised.
# The pair must describe one build, or a stale sidecar would advertise a version
# the file isn't.
if recorded != size:
return None
@@ -225,22 +165,17 @@ def _read(root: Path, platform: Platform) -> dict | None:
# sorts like one, which is not how "is this newer" works.
"version_code": code,
"size": size,
# Computed by CI over the same bytes it uploaded, so a client can tell a
# truncated download from a complete one BEFORE handing it to an installer.
# Not a trust anchor — the signature is that.
# Lets a client spot a truncated download before installing it. Not a trust
# anchor; the signature is.
"sha256": digest,
# A PATH, never an absolute URL: the client joins it to the base it is
# already linked to, so a compromised or misconfigured server cannot
# redirect the download somewhere else. `core/src/sync/client.rs` relies on
# this and says so.
# A path, never an absolute URL: the client joins it to the server it is
# linked to, so a misconfigured server can't redirect the download.
"url": platform.download_path,
}
if platform.signed:
# The signature travels WITH the metadata rather than behind its own route.
# It is ~100 bytes, it is public wherever these bundles are published, and
# the updater needs the version and the signature in the same breath — one
# request that cannot return a signature belonging to a different build.
# The signature travels with the metadata, so one response can't pair a
# version with another build's signature.
try:
sig = (root / platform.signature).read_text(encoding="utf-8").strip()
except OSError:
@@ -255,14 +190,11 @@ def _read(root: Path, platform: Platform) -> dict | None:
def _resolve(platform: Platform) -> tuple[Path, dict] | None:
"""Which directory this server serves a platform from, and what is in it.
The operator's drop-in beats the baked copy — someone who deliberately put a
build on the volume wants that build, older or not. A directory holding a
broken or half-copied pair does NOT shadow the image: it simply is not a
client, so the search moves on.
A drop-in on the volume beats the baked copy, older or not. A broken or
half-copied pair there doesn't shadow the image; the search moves on.
"""
# Both wrapped in Path(), not just the first. The asymmetry was arbitrary and it
# fails quietly: `_read` catches the TypeError a str `/` str raises and reports
# "no client here", so a directory that is perfectly fine reads as empty.
# Both must be Paths: `_read` swallows the TypeError of str / str and would report
# a good directory as empty.
for root in (Path(Config.client_root()), Path(BAKED_ROOT)):
release = _read(root, platform)
if release is not None:
@@ -282,9 +214,7 @@ def release(platform_id: str) -> dict | None:
def releases() -> dict[str, dict]:
"""Every client this server can hand out, keyed by platform id.
Platforms it holds nothing for are ABSENT rather than present-and-null, so a
caller can test for the key instead of distinguishing "no build" from "a
server that never had this platform".
A platform with no build is absent, not null.
"""
found = {p.id: _resolve(p) for p in PLATFORMS}
return {pid: r[1] for pid, r in found.items() if r is not None}
@@ -295,9 +225,7 @@ def advertisement() -> dict:
whole table, which the web UI renders the downloads section from. Phones ask
`/api/client/android` for their own update instead.
ABSENT rather than empty when the server holds none, so a client testing for the
key gets one unambiguous answer instead of having to distinguish "no client" from
"an old server that never had this field".
Absent, not empty, when the server holds none.
"""
found = releases()
return {"clients": found} if found else {}
@@ -305,11 +233,7 @@ def advertisement() -> dict:
@bp.get("/api/client")
async def client_index():
"""Everything this server holds, in one request.
The downloads UI needs all five to decide what to lead with, and five requests
to answer one question is five chances to render half a page.
"""
"""Everything this server holds, in one request, for the downloads UI."""
return jsonify({"clients": releases()})
@@ -317,12 +241,8 @@ async def client_index():
async def client_metadata(platform_id: str):
"""Version and digest without the payload. What an updater polls.
Public, because a client has to be able to ask "is there something newer?"
cheaply — before it has a token, in the case of a first pairing.
An unknown platform and a platform with no build both 404. They are the same
answer to the caller ("not here"), and distinguishing them would only tell an
unauthenticated stranger which platforms this build of the server knows about.
Public, so a client can ask "is there something newer?" before it has a token.
An unknown platform and one with no build both 404.
"""
found = release(platform_id)
if found is None:
@@ -334,25 +254,17 @@ async def client_metadata(platform_id: str):
async def client_update_manifest(platform_id: str):
"""The same build, in the shape the desktop's in-app updater reads.
Tauri's updater takes a manifest, not this module's metadata: `version` is the
ordering key it compares (the desktop's `1.0.<minutes>`, i.e. `version_code`),
and `url` + `signature` at the top level are its single-build form. So this is a
translation of `release()`, never a second description of the build.
A translation of `release()` into Tauri's single-build manifest: `version` is
the desktop's `version_code`, with `url` and `signature` at the top level.
Only for a SIGNED platform. The updater verifies the signature against the key
baked into the app before it replaces anything, so a server mirrors builds here
and cannot sign them: the trust anchor does not move with the feed. A platform
with no signature has no manifest, and a server holding no AppImage 404s, which
the desktop reports as "this server has no update to offer", never as "up to
date".
Signed platforms only. The updater checks the signature against the key built
into the app, so a server can mirror builds but not sign them. No AppImage: 404,
which the desktop shows as "no update to offer", not "up to date".
`url` must be absolute for the updater, which is the one place this module
hands out more than a path. It is built from the address THIS request arrived
on, not from Settings, because the updater sends the same Authorization header
to it that it sent here: the token goes back to the host that was asked, and
nowhere else.
`url` is absolute, built from the address this request arrived on rather than
Settings, because the updater sends the same Authorization header there.
Public, like the metadata above. The bytes `url` names are not.
Public; the bytes `url` names are not.
"""
platform = BY_ID.get(platform_id)
if platform is None or not platform.signed:
@@ -376,13 +288,9 @@ async def client_update_manifest(platform_id: str):
async def client_download(platform_id: str):
"""The bytes themselves.
Authenticated — by session cookie from a browser, or by device bearer token
from a client updating itself; `login_required` accepts either. The metadata
above is public; the bytes are not for anyone who can reach the port.
Throttled per account (Settings → Security), because each answer is tens of
megabytes: one account or a leaked device token looping here could fill the
server's uplink (family idea #5103, practice 6).
Needs a session or a device token. Throttled per account (Settings → Security),
because each answer is tens of megabytes and a looping client could fill the
uplink.
"""
platform = BY_ID.get(platform_id)
if platform is None:
@@ -406,23 +314,16 @@ async def client_download(platform_id: str):
async def send_artifact(root: Path, platform: Platform, found: dict) -> Response:
"""The artifact, with its sha256 as the ETag and Range honoured against it.
Quart's own ETag is the file's mtime, size and path. That changes when a
container restarts on the same bytes, so a phone resuming a download across a
redeploy would be told its partial copy is stale. It also says nothing about
content. The sidecar's sha256 is the identity the client already checks, so
`If-Range` is answered against the bytes themselves (family idea #5103,
practice 6).
So Quart's ETag and conditional handling are both turned off, and the
conditional pass runs here after the right ETag is set. Run before it, Range
and If-None-Match would be judged against the wrong one.
Quart's ETag is mtime, size and path, which changes when a container restarts on
the same bytes, so a resumed download would be told its partial copy is stale.
Quart's ETag and conditional handling are off, and the conditional pass runs here
after the sha256 ETag is set.
"""
response = await send_from_directory(
root, platform.artifact, mimetype=platform.mimetype, add_etags=False, conditional=False
)
response.set_etag(found["sha256"])
# Only behind a login, so no shared cache may keep a copy (family idea #5105,
# practice 11). Quart marks every file it sends `public`.
# Behind a login, so no shared cache may keep a copy. Quart marks files `public`.
response.cache_control.public = False
response.cache_control.private = True
# Without this some browsers try to render it, and Android's download handler
+4 -13
View File
@@ -1,15 +1,8 @@
from __future__ import annotations
# The colour palette, and the one place that clamps to it.
#
# It lived on `models/note.py` until M315, when a note stopped having a colour. A
# palette defined on the model that lost one would be a standing invitation to put the
# column back; here it reads as what it now is — a LABEL's vocabulary, shared with the
# import path that still names a colour.
#
# Keys, not tints. The actual colours live in each client (frontend/src/notes/colors.ts
# and NoteTint.kt), so they can be retuned without a schema migration — which M315 spent
# two steps doing.
# The label colour palette, and the one place that clamps to it. Keys, not tints: the
# colours live in each client (frontend/src/notes/colors.ts, NoteTint.kt), so they can
# be retuned without a migration.
NOTE_COLORS = {
"default",
"red",
@@ -29,8 +22,6 @@ __all__ = ["NOTE_COLORS", "normalize_color"]
def normalize_color(color: object) -> str:
"""Return `color` if it's a known palette key, else the default.
`default` is no longer something anybody can CHOOSE — nothing offers a colour
picker since M315 — but it is still where unrecognised input has to land, so this
fallback is unreachable by choice rather than dead.
Nothing offers `default` as a choice; it is where unrecognised input lands.
"""
return color if color in NOTE_COLORS else "default"
+3 -7
View File
@@ -3,9 +3,7 @@ from __future__ import annotations
import asyncio
from datetime import datetime, timedelta
# Small, dependency-free value coercions shared across the blueprints. Kept in one
# place so the "parse an ISO date" / "is this flag truthy" logic has a single
# definition instead of a near-identical copy per module.
# Small value coercions shared across the blueprints.
def iso(dt: datetime | None) -> str | None:
@@ -65,10 +63,8 @@ def detach(running: set[asyncio.Task], coro) -> asyncio.Task:
def expired_before(now: datetime, retention_days: int) -> datetime | None:
"""The cutoff: anything older than this has expired. `None` = retention is off.
Kept separate from the query so the window arithmetic — including the two ways
to say "never" (0 and negative, the latter reachable by typing a stray minus in
Settings) — is testable without a database. Trash and the audit log both expire
through it.
0 and negative both mean "never". Separate from the query so it is testable
without a database. Trash and the audit log both expire through it.
"""
if retention_days <= 0:
return None
+10 -19
View File
@@ -7,19 +7,14 @@ from pathlib import Path
class Config:
"""Bootstrap configuration.
For a basic install, ``INKWELL_DATABASE_URL`` is the ONLY required env var —
every other tunable lives in the DB-backed Settings UI (rule 25). The only other
env var is an optional "break-glass" item:
``INKWELL_DATABASE_URL`` is the only required env var; everything else is in
the Settings UI, stored in the database. Optional:
- ``INKWELL_SECRET_KEY`` — optional override for the cookie-signing secret.
If unset, a key is generated and persisted in the DB (see
``inkwell.settings.load_or_create_secret_key``), so sessions survive
restarts with no volume required.
- ``INKWELL_SECRET_KEY`` overrides the cookie-signing secret. Unset, a key is
generated and stored in the database, so sessions survive restarts.
Uploaded media lives under ``DATA_DIR`` — a fixed, authoritative path
(``/var/inkwell``), intentionally NOT configurable (a mutable data path only
invites breakage). Mount a volume there if you want uploads to persist across
container recreation; a text-notes-only install never writes to it.
Uploads live under ``DATA_DIR`` (``/var/inkwell``), which is fixed. Mount a
volume there to keep them across container recreation.
"""
DATA_DIR = "/var/inkwell"
@@ -39,15 +34,11 @@ class Config:
@classmethod
def client_root(cls) -> Path:
"""Where an operator DROPS IN clients for this server to hand out.
"""Where client builds can be dropped in to override the image's own.
One directory for every platform; `client_dist.py` picks files out of it by
name. Under DATA_DIR because it is a mounted volume: a build placed here
survives container recreation, and it beats the copy baked into the image,
which is the whole point of the directory existing.
Empty is the ordinary case — the image ships its own set and most operators
never touch this.
One directory for every platform; `client_dist.py` picks files by name. On
the data volume, so a build placed here survives container recreation.
Usually empty.
"""
return Path(cls.DATA_DIR) / "client"
+2 -4
View File
@@ -24,10 +24,8 @@ async def _probe(url: str) -> None:
async def wait_for_db(attempts: int = ATTEMPTS, delay: float = DELAY_SECONDS) -> bool:
"""Poll the database until it accepts a connection, up to `attempts` tries.
Startup runs `alembic upgrade head` as the container's first DB touch, so a
database that is a moment slow to accept connections would otherwise crash the
container. This gives it a short window to come up, verifying once per `delay`
seconds, instead of failing on the first missed connection.
Runs before `alembic upgrade head`, so a database that is slow to come up delays
the start instead of failing it.
"""
url = Config.DATABASE_URL
for attempt in range(1, attempts + 1):
+1 -1
View File
@@ -1,4 +1,4 @@
"""Admin routes over groups: named sets of people a note can be shared with (#5177).
"""Admin routes over groups: named sets of people a note can be shared with.
A group is the instance's, not anyone's: the admin makes it and says who is in it,
and every member of the instance can share a note with it. Sharing with a group
+14 -29
View File
@@ -2,28 +2,16 @@
curl -fsSL https://notes.example.com/install.sh | sh
## Why the server templates it
`curl | sh` gives a script no way to learn where it came from, so the server writes
its own address into the copy it hands out.
`curl | sh` gives a script no way to learn where it came from: no argv, no referrer,
nothing. So the server writes its own address into the copy it hands out, and the
script installs from there without being told. Anything else means a self-hoster
editing a URL by hand, which is the friction `client_dist.py` exists to remove.
Arbitrary text in a piped script is the whole attack, so exactly one variable is
substituted, `INKWELL_SERVER_DEFAULT`, with a value that passed [SAFE_ADDRESS]
(nothing a shell reads as more than characters). The script checks the shape again.
An address that can't pass is refused.
## Which way it fails
A server that can put arbitrary text into a script a person pipes to `sh` is the
whole attack. So the substitution is ONE variable, `INKWELL_SERVER_DEFAULT`, given a value
that has already passed [SAFE_ADDRESS] — no quote, no space, no `$`, nothing a shell
reads as anything but characters — and the script checks the shape again before it
uses it. If the address cannot pass, the route refuses rather than serving a script
that would install from somewhere else.
## Where the script comes from
ONE script. The forge serves `desktop/packaging/install.sh` raw, and that is how the
operator installs. The image carries a copy beside this module (the Dockerfile copies
it in), and a source checkout reads the original. Same bytes either way; this module
changes exactly one line of them.
The script is `desktop/packaging/install.sh`; the image carries a copy beside this
module, and a source checkout reads the original.
"""
from __future__ import annotations
@@ -47,11 +35,10 @@ SOURCE_SCRIPT = Path(__file__).resolve().parents[2] / "desktop" / "packaging" /
# `render` refuses a script that does not, rather than serving one with no address in.
PLACEHOLDER = 'INKWELL_SERVER_DEFAULT=""'
# What a server address may look like before it goes into a shell script: a scheme,
# a host name or IPv4 address, an optional port and an optional path. Deliberately
# narrower than a URL. Every character it admits is inert inside single quotes, and
# an address it turns away (an IPv6 literal, a path with `%` in it) is answered by
# setting Settings → Public address, not by widening this.
# A server address safe to put in a shell script: scheme, host name or IPv4, optional
# port and path. Narrower than a URL on purpose; every character is inert inside
# single quotes. An address it refuses (IPv6, `%` in a path) is fixed by setting
# Settings → Public address.
SAFE_ADDRESS = re.compile(r"https?://[A-Za-z0-9.-]+(:[0-9]{1,5})?(/[A-Za-z0-9._~/-]*)?")
@@ -68,10 +55,8 @@ def script_text() -> str | None:
def server_address(public_url: str, scheme: str, host: str) -> str | None:
"""Where the installer should fetch from, or None if no safe answer exists.
Settings → Public address when it is set: it is the address the operator says
people use, and behind a reverse proxy the request's own Host may be an internal
one. Otherwise the address this request arrived on, which is the one the person
running `curl` just reached.
Settings → Public address when set, since behind a proxy the request's Host may
be internal. Otherwise the address this request arrived on.
"""
address = (public_url or f"{scheme}://{host}").rstrip("/")
return address if SAFE_ADDRESS.fullmatch(address) else None
+3 -7
View File
@@ -1,16 +1,12 @@
"""Invites: an admin lets one person register while registration is closed (#5172).
Before these, adding a second person meant turning registration back on for the
whole internet while they signed up, with no record of who was let in (#2939 §1).
"""Invites: an admin lets one person register while registration is closed.
An invite is a link carrying a random token. It works once, until it expires, and
optionally only for one email address. Only the token's hash is kept, so the link is
shown once, when it is made; an admin who loses it revokes it and makes another.
This module is the invite itself: its status, its wire shape, and redemption, which
This module is the invite itself: its status, wire shape, and redemption, which
`auth.register` calls inside the transaction that creates the account. The admin
routes are in `invites_api.py`, the same split as `settings` and `settings_api`, and
for the same reason: the routes need `auth`, and `auth` needs this.
routes are in `invites_api.py`, because they need `auth` and `auth` needs this.
"""
from __future__ import annotations
+1 -1
View File
@@ -1,4 +1,4 @@
"""Admin routes for invites: make one, list them, revoke one (#5172).
"""Admin routes for invites: make one, list them, revoke one.
What an invite is, and how it is redeemed, is in `invites.py`.
"""
+4 -6
View File
@@ -1,10 +1,8 @@
"""Shared note↔label membership reconciliation. The "set a note's MANUAL (picker)
labels, leave the tag-sourced ones alone" logic was duplicated line-for-line between
the labels-picker API (notes.set_note_labels) and sync push (sync._apply_note_manual_labels).
Single home so both stay in lockstep. via_tag=True rows track the body #tags and are
governed by _lift_and_reconcile_tags — this function never touches them.
"""Note↔label membership. Sets a note's manual (picker) labels for both the labels
API and sync push, leaving via_tag=True rows, which track the body's #tags, to
_lift_and_reconcile_tags.
Also the one definition of a LIVE tag and of deleting one (#5382)."""
Also the one definition of a live tag and of deleting one."""
from __future__ import annotations
from datetime import datetime, timezone
+15 -27
View File
@@ -33,13 +33,11 @@ async def _label_note_count(db, label_id) -> int:
async def _merge_into(db, source: Label, target: Label) -> None:
"""Move every note tagged with `source` onto `target`, then tombstone `source`.
Shared by the explicit `/merge` route and by a rename that lands on a name
another tag already holds — those are the same operation, and having one body
is what stops them drifting into two answers for one question.
Used by `/merge` and by a rename onto a name another tag holds, which are the
same operation.
Note-body `#tags` are NOT rewritten, so a note whose body still literally
contains the source `#tag` will re-mint that tag on its next edit. Retiring a
tag means editing it out of the text; a known, documented nuance.
Body `#tags` are not rewritten, so a note still containing the source `#tag`
re-creates it on its next edit; retiring a tag means editing it out of the text.
"""
# Notes already carrying the target: a note can't hold the same label twice
# (composite PK), so the source attachment there is just dropped as a dup.
@@ -81,9 +79,8 @@ async def list_labels():
)
).all()
) if labels else {}
# When each was last used: the newest edit among the notes carrying it, trash
# aside. The rail shows the few most recent (#5931), so a tag in daily use
# stays in reach however many others there are.
# When each was last used: the newest edit among its notes, trash aside. The
# rail shows the most recent few, so a tag in daily use stays in reach.
last_used = dict(
(
await db.execute(
@@ -114,10 +111,9 @@ async def create_label():
if not name:
return json_error("tag name is required", 400)
async with session_scope() as db:
# Idempotent: creating an existing tag just returns it. Case-INSENSITIVE,
# like the clients' `find_or_create_label` — a case-sensitive match here was
# the path that MINTED the "Groceries" beside "groceries" pair that no synced
# client can hold, since their `labels` index is unique on `lower(name)`.
# Creating an existing tag returns it. Case-insensitive, like the clients:
# their `labels` index is unique on `lower(name)`, so they can't hold both
# "Groceries" and "groceries".
existing = await db.scalar(
select(Label).where(named(g.user_id, name))
)
@@ -146,11 +142,8 @@ async def update_label(label_id: str):
if label is None:
return not_found()
if has_name:
# Case-INSENSITIVE, matching the clients' `find_or_create_label` and the
# local store's `lower(name)` unique index. The Postgres constraint here
# is on the raw name, so the database would happily hold "Groceries"
# beside "groceries" — but no synced client can store both, so letting
# one be made is letting a pull fail later on a phone.
# Case-insensitive, like the clients' unique `lower(name)` index. Postgres
# would accept both spellings, and a phone's pull would then fail.
clash = await db.scalar(
select(Label).where(
named(g.user_id, name),
@@ -158,12 +151,9 @@ async def update_label(label_id: str):
)
)
if clash is not None:
# Renaming onto an existing tag MERGES the two rather than failing:
# typing an existing tag's name onto this one says they are the same
# thing. The OLDER row survives and takes the new spelling — age is
# the one property that does not depend on which of the two the
# caller happened to be renaming, so A→B and B→A agree. Ties go to
# the incumbent. Mirrors `store::rename_label` exactly.
# Renaming onto an existing tag merges the two. The older row survives
# and takes the new spelling, so A→B and B→A agree; ties go to the
# incumbent. Mirrors `store::rename_label`.
if clash.created_at <= label.created_at:
survivor, doomed = clash, label
else:
@@ -195,9 +185,7 @@ async def delete_label(label_id: str):
async def merge_label(label_id: str):
"""Merge `label_id` (source) INTO the label given by body {"into": <id>}: move
every note tagged with the source onto the target, then tombstone the source. Both
must be owned by the caller. Note-body `#tags` are NOT rewritten, so a note whose
body still literally contains the source #tag will re-mint that label on its next
edit — retire a tag by editing it out of the text (a known, documented nuance)."""
must be owned by the caller. Body `#tags` are not rewritten (see `merge_into`)."""
data = await request.get_json(silent=True) or {}
into = data.get("into")
async with session_scope() as db:
+7 -10
View File
@@ -1,18 +1,15 @@
"""What the server's emails say, and the one layout they all share (#5917).
"""What the server's emails say, and the one layout they all share.
Each email is a function returning an `Email`: a subject, a plain-text part and an
HTML part. `mailer.send` sends both as one multipart/alternative message, so a client
that shows HTML gets the branded version and one that doesn't still gets every word
and the link.
The HTML is deliberately email-shaped rather than web-shaped. It uses tables and
inline styles, because many mail clients drop `<style>` blocks and ignore flexbox.
It fetches nothing: the app sends with no outbound network beyond the mail server
(rule 164), and a remote image is a tracking pixel anyway. The one image, the
Inkwell logo, travels inside the message as an inline part (`cid:`), which mail
clients show without asking, unlike a remote image; SVG and `data:` images are
dropped by too many clients to rely on. Everything taken from settings or from a person (the site name, an address,
a link) is escaped before it goes into the HTML.
The HTML uses tables and inline styles, because many mail clients drop `<style>`
blocks and ignore flexbox. It loads nothing remote. The logo travels inside the
message as an inline part (`cid:`), which clients show without asking; SVG and
`data:` images are dropped by too many clients. Everything taken from settings or a
person (site name, address, link) is escaped.
Links come from Settings → General → Public address, never from the request: a
forged Host header would otherwise mail a stranger a link to the forger's site.
@@ -64,7 +61,7 @@ def _render(
site: str, subject: str, heading: str, lines: list[str], button: _Button | None, footer: str
) -> Email:
"""One email in the shared layout: the site's name, a heading, the message, a
button for the link, and a footer saying who sent it and why (preference 194).
button for the link, and a footer saying who sent it and why.
`lines` are paragraphs of plain text. The button goes after the first one, and the
text part spells its link out in full, since that is the only way a text-only
+4 -5
View File
@@ -1,10 +1,9 @@
"""Sending email over the SMTP server set in Settings → Email (#5266).
"""Sending email over the SMTP server set in Settings → Email.
Used for password reset links, invites, and the admin's test message. What each one
says is in `mail_templates`; this module only delivers. Email is an optional
feature (rule 164): with no server set, `mail_settings` answers None and everything
that would send says it is not available. A send that fails is logged and never
takes anything else down with it.
says is in `mail_templates`; this module only delivers. Email is optional: with no
server set, `mail_settings` answers None and everything that would send says it is
not available. A failed send is logged and takes nothing else down.
stdlib `smtplib`, run on a worker thread, so the event loop never waits on a mail
server and nothing new is installed.
+1 -1
View File
@@ -12,7 +12,7 @@ from . import Base
class AuditEvent(Base):
"""One thing that happened to an account: a sign-in, a failed one, a password
changed, a device linked (#2939 §5). Written by `audit.record`, read by admins in
changed, a device linked. Written by `audit.record`, read by admins in
Settings → Activity, and deleted after `audit_retention_days`.
"""
+1 -1
View File
@@ -12,7 +12,7 @@ from . import Base
class Invite(Base):
"""A single-use, expiring link an admin hands to one person so they can register
while registration is closed (#5172).
while registration is closed.
Only the token's SHA-256 hash is stored, as for device tokens; the link is shown
once, when the invite is made. Rows are never deleted by the app: a used, expired
+4 -4
View File
@@ -12,8 +12,8 @@ from . import Base
class Label(Base):
__tablename__ = "labels"
# A name is unique among LIVE tags only: a tombstone keeps its row for the change
# feed but gives its name back, so `#grocery` can be made again (#5382, 0040).
# A name is unique among live tags only: a tombstone keeps its row for the change
# feed but gives its name back, so `#grocery` can be made again.
__table_args__ = (
Index(
"uq_labels_owner_name_live",
@@ -36,8 +36,8 @@ class Label(Base):
updated_at: Mapped[datetime] = mapped_column(
DateTime(timezone=True), nullable=False, server_default=func.now(), onupdate=func.now()
)
# Sync (M8): monotonic per-row revision (from sync_revision_seq via DB trigger) so a
# label rename/recolor/merge/delete propagates to native clients independently of notes.
# Monotonic per-row revision (sync_revision_seq, set by a DB trigger), so a rename,
# recolour, merge or delete reaches native clients independently of notes.
sync_revision: Mapped[int | None] = mapped_column(BigInteger(), nullable=True)
# Tombstone: set means deleted. The row stays so the change feed can carry the
# delete to devices; every read of the catalog skips it (`labeling.live`).
+6 -8
View File
@@ -22,11 +22,9 @@ class Note(Base):
owner_id: Mapped[uuid.UUID] = mapped_column(
UUID(as_uuid=True), ForeignKey("users.id", ondelete="CASCADE"), nullable=False
)
# The note's NAME: its first non-empty body line, else its first checklist item
# (see notes.derive_display_title). There is no title field to prefer — a note is
# a body plus optional items, and this is simply the first thing written in it.
# Persisted so search results and export filenames have something to say, and so
# the full-text vector can weight it above the rest of the body.
# The note's name: its first non-empty body line (notes.derive_display_title).
# Stored for search results and export filenames, and so full-text search can
# weight it above the rest of the body.
display_title: Mapped[str] = mapped_column(Text(), nullable=False, server_default="")
body: Mapped[str] = mapped_column(Text(), nullable=False, server_default="")
# Manual drag order (higher = earlier); 0 until the user reorders.
@@ -44,9 +42,9 @@ class Note(Base):
updated_at: Mapped[datetime] = mapped_column(
DateTime(timezone=True), nullable=False, server_default=func.now(), onupdate=func.now()
)
# Sync (M8): a monotonic per-row revision drawn from sync_revision_seq and assigned
# by a DB trigger on every insert/update — the delta cursor native clients pull
# against. Nullable in the ORM because the trigger populates it server-side.
# Monotonic per-row revision from sync_revision_seq, set by a DB trigger on every
# write: the cursor native clients pull against. Nullable in the ORM because the
# trigger fills it.
sync_revision: Mapped[int | None] = mapped_column(BigInteger(), nullable=True)
# Hard-delete tombstone: non-null => permanently deleted (content cleared), kept so
# offline clients learn the row is gone. Distinct from deleted_at (= recoverable trash).
+1 -2
View File
@@ -25,7 +25,6 @@ class NoteAttachment(Base):
filename: Mapped[str | None] = mapped_column(Text(), nullable=True)
mime: Mapped[str] = mapped_column(Text(), nullable=False)
size: Mapped[int] = mapped_column(BigInteger(), nullable=False)
# Content hash (sha256 hex) for client-side dedupe + integrity over sync. Nullable
# for rows created before this column (no backfill).
# sha256 hex, for client-side dedupe and integrity over sync. Null on older rows.
sha256: Mapped[str | None] = mapped_column(Text(), nullable=True)
created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False, server_default=func.now())
+1 -1
View File
@@ -11,7 +11,7 @@ from . import Base
class NoteUserState(Base):
"""A recipient's own pin, archive and place for a note shared with them (#5176).
"""A recipient's own pin, archive and place for a note shared with them.
The owner's live on the note itself. `position` null means "where the owner put
it", so a recipient who only pins a note doesn't freeze its order. `updated_at`
+2 -2
View File
@@ -11,8 +11,8 @@ from . import Base
class PasswordReset(Base):
"""A one-time link an admin makes so a person who forgot their password can set
a new one (#5173). There is no mail path; the admin hands the link over.
"""A one-time link for setting a new password, made by an admin or emailed by
"forgot password".
Only the token's SHA-256 hash is stored, as for invites and device tokens. Making
a new link for an account deletes its unused ones, so only the newest works.
+1 -1
View File
@@ -11,7 +11,7 @@ from . import Base
class Setting(Base):
"""A single persisted key/value setting. Values are JSON-encoded text; the
typed defaults + metadata live in the code registry (inkwell.settings), so
an empty table means "all defaults" (rule 26)."""
an empty table means "all defaults"."""
__tablename__ = "settings"
+3 -3
View File
@@ -15,9 +15,9 @@ class Share(Base):
A resource's OWNER is tracked on the resource row itself (its owner_id column);
this table records grants BEYOND the owner — to a single user or to a whole
group. It is polymorphic (resource_type + resource_id) so every future
shareable entity (notes first, in M1) reuses one table and one ACL predicate.
Exactly one of shared_with_user_id / shared_with_group_id is set.
group. Polymorphic (resource_type + resource_id), so any shareable entity uses
one table and one ACL predicate. Exactly one of shared_with_user_id /
shared_with_group_id is set.
"""
__tablename__ = "shares"
+1 -1
View File
@@ -11,7 +11,7 @@ from . import Base
class ShareRevocation(Base):
"""A note that someone lost access to, for their devices to delete (#5175).
"""A note that someone lost access to, for their devices to delete.
The change feed only carries notes a person can see, so a share ending would
otherwise leave a stale copy on every device that had synced it. The revision
+1 -1
View File
@@ -18,7 +18,7 @@ class User(Base):
email: Mapped[str] = mapped_column(CITEXT(), nullable=False, unique=True)
# First registered user becomes admin (see auth.register); admin gates Settings.
is_admin: Mapped[bool] = mapped_column(Boolean(), nullable=False, server_default=func.false())
# Nullable: leaves room for external-identity-only accounts later (rule 26).
# Nullable, for accounts that sign in only through an external identity.
password_hash: Mapped[str | None] = mapped_column(Text(), nullable=True)
display_name: Mapped[str] = mapped_column(Text(), nullable=False)
# Moved on by a password reset to sign the account out everywhere: every session
+1 -1
View File
@@ -1,4 +1,4 @@
"""How a note sits on one person's board: pinned, archived, and where (#5176).
"""How a note sits on one person's board: pinned, archived, and where.
For the owner that is the note's own columns. For someone it is shared with it is
their `NoteUserState` row, so the owner pinning a shared shopping list doesn't
+20 -38
View File
@@ -72,11 +72,9 @@ async def list_notes():
label_params = request.args.getlist("label")
has_attachment = coerce_bool(request.args.get("has_attachment"))
query_text = (request.args.get("q") or "").strip()
# Optional creation-date range — the "browse by when" / Timeline lens. Both bounds
# are ISO-8601 instants forming a HALF-OPEN interval [created_after, created_before),
# so a client can pass local day-boundaries (start-of-day .. start-of-next-day)
# without off-by-one. `sort=created` orders newest-captured first for a chronological
# timeline; the default keeps the board's pinned/position/updated order.
# Optional creation-date range, half-open [created_after, created_before), so a
# client can pass local day boundaries. `sort=created` is newest first; the
# default is the board's pinned/position/updated order.
after_param = request.args.get("created_after")
before_param = request.args.get("created_before")
sort = request.args.get("sort")
@@ -85,7 +83,7 @@ async def list_notes():
return json_error("invalid shared", 400)
async with session_scope() as db:
# Through the viewer's own pin, archive and place: on a note shared with them
# those are theirs, not the owner's (#5176).
# those are theirs, not the owner's.
stmt = join_state(select(Note), g.user_id).where(note_visible())
stmt = apply_filter(stmt, filter_name, archived_for(g.user_id))
# Trash is the owner's: a note its owner trashed leaves a recipient's board
@@ -112,10 +110,8 @@ async def list_notes():
return json_error("invalid created_before", 400)
stmt = stmt.where(Note.created_at < before_dt)
if query_text:
# Full-text match over the note's name + body (generated tsvector,
# migrations 0005/0026), ranked. This is the ONLY text search now: the
# separate facet-less `/search` route was removed because landing on it
# was the one place you could not also narrow by tag (note 2930).
# Ranked full-text match over the note's name and body (a generated
# tsvector). The only text search, so it combines with every other filter.
tsquery = func.websearch_to_tsquery("english", query_text)
search_col = literal_column("notes.search_vector")
stmt = stmt.where(search_col.op("@@")(tsquery)).order_by(
@@ -168,8 +164,7 @@ async def complete_reminder(note_id: str):
return jsonify(await _serialize_note(db, note, g.user_id))
# The longest snooze: 30 days. The core clamps to the same range
# (`store::SNOOZE_MAX_MINUTES`), so a device and the web agree (#5386).
# The longest snooze: 30 days, the same as the core's `store::SNOOZE_MAX_MINUTES`.
SNOOZE_MAX_MINUTES = 60 * 24 * 30
@@ -283,9 +278,8 @@ async def import_notes():
if len(zf.infolist()) > IMPORT_MAX_ENTRIES:
return json_error("that archive has too many files to import", 413)
# Bound total decompressed bytes so a zip bomb can't exhaust memory/disk. Any entry
# (or the cumulative total) blowing the budget aborts the whole import — nothing is
# committed, since the session rolls back when the exception exits its block.
# Bound decompressed bytes so a zip bomb can't exhaust memory or disk. Going over
# aborts the whole import; the session rolls back, so nothing is committed.
budget = _ImportBudget()
try:
specs, source = _read_import_specs(zf, budget)
@@ -321,10 +315,7 @@ async def import_notes():
async def list_titles():
"""Owner's non-trashed notes, keyed by their display NAME.
Survived the removal of [[wiki-links]] (note 2897) because it was serving two
different things, and only one of them was linking. This is what the command
palette lists so someone can jump to a note by name — which is recall, the thing
this app is actually for. The `[[` autocomplete that also read it is gone.
What the command palette lists, so someone can jump to a note by name.
"""
async with session_scope() as db:
rows = (
@@ -367,7 +358,7 @@ async def reorder_notes():
note = visible.get(nid)
if note is None:
continue
# A note shared with the caller moves on their board only (#5176).
# A note shared with the caller moves on their board only.
if note.owner_id == g.user_id:
note.position = total - index
else:
@@ -386,8 +377,8 @@ async def create_note():
if is_empty_note(body, item_texts):
return json_error("note is empty", 400)
async with session_scope() as db:
# Items still arrive separately — a client holds a list, not a blob — but they
# are folded into the body, which is where a checklist lives now (M304).
# Items may arrive separately, from a client holding a list, and are folded
# into the body, where a checklist lives.
for text in item_texts:
body = append_item(body, text)
note = Note(owner_id=g.user_id, body="", display_title="", position=await top_position(db, g.user_id) + 1)
@@ -418,9 +409,8 @@ async def update_note(note_id: str):
owned = note.owner_id == g.user_id
if not owned:
# Someone the note is shared with may pin and archive it on their own
# board (#5176), and change its text if it was shared at `edit`; nothing
# else. They can already read it, so saying so is no oracle. Text from a
# view share 404s, as it did before they could do anything at all.
# board, and change its text if shared at `edit`; nothing else. Text from
# a view share 404s.
if set(data) - {"body", *OWN_FIELDS}:
return json_error("only the owner can change that", 403)
if "body" in data and await _get_editable(db, note_id) is None:
@@ -576,11 +566,6 @@ async def update_item(note_id: str, item_id: str):
return await _rewrite_body(db, note, body)
# The reorder route is gone with M304. Reordering a checklist is moving a line, which
# is something a text editor already does and no client ever called this for — the
# only reference to it in the tree was a test asserting the route existed.
@bp.post("/<note_id>/attachments")
@login_required
async def upload_attachment(note_id: str):
@@ -621,7 +606,7 @@ async def get_attachment(note_id: str, att_id: str):
if aid is None:
return not_found()
async with session_scope() as db:
# Owner OR shared may view (rule 47). A purged note's files are gone with it.
# Owner or a share may view. A purged note's files are gone with it.
note = await _get_visible(db, note_id)
if note is None:
return not_found()
@@ -638,9 +623,8 @@ async def get_attachment(note_id: str, att_id: str):
response = await send_file(str(file_path), mimetype=mime)
response.headers["X-Content-Type-Options"] = "nosniff"
response.headers["Cache-Control"] = "private, max-age=86400"
# Only trusted raster image types render inline; everything else — notably
# image/svg+xml, which can carry script — downloads, so a shared note's attachment
# can't execute script in a viewer's session (rule 47 = shares).
# Only trusted raster image types render inline. Everything else downloads,
# notably SVG, which can carry script into a viewer's session.
disposition = "inline" if mime in ALLOWED_IMAGE_MIMES else "attachment"
response.headers["Content-Disposition"] = f'{disposition}; filename="{_header_filename(filename)}"'
return response
@@ -722,10 +706,8 @@ async def delete_note(note_id: str):
return not_found()
if note.deleted_at is None:
return json_error("note must be trashed before permanent delete", 409)
# A tombstone, not a dropped row. Deleting the row outright would leave the
# server with no record the note ever existed, so a linked device that was
# offline at the time would keep its copy forever — and push it back the
# next time it was edited. The delete has to be something clients can LEARN.
# A tombstone, not a dropped row, so a device that was offline learns of the
# delete instead of keeping its copy and pushing it back.
await purge_note(db, note)
await db.commit()
return jsonify({"ok": True})
+2 -3
View File
@@ -2,7 +2,6 @@ from __future__ import annotations
from quart import Blueprint
# The notes blueprint lives in its own tiny module so every route module in the
# package can `from ._bp import bp` without importing the package __init__ (which
# imports the route modules) — i.e. no import cycle.
# In its own module so route modules can `from ._bp import bp` without an import cycle
# through the package __init__.
bp = Blueprint("notes", __name__, url_prefix="/api/notes")
+6 -10
View File
@@ -1,16 +1,12 @@
"""Writing a note's text — the one path every surface goes through.
A body edit is never only the body. It may keep a revision of the old text, it
renames the note, it lifts and reconciles `#tags`, and once committed it queues link
previews. That sequence used to be written out at each place a body could change —
PATCH, the item routes, restoring a revision and sync push — and the copies drifted:
restoring a revision never queued previews, so a restored link stayed a bare URL
(#5164). Now a caller states how the old text should be kept and this does the rest.
A body edit may keep a revision of the old text, renames the note, and lifts and
reconciles `#tags`. PATCH, the item routes, restoring a revision and sync push all
come through here; the caller says how the old text is kept.
Previews are the one part that cannot happen here. They are queued AFTER the commit
(`unfurl_queue.schedule`), because the fetch reads the note in a session of its own
and a note it cannot see yet gets no preview. So this returns whether the text
changed, and the caller queues them once its transaction is committed.
Link previews are queued by the caller after its commit (`unfurl_queue.schedule`),
because the fetch reads the note in its own session. So this returns whether the
text changed.
"""
from __future__ import annotations
+12 -25
View File
@@ -1,18 +1,11 @@
"""Checklist items — a note's body IS its checklist (M304).
"""Checklist items: a note's body is its checklist.
A `- [ ] milk` line is the item. There is no `note_items` table beside the body any
more, which is what lets a list sit BETWEEN two paragraphs: rows had a position in a
table and no position in the text, so a separate list could only ever render after
the prose no matter how it was styled.
A `- [ ] milk` line is the item, so a list can sit between two paragraphs. Unlike
tags, items are stored nowhere else: nothing queries them, so `parse_items` is the
whole storage layer and the rewriters below are how one is edited.
The same shape as `tags.py`, one strength further along. Tags are derived from the
body too, but they MATERIALISE into `note_labels` rows because the board queries by
label. Items materialise into nothing, because nothing queries them — their only
readers are the card, the editor and `display_title`. So `parse_items` is the whole
storage layer for a checklist, and the rewriters below are how one is edited.
THE GRAMMAR IS SHARED. Three implementations exist and they have to agree, because a
difference between any two of them is a checklist that changes shape when it syncs:
The grammar has three implementations that must agree, or a checklist changes shape
when it syncs:
core/src/local/derive.rs the native clients (desktop + Android)
src/inkwell/notes/checklist.py this file, the server
@@ -21,12 +14,9 @@ difference between any two of them is a checklist that changes shape when it syn
optional indent, `-` or `*`, one-or-more spaces, `[ ]`/`[x]`/`[X]`,
then either end-of-line or one-or-more spaces and the text.
`*` is accepted because markdown.ts already takes it for a plain bullet, and a rule
that allowed `* item` but not `* [ ] item` would be one nobody could guess. `- [ ]`
with nothing after it IS an item with empty text — that is what pressing Enter on a
list leaves behind, and refusing to parse it would make a half-typed list stop being
a list. `- [X]` parses as checked and renders back lowercase, so one canonical form
survives a round trip.
`*` is accepted because it is a plain bullet too. `- [ ]` alone is an item with empty
text, which is what Enter on a list leaves behind. `- [X]` parses as checked and
renders back lowercase, so one canonical form survives a round trip.
"""
from __future__ import annotations
@@ -101,12 +91,9 @@ def set_item_checked(body: str, index: int, checked: bool) -> str:
def append_item(body: str, text: str, checked: bool = False) -> str:
"""Add an item at the end of the body.
A blank line between prose and the list, nothing between consecutive items —
the layout `import_export._note_markdown` has always used when writing a checklist
out. That is not cosmetic: it is what the Alembic migration folds existing
`note_items` rows into AND what `derive::append_item` produces on every client, so
all three land on identical bodies. An export taken before the migration and one
taken after therefore differ in nothing.
A blank line between prose and the list, nothing between consecutive items: the
same layout as the export and the core's `derive::append_item`, so every surface
lands on identical bodies.
"""
line = render_item(text.strip(), checked)
trimmed = body.rstrip("\n")
+15 -25
View File
@@ -33,21 +33,13 @@ DISPLAY_TITLE_CAP = 200
def derive_display_title(body: str | None) -> str:
"""The note's display NAME: the first line of the body that says anything.
There is no explicit title to prefer any more (M13 step 3) — a note is a body, and
its name is simply the first thing written in it. Persisted as notes.display_title
so search results and export filenames have something to say.
Stored as notes.display_title for search results and export filenames. A task
line is named without its marker ("milk", not "- [ ] milk"), and an empty item
is skipped.
The old `first_item` fallback is gone with M304: items ARE body lines now, so a
list-only note is named by its first item without anyone having to arrange it. What
replaced the fallback is stripping the task marker — calling that note "- [ ] milk"
would show someone the storage instead of the note — and skipping an EMPTY item, so
a half-typed list does not leave a note with no name.
Lines split on "\n" alone, as every other reading of the grammar does;
`splitlines` also broke on a lone "\r" or a U+2028, so a phone and the server named
the same note differently. Mirrors `derive::display_title` in the core, and both
run the `display_titles` cases in core/testdata/grammar.json. Deterministic — a
literal first line, never generated.
Lines split on "\n" alone, like every other reader of the grammar (`splitlines`
would also break on "\r" and U+2028). Mirrors `derive::display_title` in the
core; both run the `display_titles` cases in core/testdata/grammar.json.
"""
for line in (body or "").split("\n"):
stripped = strip_marker(line.strip()).strip()
@@ -59,9 +51,8 @@ def derive_display_title(body: str | None) -> str:
def is_empty_note(body: str | None, items: list | None = None) -> bool:
"""Nothing worth keeping: no body text and no checklist items.
`items` is still a separate argument because create still accepts them separately —
the importer holds a list, not a blob — and they are folded into the body only
after this check has decided the note is worth making at all.
`items` is separate because create accepts them separately (the importer holds a
list); they are folded into the body after this check.
"""
return not (body or "").strip() and not items
@@ -77,7 +68,7 @@ def apply_filter(stmt, filter_name: str, archived):
"""Narrow a notes query to one board view.
`archived` is the viewer's archive flag (`note_state.archived_for`): a note
someone shared with you is archived on your board when YOU archived it (#5176).
someone shared with you is archived on your board when you archived it.
Every branch excludes purge tombstones — content-less rows kept only so the sync
feed can tell offline clients a note is gone (see `retention.purge_note`). The
@@ -121,12 +112,11 @@ async def _fetch_note(db, note_id: str, gate) -> Note | None:
async def _get_owned(db, note_id: str) -> Note | None:
"""Fetch a note the current user OWNS, for every change only an owner may make:
trash, delete, labels, reminders, attachments, previews, history and sharing
(#5174). Pin and archive are everyone's own (#5176).
trash, delete, labels, reminders, attachments, previews, history and sharing.
Pin and archive are everyone's own.
Someone the note is shared with gets None here, and so a 404, even when they can
read the note: a 403 would confirm the note exists to a caller who only has its
id (#1984).
Someone the note is shared with gets None, and so a 404, even when they can read
it: a 403 would confirm the note exists to a caller who only has its id.
A purged note reads as absent: the REST API must treat it as gone, so opening,
editing or restoring one 404s. The sync push path looks rows up directly rather
@@ -138,14 +128,14 @@ async def _get_owned(db, note_id: str) -> Note | None:
async def _get_visible(db, note_id: str) -> Note | None:
"""Fetch a note the current user can see: theirs, or shared with them at any
level. What a recipient may do to it from here is their own pin, archive and
place (#5176); `_get_editable` and `_get_owned` gate the rest."""
place; `_get_editable` and `_get_owned` gate the rest."""
return await _fetch_note(db, note_id, note_visible())
async def _get_editable(db, note_id: str) -> Note | None:
"""Fetch a note the current user may change the TEXT of: its body and its
checklist. The owner, or someone it is shared with at `edit`; a view share gets
None, and so a 404, like a stranger (#5174)."""
None, and so a 404, like a stranger."""
return await _fetch_note(db, note_id, note_visible(EDIT))
+10 -25
View File
@@ -42,22 +42,16 @@ def _note_markdown(note: Note, labels: list) -> str:
fm.append(f"updated: {iso(note.updated_at) or ''}")
fm.append("---")
fm.append("")
# Body and checklist are no longer alternatives — a note can carry both, so both
# are written, body first, with a blank line between them when there is.
if note.body:
fm.append(note.body)
# No separate items block any more. The body already ends with those exact lines
# (M304) — this function is where their layout was decided, and appending them a
# second time would double every checklist in an export.
# The checklist is part of the body, so it is not written a second time.
return "\n".join(fm) + "\n"
# --- Import: Inkwell's own export (round-trip) OR a Google Keep Takeout zip ---
# The `app` value an export's notes.json carries. "thoughtsync" is what every export
# written before the rename to Inkwell says (Scribe note 5071), and those zips are
# people's backups, so they keep importing. That is the one reason the old name is
# still read here; nothing writes it any more.
# The `app` values an Inkwell export's notes.json may carry. Exports from when the app
# was called ThoughtSync say "thoughtsync", and still import.
NATIVE_APP_MARKERS = ("inkwell", "thoughtsync")
# Reverse of ALLOWED_IMAGE_MIMES, for inferring an attachment's mime from its
@@ -106,9 +100,7 @@ def _keep_spec(kn: dict, keep_dir: str) -> dict:
"""Normalize one Google Keep note (Takeout <note>.json) into the common import
spec. `keep_dir` is the note JSON's folder, used to resolve attachment paths."""
list_content = kn.get("listContent") if isinstance(kn.get("listContent"), list) else []
# Keep's own notes are one or the other, but its text was being DISCARDED whenever
# a note also had list content, because the target model could only hold one.
# It can hold both now, so both are kept.
# A Keep note may carry both text and list content; both are kept.
body = kn.get("textContent") or ""
# Keep stores link annotations (e.g. shared URLs) separately from the text —
# fold any URLs into the body so the content survives the move.
@@ -247,28 +239,21 @@ async def _create_imported_note(
name derivation + tag reconciliation as create_note. Returns False (nothing
written) when the spec is empty."""
body = spec.get("body") or ""
# An imported title becomes the note's FIRST BODY LINE.
#
# Inkwell has no title field any more (M13 step 3), but the things people
# import from do — Keep notes carry one, and so does any export taken before this.
# Dropping it would silently lose text someone wrote; folding it into the body puts
# it exactly where a name now lives, so the note comes in named the way it was.
# Skipped when the body already opens with that line, so re-importing an export
# this code produced doesn't stack duplicates.
# An imported title (Keep has them) becomes the first body line, which is where a
# note's name lives. Skipped when the body already opens with it, so re-importing
# an export doesn't stack duplicates.
title = (spec.get("title") or "").strip()
if title and body.lstrip().split("\n", 1)[0].strip() != title:
body = f"{title}\n{body}" if body.strip() else title
items = spec.get("items") or []
item_texts = [t for t in ((it.get("text") or "").strip() for it in items) if t]
# A note that is only a photo is still a note — Keep has plenty of them, and they
# used to be skipped as empty. The core's importer keeps them too (portable.rs).
# A note that is only a photo is still a note, as in the core's importer.
if is_empty_note(body, item_texts) and not spec.get("attachments"):
return False
# Items fold into the body, which is where a checklist lives now (M304). Done
# before the Note is built so display_title and _lift_and_reconcile_tags both see the
# finished text — an imported item can carry a #tag like any other line.
# Items fold into the body before the Note is built, so display_title and
# _lift_and_reconcile_tags see the finished text; an item can carry a #tag.
for it in items:
text = (it.get("text") or "").strip()
if text:
+6 -10
View File
@@ -36,15 +36,11 @@ async def _labels_for_notes(db, note_ids: list) -> dict:
def items_of(body: str | None) -> list[dict]:
"""The note's checklist, read out of its body. No query, because there is no table.
Still emitted in the payload after M304, and that is not a second source of truth:
it is DERIVED on the way out, so it cannot disagree with the body it came from. It
saves every consumer that only wants to draw checkboxes from carrying a parser, and
the ones that do carry one (the native clients, the browser) are free to ignore it
and read the body.
Derived on the way out, so it can't disagree with the body; it spares a consumer
that only draws checkboxes from carrying a parser.
The id is the item's ORDINAL, which is what the rewriters in `checklist.py` take,
so a client holding one can act on it directly. It also shifts when an item is
removed — every mutation returns the reloaded note for exactly that reason.
The id is the item's ordinal, which the rewriters in `checklist.py` take. It
shifts when an item is removed, so every mutation returns the reloaded note.
"""
return [
{"id": str(i), "text": item.text, "checked": item.checked, "position": i}
@@ -108,7 +104,7 @@ async def _previews_for_notes(db, note_ids: list) -> dict:
async def _sharing_for_notes(db, notes: list, viewer) -> dict:
"""note_id -> how `viewer` holds the note (#5174): `permission` (owner, edit or
"""note_id -> how `viewer` holds the note: `permission` (owner, edit or
view), `shared` (whether it is shared at all, as far as the viewer may know) and
`shared_by` (the owner, for a note someone shared with them)."""
foreign = [n for n in notes if n.owner_id != viewer]
@@ -142,7 +138,7 @@ async def _serialize_notes(db, notes: list, viewer=None) -> list:
"""The API shape of each note. With `viewer`, each also says how the viewer holds
it, and a note someone shared with them comes without the owner's labels: labels
are personal, and a recipient's board files nothing under someone else's tags.
For the same reason its pin, archive and position are the viewer's own (#5176).
For the same reason its pin, archive and position are the viewer's own.
Sync passes no viewer to a client that asked only for its own notes."""
ids = [n.id for n in notes]
labels_map = await _labels_for_notes(db, ids)
+21 -57
View File
@@ -1,16 +1,6 @@
"""#tags — parsing note bodies, LIFTING the standalone ones out of the text, and
keeping the still-in-text ones in sync with the note_labels rows. Manual (picker)
labels are NOT touched here (see the labeling module).
A tag used to be shown twice: once as the `#todo` you typed and once as a chip. The
chip moved to the top of the card (M311) and the text now goes, but only when the tag
was standing on its own — see `split_body_tags` for the rule and why it is the
conservative one.
Was `links.py`, and also owned `[[wiki-links]]` until they were removed (note 2897):
this app is an intermediary surface for capture and recall, and a linking system is
organization, which is not what it is for. A file called links.py holding no links
would have been exactly the kind of drift that removal was meant to end.
"""#tags: parsing note bodies, lifting the standalone ones out of the text (they show
as chips), and keeping the in-text ones in sync with the note_labels rows. Manual
(picker) labels are not touched here (see the labeling module).
"""
from __future__ import annotations
@@ -26,22 +16,15 @@ from .helpers import derive_display_title
# A #tag: `#` at the start of a line or after whitespace, then a LETTER, then
# letters, digits, `_` and `-`. The same rule as `line_tags` in
# core/src/local/derive.rs and the web's markdown.ts, and all three run the cases in
# core/testdata/grammar.json.
#
# The three used to differ. The core took a tag after any punctuation, so `(#todo)`
# was a label on the phone and plain text here, and the `/#section` of a pasted URL
# became a label too. This side let a tag start with a digit or `_` (`#1st`). So a
# note's labels changed every time it synced. The shared rule is the strict one:
# nothing is a tag now that was not already one everywhere (#5166).
# core/testdata/grammar.json; a difference would change a note's labels on sync. So
# `(#todo)`, `/#section` in a URL and `#1st` are not tags.
#
# `[^\W\d_]` is "a word character that is not a digit or `_`", which is a letter.
_TAG_RE = re.compile(r"(?<!\S)#([^\W\d_][\w-]*)")
# A fence opens or closes a code block. A `#tag` inside one is CODE — the shell
# comment in a snippet someone pasted — and lifting it would delete a line of their
# example. It still becomes a label, because it always has and that is a separate
# question from whether the text may be touched.
# A fence opens or closes a code block. A `#tag` inside one is code (a pasted shell
# comment), so it is never lifted out of the text, though it still becomes a label.
_FENCE_RE = re.compile(r"^\s*(?:```|~~~)")
@@ -72,17 +55,12 @@ def split_body_tags(body: str | None) -> tuple[list[str], list[str], str]:
THE RULE: a line containing nothing but tags and whitespace is removed. Anything
else is left exactly as written.
That is deliberately the conservative reading of "standalone". The looser one —
also stripping a trailing tag off a prose line — was rejected because a trailing
tag is ambiguous and the text does not say which it is: `buy milk #grocery` is
filing, `remember to call #mom` is the sentence's object, and lifting the second
leaves "remember to call". Mangling a sentence to save a duplicate chip is a bad
trade. A tag sharing a line with words keeps its words.
A trailing tag on a prose line stays: `buy milk #grocery` is filing, but
`remember to call #mom` is the sentence's object, and the text can't tell them
apart.
`standalone` tags become ORDINARY labels — the text no longer backs them, so
nothing can derive them any more, and the way to remove one becomes the chip's ×
rather than deleting the text. `inline` tags stay derived exactly as before. That
is the whole meaning of `via_tag` after this change: backed by text still present.
`standalone` tags become ordinary labels, removed with the chip's ×. `inline`
tags stay derived (`via_tag`: backed by text still present).
"""
if not body:
return [], [], body or ""
@@ -108,8 +86,7 @@ def split_body_tags(body: str | None) -> tuple[list[str], list[str], str]:
standalone.extend(names)
lifted = re.sub(r"\n{3,}", "\n\n", "\n".join(kept)).strip("\n")
if body.strip() and not lifted.strip():
# The note was NOTHING but tags. Lifting would leave a blank card, which is a
# worse outcome than a duplicated chip — so leave it alone and let its tags
# The note was nothing but tags. Lifting would leave a blank card, so its tags
# stay derived.
return [], _dedupe(standalone + inline), body
# A tag that ALSO appears in prose stays derived: the prose copy still backs it,
@@ -138,24 +115,12 @@ async def _lift_and_reconcile_tags(db, note: Note) -> None:
"""Attach the note's tag labels, LIFT its standalone tags out of the body, and
re-derive display_title if the body moved.
NAMED FOR THE MUTATION. It used to be `_reconcile_tags` and only touched rows; it
now rewrites `note.body`, and a caller that does not expect that will compute a
display_title from text this function is about to delete.
It rewrites `note.body`, so the title is re-derived here rather than at each
call site, where it would be computed from text about to be deleted.
Which is why the lift and the re-derivation both live HERE rather than at the
seven call sites that would each have to remember. Spreading a derived-value
update across every place a body can be written is precisely the failure #2965
named about label minting: "easy to miss, and it is the common one".
The two kinds of tag are handled differently, and that difference IS what
`via_tag` means from here on — backed by text still in the body:
standalone lifted out, attached as an ORDINARY label (via_tag=False). Nothing
derives it any more because nothing is left to derive it from, and
the way to remove it becomes the chip's × — which both editors
already offer for exactly this class of label.
inline left in place, attached via_tag=True, and still detached when its
text goes. Unchanged from before.
standalone lifted out, attached as an ordinary label (via_tag=False), removed
with the chip's ×.
inline left in place, attached via_tag=True, detached when its text goes.
"""
standalone, inline, lifted = split_body_tags(note.body)
standalone_ids: set = set()
@@ -171,10 +136,9 @@ async def _lift_and_reconcile_tags(db, note: Note) -> None:
if not r.via_tag:
attached.add(r.label_id) # manual already: a #tag of the same name changes nothing
elif r.label_id in standalone_ids:
# It GRADUATED. The text backing it is about to be deleted, so the row has
# to become the record instead. This must run BEFORE the detach below, or
# the same row would be dropped for no longer being in the body — which is
# the bug that makes a naive lift delete every tag it touches.
# The text backing it is about to go, so the row becomes the record. Must
# run before the detach below, or the row would be dropped for no longer
# being in the body.
r.via_tag = False
attached.add(r.label_id)
elif r.label_id in inline_ids:
+5 -5
View File
@@ -1,17 +1,17 @@
"""Password resets: an admin makes a one-hour link for an account (#5173).
"""Password resets: an admin makes a one-hour link for an account.
Before this, a forgotten password needed a hand on the database (#2939 §2). This reuses what invites established: a random token, only its hash
kept, the link shown once to the admin who hands it over.
A random token with only its hash kept, as for invites; the link is shown once to the
admin who hands it over.
Using the link sets a new password and signs the account out everywhere. Its web
sessions end because the account's `session_epoch` moves on (see `auth`), and its
device tokens are deleted, so each linked app has to sign in again.
A person can also ask for one themselves when the server can send email (#5266):
A person can also ask for one themselves when the server can send email:
`mail_reset_link` makes the same link and mails it to the account's address.
This module is the reset itself; `auth.reset_password` redeems it and the admin route
is in `accounts_api.py`, the same split as invites and for the same reason.
is in `accounts_api.py`.
"""
from __future__ import annotations
+10 -24
View File
@@ -1,27 +1,15 @@
"""Reading what the proxies in front of this app say about a request.
Two headers carry information the app cannot see for itself — who the client is
(`X-Forwarded-For`) and whether they arrived over TLS (`X-Forwarded-Proto`) — and both
are trusted by the same rule, so the rule lives in one place. Writing it twice is
precisely how issue 2183 happened: two places holding one decision, and only one of
them updated.
`X-Forwarded-For` (who the client is) and `X-Forwarded-Proto` (whether they arrived
over TLS) are trusted by one rule, kept here.
## The rule
A forwarding header grows left to right: each hop appends what it saw, so a caller's
forged entries sit to the left of what our proxies wrote. So we count in from the
right by the **Trusted proxy hops** setting (default 1), and a forged prefix is never
selected.
A forwarding header grows LEFT to RIGHT. Each hop appends what IT saw, so the
rightmost entries are the ones our own infrastructure wrote, and anything a caller
sent arrives to the LEFT of those.
That inverts the intuitive reading. The leftmost entry is nominally "the original
client" — and is exactly the one a caller can forge, by sending the header themselves.
So we count in from the right by the number of proxies we actually run
(the **Trusted proxy hops** setting, default 1), and a forged prefix can never be
selected no matter how much of it there is.
Too HIGH a hop count is the dangerous direction: it starts believing entries no proxy
of ours wrote. Too low just means several callers share a bucket. So when the header
is shorter than configured — fewer proxies than expected — we fall back to the socket
address rather than reaching further left.
Too high a count is the dangerous direction: it believes entries no proxy of ours
wrote. When the header is shorter than configured, the socket address is used.
"""
from __future__ import annotations
@@ -61,10 +49,8 @@ def client_address() -> str:
def is_https() -> bool:
"""Whether this request reached us over TLS — directly, or via a trusted proxy.
Shared by the session cookie's `Secure` flag and by HSTS, because they are the same
question. Read with the same hop count as the address: a caller who sets
`X-Forwarded-Proto: https` on a plain-HTTP request puts it to the left of whatever
our proxy appended, so it is not what gets read.
For the session cookie's `Secure` flag and HSTS. Read with the same hop count as
the address, so a forged `X-Forwarded-Proto: https` is never what gets read.
"""
if not has_request_context():
return False
+22 -48
View File
@@ -4,37 +4,22 @@ The credential endpoints are rate-limited: login, register, and the native
device-link exchange. Everything else already needs a session or a device token to
reach, so an attacker has to get through one of these three first.
The one exception is the app downloads (`client_dist`): signed in, but each one is
tens of megabytes, so one account or a leaked device token looping on it could fill
the server's uplink (family idea #5103, practice 6).
App downloads (`client_dist`) are throttled too: each is tens of megabytes, so a
looping client could fill the server's uplink.
## Why in-process is enough here, and where that stops being true
State is per-process, which matches the image's single hypercorn worker. With several
workers the effective limit would multiply; the fix would be a shared store.
State lives in module-level dicts, so it is per-process. That is correct for how
this image actually serves — one hypercorn worker (see the Dockerfile, and the
same assumption the trash sweeper documents in app.py). If that ever gains
``--workers N``, each worker would keep its own counters and the effective limit
would multiply by N; the fix then is a shared store (the Postgres connection is
already there), not a bigger number here.
Every attempt counts against both the account tried and the caller's address, and
either can refuse it:
## Two keys, on purpose
- **The account** can't be forged, and stops credential stuffing against one email
from any number of addresses.
- **The address** bounds one source spraying many accounts. Read via `proxy.py`, so
it is only as good as the Trusted proxy hops setting.
Every attempt is counted against BOTH the account being tried and the address it
came from, and either one can refuse it:
- **The account** is the key that matters, and the key that cannot be forged. It
is what stops credential stuffing against one known email, no matter how many
addresses the attempts arrive from.
- **The address** bounds the damage from one source spraying many accounts. It is
read from ``X-Forwarded-For``, counting in from the RIGHT by
the **Trusted proxy hops** setting so that only entries our own proxies wrote are
believed — a forged header lands to the left of those and is never selected. It is
still the weaker of the two keys, because it depends on that setting matching the
deployment; the account key depends on nothing.
Counting is by failure for the sign-in routes and by attempt for registration: a
correct password should never move someone closer to being locked out, but every
registration is a row in the users table whether it succeeds or not.
Sign-in counts failures, so a correct password never moves someone closer to a
lockout; registration counts attempts.
"""
from __future__ import annotations
@@ -45,26 +30,18 @@ from collections.abc import Callable
from .settings import live
# Never let the bookkeeping become the denial of service: an attacker rotating an
# address could otherwise mint an unbounded number of buckets. Well above any real
# deployment's distinct-caller count, so a legitimate instance never reaches it; when
# it is reached the oldest buckets are dropped, which at worst forgives some attempts.
#
# Not a setting: it protects the limiter from itself rather than the app from a
# caller, and there is no operator judgment to apply to it.
# A cap on tracked keys, so rotating addresses can't grow the limiter without bound.
# Far above any real caller count; past it the oldest buckets are dropped.
MAX_BUCKETS = 10_000
class SlidingWindow:
"""Counts events per key over a trailing window.
Sliding rather than a fixed window because a fixed one lets twice the limit
through across a boundary — 10 at 14:59 and 10 at 15:00 — which for a login
limiter is the difference between the number meaning something and not.
Sliding, because a fixed window lets twice the limit through across a boundary.
The limit and window are SUPPLIERS, not values, so an admin saving a new number in
Settings takes effect on the next attempt instead of the next deploy. They are read
per call, which is a dict lookup — the settings cache never touches the database.
The limit and window are suppliers, read per call from the settings cache, so a
change in Settings applies to the next attempt.
"""
def __init__(self, limit: Callable[[], int], window_s: Callable[[], float]) -> None:
@@ -108,9 +85,8 @@ class SlidingWindow:
self._prune(key, now).append(now)
def forget(self, key: str) -> None:
"""Drop a key's history. Used after a successful sign-in, so someone who
fumbled a password twice and then got it right starts clean rather than
carrying those two for the next quarter of an hour."""
"""Drop a key's history. Used after a successful sign-in, so earlier failures
don't carry over."""
self._hits.pop(key, None)
def clear(self) -> None:
@@ -131,16 +107,14 @@ register_by_address = SlidingWindow(
lambda: live("register_limit_per_address"), _minutes("register_window_minutes")
)
# Reset emails, counted per address typed in whether or not it has an account, so
# the cap can't be used to learn which ones do (#5266).
# the cap can't be used to learn which ones do.
reset_mail_by_account = SlidingWindow(
lambda: live("reset_emails_per_account"), _minutes("signin_window_minutes")
)
# App downloads, per account. Counted per REQUEST, so resuming a broken download
# counts each piece; the setting's default leaves room for that. The window is fixed
# at an hour because the setting's label says "per hour", and one number is easier
# to judge than two.
# App downloads, per account, per hour. Counted per request, so each piece of a
# resumed download counts; the default leaves room for that.
downloads_by_account = SlidingWindow(lambda: live("client_downloads_per_hour"), lambda: 3600.0)
+2 -3
View File
@@ -4,9 +4,8 @@ import uuid
from quart import jsonify
# The app's single JSON error shape + the two guards that pair with it, so every
# blueprint returns errors and parses path ids the same way instead of hand-building
# `jsonify({"error": ...}), code` and try/except uuid blocks at ~35 call sites.
# The app's JSON error shape and the two guards that pair with it, so every blueprint
# returns errors and parses path ids the same way.
def json_error(message: str, status: int):
+1 -1
View File
@@ -1,4 +1,4 @@
"""Bring a fresh Inkwell back from a backup file (#5921).
"""Bring a fresh Inkwell back from a backup file.
docker compose exec app python -m inkwell.restore /backups/inkwell-20261016T020000Z.tar
+19 -38
View File
@@ -2,21 +2,14 @@
Two things live here, deliberately together:
**`purge_note`** — the single definition of destroying a note. Three callers reach
permanent deletion by different routes (the user's Delete forever in the web UI,
a client's `op=delete` over sync, and the sweeper below), and if each had its own
idea of what to tear down they would drift — one would forget the files, another
the revision history, and "permanently deleted" would quietly mean three different
things depending on how you got there.
**`purge_note`** is the one definition of destroying a note, used by Delete forever
in the web UI, a client's `op=delete` over sync, and the sweeper.
**The sweeper** — trash that nobody empties is not free: a trashed note keeps its
attachment BYTES on disk for as long as it sits there. So trash expires. The window
is the `trash_retention_days` setting (default 30, `0` = keep forever), re-read on
every pass so a change in admin Settings takes effect without a restart.
**The sweeper** expires trash, which still holds attachment bytes on disk. The window
is the `trash_retention_days` setting (default 30, `0` = forever), re-read each pass.
A purged note is not a deleted ROW — it's a content-less tombstone. That's what lets
an offline client that reappears next month learn the note is gone instead of
faithfully resurrecting it on the next push.
A purged note is a content-less tombstone, not a deleted row, so an offline client
that comes back learns the note is gone instead of pushing it back.
"""
from __future__ import annotations
@@ -43,17 +36,14 @@ from .storage import unlink_media
logger = logging.getLogger(__name__)
# How often the sweeper wakes. Retention is measured in days, so anything under
# "a few times a day" buys nothing but load — a note trashed at 09:00 expiring at
# 14:00 rather than 09:00 thirty days later is not a difference anyone can feel.
# How often the sweeper wakes. Retention is in days, so more often only adds load.
SWEEP_INTERVAL_SECONDS = 6 * 60 * 60
# Let the app finish booting (migrations, first requests) before the first sweep.
SWEEP_STARTUP_DELAY_SECONDS = 60
# Rows purged per transaction. A long-neglected install could have thousands of
# expired notes on the first sweep; committing in batches keeps that from becoming
# one enormous transaction holding locks while it deletes files.
# Rows purged per transaction, so a large first sweep isn't one long transaction
# holding locks while it deletes files.
SWEEP_BATCH = 200
@@ -61,11 +51,9 @@ async def purge_note(db, note: Note, edited_at: datetime | None = None) -> None:
"""Turn a note into a content-less tombstone: delete its children (and the
attachment files on disk), clear its content, stamp `purged_at`.
The row survives on purpose — offline clients read it off the delta feed and
learn the note is gone. Everything that carries the note's CONTENT goes, and
that includes history: a revision row holds the full body, so leaving revisions
behind would mean the text of a "permanently deleted" note is still on the
server, recoverable by anyone who can read the table.
The row survives so offline clients learn of it from the delta feed. Everything
carrying the note's content goes, revisions included, since each holds a full
body.
"""
atts = (await db.scalars(select(NoteAttachment).where(NoteAttachment.note_id == note.id))).all()
for a in atts:
@@ -80,12 +68,8 @@ async def purge_note(db, note: Note, edited_at: datetime | None = None) -> None:
await db.execute(sa_delete(Share).where(Share.resource_type == "note", Share.resource_id == note.id))
note.body = ""
note.display_title = ""
# `deleted_at` deliberately SURVIVES. It's still true — that is when the note was
# deleted — and keeping it means every ordinary query, present and future, that
# says "not trashed" (`deleted_at IS NULL`) excludes tombstones for free. Clearing
# it would leave a content-less row looking like a perfectly normal active note,
# and it would surface on the board as a blank card. Only the Trash view, which
# asks for `deleted_at IS NOT NULL`, has to name `purged_at` explicitly.
# `deleted_at` stays, so every "not trashed" query (`deleted_at IS NULL`) excludes
# tombstones for free. Only the Trash view has to name `purged_at`.
note.remind_at = None
note.purged_at = datetime.now(timezone.utc)
if edited_at is not None:
@@ -95,9 +79,8 @@ async def purge_note(db, note: Note, edited_at: datetime | None = None) -> None:
async def sweep_expired_trash(db, retention_days: int, *, now: datetime | None = None) -> int:
"""Purge every note whose trash has expired. Returns how many were purged.
Runs across ALL owners — it's a server-wide policy, not a per-user action, and
the sweeper has no session to scope it by (rule 47 is about honoring the ACL on
user-initiated reads, not about exempting rows from server maintenance).
Across all owners: a server-wide policy, not a user's read, so the ACL doesn't
apply.
"""
cutoff = expired_before(now or datetime.now(timezone.utc), retention_days)
if cutoff is None:
@@ -110,9 +93,8 @@ async def sweep_expired_trash(db, retention_days: int, *, now: datetime | None =
.where(
Note.deleted_at.is_not(None),
Note.deleted_at < cutoff,
# Already a tombstone. Without this the purge would re-run on
# every sweep forever, bumping sync_revision each time and
# handing clients an endless stream of "news" about one note.
# Already a tombstone. Purging again would bump sync_revision
# every sweep and send clients endless changes.
Note.purged_at.is_(None),
)
.order_by(Note.deleted_at)
@@ -140,8 +122,7 @@ async def sweep_once() -> int:
async def run_sweeper() -> None:
"""The background loop. Started in `before_serving`, cancelled on shutdown.
A sweep failure (DB blip, unreadable media directory) must never take the loop
down with it — the next pass simply finds the same expired rows and tries again.
A failed sweep never ends the loop; the next pass tries the same rows again.
"""
await asyncio.sleep(SWEEP_STARTUP_DELAY_SECONDS)
while True:
+8 -22
View File
@@ -1,25 +1,13 @@
"""When a body change is worth keeping a version of.
A note's body used to snapshot into `note_revisions` on EVERY write, which made a
write expensive — and the clients compensated by writing as rarely as they could
get away with, saving only when an editor closed. That is durability paying for
version history: a crash mid-session lost everything typed, so that the revision
list would stay tidy. The safety property is worth more than the feature it was
subsidising.
A body change is snapshotted only if it is the first of an editing session, so a
client may autosave as often as it likes. A snapshot stores the body as it was
before the edit, so each sitting yields one revision: the note as you found it.
The rule here breaks that trade. A body change earns a snapshot only if it is the
first one of an editing session, so a client may write as often as it likes.
It is a time rule rather than a protocol flag, and `sync.py` applies pushed bodies
through the same check.
Session granularity falls out of the window rather than being declared. A snapshot
stores the body as it was BEFORE the edit, so the first write of a sitting captures
the note as you found it and every write after it inside the window adds nothing —
one revision per sitting, with no "commit" flag for a client to send and no wire
surface to carry it. That last part is why this is a time rule and not a protocol
one: `sync.py` applies pushed bodies through the same check, so a client autosaving
every second cannot make the server snapshot every second either.
Deliberately NOT applied to restoring a revision. That is a considered act rather
than a keystroke, and it snapshots unconditionally so restoring is itself undoable.
Restoring a revision always snapshots, so a restore is itself undoable.
"""
from __future__ import annotations
@@ -30,10 +18,8 @@ from sqlalchemy import select
from .models.note_revision import NoteRevision
# How long one editing session is assumed to last. A constant rather than a setting:
# it is not a preference anyone holds, and the value only has to be longer than a
# sitting and shorter than the gap between two of them. Promote it to the settings
# registry (rule 25) if that ever stops being true.
# How long one editing session is assumed to last: longer than a sitting, shorter
# than the gap between two.
REVISION_WINDOW_MINUTES = 10
+7 -10
View File
@@ -1,14 +1,11 @@
"""Credentials at rest: the SMTP password and the backup bucket's secret key (#5927).
"""Credentials at rest: the SMTP password and the backup bucket's secret key.
The family's secret.Box (#5185): a stored credential is encrypted under a key the
database does not hold, so a copy of the database (a dump, a copied volume) cannot
read the keys to anything else.
A stored credential is encrypted under a key the database does not hold, so a copy
of the database (a dump, a copied volume) can't open anything else.
- **Where the key lives.** A random key in a file under DATA_DIR, made on first use.
Not the signing secret: by default that is kept in the settings table (rule 25,
no env var needed), so a key derived from it would sit beside what it seals. The
data volume and the database volume are separate, and backups take only the
attachment files from DATA_DIR, never this file.
Not the signing secret, which by default sits in the settings table beside what it
would seal. Backups take only attachment files from DATA_DIR, never this file.
- **Per purpose, per row.** The cipher key is HKDF-derived from the file's key with
a purpose label, and each value is bound to its setting's name, so a sealed value
copied onto another setting does not open there.
@@ -112,8 +109,8 @@ def seal(plain: str, owner: str) -> str:
def unseal(stored: str, owner: str) -> str | None:
"""The plain value, or None when it can't be opened (key file lost or replaced,
or the value was copied from another setting). A value stored before sealing
existed comes back as it is; startup seals it (seal_stored_secrets)."""
or the value was copied from another setting). A value stored in the clear comes
back as it is; startup seals it (seal_stored_secrets)."""
if not is_sealed(stored):
return stored
try:
+3 -4
View File
@@ -14,10 +14,9 @@ def hash_password(password: str) -> str:
return bcrypt.hashpw(password.encode("utf-8")[:_MAX_BCRYPT_BYTES], bcrypt.gensalt()).decode("utf-8")
# A real bcrypt hash of a value nobody can present, used only to spend the time a
# verification would have. Computed once at import — generating it per call would
# cost a gensalt+hash on top of the checkpw and make the "no such user" path SLOWER
# than the real one, which is the same oracle pointing the other way.
# A real bcrypt hash of a value nobody can present, to spend the time a verification
# would. Computed once at import: per call, "no such user" would be slower than a
# real check, the same timing oracle reversed.
_DUMMY_HASH = bcrypt.hashpw(secrets.token_bytes(32), bcrypt.gensalt())
+3 -5
View File
@@ -1,8 +1,6 @@
"""Shared JSON serializers — the single home for turning ORM rows into the dict
shapes the API returns, so a field name or format is defined once. Each entity has a
BASE serializer with the fields common to every context; callers layer on the
extras their surface needs (usage count in the labels API, sync_revision/purged_at
in sync deltas). Labels and people so far."""
"""Shared JSON serializers for labels and people, so a field name or format is
defined once. Callers add what their surface needs (a usage count in the labels API,
sync_revision/purged_at in sync deltas)."""
from __future__ import annotations
from .common import iso
+21 -32
View File
@@ -33,11 +33,9 @@ class SettingDef:
label: str
description: str
group: str
# Ints only. Enforced server-side in validate_updates and passed to the UI so the
# number input carries them too. These exist because several of the security
# values have ranges where a typo is not merely wrong but dangerous — a proxy hop
# count of 50 would trust anything a caller sent, and a sign-in limit of 0 would
# lock every account out permanently.
# Ints only. Enforced in validate_updates and passed to the UI's number input.
# Some typos are dangerous: a hop count of 50 trusts anything a caller sends, and
# a sign-in limit of 0 locks every account out.
minimum: int | None = None
maximum: int | None = None
# Strings only: the values allowed, shown as a choice rather than a text field.
@@ -46,7 +44,7 @@ class SettingDef:
# value is shown, which reads fine for "Inkwell" and badly for "starttls".
choice_labels: tuple[str, ...] | None = None
# A credential. The admin API never sends its value back, only whether one is
# set, and saving it empty leaves it as it was (#5266).
# set, and saving it empty leaves it as it was.
secret: bool = False
# An http(s) address, stored without a trailing slash.
url: bool = False
@@ -128,10 +126,10 @@ REGISTRY: list[SettingDef] = [
"The server contacts the linked site; private/internal addresses are always blocked.",
"Links",
),
# --- Email (#5266) ---------------------------------------------------------------
# --- Email ---------------------------------------------------------------------
#
# Only for password reset links so far. Off until a server, a sender and the
# public address are all set (`mailer.mail_settings`).
# For reset links and invites. Off until a server, a sender and the public address
# are all set (`mailer.mail_settings`).
SettingDef("smtp_host", "string", "", "SMTP server", "Leave empty to turn email off.", "Email"),
SettingDef(
"smtp_port",
@@ -158,7 +156,7 @@ REGISTRY: list[SettingDef] = [
"smtp_password", "string", "", "SMTP password", "Kept on the server and never shown again.", "Email", secret=True
),
SettingDef("smtp_from", "string", "", "From address", "Who emails come from. Defaults to the username.", "Email"),
# --- Backups (#5918) ------------------------------------------------------------
# --- Backups --------------------------------------------------------------------
#
# A dated backup file each day, to a folder or an S3-compatible bucket
# (backup_job.py). Off until switched on, because a folder has to be mounted or a
@@ -249,9 +247,8 @@ REGISTRY: list[SettingDef] = [
),
# --- Security -----------------------------------------------------------------
#
# Read on paths too hot for a database round trip (the credential throttle checks
# them BEFORE opening a connection, which is the point of checking a throttle
# before doing expensive work), so they are cached — see `live()` below.
# Cached (see `live()` below): the credential throttle reads them before opening
# a database connection.
SettingDef(
"trusted_proxy_hops",
"int",
@@ -370,8 +367,7 @@ def _coerce(defn: SettingDef, raw: Any) -> Any:
n = int(raw)
except (ValueError, TypeError):
return defn.default
# A value saved before its bounds existed is still in the row; read it as the
# nearest bound rather than let it act (#5384).
# A stored value outside its bounds is read as the nearest bound.
if defn.minimum is not None:
n = max(n, defn.minimum)
if defn.maximum is not None:
@@ -437,11 +433,10 @@ async def get_public_config(db) -> dict:
"site_name": await get_setting(db, "site_name"),
"allow_registration": await get_setting(db, "allow_registration"),
"enable_url_unfurl": await get_setting(db, "enable_url_unfurl"),
# Server policy, not user data: clients need it to say how long a note has
# left in Trash. A native client also reads it BEFORE linking, which is why
# it belongs on the unauthenticated config rather than behind login.
# Clients show how long a note has left in Trash, and a native client reads it
# before linking, so it is on the unauthenticated config.
"trash_retention_days": await get_setting(db, "trash_retention_days"),
# Whether the sign-in screen offers "Forgot password?" (#5266).
# Whether the sign-in screen offers "Forgot password?".
"password_reset_by_email": await mail_configured(db),
"min_password_length": MIN_PASSWORD_LEN,
}
@@ -482,16 +477,11 @@ async def get_admin_settings(db) -> list[dict]:
return result
# Settings the app must be able to read WITHOUT awaiting a database.
# Settings read without awaiting a database: the credential throttle needs them
# before opening a connection, so a refused attempt costs nothing.
#
# The credential throttle consults these before opening a connection — deliberately,
# because a refused attempt is supposed to cost nothing, and the proxy hop count is
# needed to know who is even asking. A per-request query would undo both.
#
# Seeded from the registry defaults so the app works before (and without) a database —
# unit tests construct it with no Postgres at all — then refreshed from the DB at boot
# and again whenever an admin saves. Same live-update contract `session_ttl_days`
# already has in settings_api.py.
# Seeded from the registry defaults (unit tests run with no Postgres), then refreshed
# from the database at boot and whenever an admin saves.
_LIVE_KEYS = (
# Read by GET /install.sh, which writes it into the installer it serves
# (installer.py) and must not need a database to answer.
@@ -516,7 +506,7 @@ def live(key: str) -> Any:
def apply_session_ttl(app, days) -> None:
"""Sessions signed from now on last `days` days. At boot and on every admin save,
so a changed lifetime applies without a restart (rule 25)."""
so a changed lifetime applies without a restart."""
app.config["PERMANENT_SESSION_LIFETIME"] = timedelta(days=int(days))
@@ -545,9 +535,8 @@ def validate_updates(updates: dict) -> tuple[dict, str | None]:
n = int(val)
except (ValueError, TypeError):
return {}, f"{defn.label} must be a whole number"
# Rejected rather than clamped: silently accepting a number and storing a
# different one is how somebody ends up believing a protection is set to
# something it is not.
# Rejected rather than clamped, so nobody believes a protection is set to
# a number it isn't.
if defn.minimum is not None and n < defn.minimum:
return {}, f"{defn.label} must be at least {defn.minimum}"
if defn.maximum is not None and n > defn.maximum:
+4 -5
View File
@@ -46,13 +46,12 @@ async def update_settings():
# Nothing is saved: a credential is never stored unsealed by a save.
return json_error(f"Couldn't encrypt the password or key to store it. {exc}", 500)
await db.commit()
# Re-read the cached security values so a saved limit or hop count applies to
# the very next request. Unconditional: cheap, and a conditional here would be
# one more place that has to know which keys are hot.
# Re-read the cached security values so a saved limit applies to the next
# request.
await refresh_live(db)
result = await get_admin_settings(db)
# Apply the live-tunable knob without a restart (rule 25).
# Apply the session lifetime without a restart.
if "session_ttl_days" in clean:
apply_session_ttl(current_app, clean["session_ttl_days"])
@@ -63,7 +62,7 @@ async def update_settings():
@require_admin
async def test_email():
"""Send a test message to the signed-in admin with the SAVED email settings, so
"Send test email" checks what reset links will actually use (#5266). An admin is
"Send test email" checks what reset links will actually use. An admin is
shown the server's own error, which is what fixing the settings takes."""
async with session_scope() as db:
cfg = await mail_settings(db)
+2 -2
View File
@@ -1,4 +1,4 @@
"""What sharing a note does to the change feed (#5175).
"""What sharing a note does to the change feed.
The feed carries every note a person can see, so a share changing has to show up in
it: granting or changing a share moves the note past each device's cursor, and ending
@@ -72,7 +72,7 @@ async def granted(db, note_id: uuid.UUID, user_id: uuid.UUID) -> None:
await bump_note(db, note_id)
# --- Group membership (#5177) ------------------------------------------------
# --- Group membership --------------------------------------------------------
#
# A note shared with a group reaches whoever is in it NOW, so joining or leaving
# changes what that one person can see without any share changing. Their devices
+5 -6
View File
@@ -1,9 +1,8 @@
"""Sharing a note with another member of this instance (#5174).
"""Sharing a note with another member of this instance.
`acl.visible_to_user` has gated every read since M0, but nothing wrote a share. This
is the writing half: the note's owner lists, grants and revokes access, and anyone
signed in can read the member directory the Share dialog picks people from. A share
goes to one person or to a group the admin made (#5177).
The note's owner lists, grants and revokes access, and anyone signed in can read the
member directory the Share dialog picks people from. A share goes to one person or
to a group the admin made. Reads are gated by `acl.visible_to_user`.
What a share grants is in `acl` (view, or edit: the body and checklist) and is
enforced by the note routes (`notes.helpers._get_owned` / `_get_editable`).
@@ -50,7 +49,7 @@ async def _member_counts(db, group_ids: list) -> dict:
@login_required
async def directory():
"""Who on this instance a note can be shared with: everyone but you, and every
group the admin has made (#5177).
group the admin has made.
Signed-in only. Each person is a name and an email and nothing more, which is
what choosing someone takes and all a fellow member needs to know; a group is its
+4 -6
View File
@@ -1,4 +1,4 @@
"""How much attachment storage an account may use (#2939 §4).
"""How much attachment storage an account may use.
`max_attachment_mb` caps one file; this caps an account's total, so one account
can't fill the volume. The limit is Settings → General → `storage_quota_gb`;
@@ -7,11 +7,9 @@ can't fill the volume. The limit is Settings → General → `storage_quota_gb`;
What counts is every attachment on every note the account owns, trashed notes
included: their files stay on disk until the trash is emptied.
An upload that would go over the limit is answered 507 Insufficient Storage, not
413. The native clients treat a 4xx as permanent and never retry the file
(`client::upload_attachment` in the core), and a 5xx as worth another try. Over the
limit is neither: freeing space should let the file through on the next sync, with
nothing for the person to redo.
An upload over the limit is answered 507, not 413: native clients never retry a 4xx
(`client::upload_attachment` in the core), and freeing space should let the file
through on the next sync.
The one way an attachment's file leaves the disk is here too (`unlink_media`).
"""
+21 -42
View File
@@ -1,9 +1,9 @@
"""Delta-sync API for the local-first native clients (M8 sync hub).
"""Delta-sync API for the local-first native clients.
Pull: `GET /api/sync/changes?since=<cursor>` returns every note + label the caller
owns whose sync_revision advanced past the cursor, newest-revision last, paginated.
With `shares=1` it also carries the notes shared with the caller, and `revoked`: the
notes that stopped being shared with them (#5175).
notes that stopped being shared with them.
Notes and labels both draw from ONE shared sequence (sync_revision_seq), so the
cursor is a single monotonic watermark across both entity types.
@@ -56,18 +56,14 @@ MAX_PUSH = 1000 # per-batch change cap
# SYNC_PROTOCOL_VERSION what this server speaks.
# MIN_CLIENT_PROTOCOL_VERSION the oldest client protocol it still accepts.
#
# Bump SYNC_PROTOCOL_VERSION for ANY wire change. Raise the floor only for a
# BREAKING one: it hard-blocks older clients, so additive changes leave it alone.
# What each version changed, and why the floor moved or didn't, is in docs/sync.md
# ("The policy").
# Bump SYNC_PROTOCOL_VERSION for any wire change. Raise the floor only for a breaking
# one, since it blocks older clients. docs/sync.md ("The policy") lists each version.
SYNC_PROTOCOL_VERSION = 7
MIN_CLIENT_PROTOCOL_VERSION = 3
# Named capabilities beyond the base protocol. An ADDITIVE change earns a name
# here rather than a min-version bump, so a newer client meeting an older server
# can degrade to "some features unavailable" instead of refusing to sync. Clients
# test for the name, never infer a capability from a version number — that's what
# keeps feature gating independent of release lockstep.
# Named capabilities beyond the base protocol. An additive change gets a name here
# rather than a floor bump, so a newer client meeting an older server loses a feature
# instead of refusing to sync. Clients test for the name, never a version number.
SYNC_FEATURES: tuple[str, ...] = (
"notes", # note delta sync (pull + push)
"labels", # the label catalog as its own entity
@@ -83,8 +79,8 @@ SYNC_FEATURES: tuple[str, ...] = (
def protocol_advertisement() -> dict:
"""What the server publishes about the sync protocol, merged into `/api/config`.
DB-free and unauthenticated on purpose: a client has to be able to ask "can I
talk to you at all?" before it holds a device token — or even has an account.
DB-free and unauthenticated: a client asks before it holds a device token or an
account.
"""
return {
"sync_protocol_version": SYNC_PROTOCOL_VERSION,
@@ -131,9 +127,8 @@ def _page_cursor(streams: list[list[int]], since: int, limit: int) -> tuple[int,
async def changes():
since = _parse_since(request.args.get("since"))
limit = _clamp_limit(request.args.get("limit"))
# `shares=1` is a client that understands shared notes (protocol 6, `shares`).
# Anything older gets only its own notes, as before, rather than someone else's
# notes it would treat as its own: pinning them, labelling them, pushing them.
# `shares=1` is a client that understands shared notes (protocol 6). Without it a
# client gets only its own notes, since it would treat others' as its own.
with_shares = request.args.get("shares") == "1"
if with_shares:
visible = note_visible()
@@ -143,7 +138,7 @@ async def changes():
# Every note the caller can see, in any state (active/archived/trash/purged),
# since a client mirrors everything; ordered by the shared revision. On a note
# shared with the caller, pinning it moves their own state row and not the
# note, so its revision here is whichever of the two moved last (#5176).
# note, so its revision here is whichever of the two moved last.
revision = revision_for()
note_rows = (
await db.execute(
@@ -238,14 +233,6 @@ def _assign_note_fields(note: Note, ch: dict) -> None:
note.position = ch["position"]
# `_first_item_text` and `_apply_note_items` lived here until M304. Both existed for
# one reason — a checklist was a table beside the body — and both are gone with it. A
# pushed change carries its items as `- [ ] ` lines inside `body`, so applying them is
# applying the body, and naming the note is reading its first line. A client that still
# sends an `items` array is a v2 client, and the version floor below turns it away
# before any of this runs.
async def _apply_note_manual_labels(db, note: Note, ch: dict) -> None:
"""Set the note's MANUAL (picker) label memberships from client label_ids, leaving
tag-sourced (via_tag) rows to _lift_and_reconcile_tags. Only labels the caller owns count."""
@@ -307,12 +294,9 @@ async def _apply_note(db, ch: dict, previews: list[tuple[uuid.UUID, str]]) -> di
note.purged_at = None # client re-created/edited → clear the tombstone
_assign_note_fields(note, ch)
# Non-destructive LWW: the overwritten server body snapshots into history, subject
# to the same session window as a direct edit (revisions.should_snapshot). This
# path is why the window is a time rule rather than a flag on the wire: a client
# autosaving every second pushes a body change every second, and without the
# check the SERVER would snapshot each one no matter how restrained the client's
# own store was being.
# The overwritten server body snapshots into history under the same session window
# as a direct edit (revisions.should_snapshot), so a client autosaving every second
# doesn't make the server snapshot every second.
body = ch["body"] if isinstance(ch.get("body"), str) else ""
if await write_body(db, note, body, snapshot="never" if creating else "session"):
# A note pushed from a linked client gets the same link previews as one typed
@@ -327,7 +311,7 @@ async def _apply_note(db, ch: dict, previews: list[tuple[uuid.UUID, str]]) -> di
async def _apply_shared_note(db, note: Note, ch: dict, edited_at, previews: list[tuple[uuid.UUID, str]]) -> dict:
"""Apply a pushed change to a note someone else owns (#5175, #5176).
"""Apply a pushed change to a note someone else owns.
Two separate things can arrive, each under its own last-write-wins:
@@ -436,10 +420,8 @@ _CHILD_ENTITIES = {"attachment": NoteAttachment, "preview": NoteLinkPreview}
async def _apply_child_delete(db, ch: dict, removed_files: list[str]) -> dict:
"""Delete one attachment or link preview the client removed.
Not subject to last-write-wins. LWW settles which of two versions of the SAME
thing survives; a removal has no rival version — nothing can re-create an id once
it is gone — so it always applies. That is what makes a deletion stick: a removal
that lost to a newer edit of the note would come straight back on the next pull.
Not subject to last-write-wins: nothing can re-create a removed id, so there is no
rival version, and a removal that lost to a newer note edit would come back.
A row this caller cannot see answers `noop`, exactly like one that never existed,
so the reply cannot be used to learn whether someone else's id is real.
@@ -494,10 +476,8 @@ async def push():
else:
results.append({"id": ch.get("id"), "status": "rejected", "error": "unknown entity"})
await db.commit()
# After the batch commits, never inside it: the fetch reads each note in a
# session of its own, and these were scheduled mid-batch until #5164, where a
# fast fetch could look for a note that was not committed yet. Scheduled rather
# than awaited — a push must not wait on somebody else's website.
# After the batch commits: the fetch reads each note in its own session.
# Scheduled, not awaited, so a push never waits on somebody else's website.
for note_id, text in previews:
schedule_unfurls(note_id, text)
# Files go once their rows have committed, so a failed batch never leaves a row
@@ -513,8 +493,7 @@ async def put_attachment(att_id: str):
"""Upload a file a client attached while offline, under the id it chose there.
The body is the file's raw bytes; `note_id`, `filename` and `sha256` ride in the
query and the type in Content-Type. Raw rather than multipart so a native client
needs no form encoder for what is one blob and three strings.
query and the type in Content-Type, so a native client needs no form encoder.
Idempotent: re-sending an id the note already holds answers 200 and changes
nothing, which is what lets a client retry an upload whose reply it never got.
+4 -5
View File
@@ -1,8 +1,7 @@
"""SSRF-hardened link unfurling — fetch a URL server-side and extract an OG/meta
preview. Dependency-free (stdlib only), matching the project's hand-rolled ethos.
"""SSRF-hardened link unfurling: fetch a URL server-side and extract an OG/meta
preview, with the standard library only.
The server fetching arbitrary user-supplied URLs is a classic SSRF surface, so the
defenses are deliberate and layered:
Fetching user-supplied URLs is an SSRF surface, so the defences are layered:
- http/https only (no file://, gopher://, …).
- Resolve the host and require EVERY resolved address to be public — reject
private / loopback / link-local / reserved / multicast / unspecified ranges.
@@ -11,7 +10,7 @@ defenses are deliberate and layered:
can't slip through.
- At most 3 redirects, each hop re-validated the same way.
- 5s timeout, 512 KB body cap, text/html only.
No AI — this just parses OpenGraph/Twitter/`<title>` meta tags.
It parses OpenGraph, Twitter and `<title>` meta tags.
"""
from __future__ import annotations
+8 -22
View File
@@ -1,25 +1,13 @@
"""Unfurling a note's URLs in the background, after the note is already saved.
## Why this is not done inline
A save never waits on a fetch from a host nobody controls; the preview catches up.
Capture speed is the product. Unfurling is a 5-second-timeout network call to a host
nobody controls, and a note must persist the instant someone stops typing — so the
save returns first and the preview catches up. A person who pastes a link and closes
the composer has already done the thing they came to do.
It runs on the server, which sees every note from every surface. A linked desktop or
Android client gets the preview on its next pull; an unlinked one has none until it
links.
## Why it is on the server rather than in each client
The server sees every note that reaches it, from all three surfaces, so detection and
fetching live in one place instead of three. A linked desktop or Android client pushes
its note and picks the preview up on the next pull; an unlinked one has no server to
ask and simply has no preview until it links, which is the honest consequence of being
offline rather than a gap to paper over.
## What it deliberately does not do
Fail loudly. A preview that could not be fetched is not an error the person needs —
the note is fine, it just has no card. The link is still in the body, still clickable,
still searchable.
A preview that can't be fetched fails quietly: the note just has no card, and the
link is still in the body.
"""
from __future__ import annotations
@@ -39,10 +27,8 @@ from .unfurl import UnfurlError, unfurl
logger = logging.getLogger(__name__)
# Matches the frontend's detector (NoteEditor.vue) so both surfaces agree on what
# counts as a link. Trailing sentence punctuation is stripped below rather than in the
# pattern — a URL can legitimately end in most of these characters, just not when the
# sentence does.
# Matches the frontend's detector (NoteEditor.vue). Trailing sentence punctuation is
# stripped below rather than in the pattern, since a URL may end in those characters.
_URL_RE = re.compile(r"(https?://[^\s<>\"'\])]+)")
# Per note, per save. A body pasted full of links should not turn into a burst of
+2 -3
View File
@@ -6,9 +6,8 @@ from inkwell.models.user import User
def test_visible_to_user_builds_owner_or_shared_predicate():
"""DB-free smoke test: the predicate compiles and references both the direct
(shares) and group (group_members) grant paths. Functional row-visibility is
verified against a live database in M1, once notes are a real shareable
resource (family practice: DB-backed tests run against the dev image, not CI).
(shares) and group (group_members) grant paths. Row visibility against a real
database is in test_integration.py.
"""
uid = uuid.uuid4()
# User stands in as a shareable resource purely to exercise the SQL builder.
+2 -5
View File
@@ -31,11 +31,8 @@ async def test_unknown_api_route_404s(app):
# --- the version a running server reports ------------------------------------
#
# Note 3127 §5 removed version tags, so this string is the only answer to "which
# build is this?" and nothing exists to contradict it when it is wrong. That makes
# the FALLBACK the interesting case rather than the happy path: it used to be
# `__version__`, so a server run from a checkout reported `0.2.0` — a real-looking
# version naming no build anybody could obtain.
# The only answer to "which build is this?", so the fallback matters most: it must
# be "unknown", never a plausible-looking `__version__`.
async def reported_version() -> str:
+2 -3
View File
@@ -1,8 +1,7 @@
"""The channel -> release tag mapping — `packaging/channel-tag.sh`.
A channel's release tag must never be spelled like a branch. The dev channel's tag
was `dev`, which shadowed the `dev` branch: once a clone had fetched tags,
`git push origin dev` failed as an ambiguous refspec (Scribe #2184).
A channel's release tag must never be spelled like a branch: a `dev` tag shadows
the `dev` branch, and `git push origin dev` then fails as an ambiguous refspec.
These run the script itself rather than reading it, so a mapping that is correct
on the page but broken in execution still fails here.
+16 -34
View File
@@ -14,28 +14,17 @@ from inkwell.client_dist import (
)
from inkwell.config import Config
# DB-free, like the rest of this suite — the test lane runs no Postgres. That is
# why the advertisement is asserted through `advertisement()` rather than through
# `/api/config`: the route is a one-line merge of this dict into a payload whose
# other half needs a database, and testing it here tests the part that can be wrong.
#
# The routes below ARE exercised, because none opens a session: the metadata route
# only stats files, and the download's 401 is returned before any token lookup.
# DB-free, so the advertisement is asserted through `advertisement()` rather than
# `/api/config`, whose other half needs a database. The routes below are exercised:
# none opens a session.
PAYLOAD = b"not really a client, but the server only ever stats it"
# Every test that is about the MECHANISM rather than about one platform runs
# against all of them. The bugs this module can have — a sidecar describing a
# different build, a half-finished copy shadowing a good one — are not
# platform-specific, and a suite that only ever exercised Android is how the other
# four would ship untested.
# Tests about the mechanism rather than one platform run against every platform.
ALL_IDS = [p.id for p in PLATFORMS]
# `version_code` is "whatever this platform's comparator reads", and that is not one
# type. Android's install gate compares an integer; the desktop's updater compares
# Tauri's semver key. The tests carry both shapes for the same reason the module
# does — a suite that only ever wrote integers would pass while every desktop
# sidecar CI writes was being rejected.
# `version_code` is an integer for Android and Tauri's semver key for the desktop,
# so the tests write both shapes.
ANDROID_CODE = 3503708
DESKTOP_CODE = "1.0.3503707"
@@ -58,10 +47,8 @@ def coded(platform_id: str, value: int):
def _empty_baked_client(tmp_path, monkeypatch):
"""Point the baked-in copy at an empty directory.
In a source checkout `src/inkwell/client/` does not exist, so these tests
would pass anyway — but only by accident of where they are run. A built image
has real clients there, and a test that silently depends on which tree it is in
is one that will eventually lie.
A built image has real clients in `src/inkwell/client/`, so the tests must not
depend on which tree they run in.
"""
monkeypatch.setattr(client_dist, "BAKED_ROOT", tmp_path / "baked")
yield
@@ -121,8 +108,7 @@ def test_the_android_names_are_the_ones_the_image_build_writes():
with nothing checking them against each other, so a rename on one side reads as
"no Android client" on the other rather than as an error.
Phones never ask for these names; they poll `/api/client/android`. The names
changed once, with the rename to Inkwell (Scribe note 5071).
Phones never ask for these names; they poll `/api/client/android`.
"""
assert BY_ID["android"].artifact == "inkwell.apk"
assert BY_ID["android"].sidecar == "inkwell-android.json"
@@ -139,8 +125,7 @@ def test_an_absent_client_is_no_client(platform_id):
def test_a_server_holding_nothing_advertises_no_keys_at_all():
"""The KEYS are missing, not null.
A client testing for one then gets an unambiguous answer rather than having to
tell "this server has no client" apart from "this server predates the feature".
A client testing for one gets an unambiguous answer.
"""
assert releases() == {}
assert advertisement() == {}
@@ -234,7 +219,7 @@ def test_a_present_client_reports_what_a_comparator_reads(platform_id):
assert not found["url"].startswith("http")
def test_the_android_payload_still_carries_every_field_it_used_to():
def test_the_android_payload_carries_every_field_phones_read():
"""Phones in the field parse this. Fields may be ADDED — `ClientRelease` in
core/src/sync/client.rs is a plain serde struct and ignores what it does not
know — but none of these may move or change meaning."""
@@ -303,15 +288,13 @@ def test_a_dropped_in_build_beats_the_one_the_image_shipped():
"""Someone who deliberately put a build on the volume wants that build."""
place("android", root=client_dist.BAKED_ROOT, version_code=300)
place("android", version_code=99)
# Lower version and all — precedence is about intent, not about newness. An
# operator pinning an older client is doing it on purpose.
# Even at a lower version: dropping in an older client is deliberate.
assert release("android")["version_code"] == 99
def test_precedence_is_decided_per_platform_not_for_the_whole_set():
"""THE trap this table introduces. Dropping in one client must not retract the
other four — "first directory holding anything wins" would mean overriding the
APK silently takes the desktop downloads offline."""
"""Dropping in one client must not retract the other four, as "first directory
holding anything wins" would."""
for platform_id in ALL_IDS:
place(platform_id, root=client_dist.BAKED_ROOT, version_code=300)
place("android", version_code=99)
@@ -326,8 +309,7 @@ def test_precedence_is_decided_per_platform_not_for_the_whole_set():
def test_a_broken_drop_in_does_not_shadow_the_baked_copy():
"""A half-finished copy onto the volume must not take the app offline.
This is the failure the copy-order advice in docs/android-distribution.md is
about, and the server should ride it out rather than go dark.
The failure the copy-order advice in docs/android-distribution.md is about.
"""
place("android", root=client_dist.BAKED_ROOT, version_code=300)
place("android", size=999_999) # sidecar describing a different build
@@ -424,7 +406,7 @@ async def test_a_server_without_the_appimage_404s_its_updater_manifest(app):
assert resp.status_code == 404
# --- serving the bytes (#5118, family idea #5103 practice 6) -----------------
# --- serving the bytes ------------------------------------------------------
#
# `send_artifact` is called directly in a request context: the route in front of it
# needs a signed-in account, and this suite has no database to sign one in with.
+5 -7
View File
@@ -34,18 +34,16 @@ async def test_revoke_self_requires_auth(app):
def test_revoke_self_routes_to_the_self_revoke_view(app):
# The static rule must win over `/devices/<device_id>`: if "self" fell through to
# the id-keyed route it would be read as a malformed UUID and answer 404. Checked
# on the URL map, because every route behind login_required now reads the
# account's session epoch (#5173). The view's 400 for a caller with no bearer
# token is in test_integration.py.
# The static rule must win over `/devices/<device_id>`, or "self" would be read as
# a malformed UUID and 404. Checked on the URL map, because every route behind
# login_required reads the database. The view's 400 is in test_integration.py.
endpoint, _ = app.url_map.bind("localhost").match("/api/auth/devices/self", method="DELETE")
assert endpoint == "auth.revoke_own_device"
async def test_device_login_validates_input(app):
# Missing credentials → 400 BEFORE any DB access, so it's checkable in the
# DB-free unit lane (invalid-cred and success paths are operator-verified).
# Missing credentials → 400 before any DB access, so it's checkable without a
# database. The other paths are in test_integration.py.
client = app.test_client()
resp = await client.post("/api/auth/device-login", json={})
assert resp.status_code == 400
+1 -1
View File
@@ -76,7 +76,7 @@ def test_a_script_without_exactly_one_placeholder_is_not_served(script):
def test_the_public_address_setting_beats_the_request_host():
"""Behind a reverse proxy the Host this request arrived with may be an internal
one; Settings → Public address is what the operator says people use."""
one; Settings → Public address is the one people use."""
assert server_address("https://notes.example.com", "http", "inkwell:5000") == "https://notes.example.com"

Some files were not shown because too many files have changed in this diff Show More