Server comments describe the code to a stranger (#6221)
CI & Build / Python lint (push) Successful in 2s
CI & Build / Build now, or wait for Android? (push) Successful in 2s
Android / Build, or is the channel already serving this? (push) Successful in 3s
Android / Core and FFI clippy and tests (push) Skipped
Android / Kotlin + Rust (APK) (push) Skipped
Desktop (Tauri) / Build, or is the channel already serving this? (push) Successful in 2s
CI & Build / Web typecheck and unit tests (push) Successful in 9s
CI & Build / Python tests (push) Successful in 13s
Android / Build the server image (push) Skipped
Desktop (Tauri) / Web tests, clippy, Rust tests and rustfmt (push) Skipped
Desktop (Tauri) / Tauri desktop (Linux) (push) Skipped
Desktop (Tauri) / Windows installer (cross-compiled) (push) Skipped
Desktop (Tauri) / Update manifest (push) Skipped
CI & Build / integration (push) Successful in 1m30s
CI & Build / Build & push image (push) Successful in 32s
CI & Build / Python lint (push) Successful in 2s
CI & Build / Build now, or wait for Android? (push) Successful in 2s
Android / Build, or is the channel already serving this? (push) Successful in 3s
Android / Core and FFI clippy and tests (push) Skipped
Android / Kotlin + Rust (APK) (push) Skipped
Desktop (Tauri) / Build, or is the channel already serving this? (push) Successful in 2s
CI & Build / Web typecheck and unit tests (push) Successful in 9s
CI & Build / Python tests (push) Successful in 13s
Android / Build the server image (push) Skipped
Desktop (Tauri) / Web tests, clippy, Rust tests and rustfmt (push) Skipped
Desktop (Tauri) / Tauri desktop (Linux) (push) Skipped
Desktop (Tauri) / Windows installer (cross-compiled) (push) Skipped
Desktop (Tauri) / Update manifest (push) Skipped
CI & Build / integration (push) Successful in 1m30s
CI & Build / Build & push image (push) Successful in 32s
Rule 32 "Comments describe the product to a stranger" sweep over src/, alembic/ and tests/. Comments and docstrings now say what the code does and the reason as a present-tense constraint, briefly. Removed: - issue, task, note, rule, milestone and family-idea references - "used to" / before-and-after narration - quotes of decisions and incident stories from the deploying instance - paragraph-length arguments, where a line or two carries the fact Kept every hazard as a present-tense fact, e.g. timing oracles, the proxy hop direction, and migrations that must stay frozen. Also: - Three test names lose "still"/"used to"/"pre_m304" phrasing. - pyproject's description drops "FabledSword family". The history these comments carried stays in Scribe and in git log. A comment-stripping diff (Scribe snippet #6226 "codeonly.py — prove a diff changed only comments") shows no code changed apart from the renamed tests and the description string. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -4,8 +4,8 @@ Revision ID: 0001
|
||||
Revises:
|
||||
Create Date: 2026-07-19
|
||||
|
||||
The M0 identity + sharing-ACL spine. No notes yet (those arrive in M1); the
|
||||
shares table is polymorphic so notes reuse it without a schema retrofit.
|
||||
Identity and the sharing ACL. The shares table is polymorphic, so any shareable
|
||||
entity uses it.
|
||||
"""
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
|
||||
@@ -4,7 +4,7 @@ Revision ID: 0002
|
||||
Revises: 0001
|
||||
Create Date: 2026-07-19
|
||||
|
||||
The M1 capture core: the notes table the masonry board renders.
|
||||
The notes table the board renders.
|
||||
"""
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
|
||||
@@ -5,7 +5,7 @@ Revises: 0002
|
||||
Create Date: 2026-07-19
|
||||
|
||||
Adds the admin role bit and the DB-backed settings table. No rows are seeded —
|
||||
defaults live in the code registry (thoughtsync.settings), so an empty table
|
||||
defaults live in the code registry (inkwell.settings), so an empty table
|
||||
means every setting is at its default.
|
||||
"""
|
||||
from alembic import op
|
||||
|
||||
@@ -16,10 +16,8 @@ depends_on = None
|
||||
def upgrade() -> None:
|
||||
# The note's display NAME: explicit title if set, else the first body line.
|
||||
op.add_column("notes", sa.Column("display_title", sa.Text(), nullable=False, server_default=""))
|
||||
# Best-effort backfill (kept intentionally simple + unambiguous — CI has no
|
||||
# Postgres lane, so this only runs on a real deploy). The app recomputes the
|
||||
# precise value via derive_display_title on the next save (which additionally
|
||||
# skips leading blank lines). Capped at 200 chars.
|
||||
# Best-effort backfill, capped at 200 chars. The app recomputes the exact value
|
||||
# (derive_display_title, which also skips leading blank lines) on the next save.
|
||||
op.execute(
|
||||
r"""
|
||||
UPDATE notes SET display_title = LEFT(btrim(
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
"""sync revision + tombstone infrastructure (M8 sync hub, step 1)
|
||||
"""sync revision + tombstone infrastructure
|
||||
|
||||
Revision ID: 0015
|
||||
Revises: 0014
|
||||
@@ -11,9 +11,6 @@ forget to bump it. Child-row changes (items / attachments / label membership) bu
|
||||
their parent note's revision, since a note syncs as a whole. `purged_at` is the
|
||||
hard-delete tombstone (distinct from `deleted_at` = recoverable trash) so an
|
||||
offline client learns a row is gone instead of resurrecting it.
|
||||
|
||||
No API behavior changes here — this is pure infrastructure. Triggers/behavior are
|
||||
operator-verified on deploy (no Postgres CI lane).
|
||||
"""
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
"""device_tokens (M8 sync hub, step 2)
|
||||
"""device_tokens
|
||||
|
||||
Revision ID: 0016
|
||||
Revises: 0015
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
"""labels.updated_at (M8 sync hub, step 4 — LWW field for the label catalog)
|
||||
"""labels.updated_at, the last-write-wins field for the label catalog
|
||||
|
||||
Revision ID: 0017
|
||||
Revises: 0016
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
"""note_attachments.sha256 (M8 sync hub, step 5 — attachment blob sync)
|
||||
"""note_attachments.sha256, for attachment blob sync
|
||||
|
||||
Revision ID: 0018
|
||||
Revises: 0017
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
"""note_attachments.filename (any-file attachments — M6 1900)
|
||||
"""note_attachments.filename, for attachments of any file type
|
||||
|
||||
Revision ID: 0019
|
||||
Revises: 0018
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
"""note_link_previews (URL unfurl — M6 1901)
|
||||
"""note_link_previews, cached URL unfurls
|
||||
|
||||
Revision ID: 0020
|
||||
Revises: 0019
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
"""saved_filters (richer search + saved views — M6 1902)
|
||||
"""saved_filters, saved search views
|
||||
|
||||
Revision ID: 0021
|
||||
Revises: 0020
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
"""notes.recurrence (recurring reminders — M6 1908)
|
||||
"""notes.recurrence, for recurring reminders
|
||||
|
||||
Revision ID: 0022
|
||||
Revises: 0021
|
||||
|
||||
@@ -1,14 +1,11 @@
|
||||
"""note_links.target_id — resolve [[links]] to a note, not to a string (M13 step 1)
|
||||
"""note_links.target_id — resolve [[links]] to a note, not to a string
|
||||
|
||||
Revision ID: 0023
|
||||
Revises: 0022
|
||||
Create Date: 2026-08-22
|
||||
|
||||
A wiki-link stored only as normalized TEXT means a note's name IS the edge: rename
|
||||
the note and every inbound link stops matching. The old answer was to rewrite the
|
||||
`[[Old Name]]` text inside every note that linked to it — workable while an explicit
|
||||
title existed to hold still, untenable once a note's name is just its first body
|
||||
line (M13).
|
||||
A wiki-link stored only as normalized text makes a note's name the edge, so renaming
|
||||
the note breaks every inbound link. Resolving to an id survives a rename.
|
||||
|
||||
`target_norm` stays: it is what an UNRESOLVED link carries, since linking to a note
|
||||
that doesn't exist yet is a supported way to create one.
|
||||
|
||||
@@ -1,22 +1,14 @@
|
||||
"""drop note_links — [[wiki-links]] are removed (note 2897)
|
||||
"""drop note_links — [[wiki-links]] are removed
|
||||
|
||||
Revision ID: 0024
|
||||
Revises: 0023
|
||||
Create Date: 2026-08-22
|
||||
|
||||
ThoughtSync is an intermediary surface for capture and recall; a linking system is
|
||||
organization, which is not what it is for. Backlinks, the graph and the name index
|
||||
went with it.
|
||||
0023 stays in the chain: removing an applied revision would strand a database's
|
||||
alembic_version pointer.
|
||||
|
||||
0023 (which added `note_links.target_id`) is deliberately left in the chain rather
|
||||
than deleted. It shipped in an image and may already be applied, and removing an
|
||||
applied revision would strand a database's alembic_version pointer. So the column is
|
||||
dropped here along with the table it lived on, and the history stays honest about the
|
||||
fact that it existed for a day.
|
||||
|
||||
No down-migration data concern: note_links was always DERIVED from note bodies. The
|
||||
`[[text]]` is still sitting in every body it was written in; nothing a person typed is
|
||||
lost by this.
|
||||
note_links was derived from note bodies, so nothing typed is lost; the `[[text]]` is
|
||||
still in every body.
|
||||
"""
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
|
||||
@@ -1,19 +1,11 @@
|
||||
"""drop notes.kind — a checklist is something a note HAS (M13 step 2)
|
||||
"""drop notes.kind — a checklist is something a note has
|
||||
|
||||
Revision ID: 0025
|
||||
Revises: 0024
|
||||
Create Date: 2026-08-22
|
||||
|
||||
`kind` was never a type: a plain TEXT column with no enum and no CHECK, compared
|
||||
against a hardcoded ("text", "list") tuple in six places. `note_items` was always an
|
||||
ordinary child table keyed by note_id, serialization always emitted `items` whatever
|
||||
the kind, and the Android editor already toggled between the two losslessly. The
|
||||
storage has modelled "a body plus optional checkable items" the whole time; only the
|
||||
gates forbade it.
|
||||
|
||||
Nothing is lost. Items were already rows in their own table, and a note that was
|
||||
`kind = 'list'` keeps every one of them — it just stops being a different sort of
|
||||
thing from the note next to it.
|
||||
A note is a body plus optional checkable items. Items are rows in their own table,
|
||||
so a `kind = 'list'` note keeps every one.
|
||||
"""
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
@@ -29,7 +21,6 @@ def upgrade() -> None:
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
# server_default so existing rows get a value; every note comes back as 'text',
|
||||
# which is right — a restored note with items would previously have hidden its
|
||||
# body, and there is no record of which ones were once lists.
|
||||
# server_default so existing rows get a value. Every note comes back as 'text':
|
||||
# nothing records which were lists.
|
||||
op.add_column("notes", sa.Column("kind", sa.Text(), nullable=False, server_default="text"))
|
||||
|
||||
@@ -4,33 +4,14 @@ Revision ID: 0026
|
||||
Revises: 0025
|
||||
Create Date: 2026-08-22
|
||||
|
||||
M13 step 3. A note is a body plus optional checkable items; its NAME is the first
|
||||
non-empty line of that body, falling back to its first checklist item. There is no
|
||||
separate field to type into, and `display_title` (already persisted, already what
|
||||
search results and export filenames read) carries the name.
|
||||
A note's name is its first non-empty body line, carried by `display_title`.
|
||||
|
||||
## The search vector has to be rebuilt, not just left alone
|
||||
`notes.search_vector` is a stored generated column over `title` (0005, weight A), and
|
||||
Postgres won't drop a column a generated one depends on. So it is recreated over
|
||||
`display_title`, keeping the name ranked above the body. That recomputes every row
|
||||
and rebuilds the GIN index, which takes a while on a large table.
|
||||
|
||||
`notes.search_vector` is a STORED GENERATED column whose expression names `title`
|
||||
(migration 0005, weight A) — Postgres will refuse to drop a column another generated
|
||||
column depends on, and even if it didn't, the weighting would be wrong. So it is
|
||||
dropped and recreated over `display_title` instead, which keeps the original
|
||||
intent: the note's NAME ranks above the rest of its body.
|
||||
|
||||
Rebuilding a stored generated column re-computes every row, and the GIN index is
|
||||
rebuilt with it. On a personal instance that is milliseconds; it is worth knowing
|
||||
before running this against something large.
|
||||
|
||||
## What happens to existing titles
|
||||
|
||||
Nothing preserves them, deliberately: `display_title` was already derived from the
|
||||
title when one was set, so every note keeps the NAME it had. What is lost is the
|
||||
distinction between "this note has an explicit title" and "this note's first line is
|
||||
its name" — which is the distinction being removed.
|
||||
|
||||
Imports are the exception and are handled in code, not here: a Keep note's title, or
|
||||
one in an export taken before this, is folded in as the note's first body line rather
|
||||
than dropped (see `_create_imported_note`).
|
||||
`display_title` was already derived from the title, so every note keeps its name.
|
||||
"""
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
|
||||
@@ -4,64 +4,31 @@ Revision ID: 0027
|
||||
Revises: 0026
|
||||
Create Date: 2026-08-24
|
||||
|
||||
M304. A checklist item becomes a `- [ ] milk` line of `notes.body`, and `note_items`
|
||||
goes. The reason is positional, not cosmetic: a row had a position in a table and no
|
||||
position in the text, so a separate list could only ever render AFTER the prose. With
|
||||
the items in the body, a list can sit between two paragraphs — which is the thing that
|
||||
could not be built before and no amount of restyling would have delivered.
|
||||
A checklist item becomes a `- [ ] milk` line of `notes.body`, so a list can sit
|
||||
between two paragraphs, and `note_items` goes.
|
||||
|
||||
## This migration rewrites note bodies
|
||||
|
||||
Every note that has items gets its body appended to. The rules below are strict
|
||||
because rewriting somebody's text deserves it — not, as an earlier draft of this
|
||||
docstring claimed, because this instance holds imported Google Keep notes. It does
|
||||
not; note 2916's headline is that nothing here is anyone's work but the operator's
|
||||
test data. What 2916 actually says about imports is conditional — text arriving from
|
||||
another app WOULD be real, and any import path has to treat it that way — and the
|
||||
importer this migration shares a format with is one nobody here has run.
|
||||
* Rows are read before the table is dropped, in this one transaction.
|
||||
* The existing body is only appended to.
|
||||
* The layout (a blank line between prose and the list, nothing between items) is
|
||||
what the export writes and what `derive::append_item` produces, so the clients
|
||||
migrate their own stores independently and still agree with the server.
|
||||
|
||||
Careful was still the right call. It cost little, and the same care is what the rule
|
||||
demands the day someone does import something:
|
||||
The fold is inlined rather than imported from `notes/checklist.py`: a migration must
|
||||
keep producing what it produced the day it ran.
|
||||
|
||||
* Rows are read BEFORE the table is dropped, in this one transaction.
|
||||
* The existing body is never rewritten, only appended to.
|
||||
* The layout — a blank line between prose and the list, nothing between consecutive
|
||||
items — is byte-for-byte what `_note_markdown` has always exported and what
|
||||
`derive::append_item` produces on every client. All three landing on the same text
|
||||
is what lets the clients migrate their own SQLite stores independently and still
|
||||
agree with the server, with no sync required to reconcile them.
|
||||
## `updated_at` is left alone
|
||||
|
||||
## The fold is inlined on purpose
|
||||
Raw SQL, so `onupdate` never fires, and a client holding an unpushed edit keeps the
|
||||
newer `updated_at` and wins last-write-wins. The `sync_revision` trigger does fire,
|
||||
so every migrated note is pulled once and clients converge.
|
||||
|
||||
`notes/checklist.py` has this same function and this migration deliberately does not
|
||||
import it. A migration has to keep producing what it produced the day it ran; if the
|
||||
app's spacing rule ever changes, this file must not change with it.
|
||||
## The downgrade is not a true inverse
|
||||
|
||||
## `updated_at` is left alone, and that is load-bearing
|
||||
|
||||
Raw SQL, so SQLAlchemy's `onupdate` never fires. Two reasons, and the second matters
|
||||
more than the first. Every client folds the same rows the same way, so the new body is
|
||||
news to nobody. And a client holding an UNPUSHED body edit still has the newer
|
||||
`updated_at`, so when it pulls the migrated note last-write-wins keeps its edit instead
|
||||
of the migration silently winning.
|
||||
|
||||
The `notes` row's own `sync_revision` trigger (migration 0015) does fire, so every
|
||||
migrated note becomes pullable once. That is wanted: it is what makes a client whose
|
||||
local fold somehow differed converge on the server's text.
|
||||
|
||||
## The downgrade is not a true inverse, and says so
|
||||
|
||||
It recreates an empty `note_items` and leaves the bodies alone. Nothing is lost —
|
||||
every item is still there as text, which is where this migration put it — but the old
|
||||
code would show those notes as prose with no checklist. A faithful inverse is not
|
||||
possible: once the items are lines, nothing distinguishes a line this migration wrote
|
||||
from one somebody typed, and a downgrade that guessed would eat hand-written task
|
||||
lists. The real rollback is a database restore.
|
||||
|
||||
Recreating the table is not decoration, though. Migration 0015's downgrade runs
|
||||
`DROP TRIGGER IF EXISTS trg_note_items_bump_note ON note_items`, and `IF EXISTS`
|
||||
covers the trigger, not the table — against a missing table that statement errors. So
|
||||
this is what keeps the migration chain runnable all the way back down.
|
||||
It recreates an empty `note_items` and leaves the bodies alone: nothing tells a line
|
||||
this wrote from one somebody typed. The real rollback is a database restore. The
|
||||
table must exist because 0015's downgrade drops a trigger on it.
|
||||
"""
|
||||
import re
|
||||
|
||||
@@ -100,9 +67,8 @@ def upgrade():
|
||||
|
||||
for note_id, items in grouped.items():
|
||||
body = bind.execute(sa.text("SELECT body FROM notes WHERE id = :id"), {"id": note_id}).scalar()
|
||||
# An item whose note is already gone has nothing to fold into. The foreign key
|
||||
# should make this impossible; skipping costs nothing and failing here would
|
||||
# leave the database half-migrated.
|
||||
# An item whose note is gone has nothing to fold into. The foreign key should
|
||||
# prevent it; skipping avoids a half-migrated database.
|
||||
if body is None:
|
||||
continue
|
||||
for text, checked in items:
|
||||
@@ -113,9 +79,8 @@ def upgrade():
|
||||
|
||||
|
||||
def downgrade():
|
||||
# Column-for-column as migration 0006 created it, index name included: 0015's
|
||||
# downgrade names both the table and its trigger, so a near-enough copy is not
|
||||
# good enough.
|
||||
# Exactly as 0006 created it, index name included: 0015's downgrade names both
|
||||
# the table and its trigger.
|
||||
op.create_table(
|
||||
"note_items",
|
||||
sa.Column("id", UUID(as_uuid=True), primary_key=True),
|
||||
|
||||
@@ -4,73 +4,27 @@ Revision ID: 0028
|
||||
Revises: 0027
|
||||
Create Date: 2026-08-26
|
||||
|
||||
M311. A `#tag` was being shown twice — once as the text you typed and once as a chip —
|
||||
and with the chip moved to the top of the card the text is redundant. This removes it,
|
||||
but only from notes where the tag was standing on its own.
|
||||
A tag standing on its own line shows as a chip, so its text is removed, by the same
|
||||
rule as `notes/tags.py:split_body_tags`:
|
||||
|
||||
## This migration rewrites note bodies
|
||||
* A line containing nothing but tags and whitespace is removed.
|
||||
* Every other line is left exactly as written, so `remember to call #mom` stays.
|
||||
* A line inside a ``` fence is never touched.
|
||||
* A note that is nothing but tags keeps its text, rather than becoming a blank card.
|
||||
|
||||
The rule is deliberately narrow, and the same one `notes/tags.py:split_body_tags`
|
||||
applies from here on:
|
||||
The label rows flip to `via_tag = false` in the same transaction: the text backing
|
||||
them is gone, and `_lift_and_reconcile_tags` would otherwise detach them on the next
|
||||
save.
|
||||
|
||||
* A line containing nothing but tags and whitespace is REMOVED.
|
||||
* Every other line is left exactly as written.
|
||||
The transform and `_display_title` are inlined: a migration must keep producing what
|
||||
it produced the day it ran. The title is recomputed only where the body moved.
|
||||
|
||||
So `#todo` on its own line goes, and `remember to call #mom tomorrow` does not. The
|
||||
looser reading — also stripping a trailing tag off a prose line — was rejected because
|
||||
the text does not say which kind it is: `buy milk #grocery` is filing, `remember to
|
||||
call #mom` is the sentence's object, and lifting the second leaves "remember to call".
|
||||
Rewriting somebody's words to save a duplicate chip is a bad trade, and a migration is
|
||||
the worst possible place to make it.
|
||||
Raw SQL, so `onupdate` never fires and a client's unpushed edit still wins
|
||||
last-write-wins. The `sync_revision` trigger fires, so clients pull the server's text.
|
||||
|
||||
Two guards, both of which cost a note nothing:
|
||||
|
||||
* A line inside a ``` fence is never touched. A `#tag` there is a shell comment in a
|
||||
snippet somebody pasted, and deleting it would eat a line of their example.
|
||||
* A note that is NOTHING but tags keeps its text. Lifting would leave a blank card,
|
||||
which is worse than the duplication this fixes.
|
||||
|
||||
## The label rows have to graduate in the same transaction
|
||||
|
||||
A `via_tag` row means "this label is backed by text still in the body". Once the text
|
||||
is gone that is false, and leaving it true is not cosmetic: `_lift_and_reconcile_tags`
|
||||
detaches any `via_tag` row it cannot find a `#tag` for, so the note would lose the tag
|
||||
on its very next save. The flip to `via_tag = false` is what makes the label the record
|
||||
instead — and what makes the chip's × appear in both editors, which is now the only way
|
||||
to remove a tag whose text no longer exists.
|
||||
|
||||
## The transform is inlined, like 0027's
|
||||
|
||||
`split_body_tags` is deliberately NOT imported. A migration has to keep producing what
|
||||
it produced the day it ran; if the app's rule is ever loosened, this file must not
|
||||
loosen with it and start eating prose it previously left alone.
|
||||
|
||||
`_display_title` is inlined for the same reason, and is only recomputed for a note whose
|
||||
body actually moved — a note named after a `#todo` line needs a new name, and reading it
|
||||
from the app would couple this migration to a rule that has already changed once (M13).
|
||||
|
||||
## `updated_at` is left alone, and that is load-bearing
|
||||
|
||||
Raw SQL, so SQLAlchemy's `onupdate` never fires. A client holding an UNPUSHED body edit
|
||||
keeps the newer `updated_at`, so when it pulls the migrated note last-write-wins keeps
|
||||
its edit instead of the migration silently winning.
|
||||
|
||||
The `sync_revision` trigger (migration 0015) does fire, so every rewritten note becomes
|
||||
pullable once and clients converge on the server's text. That is wanted here: unlike
|
||||
0027, the clients do NOT yet apply this rule locally, so the server's copy is the only
|
||||
correct one until they do.
|
||||
|
||||
## The downgrade is not a true inverse, and says so
|
||||
|
||||
It cannot be. Nothing distinguishes a `#todo` line this migration deleted from one that
|
||||
was never there, and putting one back would be guessing at where in the note it went.
|
||||
|
||||
Nothing is lost, though, which is why that is acceptable: the tag still exists as a
|
||||
label on the note, and the chip still shows it. What a downgrade cannot restore is the
|
||||
DUPLICATE — which is the thing this migration set out to remove. Rolling the rows back
|
||||
to `via_tag = true` would be actively harmful: the text that flag claims to be backed by
|
||||
is gone, so the next save would detach the label and lose the tag for real. So the
|
||||
downgrade leaves both alone. The real rollback is a database restore.
|
||||
The downgrade does nothing. Restoring the lines would be guessing, and flipping the
|
||||
rows back to `via_tag = true` would detach every label on the next save. The real
|
||||
rollback is a database restore.
|
||||
"""
|
||||
import re
|
||||
|
||||
@@ -152,17 +106,11 @@ def upgrade():
|
||||
sa.text("UPDATE notes SET body = :body, display_title = :title WHERE id = :id"),
|
||||
{"body": lifted, "title": _display_title(lifted), "id": note_id},
|
||||
)
|
||||
# Even when the body did not move, a tag can be standalone only in the sense
|
||||
# that its line was already removed by an earlier pass — so the flip is driven
|
||||
# by the tag list, not by whether the text changed.
|
||||
# The flip is driven by the tag list, not by whether the text changed: an
|
||||
# earlier pass may already have removed the line.
|
||||
if standalone:
|
||||
bind.execute(flip, {"nid": note_id, "names": [n.lower() for n in standalone]})
|
||||
|
||||
|
||||
def downgrade():
|
||||
"""Deliberately empty — see the module docstring.
|
||||
|
||||
Restoring the deleted lines would be guessing, and flipping the rows back to
|
||||
`via_tag = true` would be worse than doing nothing: the text that flag claims backs
|
||||
them is gone, so the next save would detach the label and lose the tag for real.
|
||||
"""
|
||||
"""Deliberately empty; see the module docstring."""
|
||||
|
||||
@@ -4,50 +4,18 @@ Revision ID: 0029
|
||||
Revises: 0028
|
||||
Create Date: 2026-08-28
|
||||
|
||||
M315 step 3. A note's colour was set by a picker and read by three card renderers.
|
||||
Steps 1 and 2 stopped every one of those reads: the card is one neutral per theme and
|
||||
the only coloured thing on a board is a tag. This drops the column that nothing has
|
||||
been reading since, and the picker goes with it.
|
||||
Cards are one neutral surface per theme; the only colour on a board is a tag's.
|
||||
`labels.color` is untouched. A colour a note was given is dropped with the column.
|
||||
|
||||
`labels.color` is untouched. That is the colour that survived, and the one the whole
|
||||
milestone was about keeping.
|
||||
`saved_filters.params` may carry a `"color"` facet, so stored views are swept too. In
|
||||
Python rather than `params::jsonb - 'color'`, because Postgres has no try-cast and one
|
||||
malformed blob would abort the migration; a row that doesn't parse is left as it is.
|
||||
|
||||
## What is lost, and why that is the change rather than a cost of it
|
||||
|
||||
Any colour a note was explicitly given. There is nowhere to preserve it TO — the field
|
||||
it would be preserved in is the one being dropped — and nothing renders it, so a
|
||||
preserved value would be a column kept warm for a feature that was deliberately
|
||||
removed. A note that had a colour now takes its identity from its tags, which is what
|
||||
the operator asked for: "strip color from the cards ... and keep the color for tags
|
||||
just on the tag."
|
||||
|
||||
The palette itself is not lost. `NOTE_COLORS` moved from `models/note.py` to
|
||||
`colors.py` in the same change — labels still name a colour, and leaving the vocabulary
|
||||
defined on the model that lost one would be an invitation to put the column back.
|
||||
|
||||
## The saved-filter sweep is not optional
|
||||
|
||||
`saved_filters.params` is opaque JSON mirroring the `GET /api/notes` facet query, and
|
||||
a stored view could carry `"color": "teal"`. With the facet gone that key would sit
|
||||
there forever, and `clean_params` only guards what is written FROM here on. A view that
|
||||
silently filters on a field the app no longer has is worse than one that visibly lost a
|
||||
criterion, so the stored rows are swept too.
|
||||
|
||||
Done in Python rather than as `params::jsonb - 'color'`, deliberately. Postgres has no
|
||||
try-cast: one malformed blob would abort the whole migration, and these rows are
|
||||
somebody's saved views. `json.loads` in a try/except lets a corrupt row keep whatever it
|
||||
holds and lets every other row be fixed.
|
||||
|
||||
## Search is not affected
|
||||
|
||||
`notes.search_vector` is a stored generated column over `display_title` and `body`
|
||||
(rebuilt in 0026). It never named `color`, so unlike the title drop there is nothing
|
||||
here to tear down and recreate.
|
||||
The search vector never named `color`, so it needs no rebuild.
|
||||
|
||||
## Downgrade
|
||||
|
||||
Restores the column, empty, at its old default. The values are not recoverable — see
|
||||
above. It is the schema that comes back, not the data.
|
||||
Restores the column, empty, at its old default. The values are not recoverable.
|
||||
"""
|
||||
import json
|
||||
|
||||
@@ -71,9 +39,8 @@ def upgrade() -> None:
|
||||
try:
|
||||
parsed = json.loads(params)
|
||||
except (ValueError, TypeError):
|
||||
# A blob that does not parse cannot be edited safely. Leaving it is
|
||||
# correct: it was already unreadable by the app, and this migration is not
|
||||
# the place to decide what it should have said.
|
||||
# A blob that doesn't parse can't be edited safely; the app couldn't read
|
||||
# it either.
|
||||
continue
|
||||
if not isinstance(parsed, dict) or "color" not in parsed:
|
||||
continue
|
||||
|
||||
@@ -4,15 +4,9 @@ Revision ID: 0030
|
||||
Revises: 0029
|
||||
Create Date: 2026-10-06
|
||||
|
||||
The product is renamed ThoughtSync → Inkwell (Scribe note 5071), and the `site_name`
|
||||
registry default moves with it. On its own that only reaches servers whose settings
|
||||
table has no `site_name` row — and most servers have one, because the Settings page
|
||||
saves EVERY key on Save, not just the one that changed. An admin who opened Settings
|
||||
to flip registration off has persisted `"ThoughtSync"` without ever choosing it.
|
||||
|
||||
So the row is rewritten, but only when it holds exactly the old default. A name the
|
||||
admin actually typed is theirs and is left alone; the old default is the one value we
|
||||
can be sure nobody chose.
|
||||
The app was renamed from ThoughtSync to Inkwell. The Settings page saves every key,
|
||||
so most servers have a stored `site_name` of the old default without anyone choosing
|
||||
it. The row is rewritten only when it holds exactly that; a typed name is left alone.
|
||||
|
||||
The match is on the stored JSON text (`settings.value` is `json.dumps(value)`), so
|
||||
`'"ThoughtSync"'` is the whole of what it looks for.
|
||||
|
||||
@@ -4,16 +4,9 @@ Revision ID: 0031
|
||||
Revises: 0030
|
||||
Create Date: 2026-10-07
|
||||
|
||||
0015 made every child table bump its parent note's `sync_revision`, so a note syncs
|
||||
as a whole. `note_link_previews` arrived later (0020) and was never added, and two
|
||||
things have been missing on every linked device since:
|
||||
|
||||
- A preview fetched in the background after a save (`unfurl_queue.py`) never reached
|
||||
a device that had already pulled the note. The note's revision was assigned when
|
||||
the TEXT was saved, before the preview existed, and nothing moved it afterwards.
|
||||
- A preview dismissed on the web stayed on every other device, for the same reason.
|
||||
|
||||
The trigger is the same function 0015 installs on the other child tables.
|
||||
A note syncs as a whole, so every child table bumps its note's `sync_revision`. With
|
||||
this, a preview fetched after a save, or dismissed on the web, reaches every device.
|
||||
The trigger is the function 0015 installs on the other child tables.
|
||||
|
||||
## Downgrade
|
||||
|
||||
|
||||
@@ -4,9 +4,8 @@ Revision ID: 0032
|
||||
Revises: 0031
|
||||
Create Date: 2026-10-07
|
||||
|
||||
Until now the only way to add a second person was to re-open registration to the
|
||||
whole internet while they signed up (#2939 §1). An invite lets one person register
|
||||
while registration stays closed. Only the token's SHA-256 hash is stored.
|
||||
An invite lets one person register while registration stays closed. Only the token's
|
||||
SHA-256 hash is stored.
|
||||
|
||||
## Downgrade
|
||||
|
||||
|
||||
@@ -4,20 +4,19 @@ Revision ID: 0033
|
||||
Revises: 0032
|
||||
Create Date: 2026-10-07
|
||||
|
||||
A forgotten password used to need a hand on the database (#2939 §2), and the app has
|
||||
no mail path to send a reset by. An admin makes a reset link for the account and
|
||||
hands it over (#5173). Only the token's SHA-256 hash is stored.
|
||||
An admin makes a reset link for the account and hands it over. Only the token's
|
||||
SHA-256 hash is stored.
|
||||
|
||||
`users.session_epoch` is what lets a reset sign the account out everywhere. Sessions
|
||||
are signed cookies held by the browser, so the server can't delete them; each one
|
||||
carries the epoch it was signed in under, and a reset moves the account's epoch on.
|
||||
It starts at 0, the value a cookie from before this migration is read as, so nobody
|
||||
is signed out by the upgrade itself.
|
||||
It starts at 0, the value a cookie without one is read as, so the upgrade signs
|
||||
nobody out.
|
||||
|
||||
## Downgrade
|
||||
|
||||
Drops the table and the column. Outstanding reset links stop working; sessions keep
|
||||
working, since nothing checks an epoch any more.
|
||||
working, since nothing checks an epoch.
|
||||
"""
|
||||
import sqlalchemy as sa
|
||||
from alembic import op
|
||||
|
||||
@@ -4,11 +4,10 @@ Revision ID: 0034
|
||||
Revises: 0033
|
||||
Create Date: 2026-10-07
|
||||
|
||||
Nothing wrote a share until #5174, so the table never needed these. Now the Share
|
||||
dialog does:
|
||||
For the Share dialog:
|
||||
- A unique index on (resource_type, resource_id, shared_with_user_id) where a user is
|
||||
the target, so sharing a note with someone twice updates the one grant rather than
|
||||
stacking two (the same for a group, ahead of step 15).
|
||||
stacking two (the same for a group).
|
||||
- A check that `permission` is `view` or `edit`.
|
||||
- An index on `shared_with_user_id` for "Shared with me".
|
||||
|
||||
|
||||
@@ -5,7 +5,7 @@ Revises: 0034
|
||||
Create Date: 2026-10-07
|
||||
|
||||
The change feed carries every note a person can see, including notes shared with
|
||||
them (#5175). When a share ends, the note stops being visible, so it simply stops
|
||||
them. When a share ends, the note stops being visible, so it simply stops
|
||||
appearing in the feed. A device that already holds a copy would keep it forever. A
|
||||
revocation is that missing signal: one row per (note, person) who lost the note,
|
||||
stamped from the same `sync_revision_seq` the notes and labels draw from, so it sits
|
||||
|
||||
@@ -4,12 +4,10 @@ Revision ID: 0036
|
||||
Revises: 0035
|
||||
Create Date: 2026-10-07
|
||||
|
||||
Pin, archive and board order are personal organization, and until now they were
|
||||
columns on `notes`, so on a shared note they could only ever mean the owner's
|
||||
(#5176). This table holds them for everyone else: one row per (note, person it is
|
||||
shared with) who has pinned, archived or moved it. The owner keeps the note's own
|
||||
columns; a recipient with no row sees the note unpinned, unarchived, in the owner's
|
||||
order.
|
||||
Pin, archive and board order are personal. The owner's are columns on `notes`; this
|
||||
table holds everyone else's, one row per (note, person it is shared with) who has
|
||||
pinned, archived or moved it. A recipient with no row sees the note unpinned,
|
||||
unarchived, in the owner's order.
|
||||
|
||||
`sync_revision` is stamped by the same `ts_set_sync_revision()` trigger the notes
|
||||
and labels use (0015), so a recipient's change reaches their own devices on the one
|
||||
|
||||
@@ -4,12 +4,8 @@ Revision ID: 0037
|
||||
Revises: 0036
|
||||
Create Date: 2026-10-07
|
||||
|
||||
Both came with the foundation (0001) for features that were never built (#5178).
|
||||
Nothing ever set `email_verified`, so it was false for every account on the server
|
||||
while the offline desktop reported it as true; nothing read or wrote `avatar_path`.
|
||||
A field whose value means nothing is worse than no field: a later feature would
|
||||
trust it. If address verification is ever built, it adds its own column, with a
|
||||
migration that says what the existing accounts are.
|
||||
Nothing set `email_verified` or read `avatar_path`, and a field whose value means
|
||||
nothing would be trusted by a later feature.
|
||||
|
||||
## Downgrade
|
||||
|
||||
|
||||
@@ -4,9 +4,7 @@ Revision ID: 0038
|
||||
Revises: 0037
|
||||
Create Date: 2026-10-08
|
||||
|
||||
Saved views were removed in #5180 (note 2897: recall first, organizing second). They
|
||||
lived only on the web; the desktop kept its own set that never synced, and Android
|
||||
had none. Tags in the drawer already give one-click recall.
|
||||
Saved views are removed; tags in the drawer give one-click recall.
|
||||
|
||||
## Downgrade
|
||||
|
||||
|
||||
@@ -4,9 +4,8 @@ Revision ID: 0039
|
||||
Revises: 0038
|
||||
Create Date: 2026-10-08
|
||||
|
||||
Credential events were only written to the app log, which is not queryable and is
|
||||
gone at the container's log rotation (#2939 §5). This keeps them in the database for
|
||||
`audit_retention_days`, for admins to read in Settings → Activity.
|
||||
Keeps credential events in the database for `audit_retention_days`, for admins to
|
||||
read in Settings → Activity; the app log isn't queryable and rotates away.
|
||||
|
||||
## Downgrade
|
||||
|
||||
|
||||
@@ -4,11 +4,9 @@ Revision ID: 0040
|
||||
Revises: 0039
|
||||
Create Date: 2026-10-08
|
||||
|
||||
Deleting a tag now leaves a tombstone row (`purged_at` set) so the change feed can
|
||||
tell linked devices about it (#5382). Before, the web deleted the row outright and
|
||||
devices never heard. A tombstone must not hold its name: creating `#grocery` again
|
||||
after deleting it has to make a live tag, on the web and from a device alike. So
|
||||
`(owner_id, name)` stays unique only where `purged_at IS NULL`.
|
||||
Deleting a tag leaves a tombstone row (`purged_at` set) so the change feed tells
|
||||
linked devices. A tombstone must not hold its name, so `(owner_id, name)` is unique
|
||||
only where `purged_at IS NULL`.
|
||||
|
||||
## Downgrade
|
||||
|
||||
|
||||
+1
-1
@@ -5,7 +5,7 @@ build-backend = "setuptools.build_meta"
|
||||
[project]
|
||||
name = "inkwell"
|
||||
version = "0.2.0"
|
||||
description = "Self-hosted personal thought-capture web app (FabledSword family)"
|
||||
description = "A self-hosted web app for capturing and finding thoughts"
|
||||
requires-python = ">=3.12"
|
||||
dependencies = [
|
||||
"quart>=0.19",
|
||||
|
||||
+3
-14
@@ -1,16 +1,5 @@
|
||||
"""Inkwell — self-hosted personal thought-capture web app (FabledSword family)."""
|
||||
"""Inkwell — a self-hosted web app for capturing and finding thoughts."""
|
||||
|
||||
# PACKAGING METADATA, and nothing else. Not the version any running server reports.
|
||||
#
|
||||
# A built image carries APP_VERSION in the environment, derived from the server's
|
||||
# own shipped file set (packaging/version.sh); `app.py` reads that and reports an
|
||||
# explicit "unknown" when it is absent, so this string never reaches a user and
|
||||
# bumping it changes nothing anybody sees.
|
||||
#
|
||||
# It exists because a Python package needs a version and "unknown" is not a legal
|
||||
# one here. It used to double as app.py's fallback, which meant a server run from a
|
||||
# checkout confidently reported `0.2.0` — a real-looking version naming no build
|
||||
# that exists. Note 3127 §5 is why that matters more than it reads: with version
|
||||
# tags gone, a build's self-report is the only answer to "which build is this?",
|
||||
# and there is nothing left to catch it lying.
|
||||
# Packaging metadata only. A running server reports APP_VERSION from the image, or
|
||||
# "unknown"; this string is never shown and bumping it changes nothing.
|
||||
__version__ = "0.2.0"
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
"""Admin routes over the instance's accounts: list them, make a password reset link
|
||||
for one (#5173), and read what has happened to them (the audit log, #2939 §5).
|
||||
for one, and read what has happened to them (the audit log).
|
||||
|
||||
What a reset link is, and how it is used, is in `password_resets.py`.
|
||||
"""
|
||||
|
||||
+5
-6
@@ -7,9 +7,9 @@ from sqlalchemy import ColumnElement, exists, or_, select
|
||||
from .models.group import GroupMember
|
||||
from .models.share import Share
|
||||
|
||||
# What a share grants (#5174). `edit` is the note's body and checklist; everything
|
||||
# else about a note stays its owner's. Ordered weakest first, so the strongest of a
|
||||
# person's grants is the one with the highest index.
|
||||
# What a share grants. `edit` covers the note's body and checklist; everything else
|
||||
# stays the owner's. Ordered weakest first, so a person's strongest grant has the
|
||||
# highest index.
|
||||
VIEW = "view"
|
||||
EDIT = "edit"
|
||||
PERMISSIONS = (VIEW, EDIT)
|
||||
@@ -26,9 +26,8 @@ def visible_to_user(
|
||||
"""Boolean SQL predicate: is this resource visible to ``user_id``?
|
||||
|
||||
A resource is visible if the user OWNS it, OR it is shared DIRECTLY with the
|
||||
user, OR it is shared with a GROUP the user belongs to. Scope every read (and
|
||||
mutation) of shareable user data through this — never assume a single operator
|
||||
(family rule 47).
|
||||
user, OR it is shared with a GROUP the user belongs to. Scope every read and
|
||||
mutation of shareable user data through this; the server is multi-user.
|
||||
|
||||
With ``permission``, only shares at that level count: ``permission=EDIT`` is
|
||||
"may this user change it", which the owner always may.
|
||||
|
||||
+32
-69
@@ -41,13 +41,9 @@ from .settings_api import bp as settings_bp
|
||||
from .shares_api import bp as shares_bp
|
||||
from .sync import bp as sync_bp, protocol_advertisement
|
||||
|
||||
# Without this, `logger.info` from this package goes nowhere: hypercorn configures its
|
||||
# own access/error loggers and leaves the root logger at WARNING, so the credential
|
||||
# events in auth.py would be invisible in `docker compose logs` — which is exactly
|
||||
# where they are read live. (Settings → Activity keeps most of them, `audit.py`.)
|
||||
#
|
||||
# `force=False` (the default) so a host that has already configured logging keeps its
|
||||
# own setup; LOG_LEVEL lets an operator turn it up without a code change.
|
||||
# Hypercorn leaves the root logger at WARNING, which would hide this package's INFO
|
||||
# credential events from `docker compose logs`. A host that already configured
|
||||
# logging keeps its own setup.
|
||||
logging.basicConfig(
|
||||
level=os.environ.get("INKWELL_LOG_LEVEL", "INFO").upper(),
|
||||
format="%(asctime)s %(levelname)s %(name)s %(message)s",
|
||||
@@ -65,10 +61,8 @@ class _AutoSecureSessionInterface(SecureCookieSessionInterface):
|
||||
"""Mark the session cookie `Secure` whenever the request arrived over HTTPS —
|
||||
directly, or via a TLS-terminating reverse proxy that sets X-Forwarded-Proto.
|
||||
|
||||
Auto-detecting per request (rather than a fixed SESSION_COOKIE_SECURE flag)
|
||||
hardens the cookie on HTTPS deployments without breaking a plain-HTTP install on
|
||||
a trusted LAN, where a hard-forced Secure flag would stop the browser from ever
|
||||
sending the cookie back — i.e. silently break login. No configuration required.
|
||||
Per request rather than a fixed flag, because a forced Secure cookie is never sent
|
||||
back over plain HTTP, which would break sign-in on a LAN install.
|
||||
"""
|
||||
|
||||
def get_cookie_secure(self, app: Quart) -> bool:
|
||||
@@ -83,19 +77,8 @@ def create_app() -> Quart:
|
||||
# Ephemeral/env secret so the app (and DB-free unit tests) construct without a
|
||||
# database. before_serving swaps in the real, DB-persisted key before serving.
|
||||
app.secret_key = Config.secret_key_env() or secrets.token_urlsafe(48)
|
||||
# The RUNNING build, or an explicit "unknown" — never the packaging fallback.
|
||||
#
|
||||
# This read `os.environ.get("APP_VERSION", __version__)`, so a server started
|
||||
# from a checkout reported `0.2.0`: a real-looking version that names no build
|
||||
# anybody could get. `__init__.py` already claimed the honest answer was
|
||||
# "APP_VERSION being missing, which app.py already handles" — it did not, and a
|
||||
# comment asserting a behaviour two files away from the code is how that stayed
|
||||
# true-sounding for months.
|
||||
#
|
||||
# It matters more than it used to. Note 3127 §5 removed version tags, so this
|
||||
# string is the only answer to "which build is this?" and nothing exists to
|
||||
# contradict it when it is wrong. `__version__` stays where it belongs, as
|
||||
# packaging metadata, which is the one place "unknown" is not a legal value.
|
||||
# The running build, or "unknown". Never `__version__`: that names no build, and
|
||||
# this string is the only answer to "which build is this?".
|
||||
app.config["APP_VERSION"] = os.environ.get("APP_VERSION") or "unknown"
|
||||
app.config["SESSION_COOKIE_HTTPONLY"] = True
|
||||
app.config["SESSION_COOKIE_SAMESITE"] = "Lax"
|
||||
@@ -130,14 +113,11 @@ def create_app() -> Quart:
|
||||
apply_session_ttl(app, await get_setting(db, "session_ttl_days"))
|
||||
except (ValueError, TypeError, KeyError):
|
||||
pass
|
||||
# The security settings the throttle and the proxy trust read on hot
|
||||
# paths. Cached rather than queried per request; until this runs they
|
||||
# hold their registry defaults, which is the correct behaviour for a
|
||||
# server that has not finished starting.
|
||||
# Security settings read on hot paths, cached rather than queried per
|
||||
# request. Until this runs they hold their registry defaults.
|
||||
await refresh_live(db)
|
||||
# Credentials stored before sealing existed are sealed now (#5927). A data
|
||||
# folder that can't be written leaves them as they are; it never stops the
|
||||
# server starting.
|
||||
# Seal any credentials still stored in the clear. A data folder that can't
|
||||
# be written leaves them as they are; it never stops the server starting.
|
||||
try:
|
||||
if count := await seal_stored_secrets(db):
|
||||
await db.commit()
|
||||
@@ -145,11 +125,9 @@ def create_app() -> Quart:
|
||||
except SealError as exc:
|
||||
await db.rollback()
|
||||
logger.warning("stored credentials left unsealed: %s", exc)
|
||||
# Expire old trash in the background (retention.py). One task per process is
|
||||
# correct because the image serves with a single hypercorn worker (Dockerfile);
|
||||
# if that ever gains `--workers`, this needs a lock so N workers don't each
|
||||
# sweep. Duplicate sweeps would be harmless but wasteful — a purged row is
|
||||
# skipped by `purged_at IS NULL` — so this is about load, not correctness.
|
||||
# Expire old trash in the background (retention.py). One task per process,
|
||||
# because the image runs one hypercorn worker; with several, each would sweep,
|
||||
# which is wasteful but harmless.
|
||||
app.config["TRASH_SWEEPER"] = asyncio.create_task(run_sweeper())
|
||||
# The daily backup (backup_job.py), on the same single-worker footing. It also
|
||||
# takes a Postgres advisory lock, so a second process could not double it.
|
||||
@@ -172,23 +150,16 @@ def create_app() -> Quart:
|
||||
None of these change how the app behaves for a legitimate caller; they narrow
|
||||
what a browser will do if something else goes wrong.
|
||||
|
||||
The CSP is the substantive one. `script-src 'self'` means that even if some
|
||||
future path did manage to reflect user text into the page, the browser would
|
||||
refuse to run it — the app has no inline scripts and no third-party scripts,
|
||||
so nothing legitimate is given up. The exceptions are honest ones:
|
||||
- `style-src 'unsafe-inline'` — Vue writes inline styles itself (`v-show`
|
||||
toggling display, TransitionGroup's FLIP setting transforms). Inline
|
||||
STYLE is not an execution primitive the way inline script is.
|
||||
- `img-src https: http:` — link previews render the remote og:image of
|
||||
whatever was linked, which is an arbitrary host by definition. Both
|
||||
schemes, because a LAN install is served over http and would otherwise
|
||||
lose every preview image; on an https instance the browser blocks the
|
||||
http ones as mixed content anyway, so naming it concedes nothing.
|
||||
- `blob:`/`data:` — attachment previews and the desktop's blob URI scheme.
|
||||
The CSP allows only the app's own scripts; it has no inline or third-party
|
||||
ones. The exceptions:
|
||||
- `style-src 'unsafe-inline'`: Vue writes inline styles (v-show, FLIP
|
||||
transitions). Inline style is not an execution primitive.
|
||||
- `img-src https: http:`: link previews show the linked page's image, from
|
||||
any host. http too, for LAN installs; an https page blocks those anyway.
|
||||
- `blob:`/`data:`: attachment previews and the desktop's blob URIs.
|
||||
|
||||
`frame-ancestors 'none'` replaces the older X-Frame-Options and is what stops
|
||||
the app being framed for clickjacking; `form-action 'self'` stops a form from
|
||||
being pointed at another origin.
|
||||
`frame-ancestors 'none'` prevents clickjacking; `form-action 'self'` keeps
|
||||
forms from posting to another origin.
|
||||
"""
|
||||
response.headers.setdefault(
|
||||
"Content-Security-Policy",
|
||||
@@ -204,9 +175,7 @@ def create_app() -> Quart:
|
||||
"media-src 'self' blob:; "
|
||||
"connect-src 'self'",
|
||||
)
|
||||
# Content-type sniffing turns a file we said was text into whatever the bytes
|
||||
# look like. The attachment route already sets this; every other response
|
||||
# deserves it too.
|
||||
# Stop the browser sniffing a response into a different content type.
|
||||
response.headers.setdefault("X-Content-Type-Options", "nosniff")
|
||||
# Note titles and label names end up in the URL of a search or a label lens,
|
||||
# and a full Referer would hand them to any site a link preview points at.
|
||||
@@ -216,14 +185,9 @@ def create_app() -> Quart:
|
||||
response.headers.setdefault(
|
||||
"Permissions-Policy", "camera=(), microphone=(), geolocation=(), interest-cohort=()"
|
||||
)
|
||||
# HSTS only where the request already arrived over TLS — the same detection
|
||||
# the session cookie uses. Sending it on a plain-HTTP LAN install would tell
|
||||
# the browser to refuse the only scheme that install serves.
|
||||
#
|
||||
# Scoped to this host: no `includeSubDomains` and no `preload`, both of which
|
||||
# commit domains this app does not own. A browser still remembers the policy
|
||||
# for up to a year after the header stops being sent, which is the point of
|
||||
# it — worth knowing before putting a hostname behind TLS temporarily.
|
||||
# HSTS only on requests that arrived over TLS: on a plain-HTTP install it would
|
||||
# lock the browser out. This host only, no includeSubDomains or preload. The
|
||||
# browser keeps the policy for a year, even after the header stops.
|
||||
if is_https():
|
||||
response.headers.setdefault("Strict-Transport-Security", "max-age=31536000")
|
||||
return response
|
||||
@@ -238,13 +202,12 @@ def create_app() -> Quart:
|
||||
async with session_scope() as db:
|
||||
data = await get_public_config(db)
|
||||
data["version"] = app.config["APP_VERSION"]
|
||||
# The sync-protocol handshake (M10.6). A native client reads this BEFORE
|
||||
# linking — while it still has no token and possibly no account — to decide
|
||||
# whether it can talk to this server, and which optional features to offer.
|
||||
# The sync-protocol handshake. A native client reads this before linking, with
|
||||
# no token yet, to decide whether it can talk to this server and which
|
||||
# optional features to offer.
|
||||
data.update(protocol_advertisement())
|
||||
# Which CLIENTS this server can hand out, if any, under `clients`. Absent
|
||||
# rather than null when it has none, so the web UI hides a download instead
|
||||
# of offering a button that 404s.
|
||||
# The clients this server can hand out. Absent when it has none, so the web UI
|
||||
# hides the download.
|
||||
data.update(client_advertisement())
|
||||
return jsonify(data)
|
||||
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
"""The audit log: what happened to accounts, kept where an admin can read it (#2939 §5).
|
||||
"""The audit log: what happened to accounts, kept where an admin can read it.
|
||||
|
||||
Sign-ins, failed sign-ins, password changes and resets, devices linked and
|
||||
unlinked, invites made and revoked. Each is still written to the app log as it
|
||||
|
||||
+40
-73
@@ -32,30 +32,20 @@ from .storage import storage_limit, storage_used
|
||||
|
||||
bp = Blueprint("auth", __name__, url_prefix="/api/auth")
|
||||
|
||||
# Every credential event goes to the app log, which is the live view of whether
|
||||
# anyone is knocking: `docker compose logs`. Most also go to the audit log
|
||||
# (`audit.py`), which keeps them for admins to read in Settings → Activity.
|
||||
#
|
||||
# The attempted email is included deliberately. It is the operator's own server, and
|
||||
# "somebody failed a login" without saying against WHICH account tells you nothing you
|
||||
# can act on. Passwords, obviously, never appear.
|
||||
# Every credential event goes to the app log (`docker compose logs`); most also go to
|
||||
# the audit log (Settings → Activity). The attempted email is logged so a failure can
|
||||
# be acted on; passwords never are.
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
SESSION_KEY = "user_id"
|
||||
# The account's `session_epoch` when this session signed in. A password reset moves
|
||||
# the account's epoch on, which is how it ends sessions it can't reach: they are
|
||||
# signed cookies in other people's browsers. A cookie from before epochs existed has
|
||||
# no key and reads as 0, the epoch every account started at.
|
||||
# the epoch on, which ends sessions held as signed cookies in other browsers. A
|
||||
# cookie without the key reads as 0, every account's starting epoch.
|
||||
EPOCH_KEY = "epoch"
|
||||
|
||||
# The first account can only be created this long after the server starts (family
|
||||
# idea #5105, practice 8). Without it, a fresh server on a public address belongs to
|
||||
# whoever reaches /register first, and a new DNS name is found within minutes.
|
||||
#
|
||||
# A constant, not a Setting: there is no admin yet to change one. Anyone who misses
|
||||
# the window restarts the server, which opens it again; once an account exists the
|
||||
# window no longer matters. Measured from `create_app` (STARTED_AT), so each start
|
||||
# counts afresh.
|
||||
# How long after start the first account may be created. Otherwise a fresh server on
|
||||
# a public address belongs to whoever reaches /register first. A constant, since
|
||||
# there is no admin yet to change a setting; restarting the server reopens it.
|
||||
SETUP_WINDOW_S = 30 * 60
|
||||
SETUP_CLOSED = (
|
||||
"this server's setup window has closed. Restart the server, then create the "
|
||||
@@ -136,9 +126,8 @@ def login_required(fn):
|
||||
"""Guard: 401 unless authenticated. Accepts a web session cookie OR a device
|
||||
bearer token (native clients). Sets g.user_id for the view.
|
||||
|
||||
The session path reads one column by primary key. It used to read nothing, which
|
||||
left no way to end a session short of its expiry: not for a reset password, and
|
||||
not for a deleted account (#5173)."""
|
||||
The session path reads the account's epoch, so a password reset or a deleted
|
||||
account ends the session before its expiry."""
|
||||
|
||||
@functools.wraps(fn)
|
||||
async def wrapper(*args, **kwargs):
|
||||
@@ -182,10 +171,8 @@ def require_admin(fn):
|
||||
def _throttled(retry_after: int):
|
||||
"""The 429 every throttled credential route returns.
|
||||
|
||||
Deliberately says nothing about WHICH limit was hit or how many attempts are
|
||||
left — that would tell someone probing whether the email they guessed exists.
|
||||
`Retry-After` is standard and is the one thing a legitimate client (or person)
|
||||
genuinely needs.
|
||||
Says nothing about which limit was hit or what is left, which would reveal
|
||||
whether a guessed email exists. Only `Retry-After`.
|
||||
"""
|
||||
logger.warning("throttled credential attempt from=%s retry_after=%ss", client_address(), retry_after)
|
||||
return too_many("too many attempts — try again shortly", retry_after)
|
||||
@@ -194,10 +181,8 @@ def _throttled(retry_after: int):
|
||||
def _sign_in_block(email: str) -> int | None:
|
||||
"""Seconds to wait before this sign-in may be attempted, or None to proceed.
|
||||
|
||||
Checked BEFORE the password is verified, so a throttled attempt costs no bcrypt
|
||||
— which is the other half of what this protects: hashing is deliberately slow,
|
||||
and an unauthenticated caller who can trigger it without limit has a CPU
|
||||
exhaustion primitive, not just a guessing one.
|
||||
Checked before the password is hashed, so a throttled attempt costs no bcrypt:
|
||||
unlimited hashing would let an anonymous caller exhaust the CPU.
|
||||
"""
|
||||
address = client_address()
|
||||
waits = [
|
||||
@@ -237,9 +222,8 @@ async def register():
|
||||
if not display_name:
|
||||
display_name = email.split("@", 1)[0]
|
||||
|
||||
# Counted by attempt rather than by failure: a rejected registration still cost a
|
||||
# round trip and a uniqueness check, and on an instance with signups open the
|
||||
# thing worth bounding is how fast accounts can appear at all.
|
||||
# Counted per attempt, not per failure: with signups open, what is bounded is how
|
||||
# fast accounts can appear.
|
||||
wait = register_by_address.retry_after(client_address())
|
||||
if wait is not None:
|
||||
return _throttled(wait)
|
||||
@@ -252,11 +236,10 @@ async def register():
|
||||
logger.warning("registration refused (setup window closed) email=%s from=%s", email, client_address())
|
||||
await audit.record(audit.REGISTRATION_REFUSED, email=email, detail="setup window closed")
|
||||
return json_error(SETUP_CLOSED, 403)
|
||||
# The first account bootstraps the admin and is allowed, inside the setup
|
||||
# window above, even when registration is otherwise closed. After that, an invite lets one person in
|
||||
# while it is closed (#5172). One that was offered is redeemed even while
|
||||
# registration is open, so the list still says who used it, and one that
|
||||
# doesn't hold is refused rather than ignored.
|
||||
# The first account becomes the admin and is allowed inside the setup window
|
||||
# even with registration closed. After that, an invite lets one person in while
|
||||
# it is closed. An offered invite is redeemed even when registration is open, so
|
||||
# the list says who used it, and an invalid one is refused, not ignored.
|
||||
invite_id = None
|
||||
if not is_first and invite:
|
||||
invite_id = await redeem(db, invite, email)
|
||||
@@ -281,17 +264,9 @@ async def register():
|
||||
)
|
||||
db.add(user)
|
||||
if is_first:
|
||||
# Registration CLOSES the moment the instance has an owner.
|
||||
#
|
||||
# Not "defaults closed" — that would still need the first person to get in
|
||||
# somehow. Closed as a CONSEQUENCE of the admin account existing, which is
|
||||
# the only formulation with no open window in it. Leaving the setting on
|
||||
# meant the gap between "my account exists" and "I remembered to turn it
|
||||
# off in Settings" was wide open, and on a public host that gap is the
|
||||
# entire exposure — it starts the moment DNS resolves.
|
||||
#
|
||||
# An admin who wants a second person sends them an invite (Settings →
|
||||
# Invites), which works with registration closed.
|
||||
# Registration closes as soon as the admin account exists, so there is no
|
||||
# window between "my account exists" and remembering to close it. More
|
||||
# people join by invite (Settings → Invites).
|
||||
await set_settings(db, {"allow_registration": False})
|
||||
if invite_id is not None:
|
||||
await db.flush()
|
||||
@@ -312,9 +287,8 @@ async def _check_credentials(db, email: str, password: str, route: str) -> User
|
||||
"""The account these credentials open, or None. Counts the failure against the
|
||||
throttle, logs it under `route`, and clears the account's history on success.
|
||||
|
||||
A missing account still pays for a hash. Without that, "no such account" returns
|
||||
in microseconds while a wrong password takes bcrypt's deliberate ~100ms, and the
|
||||
difference is a reliable oracle for which emails have accounts here.
|
||||
A missing account still pays for a hash, so response time doesn't reveal which
|
||||
emails have accounts.
|
||||
"""
|
||||
user = await db.scalar(select(User).where(User.email == email))
|
||||
if user is None or not user.password_hash:
|
||||
@@ -378,7 +352,7 @@ async def me():
|
||||
@login_required
|
||||
async def storage():
|
||||
"""How much attachment storage this account uses, and its limit (None when it
|
||||
has none). For the Account page (#2939 §4)."""
|
||||
has none). For the Account page."""
|
||||
async with session_scope() as db:
|
||||
return jsonify(
|
||||
{
|
||||
@@ -394,7 +368,7 @@ FORGOT_SENT = "If an account uses that address, a reset link is on its way. It w
|
||||
|
||||
@bp.post("/forgot-password")
|
||||
async def forgot_password():
|
||||
"""Email a reset link to whoever owns this address (#5266).
|
||||
"""Email a reset link to whoever owns this address.
|
||||
|
||||
Says the same thing, at the same speed, whether or not the address has an
|
||||
account: the link is made and sent off the request. Throttled per visitor address
|
||||
@@ -428,7 +402,7 @@ async def forgot_password():
|
||||
|
||||
@bp.post("/reset-password")
|
||||
async def reset_password():
|
||||
"""Set a new password with a reset link an admin made (#5173), and sign the
|
||||
"""Set a new password with a reset link, and sign the
|
||||
account out everywhere else: every other session, and every linked device.
|
||||
|
||||
The browser that used the link is signed in under the new password, since holding
|
||||
@@ -497,8 +471,8 @@ def _stay_signed_in(user: User) -> None:
|
||||
@login_required
|
||||
async def change_password():
|
||||
"""Change the account's password, and sign it out everywhere else: every other
|
||||
web session, and every linked device (family idea #5105, practice 4). The
|
||||
browser that changed it stays signed in.
|
||||
web session, and every linked device. The browser that changed it stays signed
|
||||
in.
|
||||
|
||||
The current password is required, so a session left open on someone else's
|
||||
screen can't take the account over. A wrong one counts against the sign-in
|
||||
@@ -536,8 +510,8 @@ async def change_password():
|
||||
@login_required
|
||||
async def sign_out_elsewhere():
|
||||
"""Sign the account out of every other browser and unlink every device, keeping
|
||||
this one signed in (family idea #5105, practice 4). For a session or a device
|
||||
the person doesn't recognise, or one on a machine they no longer have."""
|
||||
this one signed in. For a session or device the person doesn't recognise, or a
|
||||
machine they no longer have."""
|
||||
async with session_scope() as db:
|
||||
user = await db.get(User, g.user_id)
|
||||
if user is None:
|
||||
@@ -551,7 +525,7 @@ async def sign_out_elsewhere():
|
||||
return jsonify({"ok": True, "devices_unlinked": unlinked})
|
||||
|
||||
|
||||
# --- Device (bearer) tokens for native clients — M8 sync hub ---
|
||||
# --- Device (bearer) tokens for native clients ---
|
||||
|
||||
|
||||
async def _email_of(db) -> str | None:
|
||||
@@ -592,9 +566,8 @@ async def device_login():
|
||||
if not email or not password:
|
||||
return json_error("email and password are required", 400)
|
||||
|
||||
# Same budget as the web sign-in, and the SAME counters — this route hands out a
|
||||
# long-lived bearer token, so leaving it unthrottled would just move the guessing
|
||||
# here from /login.
|
||||
# The same counters as the web sign-in: this route hands out a long-lived token,
|
||||
# so an unthrottled one would just move the guessing here.
|
||||
wait = _sign_in_block(email)
|
||||
if wait is not None:
|
||||
return _throttled(wait)
|
||||
@@ -643,21 +616,15 @@ async def list_devices():
|
||||
async def revoke_own_device():
|
||||
"""Revoke the device token presented on THIS request.
|
||||
|
||||
What makes "unlink" on a native client actually stop its access. The client
|
||||
can't use the id-keyed route below, because it doesn't reliably know its own
|
||||
device id: a token pasted from the web app arrives without one, and `/me`
|
||||
describes the user, not the device row. Identifying the row by the presented
|
||||
token needs nothing the caller doesn't already hold, so it works for both ways
|
||||
a client can be linked.
|
||||
How "unlink" on a native client ends its access. A client doesn't reliably know
|
||||
its own device id (a token pasted from the web app arrives without one), so the
|
||||
row is found by the token it presents.
|
||||
|
||||
Declared above the `<device_id>` rule for reading order only — Werkzeug ranks a
|
||||
static rule ahead of a converter regardless of registration order.
|
||||
Werkzeug ranks this static rule ahead of `<device_id>` whatever the order.
|
||||
"""
|
||||
token = _bearer_token()
|
||||
if token is None:
|
||||
# A session-cookie caller holds no device token, so "revoke the one I'm
|
||||
# using" is meaningless rather than merely unauthorized. The web app
|
||||
# revokes by id.
|
||||
# A session caller holds no device token to revoke; the web app revokes by id.
|
||||
return json_error("no device token was presented", 400)
|
||||
# Owner-scoped like every other device route. The hash already pins a single
|
||||
# row; the guarantee shouldn't rest on one column.
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
"""Admin routes for Settings → Backups (#5920): where backups stand, Back up now, and
|
||||
"""Admin routes for Settings → Backups: where backups stand, Back up now, and
|
||||
Test destination. What a backup holds and when it runs is in `backup_job`."""
|
||||
from __future__ import annotations
|
||||
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
"""What goes into a backup, and putting it back (#5918, #5921).
|
||||
"""What goes into a backup, and putting it back.
|
||||
|
||||
Every table the models define is exported, so a table added later is backed up
|
||||
without anyone remembering to add it here. The few that are left out are named in
|
||||
@@ -74,9 +74,8 @@ def _decoder(column):
|
||||
async def export_contents(db) -> Contents:
|
||||
"""Every exported table's rows in primary-key order, and the attachment files.
|
||||
|
||||
Primary-key order is load-bearing: Postgres returns rows in heap order, and an
|
||||
UPDATE moves a row to the end of the heap, so without ORDER BY an unchanged table
|
||||
would come out shuffled after any edit and nothing would deduplicate."""
|
||||
Primary-key order keeps an unchanged table byte-identical between runs: heap order
|
||||
shifts whenever a row is updated, and then nothing deduplicates."""
|
||||
tables: dict[str, list[dict]] = {}
|
||||
for table in exported_tables():
|
||||
query = select(table).order_by(*table.primary_key.columns)
|
||||
@@ -143,7 +142,7 @@ async def restore_tables(db, members: Iterable[Member]) -> dict[str, int]:
|
||||
for member in members:
|
||||
if member.kind == "table":
|
||||
if member.name not in by_name:
|
||||
# A table this version no longer has. Nothing to put it in.
|
||||
# A table this version doesn't have. Nothing to put it in.
|
||||
logger.warning("restore: skipping table %s, which this version doesn't have", member.name)
|
||||
member.fileobj.read()
|
||||
continue
|
||||
|
||||
@@ -1,10 +1,9 @@
|
||||
"""Where the daily backup goes, and getting a backup file there (#5919).
|
||||
"""Where the daily backup goes, and getting a backup file there.
|
||||
|
||||
Two kinds of destination, both set in Settings → Backups:
|
||||
|
||||
- **folder**: a directory inside the container, usually a mounted share. A backup
|
||||
service that watches the share (Fabled Archives' folder mode, say) or a person
|
||||
takes the files from there.
|
||||
- **folder**: a directory inside the container, usually a mounted share, from which
|
||||
a backup service or a person takes the files.
|
||||
- **s3**: any S3-compatible bucket, addressed by path. Inkwell only puts and lists,
|
||||
so a key that may only add objects is enough.
|
||||
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
"""The full backup as one file: an uncompressed tar per run (#5918).
|
||||
"""The full backup as one file: an uncompressed tar per run.
|
||||
|
||||
A scheduled backup lands somewhere that keeps a history of them: a folder a backup
|
||||
service watches, or an S3 bucket. From there it is usually copied off-site by a
|
||||
@@ -25,9 +25,6 @@ that whatever did not change between two runs is the same bytes both times.
|
||||
can thin the history on its own schedule.
|
||||
- **Written atomically** through a dot-named `.part` file, which a folder watcher
|
||||
ignores until the rename.
|
||||
|
||||
The shape is Scribe's (snippet #5875), with attachment files added: in a notes app
|
||||
the files are as much the user's data as the rows.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
|
||||
@@ -1,12 +1,12 @@
|
||||
"""The daily backup: when it runs, running it, and what happened (#5920).
|
||||
"""The daily backup: when it runs, running it, and what happened.
|
||||
|
||||
- **When.** A tick every few minutes asks whether a backup is due: switched on, and
|
||||
no good backup since the most recent scheduled hour. A server that was down at
|
||||
that hour backs up as soon as it is back. Switching backups on runs the first one
|
||||
within a tick, which is also the quickest proof the destination works.
|
||||
- **Failure is "not yet".** A failed run is tried again after an hour, then two,
|
||||
then four, then waits for the next day's hour, rather than at once or never
|
||||
(rule 193). The destination may simply be restarting.
|
||||
then four, then waits for the next day's hour, rather than at once or never. The
|
||||
destination may simply be restarting.
|
||||
- **One at a time.** An in-process lock, plus a Postgres advisory lock in case a
|
||||
second server process is ever started against the same database. Back up now
|
||||
while one is running is answered 409.
|
||||
|
||||
@@ -1,14 +1,12 @@
|
||||
"""A small S3 client for sending backups: AWS Signature Version 4, path-style
|
||||
addressing (endpoint/bucket/key), and the two calls a backup needs.
|
||||
|
||||
Ported from FabledSteward's and FabledScribe's backup clients (Scribe snippet #5679),
|
||||
which are tested against the same AWS examples as tests/test_backup_s3.py. Those use
|
||||
httpx. This one uses the standard library's `http.client`, on a worker thread,
|
||||
because Inkwell has no HTTP client dependency and two calls don't justify adding one.
|
||||
Uses the standard library's `http.client` on a worker thread, since two calls don't
|
||||
justify an HTTP client dependency. The signing is checked against AWS's published
|
||||
examples (tests/test_backup_s3.py).
|
||||
|
||||
Inkwell puts and lists; it never deletes. The bucket keeps the history and thins it
|
||||
by its own policy, so a key that may only add objects is enough. It is also the safer
|
||||
key to give a service: a stolen one cannot wipe the history it writes into.
|
||||
It puts and lists, never deletes, so an add-only key is enough; a stolen one can't
|
||||
wipe the history the bucket keeps.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
@@ -128,15 +126,15 @@ def _says(status: int, reason: str, body: bytes) -> str:
|
||||
}
|
||||
said = hints.get(code) or message or reason
|
||||
if status == 404 and not code:
|
||||
# Seen live: an address without its port lands on port 80 and gets a plain
|
||||
# 404 from whatever answers there (Scribe #5871).
|
||||
# An address without its port lands on port 80 and gets a plain 404 from
|
||||
# whatever answers there.
|
||||
said = "nothing answered as S3 at that address; check the port"
|
||||
return f"HTTP {status}{f' {code}' if code else ''}: {said}."
|
||||
|
||||
|
||||
class _Throttled:
|
||||
"""A file wrapper that reads no faster than `limit` bytes a second, so an upload
|
||||
leaves room on the connection for everything else (rule 193)."""
|
||||
leaves room on the connection for everything else."""
|
||||
|
||||
def __init__(self, f: IO[bytes], limit: int):
|
||||
self._f = f
|
||||
|
||||
+63
-162
@@ -1,73 +1,33 @@
|
||||
"""The server hands out the clients it is in step with.
|
||||
|
||||
## Why the server, and not a release page
|
||||
|
||||
The Fabled-Git instance is private (issue 2091), so `install.sh` already cannot
|
||||
fetch for anyone but the operator — and a self-hoster should not need an account
|
||||
on someone else's forge to get the app for their own notes. The server they
|
||||
already trust with the notes is the obvious place to get the client from.
|
||||
|
||||
It also keeps the two in step by construction. Client and server already
|
||||
negotiate a sync protocol version before linking, so a server that also serves
|
||||
the client cannot hand out a phone it is unable to talk to.
|
||||
|
||||
That argument was never Android-specific, which is why this module now serves a
|
||||
TABLE of platforms rather than the one it was written for.
|
||||
A self-hoster gets the apps from the server that already holds their notes, with no
|
||||
account anywhere else. Client and server negotiate a sync protocol before linking,
|
||||
so a server never hands out a client it can't talk to.
|
||||
|
||||
## Where the files come from
|
||||
|
||||
Two places, checked in that order, per platform:
|
||||
Per platform, the first of:
|
||||
|
||||
1. `DATA_DIR/client/` — the mounted volume that already holds attachments. An
|
||||
operator who wants a SPECIFIC build drops it there and it wins.
|
||||
2. the copy baked into the image at build time — CI fetches the newest published
|
||||
build of each client into every image, so `:dev` and `:latest` both carry a
|
||||
full set and `docker compose pull` delivers new ones with nothing copied by
|
||||
hand.
|
||||
1. `DATA_DIR/client/`, on the mounted volume: a build dropped there wins.
|
||||
2. The copy baked into the image, inside the package rather than under DATA_DIR,
|
||||
which a volume mount would hide. CI bakes the newest build of each client into
|
||||
every image.
|
||||
|
||||
The precedence is the point: the image is the default, and a person who wants to
|
||||
override it should not have to fight it. The baked copy sits inside the package
|
||||
rather than under DATA_DIR because DATA_DIR is a volume mount, and anything the
|
||||
image wrote there would be hidden the moment one is attached.
|
||||
|
||||
**Precedence is decided per platform, not for the set.** A drop-in `.deb` does
|
||||
not shadow the baked APK. The alternative — first directory holding anything
|
||||
wins — would mean replacing one client silently retracts the other four.
|
||||
Precedence is per platform: a drop-in `.deb` doesn't shadow the baked APK.
|
||||
|
||||
## What a platform needs on disk
|
||||
|
||||
Two files, and both must be present:
|
||||
|
||||
- the artifact, at a FIXED name (`inkwell.deb`, not
|
||||
`Inkwell_2026.08.30.0307_amd64.deb`)
|
||||
- its sidecar, `{version_name, version_code, size, sha256}`
|
||||
|
||||
**Fixed names, and the version only in the sidecar.** A version-stamped filename
|
||||
would force this module to glob, and a glob over a directory an operator can drop
|
||||
files into is how you serve the older of two builds — `write-manifest.sh` carries
|
||||
a comment about exactly that, from the time it advertised a new version while
|
||||
pointing at an old binary.
|
||||
|
||||
The sidecar exists because a version is not reliably readable from the artifact:
|
||||
an APK keeps it in a binary AXML manifest Python cannot parse without the Android
|
||||
build tools, and a `.deb` or an AppImage would each need a different unpacker. CI
|
||||
writes the sidecar where the real value is already known.
|
||||
|
||||
The AppImage needs a THIRD file, `inkwell.AppImage.sig`. That is the minisign
|
||||
signature the desktop updater verifies before replacing the running binary, and a
|
||||
bundle that cannot be verified cannot be offered as an update — so a missing
|
||||
signature makes the AppImage absent rather than merely unsigned.
|
||||
- the artifact, at a fixed name (`inkwell.deb`). A version in the filename would
|
||||
mean globbing, and a glob can pick the older of two builds.
|
||||
- its sidecar, `{version_name, version_code, size, sha256}`, written by CI, because
|
||||
a version can't be read reliably from an APK, a `.deb` or an AppImage.
|
||||
- for the AppImage, `inkwell.AppImage.sig`: the signature the desktop updater
|
||||
verifies. Without it the AppImage counts as absent.
|
||||
|
||||
## Absence is normal
|
||||
|
||||
A server with no client for a platform advertises none, and the web UI hides that
|
||||
download rather than offering a button that 404s. Same for a mismatched pair: if
|
||||
the sidecar's recorded size does not match the file on disk, the two did not
|
||||
arrive together, and the server says it has nothing rather than serving one build
|
||||
while describing another.
|
||||
|
||||
This is the ONLY state available before a platform's first build has ever
|
||||
published, so it is an ordinary answer and never an error.
|
||||
A platform with no build, or whose sidecar size doesn't match the file, is simply
|
||||
not advertised, and the web UI hides that download.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
@@ -90,29 +50,17 @@ class Platform:
|
||||
"""One installable client, and where its files sit."""
|
||||
|
||||
id: str
|
||||
# What a person calls it. Named for the DISTRO rather than the package format
|
||||
# ("Debian / Ubuntu", not ".deb") — someone knows which system they run and
|
||||
# does not necessarily know which packaging it uses.
|
||||
# What a person calls it: the system ("Debian / Ubuntu"), not the package format.
|
||||
label: str
|
||||
artifact: str
|
||||
sidecar: str
|
||||
mimetype: str
|
||||
# An updater-verifiable bundle: `<artifact>.sig` must be present too, and its
|
||||
# contents are published with the metadata. Only the AppImage, because it is
|
||||
# the only bundle that can replace itself in place — a package-manager install
|
||||
# cannot, by design (see the desktop's update.rs).
|
||||
# An updater-verifiable bundle: `<artifact>.sig` must be present, and is published
|
||||
# with the metadata. Only the AppImage can replace itself in place.
|
||||
signed: bool = False
|
||||
# Whether this platform's ordering key is an INTEGER.
|
||||
#
|
||||
# `version_code` is "whatever this platform's comparator reads", and that is not
|
||||
# one type. Android's is an int because Android's own install gate compares one,
|
||||
# and it must stay a JSON number — `ClientRelease` in core/src/sync/client.rs
|
||||
# declares it `i64` and a string would fail to deserialize on every phone in the
|
||||
# field. The desktop's is Tauri's semver key, `1.0.<minutes>`, which is the value
|
||||
# its updater compares and is not an integer at all.
|
||||
#
|
||||
# Coercing everything to int was inherited from the days when Android was the
|
||||
# only platform, and would have rejected every desktop sidecar written.
|
||||
# Whether this platform's ordering key (`version_code`) is an integer. Android's
|
||||
# is, and must stay a JSON number: core/src/sync/client.rs reads it as `i64`. The
|
||||
# desktop's is Tauri's semver key, `1.0.<minutes>`.
|
||||
code_is_int: bool = True
|
||||
|
||||
@property
|
||||
@@ -124,17 +72,14 @@ class Platform:
|
||||
return f"/api/client/{self.id}/download"
|
||||
|
||||
|
||||
# ONE definition of what this server can hand out. Every route, the /api/config
|
||||
# advertisement and the tests all read this table; adding a platform is adding a
|
||||
# row.
|
||||
# What this server can hand out. Every route, /api/config and the tests read this
|
||||
# table; adding a platform is adding a row.
|
||||
PLATFORMS: tuple[Platform, ...] = (
|
||||
Platform(
|
||||
id="android",
|
||||
label="Android",
|
||||
# UNCHANGED, and it must stay unchanged: the Android lane publishes these
|
||||
# exact names, CI bakes them in under them, and clients in the field poll
|
||||
# `/api/client/android`. Renaming them to match the pattern below would
|
||||
# buy tidiness and strand every installed phone.
|
||||
# Fixed: installed phones poll `/api/client/android`, and the Android lane
|
||||
# publishes these exact names.
|
||||
artifact="inkwell.apk",
|
||||
sidecar="inkwell-android.json",
|
||||
mimetype="application/vnd.android.package-archive",
|
||||
@@ -160,9 +105,7 @@ PLATFORMS: tuple[Platform, ...] = (
|
||||
label="Other Linux (AppImage)",
|
||||
artifact="inkwell.AppImage",
|
||||
sidecar="inkwell-linux-appimage.json",
|
||||
# No registered type for an AppImage, and guessing one buys nothing: it is
|
||||
# served as an attachment either way, and octet-stream is the answer that
|
||||
# cannot be wrong.
|
||||
# AppImage has no registered type; it is served as an attachment either way.
|
||||
mimetype="application/octet-stream",
|
||||
signed=True,
|
||||
code_is_int=False,
|
||||
@@ -189,10 +132,8 @@ bp = Blueprint("client_dist", __name__)
|
||||
def _read(root: Path, platform: Platform) -> dict | None:
|
||||
"""One platform's build in one directory, or None.
|
||||
|
||||
Never raises. A missing directory, an unreadable sidecar, malformed JSON, a
|
||||
sidecar that describes a different file and — for a signed bundle — a missing
|
||||
signature are all the same answer to the only question being asked, "is there
|
||||
a client here I can honestly offer?", and that answer is no.
|
||||
Never raises. A missing directory, an unreadable or malformed sidecar, a sidecar
|
||||
describing a different file, or a missing signature all mean "no client here".
|
||||
"""
|
||||
try:
|
||||
size = (root / platform.artifact).stat().st_size
|
||||
@@ -211,9 +152,8 @@ def _read(root: Path, platform: Platform) -> dict | None:
|
||||
# not a version, and it would render as a blank on the download card.
|
||||
return None
|
||||
|
||||
# The pair has to describe one build. A sidecar left behind by a previous
|
||||
# release would otherwise advertise a version this server cannot serve, and the
|
||||
# client would download something other than what it was promised.
|
||||
# The pair must describe one build, or a stale sidecar would advertise a version
|
||||
# the file isn't.
|
||||
if recorded != size:
|
||||
return None
|
||||
|
||||
@@ -225,22 +165,17 @@ def _read(root: Path, platform: Platform) -> dict | None:
|
||||
# sorts like one, which is not how "is this newer" works.
|
||||
"version_code": code,
|
||||
"size": size,
|
||||
# Computed by CI over the same bytes it uploaded, so a client can tell a
|
||||
# truncated download from a complete one BEFORE handing it to an installer.
|
||||
# Not a trust anchor — the signature is that.
|
||||
# Lets a client spot a truncated download before installing it. Not a trust
|
||||
# anchor; the signature is.
|
||||
"sha256": digest,
|
||||
# A PATH, never an absolute URL: the client joins it to the base it is
|
||||
# already linked to, so a compromised or misconfigured server cannot
|
||||
# redirect the download somewhere else. `core/src/sync/client.rs` relies on
|
||||
# this and says so.
|
||||
# A path, never an absolute URL: the client joins it to the server it is
|
||||
# linked to, so a misconfigured server can't redirect the download.
|
||||
"url": platform.download_path,
|
||||
}
|
||||
|
||||
if platform.signed:
|
||||
# The signature travels WITH the metadata rather than behind its own route.
|
||||
# It is ~100 bytes, it is public wherever these bundles are published, and
|
||||
# the updater needs the version and the signature in the same breath — one
|
||||
# request that cannot return a signature belonging to a different build.
|
||||
# The signature travels with the metadata, so one response can't pair a
|
||||
# version with another build's signature.
|
||||
try:
|
||||
sig = (root / platform.signature).read_text(encoding="utf-8").strip()
|
||||
except OSError:
|
||||
@@ -255,14 +190,11 @@ def _read(root: Path, platform: Platform) -> dict | None:
|
||||
def _resolve(platform: Platform) -> tuple[Path, dict] | None:
|
||||
"""Which directory this server serves a platform from, and what is in it.
|
||||
|
||||
The operator's drop-in beats the baked copy — someone who deliberately put a
|
||||
build on the volume wants that build, older or not. A directory holding a
|
||||
broken or half-copied pair does NOT shadow the image: it simply is not a
|
||||
client, so the search moves on.
|
||||
A drop-in on the volume beats the baked copy, older or not. A broken or
|
||||
half-copied pair there doesn't shadow the image; the search moves on.
|
||||
"""
|
||||
# Both wrapped in Path(), not just the first. The asymmetry was arbitrary and it
|
||||
# fails quietly: `_read` catches the TypeError a str `/` str raises and reports
|
||||
# "no client here", so a directory that is perfectly fine reads as empty.
|
||||
# Both must be Paths: `_read` swallows the TypeError of str / str and would report
|
||||
# a good directory as empty.
|
||||
for root in (Path(Config.client_root()), Path(BAKED_ROOT)):
|
||||
release = _read(root, platform)
|
||||
if release is not None:
|
||||
@@ -282,9 +214,7 @@ def release(platform_id: str) -> dict | None:
|
||||
def releases() -> dict[str, dict]:
|
||||
"""Every client this server can hand out, keyed by platform id.
|
||||
|
||||
Platforms it holds nothing for are ABSENT rather than present-and-null, so a
|
||||
caller can test for the key instead of distinguishing "no build" from "a
|
||||
server that never had this platform".
|
||||
A platform with no build is absent, not null.
|
||||
"""
|
||||
found = {p.id: _resolve(p) for p in PLATFORMS}
|
||||
return {pid: r[1] for pid, r in found.items() if r is not None}
|
||||
@@ -295,9 +225,7 @@ def advertisement() -> dict:
|
||||
whole table, which the web UI renders the downloads section from. Phones ask
|
||||
`/api/client/android` for their own update instead.
|
||||
|
||||
ABSENT rather than empty when the server holds none, so a client testing for the
|
||||
key gets one unambiguous answer instead of having to distinguish "no client" from
|
||||
"an old server that never had this field".
|
||||
Absent, not empty, when the server holds none.
|
||||
"""
|
||||
found = releases()
|
||||
return {"clients": found} if found else {}
|
||||
@@ -305,11 +233,7 @@ def advertisement() -> dict:
|
||||
|
||||
@bp.get("/api/client")
|
||||
async def client_index():
|
||||
"""Everything this server holds, in one request.
|
||||
|
||||
The downloads UI needs all five to decide what to lead with, and five requests
|
||||
to answer one question is five chances to render half a page.
|
||||
"""
|
||||
"""Everything this server holds, in one request, for the downloads UI."""
|
||||
return jsonify({"clients": releases()})
|
||||
|
||||
|
||||
@@ -317,12 +241,8 @@ async def client_index():
|
||||
async def client_metadata(platform_id: str):
|
||||
"""Version and digest without the payload. What an updater polls.
|
||||
|
||||
Public, because a client has to be able to ask "is there something newer?"
|
||||
cheaply — before it has a token, in the case of a first pairing.
|
||||
|
||||
An unknown platform and a platform with no build both 404. They are the same
|
||||
answer to the caller ("not here"), and distinguishing them would only tell an
|
||||
unauthenticated stranger which platforms this build of the server knows about.
|
||||
Public, so a client can ask "is there something newer?" before it has a token.
|
||||
An unknown platform and one with no build both 404.
|
||||
"""
|
||||
found = release(platform_id)
|
||||
if found is None:
|
||||
@@ -334,25 +254,17 @@ async def client_metadata(platform_id: str):
|
||||
async def client_update_manifest(platform_id: str):
|
||||
"""The same build, in the shape the desktop's in-app updater reads.
|
||||
|
||||
Tauri's updater takes a manifest, not this module's metadata: `version` is the
|
||||
ordering key it compares (the desktop's `1.0.<minutes>`, i.e. `version_code`),
|
||||
and `url` + `signature` at the top level are its single-build form. So this is a
|
||||
translation of `release()`, never a second description of the build.
|
||||
A translation of `release()` into Tauri's single-build manifest: `version` is
|
||||
the desktop's `version_code`, with `url` and `signature` at the top level.
|
||||
|
||||
Only for a SIGNED platform. The updater verifies the signature against the key
|
||||
baked into the app before it replaces anything, so a server mirrors builds here
|
||||
and cannot sign them: the trust anchor does not move with the feed. A platform
|
||||
with no signature has no manifest, and a server holding no AppImage 404s, which
|
||||
the desktop reports as "this server has no update to offer", never as "up to
|
||||
date".
|
||||
Signed platforms only. The updater checks the signature against the key built
|
||||
into the app, so a server can mirror builds but not sign them. No AppImage: 404,
|
||||
which the desktop shows as "no update to offer", not "up to date".
|
||||
|
||||
`url` must be absolute for the updater, which is the one place this module
|
||||
hands out more than a path. It is built from the address THIS request arrived
|
||||
on, not from Settings, because the updater sends the same Authorization header
|
||||
to it that it sent here: the token goes back to the host that was asked, and
|
||||
nowhere else.
|
||||
`url` is absolute, built from the address this request arrived on rather than
|
||||
Settings, because the updater sends the same Authorization header there.
|
||||
|
||||
Public, like the metadata above. The bytes `url` names are not.
|
||||
Public; the bytes `url` names are not.
|
||||
"""
|
||||
platform = BY_ID.get(platform_id)
|
||||
if platform is None or not platform.signed:
|
||||
@@ -376,13 +288,9 @@ async def client_update_manifest(platform_id: str):
|
||||
async def client_download(platform_id: str):
|
||||
"""The bytes themselves.
|
||||
|
||||
Authenticated — by session cookie from a browser, or by device bearer token
|
||||
from a client updating itself; `login_required` accepts either. The metadata
|
||||
above is public; the bytes are not for anyone who can reach the port.
|
||||
|
||||
Throttled per account (Settings → Security), because each answer is tens of
|
||||
megabytes: one account or a leaked device token looping here could fill the
|
||||
server's uplink (family idea #5103, practice 6).
|
||||
Needs a session or a device token. Throttled per account (Settings → Security),
|
||||
because each answer is tens of megabytes and a looping client could fill the
|
||||
uplink.
|
||||
"""
|
||||
platform = BY_ID.get(platform_id)
|
||||
if platform is None:
|
||||
@@ -406,23 +314,16 @@ async def client_download(platform_id: str):
|
||||
async def send_artifact(root: Path, platform: Platform, found: dict) -> Response:
|
||||
"""The artifact, with its sha256 as the ETag and Range honoured against it.
|
||||
|
||||
Quart's own ETag is the file's mtime, size and path. That changes when a
|
||||
container restarts on the same bytes, so a phone resuming a download across a
|
||||
redeploy would be told its partial copy is stale. It also says nothing about
|
||||
content. The sidecar's sha256 is the identity the client already checks, so
|
||||
`If-Range` is answered against the bytes themselves (family idea #5103,
|
||||
practice 6).
|
||||
|
||||
So Quart's ETag and conditional handling are both turned off, and the
|
||||
conditional pass runs here after the right ETag is set. Run before it, Range
|
||||
and If-None-Match would be judged against the wrong one.
|
||||
Quart's ETag is mtime, size and path, which changes when a container restarts on
|
||||
the same bytes, so a resumed download would be told its partial copy is stale.
|
||||
Quart's ETag and conditional handling are off, and the conditional pass runs here
|
||||
after the sha256 ETag is set.
|
||||
"""
|
||||
response = await send_from_directory(
|
||||
root, platform.artifact, mimetype=platform.mimetype, add_etags=False, conditional=False
|
||||
)
|
||||
response.set_etag(found["sha256"])
|
||||
# Only behind a login, so no shared cache may keep a copy (family idea #5105,
|
||||
# practice 11). Quart marks every file it sends `public`.
|
||||
# Behind a login, so no shared cache may keep a copy. Quart marks files `public`.
|
||||
response.cache_control.public = False
|
||||
response.cache_control.private = True
|
||||
# Without this some browsers try to render it, and Android's download handler
|
||||
|
||||
+4
-13
@@ -1,15 +1,8 @@
|
||||
from __future__ import annotations
|
||||
|
||||
# The colour palette, and the one place that clamps to it.
|
||||
#
|
||||
# It lived on `models/note.py` until M315, when a note stopped having a colour. A
|
||||
# palette defined on the model that lost one would be a standing invitation to put the
|
||||
# column back; here it reads as what it now is — a LABEL's vocabulary, shared with the
|
||||
# import path that still names a colour.
|
||||
#
|
||||
# Keys, not tints. The actual colours live in each client (frontend/src/notes/colors.ts
|
||||
# and NoteTint.kt), so they can be retuned without a schema migration — which M315 spent
|
||||
# two steps doing.
|
||||
# The label colour palette, and the one place that clamps to it. Keys, not tints: the
|
||||
# colours live in each client (frontend/src/notes/colors.ts, NoteTint.kt), so they can
|
||||
# be retuned without a migration.
|
||||
NOTE_COLORS = {
|
||||
"default",
|
||||
"red",
|
||||
@@ -29,8 +22,6 @@ __all__ = ["NOTE_COLORS", "normalize_color"]
|
||||
def normalize_color(color: object) -> str:
|
||||
"""Return `color` if it's a known palette key, else the default.
|
||||
|
||||
`default` is no longer something anybody can CHOOSE — nothing offers a colour
|
||||
picker since M315 — but it is still where unrecognised input has to land, so this
|
||||
fallback is unreachable by choice rather than dead.
|
||||
Nothing offers `default` as a choice; it is where unrecognised input lands.
|
||||
"""
|
||||
return color if color in NOTE_COLORS else "default"
|
||||
|
||||
@@ -3,9 +3,7 @@ from __future__ import annotations
|
||||
import asyncio
|
||||
from datetime import datetime, timedelta
|
||||
|
||||
# Small, dependency-free value coercions shared across the blueprints. Kept in one
|
||||
# place so the "parse an ISO date" / "is this flag truthy" logic has a single
|
||||
# definition instead of a near-identical copy per module.
|
||||
# Small value coercions shared across the blueprints.
|
||||
|
||||
|
||||
def iso(dt: datetime | None) -> str | None:
|
||||
@@ -65,10 +63,8 @@ def detach(running: set[asyncio.Task], coro) -> asyncio.Task:
|
||||
def expired_before(now: datetime, retention_days: int) -> datetime | None:
|
||||
"""The cutoff: anything older than this has expired. `None` = retention is off.
|
||||
|
||||
Kept separate from the query so the window arithmetic — including the two ways
|
||||
to say "never" (0 and negative, the latter reachable by typing a stray minus in
|
||||
Settings) — is testable without a database. Trash and the audit log both expire
|
||||
through it.
|
||||
0 and negative both mean "never". Separate from the query so it is testable
|
||||
without a database. Trash and the audit log both expire through it.
|
||||
"""
|
||||
if retention_days <= 0:
|
||||
return None
|
||||
|
||||
+10
-19
@@ -7,19 +7,14 @@ from pathlib import Path
|
||||
class Config:
|
||||
"""Bootstrap configuration.
|
||||
|
||||
For a basic install, ``INKWELL_DATABASE_URL`` is the ONLY required env var —
|
||||
every other tunable lives in the DB-backed Settings UI (rule 25). The only other
|
||||
env var is an optional "break-glass" item:
|
||||
``INKWELL_DATABASE_URL`` is the only required env var; everything else is in
|
||||
the Settings UI, stored in the database. Optional:
|
||||
|
||||
- ``INKWELL_SECRET_KEY`` — optional override for the cookie-signing secret.
|
||||
If unset, a key is generated and persisted in the DB (see
|
||||
``inkwell.settings.load_or_create_secret_key``), so sessions survive
|
||||
restarts with no volume required.
|
||||
- ``INKWELL_SECRET_KEY`` overrides the cookie-signing secret. Unset, a key is
|
||||
generated and stored in the database, so sessions survive restarts.
|
||||
|
||||
Uploaded media lives under ``DATA_DIR`` — a fixed, authoritative path
|
||||
(``/var/inkwell``), intentionally NOT configurable (a mutable data path only
|
||||
invites breakage). Mount a volume there if you want uploads to persist across
|
||||
container recreation; a text-notes-only install never writes to it.
|
||||
Uploads live under ``DATA_DIR`` (``/var/inkwell``), which is fixed. Mount a
|
||||
volume there to keep them across container recreation.
|
||||
"""
|
||||
|
||||
DATA_DIR = "/var/inkwell"
|
||||
@@ -39,15 +34,11 @@ class Config:
|
||||
|
||||
@classmethod
|
||||
def client_root(cls) -> Path:
|
||||
"""Where an operator DROPS IN clients for this server to hand out.
|
||||
"""Where client builds can be dropped in to override the image's own.
|
||||
|
||||
One directory for every platform; `client_dist.py` picks files out of it by
|
||||
name. Under DATA_DIR because it is a mounted volume: a build placed here
|
||||
survives container recreation, and it beats the copy baked into the image,
|
||||
which is the whole point of the directory existing.
|
||||
|
||||
Empty is the ordinary case — the image ships its own set and most operators
|
||||
never touch this.
|
||||
One directory for every platform; `client_dist.py` picks files by name. On
|
||||
the data volume, so a build placed here survives container recreation.
|
||||
Usually empty.
|
||||
"""
|
||||
return Path(cls.DATA_DIR) / "client"
|
||||
|
||||
|
||||
@@ -24,10 +24,8 @@ async def _probe(url: str) -> None:
|
||||
async def wait_for_db(attempts: int = ATTEMPTS, delay: float = DELAY_SECONDS) -> bool:
|
||||
"""Poll the database until it accepts a connection, up to `attempts` tries.
|
||||
|
||||
Startup runs `alembic upgrade head` as the container's first DB touch, so a
|
||||
database that is a moment slow to accept connections would otherwise crash the
|
||||
container. This gives it a short window to come up, verifying once per `delay`
|
||||
seconds, instead of failing on the first missed connection.
|
||||
Runs before `alembic upgrade head`, so a database that is slow to come up delays
|
||||
the start instead of failing it.
|
||||
"""
|
||||
url = Config.DATABASE_URL
|
||||
for attempt in range(1, attempts + 1):
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
"""Admin routes over groups: named sets of people a note can be shared with (#5177).
|
||||
"""Admin routes over groups: named sets of people a note can be shared with.
|
||||
|
||||
A group is the instance's, not anyone's: the admin makes it and says who is in it,
|
||||
and every member of the instance can share a note with it. Sharing with a group
|
||||
|
||||
+14
-29
@@ -2,28 +2,16 @@
|
||||
|
||||
curl -fsSL https://notes.example.com/install.sh | sh
|
||||
|
||||
## Why the server templates it
|
||||
`curl | sh` gives a script no way to learn where it came from, so the server writes
|
||||
its own address into the copy it hands out.
|
||||
|
||||
`curl | sh` gives a script no way to learn where it came from: no argv, no referrer,
|
||||
nothing. So the server writes its own address into the copy it hands out, and the
|
||||
script installs from there without being told. Anything else means a self-hoster
|
||||
editing a URL by hand, which is the friction `client_dist.py` exists to remove.
|
||||
Arbitrary text in a piped script is the whole attack, so exactly one variable is
|
||||
substituted, `INKWELL_SERVER_DEFAULT`, with a value that passed [SAFE_ADDRESS]
|
||||
(nothing a shell reads as more than characters). The script checks the shape again.
|
||||
An address that can't pass is refused.
|
||||
|
||||
## Which way it fails
|
||||
|
||||
A server that can put arbitrary text into a script a person pipes to `sh` is the
|
||||
whole attack. So the substitution is ONE variable, `INKWELL_SERVER_DEFAULT`, given a value
|
||||
that has already passed [SAFE_ADDRESS] — no quote, no space, no `$`, nothing a shell
|
||||
reads as anything but characters — and the script checks the shape again before it
|
||||
uses it. If the address cannot pass, the route refuses rather than serving a script
|
||||
that would install from somewhere else.
|
||||
|
||||
## Where the script comes from
|
||||
|
||||
ONE script. The forge serves `desktop/packaging/install.sh` raw, and that is how the
|
||||
operator installs. The image carries a copy beside this module (the Dockerfile copies
|
||||
it in), and a source checkout reads the original. Same bytes either way; this module
|
||||
changes exactly one line of them.
|
||||
The script is `desktop/packaging/install.sh`; the image carries a copy beside this
|
||||
module, and a source checkout reads the original.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
@@ -47,11 +35,10 @@ SOURCE_SCRIPT = Path(__file__).resolve().parents[2] / "desktop" / "packaging" /
|
||||
# `render` refuses a script that does not, rather than serving one with no address in.
|
||||
PLACEHOLDER = 'INKWELL_SERVER_DEFAULT=""'
|
||||
|
||||
# What a server address may look like before it goes into a shell script: a scheme,
|
||||
# a host name or IPv4 address, an optional port and an optional path. Deliberately
|
||||
# narrower than a URL. Every character it admits is inert inside single quotes, and
|
||||
# an address it turns away (an IPv6 literal, a path with `%` in it) is answered by
|
||||
# setting Settings → Public address, not by widening this.
|
||||
# A server address safe to put in a shell script: scheme, host name or IPv4, optional
|
||||
# port and path. Narrower than a URL on purpose; every character is inert inside
|
||||
# single quotes. An address it refuses (IPv6, `%` in a path) is fixed by setting
|
||||
# Settings → Public address.
|
||||
SAFE_ADDRESS = re.compile(r"https?://[A-Za-z0-9.-]+(:[0-9]{1,5})?(/[A-Za-z0-9._~/-]*)?")
|
||||
|
||||
|
||||
@@ -68,10 +55,8 @@ def script_text() -> str | None:
|
||||
def server_address(public_url: str, scheme: str, host: str) -> str | None:
|
||||
"""Where the installer should fetch from, or None if no safe answer exists.
|
||||
|
||||
Settings → Public address when it is set: it is the address the operator says
|
||||
people use, and behind a reverse proxy the request's own Host may be an internal
|
||||
one. Otherwise the address this request arrived on, which is the one the person
|
||||
running `curl` just reached.
|
||||
Settings → Public address when set, since behind a proxy the request's Host may
|
||||
be internal. Otherwise the address this request arrived on.
|
||||
"""
|
||||
address = (public_url or f"{scheme}://{host}").rstrip("/")
|
||||
return address if SAFE_ADDRESS.fullmatch(address) else None
|
||||
|
||||
@@ -1,16 +1,12 @@
|
||||
"""Invites: an admin lets one person register while registration is closed (#5172).
|
||||
|
||||
Before these, adding a second person meant turning registration back on for the
|
||||
whole internet while they signed up, with no record of who was let in (#2939 §1).
|
||||
"""Invites: an admin lets one person register while registration is closed.
|
||||
|
||||
An invite is a link carrying a random token. It works once, until it expires, and
|
||||
optionally only for one email address. Only the token's hash is kept, so the link is
|
||||
shown once, when it is made; an admin who loses it revokes it and makes another.
|
||||
|
||||
This module is the invite itself: its status, its wire shape, and redemption, which
|
||||
This module is the invite itself: its status, wire shape, and redemption, which
|
||||
`auth.register` calls inside the transaction that creates the account. The admin
|
||||
routes are in `invites_api.py`, the same split as `settings` and `settings_api`, and
|
||||
for the same reason: the routes need `auth`, and `auth` needs this.
|
||||
routes are in `invites_api.py`, because they need `auth` and `auth` needs this.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
"""Admin routes for invites: make one, list them, revoke one (#5172).
|
||||
"""Admin routes for invites: make one, list them, revoke one.
|
||||
|
||||
What an invite is, and how it is redeemed, is in `invites.py`.
|
||||
"""
|
||||
|
||||
@@ -1,10 +1,8 @@
|
||||
"""Shared note↔label membership reconciliation. The "set a note's MANUAL (picker)
|
||||
labels, leave the tag-sourced ones alone" logic was duplicated line-for-line between
|
||||
the labels-picker API (notes.set_note_labels) and sync push (sync._apply_note_manual_labels).
|
||||
Single home so both stay in lockstep. via_tag=True rows track the body #tags and are
|
||||
governed by _lift_and_reconcile_tags — this function never touches them.
|
||||
"""Note↔label membership. Sets a note's manual (picker) labels for both the labels
|
||||
API and sync push, leaving via_tag=True rows, which track the body's #tags, to
|
||||
_lift_and_reconcile_tags.
|
||||
|
||||
Also the one definition of a LIVE tag and of deleting one (#5382)."""
|
||||
Also the one definition of a live tag and of deleting one."""
|
||||
from __future__ import annotations
|
||||
|
||||
from datetime import datetime, timezone
|
||||
|
||||
+15
-27
@@ -33,13 +33,11 @@ async def _label_note_count(db, label_id) -> int:
|
||||
async def _merge_into(db, source: Label, target: Label) -> None:
|
||||
"""Move every note tagged with `source` onto `target`, then tombstone `source`.
|
||||
|
||||
Shared by the explicit `/merge` route and by a rename that lands on a name
|
||||
another tag already holds — those are the same operation, and having one body
|
||||
is what stops them drifting into two answers for one question.
|
||||
Used by `/merge` and by a rename onto a name another tag holds, which are the
|
||||
same operation.
|
||||
|
||||
Note-body `#tags` are NOT rewritten, so a note whose body still literally
|
||||
contains the source `#tag` will re-mint that tag on its next edit. Retiring a
|
||||
tag means editing it out of the text; a known, documented nuance.
|
||||
Body `#tags` are not rewritten, so a note still containing the source `#tag`
|
||||
re-creates it on its next edit; retiring a tag means editing it out of the text.
|
||||
"""
|
||||
# Notes already carrying the target: a note can't hold the same label twice
|
||||
# (composite PK), so the source attachment there is just dropped as a dup.
|
||||
@@ -81,9 +79,8 @@ async def list_labels():
|
||||
)
|
||||
).all()
|
||||
) if labels else {}
|
||||
# When each was last used: the newest edit among the notes carrying it, trash
|
||||
# aside. The rail shows the few most recent (#5931), so a tag in daily use
|
||||
# stays in reach however many others there are.
|
||||
# When each was last used: the newest edit among its notes, trash aside. The
|
||||
# rail shows the most recent few, so a tag in daily use stays in reach.
|
||||
last_used = dict(
|
||||
(
|
||||
await db.execute(
|
||||
@@ -114,10 +111,9 @@ async def create_label():
|
||||
if not name:
|
||||
return json_error("tag name is required", 400)
|
||||
async with session_scope() as db:
|
||||
# Idempotent: creating an existing tag just returns it. Case-INSENSITIVE,
|
||||
# like the clients' `find_or_create_label` — a case-sensitive match here was
|
||||
# the path that MINTED the "Groceries" beside "groceries" pair that no synced
|
||||
# client can hold, since their `labels` index is unique on `lower(name)`.
|
||||
# Creating an existing tag returns it. Case-insensitive, like the clients:
|
||||
# their `labels` index is unique on `lower(name)`, so they can't hold both
|
||||
# "Groceries" and "groceries".
|
||||
existing = await db.scalar(
|
||||
select(Label).where(named(g.user_id, name))
|
||||
)
|
||||
@@ -146,11 +142,8 @@ async def update_label(label_id: str):
|
||||
if label is None:
|
||||
return not_found()
|
||||
if has_name:
|
||||
# Case-INSENSITIVE, matching the clients' `find_or_create_label` and the
|
||||
# local store's `lower(name)` unique index. The Postgres constraint here
|
||||
# is on the raw name, so the database would happily hold "Groceries"
|
||||
# beside "groceries" — but no synced client can store both, so letting
|
||||
# one be made is letting a pull fail later on a phone.
|
||||
# Case-insensitive, like the clients' unique `lower(name)` index. Postgres
|
||||
# would accept both spellings, and a phone's pull would then fail.
|
||||
clash = await db.scalar(
|
||||
select(Label).where(
|
||||
named(g.user_id, name),
|
||||
@@ -158,12 +151,9 @@ async def update_label(label_id: str):
|
||||
)
|
||||
)
|
||||
if clash is not None:
|
||||
# Renaming onto an existing tag MERGES the two rather than failing:
|
||||
# typing an existing tag's name onto this one says they are the same
|
||||
# thing. The OLDER row survives and takes the new spelling — age is
|
||||
# the one property that does not depend on which of the two the
|
||||
# caller happened to be renaming, so A→B and B→A agree. Ties go to
|
||||
# the incumbent. Mirrors `store::rename_label` exactly.
|
||||
# Renaming onto an existing tag merges the two. The older row survives
|
||||
# and takes the new spelling, so A→B and B→A agree; ties go to the
|
||||
# incumbent. Mirrors `store::rename_label`.
|
||||
if clash.created_at <= label.created_at:
|
||||
survivor, doomed = clash, label
|
||||
else:
|
||||
@@ -195,9 +185,7 @@ async def delete_label(label_id: str):
|
||||
async def merge_label(label_id: str):
|
||||
"""Merge `label_id` (source) INTO the label given by body {"into": <id>}: move
|
||||
every note tagged with the source onto the target, then tombstone the source. Both
|
||||
must be owned by the caller. Note-body `#tags` are NOT rewritten, so a note whose
|
||||
body still literally contains the source #tag will re-mint that label on its next
|
||||
edit — retire a tag by editing it out of the text (a known, documented nuance)."""
|
||||
must be owned by the caller. Body `#tags` are not rewritten (see `merge_into`)."""
|
||||
data = await request.get_json(silent=True) or {}
|
||||
into = data.get("into")
|
||||
async with session_scope() as db:
|
||||
|
||||
@@ -1,18 +1,15 @@
|
||||
"""What the server's emails say, and the one layout they all share (#5917).
|
||||
"""What the server's emails say, and the one layout they all share.
|
||||
|
||||
Each email is a function returning an `Email`: a subject, a plain-text part and an
|
||||
HTML part. `mailer.send` sends both as one multipart/alternative message, so a client
|
||||
that shows HTML gets the branded version and one that doesn't still gets every word
|
||||
and the link.
|
||||
|
||||
The HTML is deliberately email-shaped rather than web-shaped. It uses tables and
|
||||
inline styles, because many mail clients drop `<style>` blocks and ignore flexbox.
|
||||
It fetches nothing: the app sends with no outbound network beyond the mail server
|
||||
(rule 164), and a remote image is a tracking pixel anyway. The one image, the
|
||||
Inkwell logo, travels inside the message as an inline part (`cid:`), which mail
|
||||
clients show without asking, unlike a remote image; SVG and `data:` images are
|
||||
dropped by too many clients to rely on. Everything taken from settings or from a person (the site name, an address,
|
||||
a link) is escaped before it goes into the HTML.
|
||||
The HTML uses tables and inline styles, because many mail clients drop `<style>`
|
||||
blocks and ignore flexbox. It loads nothing remote. The logo travels inside the
|
||||
message as an inline part (`cid:`), which clients show without asking; SVG and
|
||||
`data:` images are dropped by too many clients. Everything taken from settings or a
|
||||
person (site name, address, link) is escaped.
|
||||
|
||||
Links come from Settings → General → Public address, never from the request: a
|
||||
forged Host header would otherwise mail a stranger a link to the forger's site.
|
||||
@@ -64,7 +61,7 @@ def _render(
|
||||
site: str, subject: str, heading: str, lines: list[str], button: _Button | None, footer: str
|
||||
) -> Email:
|
||||
"""One email in the shared layout: the site's name, a heading, the message, a
|
||||
button for the link, and a footer saying who sent it and why (preference 194).
|
||||
button for the link, and a footer saying who sent it and why.
|
||||
|
||||
`lines` are paragraphs of plain text. The button goes after the first one, and the
|
||||
text part spells its link out in full, since that is the only way a text-only
|
||||
|
||||
@@ -1,10 +1,9 @@
|
||||
"""Sending email over the SMTP server set in Settings → Email (#5266).
|
||||
"""Sending email over the SMTP server set in Settings → Email.
|
||||
|
||||
Used for password reset links, invites, and the admin's test message. What each one
|
||||
says is in `mail_templates`; this module only delivers. Email is an optional
|
||||
feature (rule 164): with no server set, `mail_settings` answers None and everything
|
||||
that would send says it is not available. A send that fails is logged and never
|
||||
takes anything else down with it.
|
||||
says is in `mail_templates`; this module only delivers. Email is optional: with no
|
||||
server set, `mail_settings` answers None and everything that would send says it is
|
||||
not available. A failed send is logged and takes nothing else down.
|
||||
|
||||
stdlib `smtplib`, run on a worker thread, so the event loop never waits on a mail
|
||||
server and nothing new is installed.
|
||||
|
||||
@@ -12,7 +12,7 @@ from . import Base
|
||||
|
||||
class AuditEvent(Base):
|
||||
"""One thing that happened to an account: a sign-in, a failed one, a password
|
||||
changed, a device linked (#2939 §5). Written by `audit.record`, read by admins in
|
||||
changed, a device linked. Written by `audit.record`, read by admins in
|
||||
Settings → Activity, and deleted after `audit_retention_days`.
|
||||
"""
|
||||
|
||||
|
||||
@@ -12,7 +12,7 @@ from . import Base
|
||||
|
||||
class Invite(Base):
|
||||
"""A single-use, expiring link an admin hands to one person so they can register
|
||||
while registration is closed (#5172).
|
||||
while registration is closed.
|
||||
|
||||
Only the token's SHA-256 hash is stored, as for device tokens; the link is shown
|
||||
once, when the invite is made. Rows are never deleted by the app: a used, expired
|
||||
|
||||
@@ -12,8 +12,8 @@ from . import Base
|
||||
|
||||
class Label(Base):
|
||||
__tablename__ = "labels"
|
||||
# A name is unique among LIVE tags only: a tombstone keeps its row for the change
|
||||
# feed but gives its name back, so `#grocery` can be made again (#5382, 0040).
|
||||
# A name is unique among live tags only: a tombstone keeps its row for the change
|
||||
# feed but gives its name back, so `#grocery` can be made again.
|
||||
__table_args__ = (
|
||||
Index(
|
||||
"uq_labels_owner_name_live",
|
||||
@@ -36,8 +36,8 @@ class Label(Base):
|
||||
updated_at: Mapped[datetime] = mapped_column(
|
||||
DateTime(timezone=True), nullable=False, server_default=func.now(), onupdate=func.now()
|
||||
)
|
||||
# Sync (M8): monotonic per-row revision (from sync_revision_seq via DB trigger) so a
|
||||
# label rename/recolor/merge/delete propagates to native clients independently of notes.
|
||||
# Monotonic per-row revision (sync_revision_seq, set by a DB trigger), so a rename,
|
||||
# recolour, merge or delete reaches native clients independently of notes.
|
||||
sync_revision: Mapped[int | None] = mapped_column(BigInteger(), nullable=True)
|
||||
# Tombstone: set means deleted. The row stays so the change feed can carry the
|
||||
# delete to devices; every read of the catalog skips it (`labeling.live`).
|
||||
|
||||
@@ -22,11 +22,9 @@ class Note(Base):
|
||||
owner_id: Mapped[uuid.UUID] = mapped_column(
|
||||
UUID(as_uuid=True), ForeignKey("users.id", ondelete="CASCADE"), nullable=False
|
||||
)
|
||||
# The note's NAME: its first non-empty body line, else its first checklist item
|
||||
# (see notes.derive_display_title). There is no title field to prefer — a note is
|
||||
# a body plus optional items, and this is simply the first thing written in it.
|
||||
# Persisted so search results and export filenames have something to say, and so
|
||||
# the full-text vector can weight it above the rest of the body.
|
||||
# The note's name: its first non-empty body line (notes.derive_display_title).
|
||||
# Stored for search results and export filenames, and so full-text search can
|
||||
# weight it above the rest of the body.
|
||||
display_title: Mapped[str] = mapped_column(Text(), nullable=False, server_default="")
|
||||
body: Mapped[str] = mapped_column(Text(), nullable=False, server_default="")
|
||||
# Manual drag order (higher = earlier); 0 until the user reorders.
|
||||
@@ -44,9 +42,9 @@ class Note(Base):
|
||||
updated_at: Mapped[datetime] = mapped_column(
|
||||
DateTime(timezone=True), nullable=False, server_default=func.now(), onupdate=func.now()
|
||||
)
|
||||
# Sync (M8): a monotonic per-row revision drawn from sync_revision_seq and assigned
|
||||
# by a DB trigger on every insert/update — the delta cursor native clients pull
|
||||
# against. Nullable in the ORM because the trigger populates it server-side.
|
||||
# Monotonic per-row revision from sync_revision_seq, set by a DB trigger on every
|
||||
# write: the cursor native clients pull against. Nullable in the ORM because the
|
||||
# trigger fills it.
|
||||
sync_revision: Mapped[int | None] = mapped_column(BigInteger(), nullable=True)
|
||||
# Hard-delete tombstone: non-null => permanently deleted (content cleared), kept so
|
||||
# offline clients learn the row is gone. Distinct from deleted_at (= recoverable trash).
|
||||
|
||||
@@ -25,7 +25,6 @@ class NoteAttachment(Base):
|
||||
filename: Mapped[str | None] = mapped_column(Text(), nullable=True)
|
||||
mime: Mapped[str] = mapped_column(Text(), nullable=False)
|
||||
size: Mapped[int] = mapped_column(BigInteger(), nullable=False)
|
||||
# Content hash (sha256 hex) for client-side dedupe + integrity over sync. Nullable
|
||||
# for rows created before this column (no backfill).
|
||||
# sha256 hex, for client-side dedupe and integrity over sync. Null on older rows.
|
||||
sha256: Mapped[str | None] = mapped_column(Text(), nullable=True)
|
||||
created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False, server_default=func.now())
|
||||
|
||||
@@ -11,7 +11,7 @@ from . import Base
|
||||
|
||||
|
||||
class NoteUserState(Base):
|
||||
"""A recipient's own pin, archive and place for a note shared with them (#5176).
|
||||
"""A recipient's own pin, archive and place for a note shared with them.
|
||||
|
||||
The owner's live on the note itself. `position` null means "where the owner put
|
||||
it", so a recipient who only pins a note doesn't freeze its order. `updated_at`
|
||||
|
||||
@@ -11,8 +11,8 @@ from . import Base
|
||||
|
||||
|
||||
class PasswordReset(Base):
|
||||
"""A one-time link an admin makes so a person who forgot their password can set
|
||||
a new one (#5173). There is no mail path; the admin hands the link over.
|
||||
"""A one-time link for setting a new password, made by an admin or emailed by
|
||||
"forgot password".
|
||||
|
||||
Only the token's SHA-256 hash is stored, as for invites and device tokens. Making
|
||||
a new link for an account deletes its unused ones, so only the newest works.
|
||||
|
||||
@@ -11,7 +11,7 @@ from . import Base
|
||||
class Setting(Base):
|
||||
"""A single persisted key/value setting. Values are JSON-encoded text; the
|
||||
typed defaults + metadata live in the code registry (inkwell.settings), so
|
||||
an empty table means "all defaults" (rule 26)."""
|
||||
an empty table means "all defaults"."""
|
||||
|
||||
__tablename__ = "settings"
|
||||
|
||||
|
||||
@@ -15,9 +15,9 @@ class Share(Base):
|
||||
|
||||
A resource's OWNER is tracked on the resource row itself (its owner_id column);
|
||||
this table records grants BEYOND the owner — to a single user or to a whole
|
||||
group. It is polymorphic (resource_type + resource_id) so every future
|
||||
shareable entity (notes first, in M1) reuses one table and one ACL predicate.
|
||||
Exactly one of shared_with_user_id / shared_with_group_id is set.
|
||||
group. Polymorphic (resource_type + resource_id), so any shareable entity uses
|
||||
one table and one ACL predicate. Exactly one of shared_with_user_id /
|
||||
shared_with_group_id is set.
|
||||
"""
|
||||
|
||||
__tablename__ = "shares"
|
||||
|
||||
@@ -11,7 +11,7 @@ from . import Base
|
||||
|
||||
|
||||
class ShareRevocation(Base):
|
||||
"""A note that someone lost access to, for their devices to delete (#5175).
|
||||
"""A note that someone lost access to, for their devices to delete.
|
||||
|
||||
The change feed only carries notes a person can see, so a share ending would
|
||||
otherwise leave a stale copy on every device that had synced it. The revision
|
||||
|
||||
@@ -18,7 +18,7 @@ class User(Base):
|
||||
email: Mapped[str] = mapped_column(CITEXT(), nullable=False, unique=True)
|
||||
# First registered user becomes admin (see auth.register); admin gates Settings.
|
||||
is_admin: Mapped[bool] = mapped_column(Boolean(), nullable=False, server_default=func.false())
|
||||
# Nullable: leaves room for external-identity-only accounts later (rule 26).
|
||||
# Nullable, for accounts that sign in only through an external identity.
|
||||
password_hash: Mapped[str | None] = mapped_column(Text(), nullable=True)
|
||||
display_name: Mapped[str] = mapped_column(Text(), nullable=False)
|
||||
# Moved on by a password reset to sign the account out everywhere: every session
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
"""How a note sits on one person's board: pinned, archived, and where (#5176).
|
||||
"""How a note sits on one person's board: pinned, archived, and where.
|
||||
|
||||
For the owner that is the note's own columns. For someone it is shared with it is
|
||||
their `NoteUserState` row, so the owner pinning a shared shopping list doesn't
|
||||
|
||||
@@ -72,11 +72,9 @@ async def list_notes():
|
||||
label_params = request.args.getlist("label")
|
||||
has_attachment = coerce_bool(request.args.get("has_attachment"))
|
||||
query_text = (request.args.get("q") or "").strip()
|
||||
# Optional creation-date range — the "browse by when" / Timeline lens. Both bounds
|
||||
# are ISO-8601 instants forming a HALF-OPEN interval [created_after, created_before),
|
||||
# so a client can pass local day-boundaries (start-of-day .. start-of-next-day)
|
||||
# without off-by-one. `sort=created` orders newest-captured first for a chronological
|
||||
# timeline; the default keeps the board's pinned/position/updated order.
|
||||
# Optional creation-date range, half-open [created_after, created_before), so a
|
||||
# client can pass local day boundaries. `sort=created` is newest first; the
|
||||
# default is the board's pinned/position/updated order.
|
||||
after_param = request.args.get("created_after")
|
||||
before_param = request.args.get("created_before")
|
||||
sort = request.args.get("sort")
|
||||
@@ -85,7 +83,7 @@ async def list_notes():
|
||||
return json_error("invalid shared", 400)
|
||||
async with session_scope() as db:
|
||||
# Through the viewer's own pin, archive and place: on a note shared with them
|
||||
# those are theirs, not the owner's (#5176).
|
||||
# those are theirs, not the owner's.
|
||||
stmt = join_state(select(Note), g.user_id).where(note_visible())
|
||||
stmt = apply_filter(stmt, filter_name, archived_for(g.user_id))
|
||||
# Trash is the owner's: a note its owner trashed leaves a recipient's board
|
||||
@@ -112,10 +110,8 @@ async def list_notes():
|
||||
return json_error("invalid created_before", 400)
|
||||
stmt = stmt.where(Note.created_at < before_dt)
|
||||
if query_text:
|
||||
# Full-text match over the note's name + body (generated tsvector,
|
||||
# migrations 0005/0026), ranked. This is the ONLY text search now: the
|
||||
# separate facet-less `/search` route was removed because landing on it
|
||||
# was the one place you could not also narrow by tag (note 2930).
|
||||
# Ranked full-text match over the note's name and body (a generated
|
||||
# tsvector). The only text search, so it combines with every other filter.
|
||||
tsquery = func.websearch_to_tsquery("english", query_text)
|
||||
search_col = literal_column("notes.search_vector")
|
||||
stmt = stmt.where(search_col.op("@@")(tsquery)).order_by(
|
||||
@@ -168,8 +164,7 @@ async def complete_reminder(note_id: str):
|
||||
return jsonify(await _serialize_note(db, note, g.user_id))
|
||||
|
||||
|
||||
# The longest snooze: 30 days. The core clamps to the same range
|
||||
# (`store::SNOOZE_MAX_MINUTES`), so a device and the web agree (#5386).
|
||||
# The longest snooze: 30 days, the same as the core's `store::SNOOZE_MAX_MINUTES`.
|
||||
SNOOZE_MAX_MINUTES = 60 * 24 * 30
|
||||
|
||||
|
||||
@@ -283,9 +278,8 @@ async def import_notes():
|
||||
if len(zf.infolist()) > IMPORT_MAX_ENTRIES:
|
||||
return json_error("that archive has too many files to import", 413)
|
||||
|
||||
# Bound total decompressed bytes so a zip bomb can't exhaust memory/disk. Any entry
|
||||
# (or the cumulative total) blowing the budget aborts the whole import — nothing is
|
||||
# committed, since the session rolls back when the exception exits its block.
|
||||
# Bound decompressed bytes so a zip bomb can't exhaust memory or disk. Going over
|
||||
# aborts the whole import; the session rolls back, so nothing is committed.
|
||||
budget = _ImportBudget()
|
||||
try:
|
||||
specs, source = _read_import_specs(zf, budget)
|
||||
@@ -321,10 +315,7 @@ async def import_notes():
|
||||
async def list_titles():
|
||||
"""Owner's non-trashed notes, keyed by their display NAME.
|
||||
|
||||
Survived the removal of [[wiki-links]] (note 2897) because it was serving two
|
||||
different things, and only one of them was linking. This is what the command
|
||||
palette lists so someone can jump to a note by name — which is recall, the thing
|
||||
this app is actually for. The `[[` autocomplete that also read it is gone.
|
||||
What the command palette lists, so someone can jump to a note by name.
|
||||
"""
|
||||
async with session_scope() as db:
|
||||
rows = (
|
||||
@@ -367,7 +358,7 @@ async def reorder_notes():
|
||||
note = visible.get(nid)
|
||||
if note is None:
|
||||
continue
|
||||
# A note shared with the caller moves on their board only (#5176).
|
||||
# A note shared with the caller moves on their board only.
|
||||
if note.owner_id == g.user_id:
|
||||
note.position = total - index
|
||||
else:
|
||||
@@ -386,8 +377,8 @@ async def create_note():
|
||||
if is_empty_note(body, item_texts):
|
||||
return json_error("note is empty", 400)
|
||||
async with session_scope() as db:
|
||||
# Items still arrive separately — a client holds a list, not a blob — but they
|
||||
# are folded into the body, which is where a checklist lives now (M304).
|
||||
# Items may arrive separately, from a client holding a list, and are folded
|
||||
# into the body, where a checklist lives.
|
||||
for text in item_texts:
|
||||
body = append_item(body, text)
|
||||
note = Note(owner_id=g.user_id, body="", display_title="", position=await top_position(db, g.user_id) + 1)
|
||||
@@ -418,9 +409,8 @@ async def update_note(note_id: str):
|
||||
owned = note.owner_id == g.user_id
|
||||
if not owned:
|
||||
# Someone the note is shared with may pin and archive it on their own
|
||||
# board (#5176), and change its text if it was shared at `edit`; nothing
|
||||
# else. They can already read it, so saying so is no oracle. Text from a
|
||||
# view share 404s, as it did before they could do anything at all.
|
||||
# board, and change its text if shared at `edit`; nothing else. Text from
|
||||
# a view share 404s.
|
||||
if set(data) - {"body", *OWN_FIELDS}:
|
||||
return json_error("only the owner can change that", 403)
|
||||
if "body" in data and await _get_editable(db, note_id) is None:
|
||||
@@ -576,11 +566,6 @@ async def update_item(note_id: str, item_id: str):
|
||||
return await _rewrite_body(db, note, body)
|
||||
|
||||
|
||||
# The reorder route is gone with M304. Reordering a checklist is moving a line, which
|
||||
# is something a text editor already does and no client ever called this for — the
|
||||
# only reference to it in the tree was a test asserting the route existed.
|
||||
|
||||
|
||||
@bp.post("/<note_id>/attachments")
|
||||
@login_required
|
||||
async def upload_attachment(note_id: str):
|
||||
@@ -621,7 +606,7 @@ async def get_attachment(note_id: str, att_id: str):
|
||||
if aid is None:
|
||||
return not_found()
|
||||
async with session_scope() as db:
|
||||
# Owner OR shared may view (rule 47). A purged note's files are gone with it.
|
||||
# Owner or a share may view. A purged note's files are gone with it.
|
||||
note = await _get_visible(db, note_id)
|
||||
if note is None:
|
||||
return not_found()
|
||||
@@ -638,9 +623,8 @@ async def get_attachment(note_id: str, att_id: str):
|
||||
response = await send_file(str(file_path), mimetype=mime)
|
||||
response.headers["X-Content-Type-Options"] = "nosniff"
|
||||
response.headers["Cache-Control"] = "private, max-age=86400"
|
||||
# Only trusted raster image types render inline; everything else — notably
|
||||
# image/svg+xml, which can carry script — downloads, so a shared note's attachment
|
||||
# can't execute script in a viewer's session (rule 47 = shares).
|
||||
# Only trusted raster image types render inline. Everything else downloads,
|
||||
# notably SVG, which can carry script into a viewer's session.
|
||||
disposition = "inline" if mime in ALLOWED_IMAGE_MIMES else "attachment"
|
||||
response.headers["Content-Disposition"] = f'{disposition}; filename="{_header_filename(filename)}"'
|
||||
return response
|
||||
@@ -722,10 +706,8 @@ async def delete_note(note_id: str):
|
||||
return not_found()
|
||||
if note.deleted_at is None:
|
||||
return json_error("note must be trashed before permanent delete", 409)
|
||||
# A tombstone, not a dropped row. Deleting the row outright would leave the
|
||||
# server with no record the note ever existed, so a linked device that was
|
||||
# offline at the time would keep its copy forever — and push it back the
|
||||
# next time it was edited. The delete has to be something clients can LEARN.
|
||||
# A tombstone, not a dropped row, so a device that was offline learns of the
|
||||
# delete instead of keeping its copy and pushing it back.
|
||||
await purge_note(db, note)
|
||||
await db.commit()
|
||||
return jsonify({"ok": True})
|
||||
|
||||
@@ -2,7 +2,6 @@ from __future__ import annotations
|
||||
|
||||
from quart import Blueprint
|
||||
|
||||
# The notes blueprint lives in its own tiny module so every route module in the
|
||||
# package can `from ._bp import bp` without importing the package __init__ (which
|
||||
# imports the route modules) — i.e. no import cycle.
|
||||
# In its own module so route modules can `from ._bp import bp` without an import cycle
|
||||
# through the package __init__.
|
||||
bp = Blueprint("notes", __name__, url_prefix="/api/notes")
|
||||
|
||||
@@ -1,16 +1,12 @@
|
||||
"""Writing a note's text — the one path every surface goes through.
|
||||
|
||||
A body edit is never only the body. It may keep a revision of the old text, it
|
||||
renames the note, it lifts and reconciles `#tags`, and once committed it queues link
|
||||
previews. That sequence used to be written out at each place a body could change —
|
||||
PATCH, the item routes, restoring a revision and sync push — and the copies drifted:
|
||||
restoring a revision never queued previews, so a restored link stayed a bare URL
|
||||
(#5164). Now a caller states how the old text should be kept and this does the rest.
|
||||
A body edit may keep a revision of the old text, renames the note, and lifts and
|
||||
reconciles `#tags`. PATCH, the item routes, restoring a revision and sync push all
|
||||
come through here; the caller says how the old text is kept.
|
||||
|
||||
Previews are the one part that cannot happen here. They are queued AFTER the commit
|
||||
(`unfurl_queue.schedule`), because the fetch reads the note in a session of its own
|
||||
and a note it cannot see yet gets no preview. So this returns whether the text
|
||||
changed, and the caller queues them once its transaction is committed.
|
||||
Link previews are queued by the caller after its commit (`unfurl_queue.schedule`),
|
||||
because the fetch reads the note in its own session. So this returns whether the
|
||||
text changed.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
|
||||
@@ -1,18 +1,11 @@
|
||||
"""Checklist items — a note's body IS its checklist (M304).
|
||||
"""Checklist items: a note's body is its checklist.
|
||||
|
||||
A `- [ ] milk` line is the item. There is no `note_items` table beside the body any
|
||||
more, which is what lets a list sit BETWEEN two paragraphs: rows had a position in a
|
||||
table and no position in the text, so a separate list could only ever render after
|
||||
the prose no matter how it was styled.
|
||||
A `- [ ] milk` line is the item, so a list can sit between two paragraphs. Unlike
|
||||
tags, items are stored nowhere else: nothing queries them, so `parse_items` is the
|
||||
whole storage layer and the rewriters below are how one is edited.
|
||||
|
||||
The same shape as `tags.py`, one strength further along. Tags are derived from the
|
||||
body too, but they MATERIALISE into `note_labels` rows because the board queries by
|
||||
label. Items materialise into nothing, because nothing queries them — their only
|
||||
readers are the card, the editor and `display_title`. So `parse_items` is the whole
|
||||
storage layer for a checklist, and the rewriters below are how one is edited.
|
||||
|
||||
THE GRAMMAR IS SHARED. Three implementations exist and they have to agree, because a
|
||||
difference between any two of them is a checklist that changes shape when it syncs:
|
||||
The grammar has three implementations that must agree, or a checklist changes shape
|
||||
when it syncs:
|
||||
|
||||
core/src/local/derive.rs the native clients (desktop + Android)
|
||||
src/inkwell/notes/checklist.py this file, the server
|
||||
@@ -21,12 +14,9 @@ difference between any two of them is a checklist that changes shape when it syn
|
||||
optional indent, `-` or `*`, one-or-more spaces, `[ ]`/`[x]`/`[X]`,
|
||||
then either end-of-line or one-or-more spaces and the text.
|
||||
|
||||
`*` is accepted because markdown.ts already takes it for a plain bullet, and a rule
|
||||
that allowed `* item` but not `* [ ] item` would be one nobody could guess. `- [ ]`
|
||||
with nothing after it IS an item with empty text — that is what pressing Enter on a
|
||||
list leaves behind, and refusing to parse it would make a half-typed list stop being
|
||||
a list. `- [X]` parses as checked and renders back lowercase, so one canonical form
|
||||
survives a round trip.
|
||||
`*` is accepted because it is a plain bullet too. `- [ ]` alone is an item with empty
|
||||
text, which is what Enter on a list leaves behind. `- [X]` parses as checked and
|
||||
renders back lowercase, so one canonical form survives a round trip.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
@@ -101,12 +91,9 @@ def set_item_checked(body: str, index: int, checked: bool) -> str:
|
||||
def append_item(body: str, text: str, checked: bool = False) -> str:
|
||||
"""Add an item at the end of the body.
|
||||
|
||||
A blank line between prose and the list, nothing between consecutive items —
|
||||
the layout `import_export._note_markdown` has always used when writing a checklist
|
||||
out. That is not cosmetic: it is what the Alembic migration folds existing
|
||||
`note_items` rows into AND what `derive::append_item` produces on every client, so
|
||||
all three land on identical bodies. An export taken before the migration and one
|
||||
taken after therefore differ in nothing.
|
||||
A blank line between prose and the list, nothing between consecutive items: the
|
||||
same layout as the export and the core's `derive::append_item`, so every surface
|
||||
lands on identical bodies.
|
||||
"""
|
||||
line = render_item(text.strip(), checked)
|
||||
trimmed = body.rstrip("\n")
|
||||
|
||||
@@ -33,21 +33,13 @@ DISPLAY_TITLE_CAP = 200
|
||||
def derive_display_title(body: str | None) -> str:
|
||||
"""The note's display NAME: the first line of the body that says anything.
|
||||
|
||||
There is no explicit title to prefer any more (M13 step 3) — a note is a body, and
|
||||
its name is simply the first thing written in it. Persisted as notes.display_title
|
||||
so search results and export filenames have something to say.
|
||||
Stored as notes.display_title for search results and export filenames. A task
|
||||
line is named without its marker ("milk", not "- [ ] milk"), and an empty item
|
||||
is skipped.
|
||||
|
||||
The old `first_item` fallback is gone with M304: items ARE body lines now, so a
|
||||
list-only note is named by its first item without anyone having to arrange it. What
|
||||
replaced the fallback is stripping the task marker — calling that note "- [ ] milk"
|
||||
would show someone the storage instead of the note — and skipping an EMPTY item, so
|
||||
a half-typed list does not leave a note with no name.
|
||||
|
||||
Lines split on "\n" alone, as every other reading of the grammar does;
|
||||
`splitlines` also broke on a lone "\r" or a U+2028, so a phone and the server named
|
||||
the same note differently. Mirrors `derive::display_title` in the core, and both
|
||||
run the `display_titles` cases in core/testdata/grammar.json. Deterministic — a
|
||||
literal first line, never generated.
|
||||
Lines split on "\n" alone, like every other reader of the grammar (`splitlines`
|
||||
would also break on "\r" and U+2028). Mirrors `derive::display_title` in the
|
||||
core; both run the `display_titles` cases in core/testdata/grammar.json.
|
||||
"""
|
||||
for line in (body or "").split("\n"):
|
||||
stripped = strip_marker(line.strip()).strip()
|
||||
@@ -59,9 +51,8 @@ def derive_display_title(body: str | None) -> str:
|
||||
def is_empty_note(body: str | None, items: list | None = None) -> bool:
|
||||
"""Nothing worth keeping: no body text and no checklist items.
|
||||
|
||||
`items` is still a separate argument because create still accepts them separately —
|
||||
the importer holds a list, not a blob — and they are folded into the body only
|
||||
after this check has decided the note is worth making at all.
|
||||
`items` is separate because create accepts them separately (the importer holds a
|
||||
list); they are folded into the body after this check.
|
||||
"""
|
||||
return not (body or "").strip() and not items
|
||||
|
||||
@@ -77,7 +68,7 @@ def apply_filter(stmt, filter_name: str, archived):
|
||||
"""Narrow a notes query to one board view.
|
||||
|
||||
`archived` is the viewer's archive flag (`note_state.archived_for`): a note
|
||||
someone shared with you is archived on your board when YOU archived it (#5176).
|
||||
someone shared with you is archived on your board when you archived it.
|
||||
|
||||
Every branch excludes purge tombstones — content-less rows kept only so the sync
|
||||
feed can tell offline clients a note is gone (see `retention.purge_note`). The
|
||||
@@ -121,12 +112,11 @@ async def _fetch_note(db, note_id: str, gate) -> Note | None:
|
||||
|
||||
async def _get_owned(db, note_id: str) -> Note | None:
|
||||
"""Fetch a note the current user OWNS, for every change only an owner may make:
|
||||
trash, delete, labels, reminders, attachments, previews, history and sharing
|
||||
(#5174). Pin and archive are everyone's own (#5176).
|
||||
trash, delete, labels, reminders, attachments, previews, history and sharing.
|
||||
Pin and archive are everyone's own.
|
||||
|
||||
Someone the note is shared with gets None here, and so a 404, even when they can
|
||||
read the note: a 403 would confirm the note exists to a caller who only has its
|
||||
id (#1984).
|
||||
Someone the note is shared with gets None, and so a 404, even when they can read
|
||||
it: a 403 would confirm the note exists to a caller who only has its id.
|
||||
|
||||
A purged note reads as absent: the REST API must treat it as gone, so opening,
|
||||
editing or restoring one 404s. The sync push path looks rows up directly rather
|
||||
@@ -138,14 +128,14 @@ async def _get_owned(db, note_id: str) -> Note | None:
|
||||
async def _get_visible(db, note_id: str) -> Note | None:
|
||||
"""Fetch a note the current user can see: theirs, or shared with them at any
|
||||
level. What a recipient may do to it from here is their own pin, archive and
|
||||
place (#5176); `_get_editable` and `_get_owned` gate the rest."""
|
||||
place; `_get_editable` and `_get_owned` gate the rest."""
|
||||
return await _fetch_note(db, note_id, note_visible())
|
||||
|
||||
|
||||
async def _get_editable(db, note_id: str) -> Note | None:
|
||||
"""Fetch a note the current user may change the TEXT of: its body and its
|
||||
checklist. The owner, or someone it is shared with at `edit`; a view share gets
|
||||
None, and so a 404, like a stranger (#5174)."""
|
||||
None, and so a 404, like a stranger."""
|
||||
return await _fetch_note(db, note_id, note_visible(EDIT))
|
||||
|
||||
|
||||
|
||||
@@ -42,22 +42,16 @@ def _note_markdown(note: Note, labels: list) -> str:
|
||||
fm.append(f"updated: {iso(note.updated_at) or ''}")
|
||||
fm.append("---")
|
||||
fm.append("")
|
||||
# Body and checklist are no longer alternatives — a note can carry both, so both
|
||||
# are written, body first, with a blank line between them when there is.
|
||||
if note.body:
|
||||
fm.append(note.body)
|
||||
# No separate items block any more. The body already ends with those exact lines
|
||||
# (M304) — this function is where their layout was decided, and appending them a
|
||||
# second time would double every checklist in an export.
|
||||
# The checklist is part of the body, so it is not written a second time.
|
||||
return "\n".join(fm) + "\n"
|
||||
|
||||
|
||||
# --- Import: Inkwell's own export (round-trip) OR a Google Keep Takeout zip ---
|
||||
|
||||
# The `app` value an export's notes.json carries. "thoughtsync" is what every export
|
||||
# written before the rename to Inkwell says (Scribe note 5071), and those zips are
|
||||
# people's backups, so they keep importing. That is the one reason the old name is
|
||||
# still read here; nothing writes it any more.
|
||||
# The `app` values an Inkwell export's notes.json may carry. Exports from when the app
|
||||
# was called ThoughtSync say "thoughtsync", and still import.
|
||||
NATIVE_APP_MARKERS = ("inkwell", "thoughtsync")
|
||||
|
||||
# Reverse of ALLOWED_IMAGE_MIMES, for inferring an attachment's mime from its
|
||||
@@ -106,9 +100,7 @@ def _keep_spec(kn: dict, keep_dir: str) -> dict:
|
||||
"""Normalize one Google Keep note (Takeout <note>.json) into the common import
|
||||
spec. `keep_dir` is the note JSON's folder, used to resolve attachment paths."""
|
||||
list_content = kn.get("listContent") if isinstance(kn.get("listContent"), list) else []
|
||||
# Keep's own notes are one or the other, but its text was being DISCARDED whenever
|
||||
# a note also had list content, because the target model could only hold one.
|
||||
# It can hold both now, so both are kept.
|
||||
# A Keep note may carry both text and list content; both are kept.
|
||||
body = kn.get("textContent") or ""
|
||||
# Keep stores link annotations (e.g. shared URLs) separately from the text —
|
||||
# fold any URLs into the body so the content survives the move.
|
||||
@@ -247,28 +239,21 @@ async def _create_imported_note(
|
||||
name derivation + tag reconciliation as create_note. Returns False (nothing
|
||||
written) when the spec is empty."""
|
||||
body = spec.get("body") or ""
|
||||
# An imported title becomes the note's FIRST BODY LINE.
|
||||
#
|
||||
# Inkwell has no title field any more (M13 step 3), but the things people
|
||||
# import from do — Keep notes carry one, and so does any export taken before this.
|
||||
# Dropping it would silently lose text someone wrote; folding it into the body puts
|
||||
# it exactly where a name now lives, so the note comes in named the way it was.
|
||||
# Skipped when the body already opens with that line, so re-importing an export
|
||||
# this code produced doesn't stack duplicates.
|
||||
# An imported title (Keep has them) becomes the first body line, which is where a
|
||||
# note's name lives. Skipped when the body already opens with it, so re-importing
|
||||
# an export doesn't stack duplicates.
|
||||
title = (spec.get("title") or "").strip()
|
||||
if title and body.lstrip().split("\n", 1)[0].strip() != title:
|
||||
body = f"{title}\n{body}" if body.strip() else title
|
||||
|
||||
items = spec.get("items") or []
|
||||
item_texts = [t for t in ((it.get("text") or "").strip() for it in items) if t]
|
||||
# A note that is only a photo is still a note — Keep has plenty of them, and they
|
||||
# used to be skipped as empty. The core's importer keeps them too (portable.rs).
|
||||
# A note that is only a photo is still a note, as in the core's importer.
|
||||
if is_empty_note(body, item_texts) and not spec.get("attachments"):
|
||||
return False
|
||||
|
||||
# Items fold into the body, which is where a checklist lives now (M304). Done
|
||||
# before the Note is built so display_title and _lift_and_reconcile_tags both see the
|
||||
# finished text — an imported item can carry a #tag like any other line.
|
||||
# Items fold into the body before the Note is built, so display_title and
|
||||
# _lift_and_reconcile_tags see the finished text; an item can carry a #tag.
|
||||
for it in items:
|
||||
text = (it.get("text") or "").strip()
|
||||
if text:
|
||||
|
||||
@@ -36,15 +36,11 @@ async def _labels_for_notes(db, note_ids: list) -> dict:
|
||||
def items_of(body: str | None) -> list[dict]:
|
||||
"""The note's checklist, read out of its body. No query, because there is no table.
|
||||
|
||||
Still emitted in the payload after M304, and that is not a second source of truth:
|
||||
it is DERIVED on the way out, so it cannot disagree with the body it came from. It
|
||||
saves every consumer that only wants to draw checkboxes from carrying a parser, and
|
||||
the ones that do carry one (the native clients, the browser) are free to ignore it
|
||||
and read the body.
|
||||
Derived on the way out, so it can't disagree with the body; it spares a consumer
|
||||
that only draws checkboxes from carrying a parser.
|
||||
|
||||
The id is the item's ORDINAL, which is what the rewriters in `checklist.py` take,
|
||||
so a client holding one can act on it directly. It also shifts when an item is
|
||||
removed — every mutation returns the reloaded note for exactly that reason.
|
||||
The id is the item's ordinal, which the rewriters in `checklist.py` take. It
|
||||
shifts when an item is removed, so every mutation returns the reloaded note.
|
||||
"""
|
||||
return [
|
||||
{"id": str(i), "text": item.text, "checked": item.checked, "position": i}
|
||||
@@ -108,7 +104,7 @@ async def _previews_for_notes(db, note_ids: list) -> dict:
|
||||
|
||||
|
||||
async def _sharing_for_notes(db, notes: list, viewer) -> dict:
|
||||
"""note_id -> how `viewer` holds the note (#5174): `permission` (owner, edit or
|
||||
"""note_id -> how `viewer` holds the note: `permission` (owner, edit or
|
||||
view), `shared` (whether it is shared at all, as far as the viewer may know) and
|
||||
`shared_by` (the owner, for a note someone shared with them)."""
|
||||
foreign = [n for n in notes if n.owner_id != viewer]
|
||||
@@ -142,7 +138,7 @@ async def _serialize_notes(db, notes: list, viewer=None) -> list:
|
||||
"""The API shape of each note. With `viewer`, each also says how the viewer holds
|
||||
it, and a note someone shared with them comes without the owner's labels: labels
|
||||
are personal, and a recipient's board files nothing under someone else's tags.
|
||||
For the same reason its pin, archive and position are the viewer's own (#5176).
|
||||
For the same reason its pin, archive and position are the viewer's own.
|
||||
Sync passes no viewer to a client that asked only for its own notes."""
|
||||
ids = [n.id for n in notes]
|
||||
labels_map = await _labels_for_notes(db, ids)
|
||||
|
||||
+21
-57
@@ -1,16 +1,6 @@
|
||||
"""#tags — parsing note bodies, LIFTING the standalone ones out of the text, and
|
||||
keeping the still-in-text ones in sync with the note_labels rows. Manual (picker)
|
||||
labels are NOT touched here (see the labeling module).
|
||||
|
||||
A tag used to be shown twice: once as the `#todo` you typed and once as a chip. The
|
||||
chip moved to the top of the card (M311) and the text now goes, but only when the tag
|
||||
was standing on its own — see `split_body_tags` for the rule and why it is the
|
||||
conservative one.
|
||||
|
||||
Was `links.py`, and also owned `[[wiki-links]]` until they were removed (note 2897):
|
||||
this app is an intermediary surface for capture and recall, and a linking system is
|
||||
organization, which is not what it is for. A file called links.py holding no links
|
||||
would have been exactly the kind of drift that removal was meant to end.
|
||||
"""#tags: parsing note bodies, lifting the standalone ones out of the text (they show
|
||||
as chips), and keeping the in-text ones in sync with the note_labels rows. Manual
|
||||
(picker) labels are not touched here (see the labeling module).
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
@@ -26,22 +16,15 @@ from .helpers import derive_display_title
|
||||
# A #tag: `#` at the start of a line or after whitespace, then a LETTER, then
|
||||
# letters, digits, `_` and `-`. The same rule as `line_tags` in
|
||||
# core/src/local/derive.rs and the web's markdown.ts, and all three run the cases in
|
||||
# core/testdata/grammar.json.
|
||||
#
|
||||
# The three used to differ. The core took a tag after any punctuation, so `(#todo)`
|
||||
# was a label on the phone and plain text here, and the `/#section` of a pasted URL
|
||||
# became a label too. This side let a tag start with a digit or `_` (`#1st`). So a
|
||||
# note's labels changed every time it synced. The shared rule is the strict one:
|
||||
# nothing is a tag now that was not already one everywhere (#5166).
|
||||
# core/testdata/grammar.json; a difference would change a note's labels on sync. So
|
||||
# `(#todo)`, `/#section` in a URL and `#1st` are not tags.
|
||||
#
|
||||
# `[^\W\d_]` is "a word character that is not a digit or `_`", which is a letter.
|
||||
_TAG_RE = re.compile(r"(?<!\S)#([^\W\d_][\w-]*)")
|
||||
|
||||
|
||||
# A fence opens or closes a code block. A `#tag` inside one is CODE — the shell
|
||||
# comment in a snippet someone pasted — and lifting it would delete a line of their
|
||||
# example. It still becomes a label, because it always has and that is a separate
|
||||
# question from whether the text may be touched.
|
||||
# A fence opens or closes a code block. A `#tag` inside one is code (a pasted shell
|
||||
# comment), so it is never lifted out of the text, though it still becomes a label.
|
||||
_FENCE_RE = re.compile(r"^\s*(?:```|~~~)")
|
||||
|
||||
|
||||
@@ -72,17 +55,12 @@ def split_body_tags(body: str | None) -> tuple[list[str], list[str], str]:
|
||||
THE RULE: a line containing nothing but tags and whitespace is removed. Anything
|
||||
else is left exactly as written.
|
||||
|
||||
That is deliberately the conservative reading of "standalone". The looser one —
|
||||
also stripping a trailing tag off a prose line — was rejected because a trailing
|
||||
tag is ambiguous and the text does not say which it is: `buy milk #grocery` is
|
||||
filing, `remember to call #mom` is the sentence's object, and lifting the second
|
||||
leaves "remember to call". Mangling a sentence to save a duplicate chip is a bad
|
||||
trade. A tag sharing a line with words keeps its words.
|
||||
A trailing tag on a prose line stays: `buy milk #grocery` is filing, but
|
||||
`remember to call #mom` is the sentence's object, and the text can't tell them
|
||||
apart.
|
||||
|
||||
`standalone` tags become ORDINARY labels — the text no longer backs them, so
|
||||
nothing can derive them any more, and the way to remove one becomes the chip's ×
|
||||
rather than deleting the text. `inline` tags stay derived exactly as before. That
|
||||
is the whole meaning of `via_tag` after this change: backed by text still present.
|
||||
`standalone` tags become ordinary labels, removed with the chip's ×. `inline`
|
||||
tags stay derived (`via_tag`: backed by text still present).
|
||||
"""
|
||||
if not body:
|
||||
return [], [], body or ""
|
||||
@@ -108,8 +86,7 @@ def split_body_tags(body: str | None) -> tuple[list[str], list[str], str]:
|
||||
standalone.extend(names)
|
||||
lifted = re.sub(r"\n{3,}", "\n\n", "\n".join(kept)).strip("\n")
|
||||
if body.strip() and not lifted.strip():
|
||||
# The note was NOTHING but tags. Lifting would leave a blank card, which is a
|
||||
# worse outcome than a duplicated chip — so leave it alone and let its tags
|
||||
# The note was nothing but tags. Lifting would leave a blank card, so its tags
|
||||
# stay derived.
|
||||
return [], _dedupe(standalone + inline), body
|
||||
# A tag that ALSO appears in prose stays derived: the prose copy still backs it,
|
||||
@@ -138,24 +115,12 @@ async def _lift_and_reconcile_tags(db, note: Note) -> None:
|
||||
"""Attach the note's tag labels, LIFT its standalone tags out of the body, and
|
||||
re-derive display_title if the body moved.
|
||||
|
||||
NAMED FOR THE MUTATION. It used to be `_reconcile_tags` and only touched rows; it
|
||||
now rewrites `note.body`, and a caller that does not expect that will compute a
|
||||
display_title from text this function is about to delete.
|
||||
It rewrites `note.body`, so the title is re-derived here rather than at each
|
||||
call site, where it would be computed from text about to be deleted.
|
||||
|
||||
Which is why the lift and the re-derivation both live HERE rather than at the
|
||||
seven call sites that would each have to remember. Spreading a derived-value
|
||||
update across every place a body can be written is precisely the failure #2965
|
||||
named about label minting: "easy to miss, and it is the common one".
|
||||
|
||||
The two kinds of tag are handled differently, and that difference IS what
|
||||
`via_tag` means from here on — backed by text still in the body:
|
||||
|
||||
standalone lifted out, attached as an ORDINARY label (via_tag=False). Nothing
|
||||
derives it any more because nothing is left to derive it from, and
|
||||
the way to remove it becomes the chip's × — which both editors
|
||||
already offer for exactly this class of label.
|
||||
inline left in place, attached via_tag=True, and still detached when its
|
||||
text goes. Unchanged from before.
|
||||
standalone lifted out, attached as an ordinary label (via_tag=False), removed
|
||||
with the chip's ×.
|
||||
inline left in place, attached via_tag=True, detached when its text goes.
|
||||
"""
|
||||
standalone, inline, lifted = split_body_tags(note.body)
|
||||
standalone_ids: set = set()
|
||||
@@ -171,10 +136,9 @@ async def _lift_and_reconcile_tags(db, note: Note) -> None:
|
||||
if not r.via_tag:
|
||||
attached.add(r.label_id) # manual already: a #tag of the same name changes nothing
|
||||
elif r.label_id in standalone_ids:
|
||||
# It GRADUATED. The text backing it is about to be deleted, so the row has
|
||||
# to become the record instead. This must run BEFORE the detach below, or
|
||||
# the same row would be dropped for no longer being in the body — which is
|
||||
# the bug that makes a naive lift delete every tag it touches.
|
||||
# The text backing it is about to go, so the row becomes the record. Must
|
||||
# run before the detach below, or the row would be dropped for no longer
|
||||
# being in the body.
|
||||
r.via_tag = False
|
||||
attached.add(r.label_id)
|
||||
elif r.label_id in inline_ids:
|
||||
|
||||
@@ -1,17 +1,17 @@
|
||||
"""Password resets: an admin makes a one-hour link for an account (#5173).
|
||||
"""Password resets: an admin makes a one-hour link for an account.
|
||||
|
||||
Before this, a forgotten password needed a hand on the database (#2939 §2). This reuses what invites established: a random token, only its hash
|
||||
kept, the link shown once to the admin who hands it over.
|
||||
A random token with only its hash kept, as for invites; the link is shown once to the
|
||||
admin who hands it over.
|
||||
|
||||
Using the link sets a new password and signs the account out everywhere. Its web
|
||||
sessions end because the account's `session_epoch` moves on (see `auth`), and its
|
||||
device tokens are deleted, so each linked app has to sign in again.
|
||||
|
||||
A person can also ask for one themselves when the server can send email (#5266):
|
||||
A person can also ask for one themselves when the server can send email:
|
||||
`mail_reset_link` makes the same link and mails it to the account's address.
|
||||
|
||||
This module is the reset itself; `auth.reset_password` redeems it and the admin route
|
||||
is in `accounts_api.py`, the same split as invites and for the same reason.
|
||||
is in `accounts_api.py`.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
|
||||
+10
-24
@@ -1,27 +1,15 @@
|
||||
"""Reading what the proxies in front of this app say about a request.
|
||||
|
||||
Two headers carry information the app cannot see for itself — who the client is
|
||||
(`X-Forwarded-For`) and whether they arrived over TLS (`X-Forwarded-Proto`) — and both
|
||||
are trusted by the same rule, so the rule lives in one place. Writing it twice is
|
||||
precisely how issue 2183 happened: two places holding one decision, and only one of
|
||||
them updated.
|
||||
`X-Forwarded-For` (who the client is) and `X-Forwarded-Proto` (whether they arrived
|
||||
over TLS) are trusted by one rule, kept here.
|
||||
|
||||
## The rule
|
||||
A forwarding header grows left to right: each hop appends what it saw, so a caller's
|
||||
forged entries sit to the left of what our proxies wrote. So we count in from the
|
||||
right by the **Trusted proxy hops** setting (default 1), and a forged prefix is never
|
||||
selected.
|
||||
|
||||
A forwarding header grows LEFT to RIGHT. Each hop appends what IT saw, so the
|
||||
rightmost entries are the ones our own infrastructure wrote, and anything a caller
|
||||
sent arrives to the LEFT of those.
|
||||
|
||||
That inverts the intuitive reading. The leftmost entry is nominally "the original
|
||||
client" — and is exactly the one a caller can forge, by sending the header themselves.
|
||||
So we count in from the right by the number of proxies we actually run
|
||||
(the **Trusted proxy hops** setting, default 1), and a forged prefix can never be
|
||||
selected no matter how much of it there is.
|
||||
|
||||
Too HIGH a hop count is the dangerous direction: it starts believing entries no proxy
|
||||
of ours wrote. Too low just means several callers share a bucket. So when the header
|
||||
is shorter than configured — fewer proxies than expected — we fall back to the socket
|
||||
address rather than reaching further left.
|
||||
Too high a count is the dangerous direction: it believes entries no proxy of ours
|
||||
wrote. When the header is shorter than configured, the socket address is used.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
@@ -61,10 +49,8 @@ def client_address() -> str:
|
||||
def is_https() -> bool:
|
||||
"""Whether this request reached us over TLS — directly, or via a trusted proxy.
|
||||
|
||||
Shared by the session cookie's `Secure` flag and by HSTS, because they are the same
|
||||
question. Read with the same hop count as the address: a caller who sets
|
||||
`X-Forwarded-Proto: https` on a plain-HTTP request puts it to the left of whatever
|
||||
our proxy appended, so it is not what gets read.
|
||||
For the session cookie's `Secure` flag and HSTS. Read with the same hop count as
|
||||
the address, so a forged `X-Forwarded-Proto: https` is never what gets read.
|
||||
"""
|
||||
if not has_request_context():
|
||||
return False
|
||||
|
||||
+22
-48
@@ -4,37 +4,22 @@ The credential endpoints are rate-limited: login, register, and the native
|
||||
device-link exchange. Everything else already needs a session or a device token to
|
||||
reach, so an attacker has to get through one of these three first.
|
||||
|
||||
The one exception is the app downloads (`client_dist`): signed in, but each one is
|
||||
tens of megabytes, so one account or a leaked device token looping on it could fill
|
||||
the server's uplink (family idea #5103, practice 6).
|
||||
App downloads (`client_dist`) are throttled too: each is tens of megabytes, so a
|
||||
looping client could fill the server's uplink.
|
||||
|
||||
## Why in-process is enough here, and where that stops being true
|
||||
State is per-process, which matches the image's single hypercorn worker. With several
|
||||
workers the effective limit would multiply; the fix would be a shared store.
|
||||
|
||||
State lives in module-level dicts, so it is per-process. That is correct for how
|
||||
this image actually serves — one hypercorn worker (see the Dockerfile, and the
|
||||
same assumption the trash sweeper documents in app.py). If that ever gains
|
||||
``--workers N``, each worker would keep its own counters and the effective limit
|
||||
would multiply by N; the fix then is a shared store (the Postgres connection is
|
||||
already there), not a bigger number here.
|
||||
Every attempt counts against both the account tried and the caller's address, and
|
||||
either can refuse it:
|
||||
|
||||
## Two keys, on purpose
|
||||
- **The account** can't be forged, and stops credential stuffing against one email
|
||||
from any number of addresses.
|
||||
- **The address** bounds one source spraying many accounts. Read via `proxy.py`, so
|
||||
it is only as good as the Trusted proxy hops setting.
|
||||
|
||||
Every attempt is counted against BOTH the account being tried and the address it
|
||||
came from, and either one can refuse it:
|
||||
|
||||
- **The account** is the key that matters, and the key that cannot be forged. It
|
||||
is what stops credential stuffing against one known email, no matter how many
|
||||
addresses the attempts arrive from.
|
||||
- **The address** bounds the damage from one source spraying many accounts. It is
|
||||
read from ``X-Forwarded-For``, counting in from the RIGHT by
|
||||
the **Trusted proxy hops** setting so that only entries our own proxies wrote are
|
||||
believed — a forged header lands to the left of those and is never selected. It is
|
||||
still the weaker of the two keys, because it depends on that setting matching the
|
||||
deployment; the account key depends on nothing.
|
||||
|
||||
Counting is by failure for the sign-in routes and by attempt for registration: a
|
||||
correct password should never move someone closer to being locked out, but every
|
||||
registration is a row in the users table whether it succeeds or not.
|
||||
Sign-in counts failures, so a correct password never moves someone closer to a
|
||||
lockout; registration counts attempts.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
@@ -45,26 +30,18 @@ from collections.abc import Callable
|
||||
from .settings import live
|
||||
|
||||
|
||||
# Never let the bookkeeping become the denial of service: an attacker rotating an
|
||||
# address could otherwise mint an unbounded number of buckets. Well above any real
|
||||
# deployment's distinct-caller count, so a legitimate instance never reaches it; when
|
||||
# it is reached the oldest buckets are dropped, which at worst forgives some attempts.
|
||||
#
|
||||
# Not a setting: it protects the limiter from itself rather than the app from a
|
||||
# caller, and there is no operator judgment to apply to it.
|
||||
# A cap on tracked keys, so rotating addresses can't grow the limiter without bound.
|
||||
# Far above any real caller count; past it the oldest buckets are dropped.
|
||||
MAX_BUCKETS = 10_000
|
||||
|
||||
|
||||
class SlidingWindow:
|
||||
"""Counts events per key over a trailing window.
|
||||
|
||||
Sliding rather than a fixed window because a fixed one lets twice the limit
|
||||
through across a boundary — 10 at 14:59 and 10 at 15:00 — which for a login
|
||||
limiter is the difference between the number meaning something and not.
|
||||
Sliding, because a fixed window lets twice the limit through across a boundary.
|
||||
|
||||
The limit and window are SUPPLIERS, not values, so an admin saving a new number in
|
||||
Settings takes effect on the next attempt instead of the next deploy. They are read
|
||||
per call, which is a dict lookup — the settings cache never touches the database.
|
||||
The limit and window are suppliers, read per call from the settings cache, so a
|
||||
change in Settings applies to the next attempt.
|
||||
"""
|
||||
|
||||
def __init__(self, limit: Callable[[], int], window_s: Callable[[], float]) -> None:
|
||||
@@ -108,9 +85,8 @@ class SlidingWindow:
|
||||
self._prune(key, now).append(now)
|
||||
|
||||
def forget(self, key: str) -> None:
|
||||
"""Drop a key's history. Used after a successful sign-in, so someone who
|
||||
fumbled a password twice and then got it right starts clean rather than
|
||||
carrying those two for the next quarter of an hour."""
|
||||
"""Drop a key's history. Used after a successful sign-in, so earlier failures
|
||||
don't carry over."""
|
||||
self._hits.pop(key, None)
|
||||
|
||||
def clear(self) -> None:
|
||||
@@ -131,16 +107,14 @@ register_by_address = SlidingWindow(
|
||||
lambda: live("register_limit_per_address"), _minutes("register_window_minutes")
|
||||
)
|
||||
# Reset emails, counted per address typed in whether or not it has an account, so
|
||||
# the cap can't be used to learn which ones do (#5266).
|
||||
# the cap can't be used to learn which ones do.
|
||||
reset_mail_by_account = SlidingWindow(
|
||||
lambda: live("reset_emails_per_account"), _minutes("signin_window_minutes")
|
||||
)
|
||||
|
||||
|
||||
# App downloads, per account. Counted per REQUEST, so resuming a broken download
|
||||
# counts each piece; the setting's default leaves room for that. The window is fixed
|
||||
# at an hour because the setting's label says "per hour", and one number is easier
|
||||
# to judge than two.
|
||||
# App downloads, per account, per hour. Counted per request, so each piece of a
|
||||
# resumed download counts; the default leaves room for that.
|
||||
downloads_by_account = SlidingWindow(lambda: live("client_downloads_per_hour"), lambda: 3600.0)
|
||||
|
||||
|
||||
|
||||
@@ -4,9 +4,8 @@ import uuid
|
||||
|
||||
from quart import jsonify
|
||||
|
||||
# The app's single JSON error shape + the two guards that pair with it, so every
|
||||
# blueprint returns errors and parses path ids the same way instead of hand-building
|
||||
# `jsonify({"error": ...}), code` and try/except uuid blocks at ~35 call sites.
|
||||
# The app's JSON error shape and the two guards that pair with it, so every blueprint
|
||||
# returns errors and parses path ids the same way.
|
||||
|
||||
|
||||
def json_error(message: str, status: int):
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
"""Bring a fresh Inkwell back from a backup file (#5921).
|
||||
"""Bring a fresh Inkwell back from a backup file.
|
||||
|
||||
docker compose exec app python -m inkwell.restore /backups/inkwell-20261016T020000Z.tar
|
||||
|
||||
|
||||
+19
-38
@@ -2,21 +2,14 @@
|
||||
|
||||
Two things live here, deliberately together:
|
||||
|
||||
**`purge_note`** — the single definition of destroying a note. Three callers reach
|
||||
permanent deletion by different routes (the user's Delete forever in the web UI,
|
||||
a client's `op=delete` over sync, and the sweeper below), and if each had its own
|
||||
idea of what to tear down they would drift — one would forget the files, another
|
||||
the revision history, and "permanently deleted" would quietly mean three different
|
||||
things depending on how you got there.
|
||||
**`purge_note`** is the one definition of destroying a note, used by Delete forever
|
||||
in the web UI, a client's `op=delete` over sync, and the sweeper.
|
||||
|
||||
**The sweeper** — trash that nobody empties is not free: a trashed note keeps its
|
||||
attachment BYTES on disk for as long as it sits there. So trash expires. The window
|
||||
is the `trash_retention_days` setting (default 30, `0` = keep forever), re-read on
|
||||
every pass so a change in admin Settings takes effect without a restart.
|
||||
**The sweeper** expires trash, which still holds attachment bytes on disk. The window
|
||||
is the `trash_retention_days` setting (default 30, `0` = forever), re-read each pass.
|
||||
|
||||
A purged note is not a deleted ROW — it's a content-less tombstone. That's what lets
|
||||
an offline client that reappears next month learn the note is gone instead of
|
||||
faithfully resurrecting it on the next push.
|
||||
A purged note is a content-less tombstone, not a deleted row, so an offline client
|
||||
that comes back learns the note is gone instead of pushing it back.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
@@ -43,17 +36,14 @@ from .storage import unlink_media
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# How often the sweeper wakes. Retention is measured in days, so anything under
|
||||
# "a few times a day" buys nothing but load — a note trashed at 09:00 expiring at
|
||||
# 14:00 rather than 09:00 thirty days later is not a difference anyone can feel.
|
||||
# How often the sweeper wakes. Retention is in days, so more often only adds load.
|
||||
SWEEP_INTERVAL_SECONDS = 6 * 60 * 60
|
||||
|
||||
# Let the app finish booting (migrations, first requests) before the first sweep.
|
||||
SWEEP_STARTUP_DELAY_SECONDS = 60
|
||||
|
||||
# Rows purged per transaction. A long-neglected install could have thousands of
|
||||
# expired notes on the first sweep; committing in batches keeps that from becoming
|
||||
# one enormous transaction holding locks while it deletes files.
|
||||
# Rows purged per transaction, so a large first sweep isn't one long transaction
|
||||
# holding locks while it deletes files.
|
||||
SWEEP_BATCH = 200
|
||||
|
||||
|
||||
@@ -61,11 +51,9 @@ async def purge_note(db, note: Note, edited_at: datetime | None = None) -> None:
|
||||
"""Turn a note into a content-less tombstone: delete its children (and the
|
||||
attachment files on disk), clear its content, stamp `purged_at`.
|
||||
|
||||
The row survives on purpose — offline clients read it off the delta feed and
|
||||
learn the note is gone. Everything that carries the note's CONTENT goes, and
|
||||
that includes history: a revision row holds the full body, so leaving revisions
|
||||
behind would mean the text of a "permanently deleted" note is still on the
|
||||
server, recoverable by anyone who can read the table.
|
||||
The row survives so offline clients learn of it from the delta feed. Everything
|
||||
carrying the note's content goes, revisions included, since each holds a full
|
||||
body.
|
||||
"""
|
||||
atts = (await db.scalars(select(NoteAttachment).where(NoteAttachment.note_id == note.id))).all()
|
||||
for a in atts:
|
||||
@@ -80,12 +68,8 @@ async def purge_note(db, note: Note, edited_at: datetime | None = None) -> None:
|
||||
await db.execute(sa_delete(Share).where(Share.resource_type == "note", Share.resource_id == note.id))
|
||||
note.body = ""
|
||||
note.display_title = ""
|
||||
# `deleted_at` deliberately SURVIVES. It's still true — that is when the note was
|
||||
# deleted — and keeping it means every ordinary query, present and future, that
|
||||
# says "not trashed" (`deleted_at IS NULL`) excludes tombstones for free. Clearing
|
||||
# it would leave a content-less row looking like a perfectly normal active note,
|
||||
# and it would surface on the board as a blank card. Only the Trash view, which
|
||||
# asks for `deleted_at IS NOT NULL`, has to name `purged_at` explicitly.
|
||||
# `deleted_at` stays, so every "not trashed" query (`deleted_at IS NULL`) excludes
|
||||
# tombstones for free. Only the Trash view has to name `purged_at`.
|
||||
note.remind_at = None
|
||||
note.purged_at = datetime.now(timezone.utc)
|
||||
if edited_at is not None:
|
||||
@@ -95,9 +79,8 @@ async def purge_note(db, note: Note, edited_at: datetime | None = None) -> None:
|
||||
async def sweep_expired_trash(db, retention_days: int, *, now: datetime | None = None) -> int:
|
||||
"""Purge every note whose trash has expired. Returns how many were purged.
|
||||
|
||||
Runs across ALL owners — it's a server-wide policy, not a per-user action, and
|
||||
the sweeper has no session to scope it by (rule 47 is about honoring the ACL on
|
||||
user-initiated reads, not about exempting rows from server maintenance).
|
||||
Across all owners: a server-wide policy, not a user's read, so the ACL doesn't
|
||||
apply.
|
||||
"""
|
||||
cutoff = expired_before(now or datetime.now(timezone.utc), retention_days)
|
||||
if cutoff is None:
|
||||
@@ -110,9 +93,8 @@ async def sweep_expired_trash(db, retention_days: int, *, now: datetime | None =
|
||||
.where(
|
||||
Note.deleted_at.is_not(None),
|
||||
Note.deleted_at < cutoff,
|
||||
# Already a tombstone. Without this the purge would re-run on
|
||||
# every sweep forever, bumping sync_revision each time and
|
||||
# handing clients an endless stream of "news" about one note.
|
||||
# Already a tombstone. Purging again would bump sync_revision
|
||||
# every sweep and send clients endless changes.
|
||||
Note.purged_at.is_(None),
|
||||
)
|
||||
.order_by(Note.deleted_at)
|
||||
@@ -140,8 +122,7 @@ async def sweep_once() -> int:
|
||||
async def run_sweeper() -> None:
|
||||
"""The background loop. Started in `before_serving`, cancelled on shutdown.
|
||||
|
||||
A sweep failure (DB blip, unreadable media directory) must never take the loop
|
||||
down with it — the next pass simply finds the same expired rows and tries again.
|
||||
A failed sweep never ends the loop; the next pass tries the same rows again.
|
||||
"""
|
||||
await asyncio.sleep(SWEEP_STARTUP_DELAY_SECONDS)
|
||||
while True:
|
||||
|
||||
@@ -1,25 +1,13 @@
|
||||
"""When a body change is worth keeping a version of.
|
||||
|
||||
A note's body used to snapshot into `note_revisions` on EVERY write, which made a
|
||||
write expensive — and the clients compensated by writing as rarely as they could
|
||||
get away with, saving only when an editor closed. That is durability paying for
|
||||
version history: a crash mid-session lost everything typed, so that the revision
|
||||
list would stay tidy. The safety property is worth more than the feature it was
|
||||
subsidising.
|
||||
A body change is snapshotted only if it is the first of an editing session, so a
|
||||
client may autosave as often as it likes. A snapshot stores the body as it was
|
||||
before the edit, so each sitting yields one revision: the note as you found it.
|
||||
|
||||
The rule here breaks that trade. A body change earns a snapshot only if it is the
|
||||
first one of an editing session, so a client may write as often as it likes.
|
||||
It is a time rule rather than a protocol flag, and `sync.py` applies pushed bodies
|
||||
through the same check.
|
||||
|
||||
Session granularity falls out of the window rather than being declared. A snapshot
|
||||
stores the body as it was BEFORE the edit, so the first write of a sitting captures
|
||||
the note as you found it and every write after it inside the window adds nothing —
|
||||
one revision per sitting, with no "commit" flag for a client to send and no wire
|
||||
surface to carry it. That last part is why this is a time rule and not a protocol
|
||||
one: `sync.py` applies pushed bodies through the same check, so a client autosaving
|
||||
every second cannot make the server snapshot every second either.
|
||||
|
||||
Deliberately NOT applied to restoring a revision. That is a considered act rather
|
||||
than a keystroke, and it snapshots unconditionally so restoring is itself undoable.
|
||||
Restoring a revision always snapshots, so a restore is itself undoable.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
@@ -30,10 +18,8 @@ from sqlalchemy import select
|
||||
|
||||
from .models.note_revision import NoteRevision
|
||||
|
||||
# How long one editing session is assumed to last. A constant rather than a setting:
|
||||
# it is not a preference anyone holds, and the value only has to be longer than a
|
||||
# sitting and shorter than the gap between two of them. Promote it to the settings
|
||||
# registry (rule 25) if that ever stops being true.
|
||||
# How long one editing session is assumed to last: longer than a sitting, shorter
|
||||
# than the gap between two.
|
||||
REVISION_WINDOW_MINUTES = 10
|
||||
|
||||
|
||||
|
||||
+7
-10
@@ -1,14 +1,11 @@
|
||||
"""Credentials at rest: the SMTP password and the backup bucket's secret key (#5927).
|
||||
"""Credentials at rest: the SMTP password and the backup bucket's secret key.
|
||||
|
||||
The family's secret.Box (#5185): a stored credential is encrypted under a key the
|
||||
database does not hold, so a copy of the database (a dump, a copied volume) cannot
|
||||
read the keys to anything else.
|
||||
A stored credential is encrypted under a key the database does not hold, so a copy
|
||||
of the database (a dump, a copied volume) can't open anything else.
|
||||
|
||||
- **Where the key lives.** A random key in a file under DATA_DIR, made on first use.
|
||||
Not the signing secret: by default that is kept in the settings table (rule 25,
|
||||
no env var needed), so a key derived from it would sit beside what it seals. The
|
||||
data volume and the database volume are separate, and backups take only the
|
||||
attachment files from DATA_DIR, never this file.
|
||||
Not the signing secret, which by default sits in the settings table beside what it
|
||||
would seal. Backups take only attachment files from DATA_DIR, never this file.
|
||||
- **Per purpose, per row.** The cipher key is HKDF-derived from the file's key with
|
||||
a purpose label, and each value is bound to its setting's name, so a sealed value
|
||||
copied onto another setting does not open there.
|
||||
@@ -112,8 +109,8 @@ def seal(plain: str, owner: str) -> str:
|
||||
|
||||
def unseal(stored: str, owner: str) -> str | None:
|
||||
"""The plain value, or None when it can't be opened (key file lost or replaced,
|
||||
or the value was copied from another setting). A value stored before sealing
|
||||
existed comes back as it is; startup seals it (seal_stored_secrets)."""
|
||||
or the value was copied from another setting). A value stored in the clear comes
|
||||
back as it is; startup seals it (seal_stored_secrets)."""
|
||||
if not is_sealed(stored):
|
||||
return stored
|
||||
try:
|
||||
|
||||
@@ -14,10 +14,9 @@ def hash_password(password: str) -> str:
|
||||
return bcrypt.hashpw(password.encode("utf-8")[:_MAX_BCRYPT_BYTES], bcrypt.gensalt()).decode("utf-8")
|
||||
|
||||
|
||||
# A real bcrypt hash of a value nobody can present, used only to spend the time a
|
||||
# verification would have. Computed once at import — generating it per call would
|
||||
# cost a gensalt+hash on top of the checkpw and make the "no such user" path SLOWER
|
||||
# than the real one, which is the same oracle pointing the other way.
|
||||
# A real bcrypt hash of a value nobody can present, to spend the time a verification
|
||||
# would. Computed once at import: per call, "no such user" would be slower than a
|
||||
# real check, the same timing oracle reversed.
|
||||
_DUMMY_HASH = bcrypt.hashpw(secrets.token_bytes(32), bcrypt.gensalt())
|
||||
|
||||
|
||||
|
||||
@@ -1,8 +1,6 @@
|
||||
"""Shared JSON serializers — the single home for turning ORM rows into the dict
|
||||
shapes the API returns, so a field name or format is defined once. Each entity has a
|
||||
BASE serializer with the fields common to every context; callers layer on the
|
||||
extras their surface needs (usage count in the labels API, sync_revision/purged_at
|
||||
in sync deltas). Labels and people so far."""
|
||||
"""Shared JSON serializers for labels and people, so a field name or format is
|
||||
defined once. Callers add what their surface needs (a usage count in the labels API,
|
||||
sync_revision/purged_at in sync deltas)."""
|
||||
from __future__ import annotations
|
||||
|
||||
from .common import iso
|
||||
|
||||
+21
-32
@@ -33,11 +33,9 @@ class SettingDef:
|
||||
label: str
|
||||
description: str
|
||||
group: str
|
||||
# Ints only. Enforced server-side in validate_updates and passed to the UI so the
|
||||
# number input carries them too. These exist because several of the security
|
||||
# values have ranges where a typo is not merely wrong but dangerous — a proxy hop
|
||||
# count of 50 would trust anything a caller sent, and a sign-in limit of 0 would
|
||||
# lock every account out permanently.
|
||||
# Ints only. Enforced in validate_updates and passed to the UI's number input.
|
||||
# Some typos are dangerous: a hop count of 50 trusts anything a caller sends, and
|
||||
# a sign-in limit of 0 locks every account out.
|
||||
minimum: int | None = None
|
||||
maximum: int | None = None
|
||||
# Strings only: the values allowed, shown as a choice rather than a text field.
|
||||
@@ -46,7 +44,7 @@ class SettingDef:
|
||||
# value is shown, which reads fine for "Inkwell" and badly for "starttls".
|
||||
choice_labels: tuple[str, ...] | None = None
|
||||
# A credential. The admin API never sends its value back, only whether one is
|
||||
# set, and saving it empty leaves it as it was (#5266).
|
||||
# set, and saving it empty leaves it as it was.
|
||||
secret: bool = False
|
||||
# An http(s) address, stored without a trailing slash.
|
||||
url: bool = False
|
||||
@@ -128,10 +126,10 @@ REGISTRY: list[SettingDef] = [
|
||||
"The server contacts the linked site; private/internal addresses are always blocked.",
|
||||
"Links",
|
||||
),
|
||||
# --- Email (#5266) ---------------------------------------------------------------
|
||||
# --- Email ---------------------------------------------------------------------
|
||||
#
|
||||
# Only for password reset links so far. Off until a server, a sender and the
|
||||
# public address are all set (`mailer.mail_settings`).
|
||||
# For reset links and invites. Off until a server, a sender and the public address
|
||||
# are all set (`mailer.mail_settings`).
|
||||
SettingDef("smtp_host", "string", "", "SMTP server", "Leave empty to turn email off.", "Email"),
|
||||
SettingDef(
|
||||
"smtp_port",
|
||||
@@ -158,7 +156,7 @@ REGISTRY: list[SettingDef] = [
|
||||
"smtp_password", "string", "", "SMTP password", "Kept on the server and never shown again.", "Email", secret=True
|
||||
),
|
||||
SettingDef("smtp_from", "string", "", "From address", "Who emails come from. Defaults to the username.", "Email"),
|
||||
# --- Backups (#5918) ------------------------------------------------------------
|
||||
# --- Backups --------------------------------------------------------------------
|
||||
#
|
||||
# A dated backup file each day, to a folder or an S3-compatible bucket
|
||||
# (backup_job.py). Off until switched on, because a folder has to be mounted or a
|
||||
@@ -249,9 +247,8 @@ REGISTRY: list[SettingDef] = [
|
||||
),
|
||||
# --- Security -----------------------------------------------------------------
|
||||
#
|
||||
# Read on paths too hot for a database round trip (the credential throttle checks
|
||||
# them BEFORE opening a connection, which is the point of checking a throttle
|
||||
# before doing expensive work), so they are cached — see `live()` below.
|
||||
# Cached (see `live()` below): the credential throttle reads them before opening
|
||||
# a database connection.
|
||||
SettingDef(
|
||||
"trusted_proxy_hops",
|
||||
"int",
|
||||
@@ -370,8 +367,7 @@ def _coerce(defn: SettingDef, raw: Any) -> Any:
|
||||
n = int(raw)
|
||||
except (ValueError, TypeError):
|
||||
return defn.default
|
||||
# A value saved before its bounds existed is still in the row; read it as the
|
||||
# nearest bound rather than let it act (#5384).
|
||||
# A stored value outside its bounds is read as the nearest bound.
|
||||
if defn.minimum is not None:
|
||||
n = max(n, defn.minimum)
|
||||
if defn.maximum is not None:
|
||||
@@ -437,11 +433,10 @@ async def get_public_config(db) -> dict:
|
||||
"site_name": await get_setting(db, "site_name"),
|
||||
"allow_registration": await get_setting(db, "allow_registration"),
|
||||
"enable_url_unfurl": await get_setting(db, "enable_url_unfurl"),
|
||||
# Server policy, not user data: clients need it to say how long a note has
|
||||
# left in Trash. A native client also reads it BEFORE linking, which is why
|
||||
# it belongs on the unauthenticated config rather than behind login.
|
||||
# Clients show how long a note has left in Trash, and a native client reads it
|
||||
# before linking, so it is on the unauthenticated config.
|
||||
"trash_retention_days": await get_setting(db, "trash_retention_days"),
|
||||
# Whether the sign-in screen offers "Forgot password?" (#5266).
|
||||
# Whether the sign-in screen offers "Forgot password?".
|
||||
"password_reset_by_email": await mail_configured(db),
|
||||
"min_password_length": MIN_PASSWORD_LEN,
|
||||
}
|
||||
@@ -482,16 +477,11 @@ async def get_admin_settings(db) -> list[dict]:
|
||||
return result
|
||||
|
||||
|
||||
# Settings the app must be able to read WITHOUT awaiting a database.
|
||||
# Settings read without awaiting a database: the credential throttle needs them
|
||||
# before opening a connection, so a refused attempt costs nothing.
|
||||
#
|
||||
# The credential throttle consults these before opening a connection — deliberately,
|
||||
# because a refused attempt is supposed to cost nothing, and the proxy hop count is
|
||||
# needed to know who is even asking. A per-request query would undo both.
|
||||
#
|
||||
# Seeded from the registry defaults so the app works before (and without) a database —
|
||||
# unit tests construct it with no Postgres at all — then refreshed from the DB at boot
|
||||
# and again whenever an admin saves. Same live-update contract `session_ttl_days`
|
||||
# already has in settings_api.py.
|
||||
# Seeded from the registry defaults (unit tests run with no Postgres), then refreshed
|
||||
# from the database at boot and whenever an admin saves.
|
||||
_LIVE_KEYS = (
|
||||
# Read by GET /install.sh, which writes it into the installer it serves
|
||||
# (installer.py) and must not need a database to answer.
|
||||
@@ -516,7 +506,7 @@ def live(key: str) -> Any:
|
||||
|
||||
def apply_session_ttl(app, days) -> None:
|
||||
"""Sessions signed from now on last `days` days. At boot and on every admin save,
|
||||
so a changed lifetime applies without a restart (rule 25)."""
|
||||
so a changed lifetime applies without a restart."""
|
||||
app.config["PERMANENT_SESSION_LIFETIME"] = timedelta(days=int(days))
|
||||
|
||||
|
||||
@@ -545,9 +535,8 @@ def validate_updates(updates: dict) -> tuple[dict, str | None]:
|
||||
n = int(val)
|
||||
except (ValueError, TypeError):
|
||||
return {}, f"{defn.label} must be a whole number"
|
||||
# Rejected rather than clamped: silently accepting a number and storing a
|
||||
# different one is how somebody ends up believing a protection is set to
|
||||
# something it is not.
|
||||
# Rejected rather than clamped, so nobody believes a protection is set to
|
||||
# a number it isn't.
|
||||
if defn.minimum is not None and n < defn.minimum:
|
||||
return {}, f"{defn.label} must be at least {defn.minimum}"
|
||||
if defn.maximum is not None and n > defn.maximum:
|
||||
|
||||
@@ -46,13 +46,12 @@ async def update_settings():
|
||||
# Nothing is saved: a credential is never stored unsealed by a save.
|
||||
return json_error(f"Couldn't encrypt the password or key to store it. {exc}", 500)
|
||||
await db.commit()
|
||||
# Re-read the cached security values so a saved limit or hop count applies to
|
||||
# the very next request. Unconditional: cheap, and a conditional here would be
|
||||
# one more place that has to know which keys are hot.
|
||||
# Re-read the cached security values so a saved limit applies to the next
|
||||
# request.
|
||||
await refresh_live(db)
|
||||
result = await get_admin_settings(db)
|
||||
|
||||
# Apply the live-tunable knob without a restart (rule 25).
|
||||
# Apply the session lifetime without a restart.
|
||||
if "session_ttl_days" in clean:
|
||||
apply_session_ttl(current_app, clean["session_ttl_days"])
|
||||
|
||||
@@ -63,7 +62,7 @@ async def update_settings():
|
||||
@require_admin
|
||||
async def test_email():
|
||||
"""Send a test message to the signed-in admin with the SAVED email settings, so
|
||||
"Send test email" checks what reset links will actually use (#5266). An admin is
|
||||
"Send test email" checks what reset links will actually use. An admin is
|
||||
shown the server's own error, which is what fixing the settings takes."""
|
||||
async with session_scope() as db:
|
||||
cfg = await mail_settings(db)
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
"""What sharing a note does to the change feed (#5175).
|
||||
"""What sharing a note does to the change feed.
|
||||
|
||||
The feed carries every note a person can see, so a share changing has to show up in
|
||||
it: granting or changing a share moves the note past each device's cursor, and ending
|
||||
@@ -72,7 +72,7 @@ async def granted(db, note_id: uuid.UUID, user_id: uuid.UUID) -> None:
|
||||
await bump_note(db, note_id)
|
||||
|
||||
|
||||
# --- Group membership (#5177) ------------------------------------------------
|
||||
# --- Group membership --------------------------------------------------------
|
||||
#
|
||||
# A note shared with a group reaches whoever is in it NOW, so joining or leaving
|
||||
# changes what that one person can see without any share changing. Their devices
|
||||
|
||||
@@ -1,9 +1,8 @@
|
||||
"""Sharing a note with another member of this instance (#5174).
|
||||
"""Sharing a note with another member of this instance.
|
||||
|
||||
`acl.visible_to_user` has gated every read since M0, but nothing wrote a share. This
|
||||
is the writing half: the note's owner lists, grants and revokes access, and anyone
|
||||
signed in can read the member directory the Share dialog picks people from. A share
|
||||
goes to one person or to a group the admin made (#5177).
|
||||
The note's owner lists, grants and revokes access, and anyone signed in can read the
|
||||
member directory the Share dialog picks people from. A share goes to one person or
|
||||
to a group the admin made. Reads are gated by `acl.visible_to_user`.
|
||||
|
||||
What a share grants is in `acl` (view, or edit: the body and checklist) and is
|
||||
enforced by the note routes (`notes.helpers._get_owned` / `_get_editable`).
|
||||
@@ -50,7 +49,7 @@ async def _member_counts(db, group_ids: list) -> dict:
|
||||
@login_required
|
||||
async def directory():
|
||||
"""Who on this instance a note can be shared with: everyone but you, and every
|
||||
group the admin has made (#5177).
|
||||
group the admin has made.
|
||||
|
||||
Signed-in only. Each person is a name and an email and nothing more, which is
|
||||
what choosing someone takes and all a fellow member needs to know; a group is its
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
"""How much attachment storage an account may use (#2939 §4).
|
||||
"""How much attachment storage an account may use.
|
||||
|
||||
`max_attachment_mb` caps one file; this caps an account's total, so one account
|
||||
can't fill the volume. The limit is Settings → General → `storage_quota_gb`;
|
||||
@@ -7,11 +7,9 @@ can't fill the volume. The limit is Settings → General → `storage_quota_gb`;
|
||||
What counts is every attachment on every note the account owns, trashed notes
|
||||
included: their files stay on disk until the trash is emptied.
|
||||
|
||||
An upload that would go over the limit is answered 507 Insufficient Storage, not
|
||||
413. The native clients treat a 4xx as permanent and never retry the file
|
||||
(`client::upload_attachment` in the core), and a 5xx as worth another try. Over the
|
||||
limit is neither: freeing space should let the file through on the next sync, with
|
||||
nothing for the person to redo.
|
||||
An upload over the limit is answered 507, not 413: native clients never retry a 4xx
|
||||
(`client::upload_attachment` in the core), and freeing space should let the file
|
||||
through on the next sync.
|
||||
|
||||
The one way an attachment's file leaves the disk is here too (`unlink_media`).
|
||||
"""
|
||||
|
||||
+21
-42
@@ -1,9 +1,9 @@
|
||||
"""Delta-sync API for the local-first native clients (M8 sync hub).
|
||||
"""Delta-sync API for the local-first native clients.
|
||||
|
||||
Pull: `GET /api/sync/changes?since=<cursor>` returns every note + label the caller
|
||||
owns whose sync_revision advanced past the cursor, newest-revision last, paginated.
|
||||
With `shares=1` it also carries the notes shared with the caller, and `revoked`: the
|
||||
notes that stopped being shared with them (#5175).
|
||||
notes that stopped being shared with them.
|
||||
Notes and labels both draw from ONE shared sequence (sync_revision_seq), so the
|
||||
cursor is a single monotonic watermark across both entity types.
|
||||
|
||||
@@ -56,18 +56,14 @@ MAX_PUSH = 1000 # per-batch change cap
|
||||
# SYNC_PROTOCOL_VERSION what this server speaks.
|
||||
# MIN_CLIENT_PROTOCOL_VERSION the oldest client protocol it still accepts.
|
||||
#
|
||||
# Bump SYNC_PROTOCOL_VERSION for ANY wire change. Raise the floor only for a
|
||||
# BREAKING one: it hard-blocks older clients, so additive changes leave it alone.
|
||||
# What each version changed, and why the floor moved or didn't, is in docs/sync.md
|
||||
# ("The policy").
|
||||
# Bump SYNC_PROTOCOL_VERSION for any wire change. Raise the floor only for a breaking
|
||||
# one, since it blocks older clients. docs/sync.md ("The policy") lists each version.
|
||||
SYNC_PROTOCOL_VERSION = 7
|
||||
MIN_CLIENT_PROTOCOL_VERSION = 3
|
||||
|
||||
# Named capabilities beyond the base protocol. An ADDITIVE change earns a name
|
||||
# here rather than a min-version bump, so a newer client meeting an older server
|
||||
# can degrade to "some features unavailable" instead of refusing to sync. Clients
|
||||
# test for the name, never infer a capability from a version number — that's what
|
||||
# keeps feature gating independent of release lockstep.
|
||||
# Named capabilities beyond the base protocol. An additive change gets a name here
|
||||
# rather than a floor bump, so a newer client meeting an older server loses a feature
|
||||
# instead of refusing to sync. Clients test for the name, never a version number.
|
||||
SYNC_FEATURES: tuple[str, ...] = (
|
||||
"notes", # note delta sync (pull + push)
|
||||
"labels", # the label catalog as its own entity
|
||||
@@ -83,8 +79,8 @@ SYNC_FEATURES: tuple[str, ...] = (
|
||||
def protocol_advertisement() -> dict:
|
||||
"""What the server publishes about the sync protocol, merged into `/api/config`.
|
||||
|
||||
DB-free and unauthenticated on purpose: a client has to be able to ask "can I
|
||||
talk to you at all?" before it holds a device token — or even has an account.
|
||||
DB-free and unauthenticated: a client asks before it holds a device token or an
|
||||
account.
|
||||
"""
|
||||
return {
|
||||
"sync_protocol_version": SYNC_PROTOCOL_VERSION,
|
||||
@@ -131,9 +127,8 @@ def _page_cursor(streams: list[list[int]], since: int, limit: int) -> tuple[int,
|
||||
async def changes():
|
||||
since = _parse_since(request.args.get("since"))
|
||||
limit = _clamp_limit(request.args.get("limit"))
|
||||
# `shares=1` is a client that understands shared notes (protocol 6, `shares`).
|
||||
# Anything older gets only its own notes, as before, rather than someone else's
|
||||
# notes it would treat as its own: pinning them, labelling them, pushing them.
|
||||
# `shares=1` is a client that understands shared notes (protocol 6). Without it a
|
||||
# client gets only its own notes, since it would treat others' as its own.
|
||||
with_shares = request.args.get("shares") == "1"
|
||||
if with_shares:
|
||||
visible = note_visible()
|
||||
@@ -143,7 +138,7 @@ async def changes():
|
||||
# Every note the caller can see, in any state (active/archived/trash/purged),
|
||||
# since a client mirrors everything; ordered by the shared revision. On a note
|
||||
# shared with the caller, pinning it moves their own state row and not the
|
||||
# note, so its revision here is whichever of the two moved last (#5176).
|
||||
# note, so its revision here is whichever of the two moved last.
|
||||
revision = revision_for()
|
||||
note_rows = (
|
||||
await db.execute(
|
||||
@@ -238,14 +233,6 @@ def _assign_note_fields(note: Note, ch: dict) -> None:
|
||||
note.position = ch["position"]
|
||||
|
||||
|
||||
# `_first_item_text` and `_apply_note_items` lived here until M304. Both existed for
|
||||
# one reason — a checklist was a table beside the body — and both are gone with it. A
|
||||
# pushed change carries its items as `- [ ] ` lines inside `body`, so applying them is
|
||||
# applying the body, and naming the note is reading its first line. A client that still
|
||||
# sends an `items` array is a v2 client, and the version floor below turns it away
|
||||
# before any of this runs.
|
||||
|
||||
|
||||
async def _apply_note_manual_labels(db, note: Note, ch: dict) -> None:
|
||||
"""Set the note's MANUAL (picker) label memberships from client label_ids, leaving
|
||||
tag-sourced (via_tag) rows to _lift_and_reconcile_tags. Only labels the caller owns count."""
|
||||
@@ -307,12 +294,9 @@ async def _apply_note(db, ch: dict, previews: list[tuple[uuid.UUID, str]]) -> di
|
||||
note.purged_at = None # client re-created/edited → clear the tombstone
|
||||
|
||||
_assign_note_fields(note, ch)
|
||||
# Non-destructive LWW: the overwritten server body snapshots into history, subject
|
||||
# to the same session window as a direct edit (revisions.should_snapshot). This
|
||||
# path is why the window is a time rule rather than a flag on the wire: a client
|
||||
# autosaving every second pushes a body change every second, and without the
|
||||
# check the SERVER would snapshot each one no matter how restrained the client's
|
||||
# own store was being.
|
||||
# The overwritten server body snapshots into history under the same session window
|
||||
# as a direct edit (revisions.should_snapshot), so a client autosaving every second
|
||||
# doesn't make the server snapshot every second.
|
||||
body = ch["body"] if isinstance(ch.get("body"), str) else ""
|
||||
if await write_body(db, note, body, snapshot="never" if creating else "session"):
|
||||
# A note pushed from a linked client gets the same link previews as one typed
|
||||
@@ -327,7 +311,7 @@ async def _apply_note(db, ch: dict, previews: list[tuple[uuid.UUID, str]]) -> di
|
||||
|
||||
|
||||
async def _apply_shared_note(db, note: Note, ch: dict, edited_at, previews: list[tuple[uuid.UUID, str]]) -> dict:
|
||||
"""Apply a pushed change to a note someone else owns (#5175, #5176).
|
||||
"""Apply a pushed change to a note someone else owns.
|
||||
|
||||
Two separate things can arrive, each under its own last-write-wins:
|
||||
|
||||
@@ -436,10 +420,8 @@ _CHILD_ENTITIES = {"attachment": NoteAttachment, "preview": NoteLinkPreview}
|
||||
async def _apply_child_delete(db, ch: dict, removed_files: list[str]) -> dict:
|
||||
"""Delete one attachment or link preview the client removed.
|
||||
|
||||
Not subject to last-write-wins. LWW settles which of two versions of the SAME
|
||||
thing survives; a removal has no rival version — nothing can re-create an id once
|
||||
it is gone — so it always applies. That is what makes a deletion stick: a removal
|
||||
that lost to a newer edit of the note would come straight back on the next pull.
|
||||
Not subject to last-write-wins: nothing can re-create a removed id, so there is no
|
||||
rival version, and a removal that lost to a newer note edit would come back.
|
||||
|
||||
A row this caller cannot see answers `noop`, exactly like one that never existed,
|
||||
so the reply cannot be used to learn whether someone else's id is real.
|
||||
@@ -494,10 +476,8 @@ async def push():
|
||||
else:
|
||||
results.append({"id": ch.get("id"), "status": "rejected", "error": "unknown entity"})
|
||||
await db.commit()
|
||||
# After the batch commits, never inside it: the fetch reads each note in a
|
||||
# session of its own, and these were scheduled mid-batch until #5164, where a
|
||||
# fast fetch could look for a note that was not committed yet. Scheduled rather
|
||||
# than awaited — a push must not wait on somebody else's website.
|
||||
# After the batch commits: the fetch reads each note in its own session.
|
||||
# Scheduled, not awaited, so a push never waits on somebody else's website.
|
||||
for note_id, text in previews:
|
||||
schedule_unfurls(note_id, text)
|
||||
# Files go once their rows have committed, so a failed batch never leaves a row
|
||||
@@ -513,8 +493,7 @@ async def put_attachment(att_id: str):
|
||||
"""Upload a file a client attached while offline, under the id it chose there.
|
||||
|
||||
The body is the file's raw bytes; `note_id`, `filename` and `sha256` ride in the
|
||||
query and the type in Content-Type. Raw rather than multipart so a native client
|
||||
needs no form encoder for what is one blob and three strings.
|
||||
query and the type in Content-Type, so a native client needs no form encoder.
|
||||
|
||||
Idempotent: re-sending an id the note already holds answers 200 and changes
|
||||
nothing, which is what lets a client retry an upload whose reply it never got.
|
||||
|
||||
@@ -1,8 +1,7 @@
|
||||
"""SSRF-hardened link unfurling — fetch a URL server-side and extract an OG/meta
|
||||
preview. Dependency-free (stdlib only), matching the project's hand-rolled ethos.
|
||||
"""SSRF-hardened link unfurling: fetch a URL server-side and extract an OG/meta
|
||||
preview, with the standard library only.
|
||||
|
||||
The server fetching arbitrary user-supplied URLs is a classic SSRF surface, so the
|
||||
defenses are deliberate and layered:
|
||||
Fetching user-supplied URLs is an SSRF surface, so the defences are layered:
|
||||
- http/https only (no file://, gopher://, …).
|
||||
- Resolve the host and require EVERY resolved address to be public — reject
|
||||
private / loopback / link-local / reserved / multicast / unspecified ranges.
|
||||
@@ -11,7 +10,7 @@ defenses are deliberate and layered:
|
||||
can't slip through.
|
||||
- At most 3 redirects, each hop re-validated the same way.
|
||||
- 5s timeout, 512 KB body cap, text/html only.
|
||||
No AI — this just parses OpenGraph/Twitter/`<title>` meta tags.
|
||||
It parses OpenGraph, Twitter and `<title>` meta tags.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
|
||||
@@ -1,25 +1,13 @@
|
||||
"""Unfurling a note's URLs in the background, after the note is already saved.
|
||||
|
||||
## Why this is not done inline
|
||||
A save never waits on a fetch from a host nobody controls; the preview catches up.
|
||||
|
||||
Capture speed is the product. Unfurling is a 5-second-timeout network call to a host
|
||||
nobody controls, and a note must persist the instant someone stops typing — so the
|
||||
save returns first and the preview catches up. A person who pastes a link and closes
|
||||
the composer has already done the thing they came to do.
|
||||
It runs on the server, which sees every note from every surface. A linked desktop or
|
||||
Android client gets the preview on its next pull; an unlinked one has none until it
|
||||
links.
|
||||
|
||||
## Why it is on the server rather than in each client
|
||||
|
||||
The server sees every note that reaches it, from all three surfaces, so detection and
|
||||
fetching live in one place instead of three. A linked desktop or Android client pushes
|
||||
its note and picks the preview up on the next pull; an unlinked one has no server to
|
||||
ask and simply has no preview until it links, which is the honest consequence of being
|
||||
offline rather than a gap to paper over.
|
||||
|
||||
## What it deliberately does not do
|
||||
|
||||
Fail loudly. A preview that could not be fetched is not an error the person needs —
|
||||
the note is fine, it just has no card. The link is still in the body, still clickable,
|
||||
still searchable.
|
||||
A preview that can't be fetched fails quietly: the note just has no card, and the
|
||||
link is still in the body.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
@@ -39,10 +27,8 @@ from .unfurl import UnfurlError, unfurl
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# Matches the frontend's detector (NoteEditor.vue) so both surfaces agree on what
|
||||
# counts as a link. Trailing sentence punctuation is stripped below rather than in the
|
||||
# pattern — a URL can legitimately end in most of these characters, just not when the
|
||||
# sentence does.
|
||||
# Matches the frontend's detector (NoteEditor.vue). Trailing sentence punctuation is
|
||||
# stripped below rather than in the pattern, since a URL may end in those characters.
|
||||
_URL_RE = re.compile(r"(https?://[^\s<>\"'\])]+)")
|
||||
|
||||
# Per note, per save. A body pasted full of links should not turn into a burst of
|
||||
|
||||
+2
-3
@@ -6,9 +6,8 @@ from inkwell.models.user import User
|
||||
|
||||
def test_visible_to_user_builds_owner_or_shared_predicate():
|
||||
"""DB-free smoke test: the predicate compiles and references both the direct
|
||||
(shares) and group (group_members) grant paths. Functional row-visibility is
|
||||
verified against a live database in M1, once notes are a real shareable
|
||||
resource (family practice: DB-backed tests run against the dev image, not CI).
|
||||
(shares) and group (group_members) grant paths. Row visibility against a real
|
||||
database is in test_integration.py.
|
||||
"""
|
||||
uid = uuid.uuid4()
|
||||
# User stands in as a shareable resource purely to exercise the SQL builder.
|
||||
|
||||
+2
-5
@@ -31,11 +31,8 @@ async def test_unknown_api_route_404s(app):
|
||||
|
||||
# --- the version a running server reports ------------------------------------
|
||||
#
|
||||
# Note 3127 §5 removed version tags, so this string is the only answer to "which
|
||||
# build is this?" and nothing exists to contradict it when it is wrong. That makes
|
||||
# the FALLBACK the interesting case rather than the happy path: it used to be
|
||||
# `__version__`, so a server run from a checkout reported `0.2.0` — a real-looking
|
||||
# version naming no build anybody could obtain.
|
||||
# The only answer to "which build is this?", so the fallback matters most: it must
|
||||
# be "unknown", never a plausible-looking `__version__`.
|
||||
|
||||
|
||||
async def reported_version() -> str:
|
||||
|
||||
@@ -1,8 +1,7 @@
|
||||
"""The channel -> release tag mapping — `packaging/channel-tag.sh`.
|
||||
|
||||
A channel's release tag must never be spelled like a branch. The dev channel's tag
|
||||
was `dev`, which shadowed the `dev` branch: once a clone had fetched tags,
|
||||
`git push origin dev` failed as an ambiguous refspec (Scribe #2184).
|
||||
A channel's release tag must never be spelled like a branch: a `dev` tag shadows
|
||||
the `dev` branch, and `git push origin dev` then fails as an ambiguous refspec.
|
||||
|
||||
These run the script itself rather than reading it, so a mapping that is correct
|
||||
on the page but broken in execution still fails here.
|
||||
|
||||
+16
-34
@@ -14,28 +14,17 @@ from inkwell.client_dist import (
|
||||
)
|
||||
from inkwell.config import Config
|
||||
|
||||
# DB-free, like the rest of this suite — the test lane runs no Postgres. That is
|
||||
# why the advertisement is asserted through `advertisement()` rather than through
|
||||
# `/api/config`: the route is a one-line merge of this dict into a payload whose
|
||||
# other half needs a database, and testing it here tests the part that can be wrong.
|
||||
#
|
||||
# The routes below ARE exercised, because none opens a session: the metadata route
|
||||
# only stats files, and the download's 401 is returned before any token lookup.
|
||||
# DB-free, so the advertisement is asserted through `advertisement()` rather than
|
||||
# `/api/config`, whose other half needs a database. The routes below are exercised:
|
||||
# none opens a session.
|
||||
|
||||
PAYLOAD = b"not really a client, but the server only ever stats it"
|
||||
|
||||
# Every test that is about the MECHANISM rather than about one platform runs
|
||||
# against all of them. The bugs this module can have — a sidecar describing a
|
||||
# different build, a half-finished copy shadowing a good one — are not
|
||||
# platform-specific, and a suite that only ever exercised Android is how the other
|
||||
# four would ship untested.
|
||||
# Tests about the mechanism rather than one platform run against every platform.
|
||||
ALL_IDS = [p.id for p in PLATFORMS]
|
||||
|
||||
# `version_code` is "whatever this platform's comparator reads", and that is not one
|
||||
# type. Android's install gate compares an integer; the desktop's updater compares
|
||||
# Tauri's semver key. The tests carry both shapes for the same reason the module
|
||||
# does — a suite that only ever wrote integers would pass while every desktop
|
||||
# sidecar CI writes was being rejected.
|
||||
# `version_code` is an integer for Android and Tauri's semver key for the desktop,
|
||||
# so the tests write both shapes.
|
||||
ANDROID_CODE = 3503708
|
||||
DESKTOP_CODE = "1.0.3503707"
|
||||
|
||||
@@ -58,10 +47,8 @@ def coded(platform_id: str, value: int):
|
||||
def _empty_baked_client(tmp_path, monkeypatch):
|
||||
"""Point the baked-in copy at an empty directory.
|
||||
|
||||
In a source checkout `src/inkwell/client/` does not exist, so these tests
|
||||
would pass anyway — but only by accident of where they are run. A built image
|
||||
has real clients there, and a test that silently depends on which tree it is in
|
||||
is one that will eventually lie.
|
||||
A built image has real clients in `src/inkwell/client/`, so the tests must not
|
||||
depend on which tree they run in.
|
||||
"""
|
||||
monkeypatch.setattr(client_dist, "BAKED_ROOT", tmp_path / "baked")
|
||||
yield
|
||||
@@ -121,8 +108,7 @@ def test_the_android_names_are_the_ones_the_image_build_writes():
|
||||
with nothing checking them against each other, so a rename on one side reads as
|
||||
"no Android client" on the other rather than as an error.
|
||||
|
||||
Phones never ask for these names; they poll `/api/client/android`. The names
|
||||
changed once, with the rename to Inkwell (Scribe note 5071).
|
||||
Phones never ask for these names; they poll `/api/client/android`.
|
||||
"""
|
||||
assert BY_ID["android"].artifact == "inkwell.apk"
|
||||
assert BY_ID["android"].sidecar == "inkwell-android.json"
|
||||
@@ -139,8 +125,7 @@ def test_an_absent_client_is_no_client(platform_id):
|
||||
def test_a_server_holding_nothing_advertises_no_keys_at_all():
|
||||
"""The KEYS are missing, not null.
|
||||
|
||||
A client testing for one then gets an unambiguous answer rather than having to
|
||||
tell "this server has no client" apart from "this server predates the feature".
|
||||
A client testing for one gets an unambiguous answer.
|
||||
"""
|
||||
assert releases() == {}
|
||||
assert advertisement() == {}
|
||||
@@ -234,7 +219,7 @@ def test_a_present_client_reports_what_a_comparator_reads(platform_id):
|
||||
assert not found["url"].startswith("http")
|
||||
|
||||
|
||||
def test_the_android_payload_still_carries_every_field_it_used_to():
|
||||
def test_the_android_payload_carries_every_field_phones_read():
|
||||
"""Phones in the field parse this. Fields may be ADDED — `ClientRelease` in
|
||||
core/src/sync/client.rs is a plain serde struct and ignores what it does not
|
||||
know — but none of these may move or change meaning."""
|
||||
@@ -303,15 +288,13 @@ def test_a_dropped_in_build_beats_the_one_the_image_shipped():
|
||||
"""Someone who deliberately put a build on the volume wants that build."""
|
||||
place("android", root=client_dist.BAKED_ROOT, version_code=300)
|
||||
place("android", version_code=99)
|
||||
# Lower version and all — precedence is about intent, not about newness. An
|
||||
# operator pinning an older client is doing it on purpose.
|
||||
# Even at a lower version: dropping in an older client is deliberate.
|
||||
assert release("android")["version_code"] == 99
|
||||
|
||||
|
||||
def test_precedence_is_decided_per_platform_not_for_the_whole_set():
|
||||
"""THE trap this table introduces. Dropping in one client must not retract the
|
||||
other four — "first directory holding anything wins" would mean overriding the
|
||||
APK silently takes the desktop downloads offline."""
|
||||
"""Dropping in one client must not retract the other four, as "first directory
|
||||
holding anything wins" would."""
|
||||
for platform_id in ALL_IDS:
|
||||
place(platform_id, root=client_dist.BAKED_ROOT, version_code=300)
|
||||
place("android", version_code=99)
|
||||
@@ -326,8 +309,7 @@ def test_precedence_is_decided_per_platform_not_for_the_whole_set():
|
||||
def test_a_broken_drop_in_does_not_shadow_the_baked_copy():
|
||||
"""A half-finished copy onto the volume must not take the app offline.
|
||||
|
||||
This is the failure the copy-order advice in docs/android-distribution.md is
|
||||
about, and the server should ride it out rather than go dark.
|
||||
The failure the copy-order advice in docs/android-distribution.md is about.
|
||||
"""
|
||||
place("android", root=client_dist.BAKED_ROOT, version_code=300)
|
||||
place("android", size=999_999) # sidecar describing a different build
|
||||
@@ -424,7 +406,7 @@ async def test_a_server_without_the_appimage_404s_its_updater_manifest(app):
|
||||
assert resp.status_code == 404
|
||||
|
||||
|
||||
# --- serving the bytes (#5118, family idea #5103 practice 6) -----------------
|
||||
# --- serving the bytes ------------------------------------------------------
|
||||
#
|
||||
# `send_artifact` is called directly in a request context: the route in front of it
|
||||
# needs a signed-in account, and this suite has no database to sign one in with.
|
||||
|
||||
@@ -34,18 +34,16 @@ async def test_revoke_self_requires_auth(app):
|
||||
|
||||
|
||||
def test_revoke_self_routes_to_the_self_revoke_view(app):
|
||||
# The static rule must win over `/devices/<device_id>`: if "self" fell through to
|
||||
# the id-keyed route it would be read as a malformed UUID and answer 404. Checked
|
||||
# on the URL map, because every route behind login_required now reads the
|
||||
# account's session epoch (#5173). The view's 400 for a caller with no bearer
|
||||
# token is in test_integration.py.
|
||||
# The static rule must win over `/devices/<device_id>`, or "self" would be read as
|
||||
# a malformed UUID and 404. Checked on the URL map, because every route behind
|
||||
# login_required reads the database. The view's 400 is in test_integration.py.
|
||||
endpoint, _ = app.url_map.bind("localhost").match("/api/auth/devices/self", method="DELETE")
|
||||
assert endpoint == "auth.revoke_own_device"
|
||||
|
||||
|
||||
async def test_device_login_validates_input(app):
|
||||
# Missing credentials → 400 BEFORE any DB access, so it's checkable in the
|
||||
# DB-free unit lane (invalid-cred and success paths are operator-verified).
|
||||
# Missing credentials → 400 before any DB access, so it's checkable without a
|
||||
# database. The other paths are in test_integration.py.
|
||||
client = app.test_client()
|
||||
resp = await client.post("/api/auth/device-login", json={})
|
||||
assert resp.status_code == 400
|
||||
|
||||
@@ -76,7 +76,7 @@ def test_a_script_without_exactly_one_placeholder_is_not_served(script):
|
||||
|
||||
def test_the_public_address_setting_beats_the_request_host():
|
||||
"""Behind a reverse proxy the Host this request arrived with may be an internal
|
||||
one; Settings → Public address is what the operator says people use."""
|
||||
one; Settings → Public address is the one people use."""
|
||||
assert server_address("https://notes.example.com", "http", "inkwell:5000") == "https://notes.example.com"
|
||||
|
||||
|
||||
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user