CI & Build / Build now, or wait for Android? (push) Successful in 4s
Android / Build, or is the channel already serving this? (push) Successful in 4s
CI & Build / Python lint (push) Successful in 2s
CI & Build / Web typecheck and unit tests (push) Successful in 12s
Desktop (Tauri) / Build, or is the channel already serving this? (push) Successful in 2s
CI & Build / Python tests (push) Successful in 13s
CI & Build / integration (push) Successful in 48s
CI & Build / Build & push image (push) Skipped
Desktop (Tauri) / Web tests, clippy, Rust tests and rustfmt (push) Successful in 4m26s
Desktop (Tauri) / Windows installer (cross-compiled) (push) Successful in 3m32s
Desktop (Tauri) / Tauri desktop (Linux) (push) Successful in 4m10s
Desktop (Tauri) / Update manifest (push) Successful in 4s
Android / Kotlin + Rust (APK) (push) Successful in 12m42s
Export and Import were a link to the server and a reject("needs a server") on the
desktop. Both now run in the core with no server:
- core/src/local/portable.rs builds the same zip the server writes (notes.json,
a Markdown file per note, each attachment this device holds) and reads either
export marker or a Google Keep Takeout zip, with the server's decompression
budget and an all-or-nothing transaction. Export saves to Downloads (no new
plugin) and the sidebar says where; Import takes the archive as raw IPC bytes.
- core/testdata/portable.json pins the format for both copies: the server runs
its Keep and native readers against it (test_portable_fixture.py) and checks
its real export's keys (test_integration.py); the core runs the same cases.
- Found on the way: both importers skipped a Keep note that is only a photo as
"empty". It now imports, on the server and in the core.
- New dependency, approved: `zip` (deflate only) plus `flate2` on its pure-Rust
backend, both already in the lockfile.
The AppImage applications-menu toggle moves from Account, which the desktop
never shows, to the Sync page; the first-run prompt now says so.
errorMessage (#5236) replaces the hand-rolled `.error ?? …` / `.message ?? e`
reads at the remaining catch sites, so a desktop failure shows its real reason.
Task #5170.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
895 lines
31 KiB
Rust
895 lines
31 KiB
Rust
//! A whole store as a zip, out and back in, with no server.
|
|
//!
|
|
//! The same archive the server writes at `GET /api/notes/export` and reads at
|
|
//! `POST /api/notes/import` (`src/inkwell/notes/import_export.py`), so a backup taken
|
|
//! on one surface restores on any other. Both copies are held to
|
|
//! `core/testdata/portable.json`; change the format there first.
|
|
//!
|
|
//! - **Export:** `notes.json` (the machine format), a Markdown file per note for
|
|
//! reading, and each attachment this device holds.
|
|
//! - **Import:** an Inkwell export (either app marker) or a Google Keep Takeout zip.
|
|
//! Additive: notes are created, never matched against existing ones. Decompression
|
|
//! is capped per entry and in total, so a zip bomb fails instead of filling memory.
|
|
|
|
use std::io::{Cursor, Read, Write};
|
|
|
|
use chrono::{DateTime, SecondsFormat, Utc};
|
|
use rusqlite::{params, Connection};
|
|
use serde::Serialize;
|
|
use serde_json::{json, Map, Value};
|
|
use zip::write::SimpleFileOptions;
|
|
use zip::{CompressionMethod, ZipArchive, ZipWriter};
|
|
|
|
use super::models::{Note, NoteCreateInput};
|
|
use super::{derive, recur, store};
|
|
use crate::sync::blobs::BlobStore;
|
|
|
|
/// The `app` an export's notes.json carries. "thoughtsync" is what every export
|
|
/// written before the rename says, and those are people's backups.
|
|
const APP_MARKERS: [&str; 2] = ["inkwell", "thoughtsync"];
|
|
|
|
const MAX_ENTRIES: usize = 10_000;
|
|
const MAX_ENTRY_BYTES: u64 = 64 * 1024 * 1024;
|
|
const MAX_TOTAL_BYTES: u64 = 512 * 1024 * 1024;
|
|
|
|
/// What an import did, in the server's words.
|
|
#[derive(Debug, Clone, Serialize, PartialEq)]
|
|
pub struct ImportSummary {
|
|
/// "inkwell" or "keep".
|
|
pub source: String,
|
|
pub imported: usize,
|
|
pub skipped: usize,
|
|
}
|
|
|
|
/// One note as either importer reads it, before anything is written. Mirrors the
|
|
/// server's "common import spec".
|
|
#[derive(Debug, Clone, Default, PartialEq)]
|
|
pub struct ImportSpec {
|
|
pub title: Option<String>,
|
|
pub body: String,
|
|
pub pinned: bool,
|
|
pub archived: bool,
|
|
pub trashed: bool,
|
|
pub remind_at: Option<String>,
|
|
pub recurrence: Option<String>,
|
|
pub created_at: Option<String>,
|
|
pub updated_at: Option<String>,
|
|
pub labels: Vec<String>,
|
|
pub items: Vec<(String, bool)>,
|
|
/// (path inside the zip, mime if the source said)
|
|
pub attachments: Vec<(String, Option<String>)>,
|
|
}
|
|
|
|
// ---- export -----------------------------------------------------------------
|
|
|
|
/// The export's file name without `.zip`, dated like the server's:
|
|
/// `inkwell-export-YYYYMMDD`.
|
|
pub fn export_stem() -> String {
|
|
format!("inkwell-export-{}", Utc::now().format("%Y%m%d"))
|
|
}
|
|
|
|
/// Every live note (not trashed) as an export archive.
|
|
pub fn export_zip(conn: &Connection, blobs: &BlobStore) -> Result<Vec<u8>, String> {
|
|
let ids: Vec<String> = {
|
|
let mut stmt = conn
|
|
.prepare("SELECT id FROM notes WHERE trashed = 0 ORDER BY created_at")
|
|
.map_err(|e| e.to_string())?;
|
|
let rows = stmt
|
|
.query_map([], |r| r.get(0))
|
|
.map_err(|e| e.to_string())?;
|
|
rows.collect::<rusqlite::Result<_>>()
|
|
.map_err(|e| e.to_string())?
|
|
};
|
|
let labels: Vec<Value> = store::list_labels(conn)
|
|
.map_err(|e| e.to_string())?
|
|
.into_iter()
|
|
.map(|l| json!({ "name": l.name, "color": l.color }))
|
|
.collect();
|
|
|
|
let mut zip = ZipWriter::new(Cursor::new(Vec::new()));
|
|
let opts = SimpleFileOptions::default().compression_method(CompressionMethod::Deflated);
|
|
let mut notes = Vec::with_capacity(ids.len());
|
|
for id in &ids {
|
|
let note = store::get_note(conn, id).map_err(|e| e.to_string())?;
|
|
let short: String = note.id.chars().take(8).collect();
|
|
|
|
let mut files = Vec::new();
|
|
for (i, att) in note.attachments.iter().enumerate() {
|
|
// A file this device hasn't downloaded yet is left out of the list as well
|
|
// as the archive, so an importer never goes looking for it.
|
|
let Some(bytes) = att.sha256.as_deref().and_then(|sha| blobs.read(sha)) else {
|
|
continue;
|
|
};
|
|
// A folder per attachment, so two files with one name in one note don't
|
|
// collide, and the importer still gets the real name from the basename.
|
|
let name = store::safe_filename(att.filename.as_deref().unwrap_or(""));
|
|
let path = format!("attachments/{short}/{i}/{name}");
|
|
write_entry(&mut zip, &path, &bytes, opts)?;
|
|
files.push(json!({ "file": path, "mime": att.mime }));
|
|
}
|
|
|
|
let md = format!("notes/{}-{short}.md", slugify(¬e.display_title));
|
|
write_entry(&mut zip, &md, note_markdown(¬e).as_bytes(), opts)?;
|
|
notes.push(json!({
|
|
"id": note.id,
|
|
"display_title": note.display_title,
|
|
"body": note.body,
|
|
"pinned": note.pinned,
|
|
"archived": note.archived,
|
|
"remind_at": note.remind_at,
|
|
"recurrence": note.recurrence,
|
|
"created_at": note.created_at,
|
|
"updated_at": note.updated_at,
|
|
"labels": note.labels.iter().map(|l| l.name.clone()).collect::<Vec<_>>(),
|
|
"attachments": files,
|
|
}));
|
|
}
|
|
|
|
let doc = json!({
|
|
"app": "inkwell",
|
|
"version": 1,
|
|
"exported_at": Utc::now().to_rfc3339_opts(SecondsFormat::Millis, true),
|
|
"labels": labels,
|
|
"notes": notes,
|
|
});
|
|
let text = serde_json::to_string_pretty(&doc).map_err(|e| e.to_string())?;
|
|
write_entry(&mut zip, "notes.json", text.as_bytes(), opts)?;
|
|
zip.finish()
|
|
.map(Cursor::into_inner)
|
|
.map_err(|e| e.to_string())
|
|
}
|
|
|
|
fn write_entry(
|
|
zip: &mut ZipWriter<Cursor<Vec<u8>>>,
|
|
path: &str,
|
|
bytes: &[u8],
|
|
opts: SimpleFileOptions,
|
|
) -> Result<(), String> {
|
|
zip.start_file(path, opts).map_err(|e| e.to_string())?;
|
|
zip.write_all(bytes).map_err(|e| e.to_string())
|
|
}
|
|
|
|
/// One note as a readable Markdown file with a small frontmatter block. notes.json is
|
|
/// the machine format; this is for a person opening the archive. Mirrors
|
|
/// `_note_markdown` on the server.
|
|
pub fn note_markdown(note: &Note) -> String {
|
|
let mut out = vec![
|
|
"---".to_string(),
|
|
format!("display_name: {}", note.display_title),
|
|
];
|
|
if !note.labels.is_empty() {
|
|
let names: Vec<&str> = note.labels.iter().map(|l| l.name.as_str()).collect();
|
|
out.push(format!("labels: [{}]", names.join(", ")));
|
|
}
|
|
if note.pinned {
|
|
out.push("pinned: true".into());
|
|
}
|
|
if note.archived {
|
|
out.push("archived: true".into());
|
|
}
|
|
if let Some(at) = ¬e.remind_at {
|
|
out.push(format!("remind_at: {at}"));
|
|
}
|
|
out.push(format!(
|
|
"created: {}",
|
|
note.created_at.as_deref().unwrap_or("")
|
|
));
|
|
out.push(format!(
|
|
"updated: {}",
|
|
note.updated_at.as_deref().unwrap_or("")
|
|
));
|
|
out.push("---".into());
|
|
out.push(String::new());
|
|
if !note.body.is_empty() {
|
|
out.push(note.body.clone());
|
|
}
|
|
out.join("\n") + "\n"
|
|
}
|
|
|
|
/// A filesystem-safe slug from a note's name, for its .md file. Mirrors `_slugify`:
|
|
/// keep word characters, spaces and hyphens; collapse runs of space, underscore and
|
|
/// hyphen into one hyphen; at most 60 characters; "note" when nothing is left.
|
|
pub fn slugify(text: &str) -> String {
|
|
let kept: String = text
|
|
.trim()
|
|
.to_lowercase()
|
|
.chars()
|
|
.filter(|c| c.is_alphanumeric() || *c == '_' || *c == '-' || c.is_whitespace())
|
|
.collect();
|
|
let mut slug = String::new();
|
|
let mut gap = false;
|
|
for c in kept.chars() {
|
|
if c.is_whitespace() || c == '_' || c == '-' {
|
|
gap = true;
|
|
} else {
|
|
if gap && !slug.is_empty() {
|
|
slug.push('-');
|
|
}
|
|
gap = false;
|
|
slug.push(c);
|
|
}
|
|
}
|
|
let slug: String = slug.chars().take(60).collect();
|
|
let slug = slug.trim_end_matches('-').to_string();
|
|
if slug.is_empty() {
|
|
"note".into()
|
|
} else {
|
|
slug
|
|
}
|
|
}
|
|
|
|
// ---- import -----------------------------------------------------------------
|
|
|
|
/// Bytes pulled out of the archive, capped per entry and across the whole import.
|
|
/// Each read stops one byte past what is left, so an oversized or size-lying entry
|
|
/// is caught mid-read rather than after it has been inflated.
|
|
struct Budget {
|
|
remaining: u64,
|
|
}
|
|
|
|
impl Budget {
|
|
fn read(
|
|
&mut self,
|
|
zip: &mut ZipArchive<Cursor<&[u8]>>,
|
|
name: &str,
|
|
) -> Result<Option<Vec<u8>>, String> {
|
|
let cap = MAX_ENTRY_BYTES.min(self.remaining);
|
|
let entry = match zip.by_name(name) {
|
|
Ok(entry) => entry,
|
|
Err(zip::result::ZipError::FileNotFound) => return Ok(None),
|
|
Err(e) => return Err(e.to_string()),
|
|
};
|
|
let mut data = Vec::new();
|
|
entry
|
|
.take(cap + 1)
|
|
.read_to_end(&mut data)
|
|
.map_err(|e| e.to_string())?;
|
|
if data.len() as u64 > cap {
|
|
return Err(TOO_LARGE.into());
|
|
}
|
|
self.remaining -= data.len() as u64;
|
|
Ok(Some(data))
|
|
}
|
|
}
|
|
|
|
const TOO_LARGE: &str = "that archive is too large to import";
|
|
|
|
/// Import an Inkwell export or a Google Keep Takeout zip into this store.
|
|
///
|
|
/// All or nothing for the notes: any failure rolls the whole import back. Attachment
|
|
/// bytes already written to the blob store stay there, which is harmless — they are
|
|
/// keyed by content and nothing points at them.
|
|
pub fn import_zip(
|
|
conn: &Connection,
|
|
blobs: &BlobStore,
|
|
bytes: &[u8],
|
|
) -> Result<ImportSummary, String> {
|
|
let mut zip = ZipArchive::new(Cursor::new(bytes))
|
|
.map_err(|_| "that file isn't a valid .zip archive".to_string())?;
|
|
if zip.len() > MAX_ENTRIES {
|
|
return Err("that archive has too many files to import".into());
|
|
}
|
|
let mut budget = Budget {
|
|
remaining: MAX_TOTAL_BYTES,
|
|
};
|
|
let (specs, source) = read_specs(&mut zip, &mut budget)?;
|
|
if specs.is_empty() {
|
|
return Err(
|
|
"no importable notes found — expected an Inkwell export or a Google Keep Takeout zip"
|
|
.into(),
|
|
);
|
|
}
|
|
|
|
let tx = conn.unchecked_transaction().map_err(|e| e.to_string())?;
|
|
let mut summary = ImportSummary {
|
|
source,
|
|
imported: 0,
|
|
skipped: 0,
|
|
};
|
|
for spec in &specs {
|
|
if create_imported(&tx, blobs, spec, &mut zip, &mut budget)? {
|
|
summary.imported += 1;
|
|
} else {
|
|
summary.skipped += 1;
|
|
}
|
|
}
|
|
tx.commit().map_err(|e| e.to_string())?;
|
|
Ok(summary)
|
|
}
|
|
|
|
fn read_specs(
|
|
zip: &mut ZipArchive<Cursor<&[u8]>>,
|
|
budget: &mut Budget,
|
|
) -> Result<(Vec<ImportSpec>, String), String> {
|
|
let names: Vec<String> = zip.file_names().map(str::to_string).collect();
|
|
for name in names.iter().filter(|n| basename(n) == "notes.json") {
|
|
let Some(raw) = budget.read(zip, name)? else {
|
|
continue;
|
|
};
|
|
let Ok(doc) = serde_json::from_slice::<Value>(&raw) else {
|
|
continue;
|
|
};
|
|
let marker = doc.get("app").and_then(Value::as_str).unwrap_or("");
|
|
if APP_MARKERS.contains(&marker) {
|
|
let specs = doc
|
|
.get("notes")
|
|
.and_then(Value::as_array)
|
|
.map(|notes| {
|
|
notes
|
|
.iter()
|
|
.filter_map(Value::as_object)
|
|
.map(native_spec)
|
|
.collect()
|
|
})
|
|
.unwrap_or_default();
|
|
return Ok((specs, "inkwell".into()));
|
|
}
|
|
}
|
|
|
|
const KEEP_KEYS: [&str; 6] = [
|
|
"textContent",
|
|
"listContent",
|
|
"isPinned",
|
|
"isArchived",
|
|
"isTrashed",
|
|
"userEditedTimestampUsec",
|
|
];
|
|
let mut specs = Vec::new();
|
|
for name in &names {
|
|
if !name.to_lowercase().ends_with(".json") || basename(name) == "notes.json" {
|
|
continue;
|
|
}
|
|
let Some(raw) = budget.read(zip, name)? else {
|
|
continue;
|
|
};
|
|
let Ok(Value::Object(note)) = serde_json::from_slice::<Value>(&raw) else {
|
|
continue;
|
|
};
|
|
if KEEP_KEYS.iter().any(|k| note.contains_key(*k)) {
|
|
specs.push(keep_spec(¬e, dirname(name)));
|
|
}
|
|
}
|
|
let source = if specs.is_empty() { "" } else { "keep" };
|
|
Ok((specs, source.into()))
|
|
}
|
|
|
|
/// One note from an Inkwell export's notes.json. Mirrors `_native_spec`.
|
|
pub fn native_spec(n: &Map<String, Value>) -> ImportSpec {
|
|
ImportSpec {
|
|
title: str_of(n, "title"),
|
|
body: str_of(n, "body").unwrap_or_default(),
|
|
pinned: bool_of(n, "pinned"),
|
|
archived: bool_of(n, "archived"),
|
|
trashed: false, // an export carries only live notes
|
|
remind_at: str_of(n, "remind_at").and_then(|s| instant(&s)),
|
|
recurrence: recur::normalize(n.get("recurrence").and_then(Value::as_str))
|
|
.map(str::to_string),
|
|
created_at: str_of(n, "created_at").and_then(|s| instant(&s)),
|
|
updated_at: str_of(n, "updated_at").and_then(|s| instant(&s)),
|
|
labels: strings(n.get("labels")),
|
|
items: objects(n.get("items"))
|
|
.map(|it| {
|
|
(
|
|
str_of(it, "text").unwrap_or_default(),
|
|
bool_of(it, "checked"),
|
|
)
|
|
})
|
|
.collect(),
|
|
attachments: objects(n.get("attachments"))
|
|
.filter_map(|a| {
|
|
Some((
|
|
str_of(a, "file").filter(|f| !f.is_empty())?,
|
|
str_of(a, "mime"),
|
|
))
|
|
})
|
|
.collect(),
|
|
}
|
|
}
|
|
|
|
/// One Google Keep note (a Takeout `<note>.json`). `dir` is the folder the JSON sits
|
|
/// in, which its attachment paths are relative to. Mirrors `_keep_spec`.
|
|
pub fn keep_spec(k: &Map<String, Value>, dir: &str) -> ImportSpec {
|
|
// Keep stores links separately from the text; fold them in so they survive.
|
|
let mut body = str_of(k, "textContent").unwrap_or_default();
|
|
let urls: Vec<String> = objects(k.get("annotations"))
|
|
.filter_map(|a| str_of(a, "url"))
|
|
.filter(|u| !u.is_empty() && !body.contains(u.as_str()))
|
|
.collect();
|
|
if !urls.is_empty() {
|
|
let extra = urls.join("\n");
|
|
body = if body.trim().is_empty() {
|
|
extra
|
|
} else {
|
|
format!("{body}\n\n{extra}")
|
|
};
|
|
}
|
|
|
|
let attachments = objects(k.get("attachments"))
|
|
.filter_map(|a| {
|
|
let fp = str_of(a, "filePath").filter(|p| !p.is_empty())?;
|
|
let file = if dir.is_empty() {
|
|
fp.clone()
|
|
} else {
|
|
format!("{dir}/{fp}")
|
|
};
|
|
let mime = str_of(a, "mimetype").or_else(|| mime_from_name(&fp).map(str::to_string));
|
|
Some((file, mime))
|
|
})
|
|
.collect();
|
|
|
|
ImportSpec {
|
|
title: str_of(k, "title"),
|
|
body,
|
|
pinned: bool_of(k, "isPinned"),
|
|
archived: bool_of(k, "isArchived"),
|
|
trashed: bool_of(k, "isTrashed"),
|
|
remind_at: None, // Keep's reminders aren't in its Takeout JSON
|
|
recurrence: None,
|
|
created_at: k.get("createdTimestampUsec").and_then(usec_instant),
|
|
updated_at: k.get("userEditedTimestampUsec").and_then(usec_instant),
|
|
labels: objects(k.get("labels"))
|
|
.filter_map(|l| str_of(l, "name"))
|
|
.collect(),
|
|
items: objects(k.get("listContent"))
|
|
.map(|li| {
|
|
(
|
|
str_of(li, "text").unwrap_or_default(),
|
|
bool_of(li, "isChecked"),
|
|
)
|
|
})
|
|
.collect(),
|
|
attachments,
|
|
}
|
|
}
|
|
|
|
/// Write one imported note. False, with nothing written, when there is nothing in it.
|
|
fn create_imported(
|
|
conn: &Connection,
|
|
blobs: &BlobStore,
|
|
spec: &ImportSpec,
|
|
zip: &mut ZipArchive<Cursor<&[u8]>>,
|
|
budget: &mut Budget,
|
|
) -> Result<bool, String> {
|
|
// An imported title becomes the first body line — Inkwell has no title field, and
|
|
// dropping it would lose text someone wrote. Skipped when the body already opens
|
|
// with it, so re-importing an export doesn't stack duplicates.
|
|
let mut body = spec.body.clone();
|
|
let title = spec.title.as_deref().unwrap_or("").trim();
|
|
let first_line = body.trim_start().split('\n').next().unwrap_or("").trim();
|
|
if !title.is_empty() && first_line != title {
|
|
body = if body.trim().is_empty() {
|
|
title.to_string()
|
|
} else {
|
|
format!("{title}\n{body}")
|
|
};
|
|
}
|
|
let items: Vec<(&str, bool)> = spec
|
|
.items
|
|
.iter()
|
|
.map(|(t, c)| (t.trim(), *c))
|
|
.filter(|(t, _)| !t.is_empty())
|
|
.collect();
|
|
// A note that is only a photo is still a note — Keep has plenty of them.
|
|
if body.trim().is_empty() && items.is_empty() && spec.attachments.is_empty() {
|
|
return Ok(false);
|
|
}
|
|
for (text, checked) in &items {
|
|
body = derive::append_item(&body, text, *checked);
|
|
}
|
|
|
|
let err = |e: rusqlite::Error| e.to_string();
|
|
let note = store::create_note(conn, &NoteCreateInput { body, items: None }).map_err(err)?;
|
|
for name in spec
|
|
.labels
|
|
.iter()
|
|
.map(|n| n.trim())
|
|
.filter(|n| !n.is_empty())
|
|
{
|
|
let label = store::create_label(conn, name).map_err(err)?;
|
|
conn.execute(
|
|
"INSERT OR IGNORE INTO note_labels (note_id, label_id, via_tag) VALUES (?1, ?2, 0)",
|
|
params![note.id, label.id],
|
|
)
|
|
.map_err(err)?;
|
|
}
|
|
for (file, mime) in &spec.attachments {
|
|
let Some(raw) = budget.read(zip, file)? else {
|
|
continue;
|
|
};
|
|
store::add_attachment(
|
|
conn,
|
|
blobs,
|
|
¬e.id,
|
|
basename(file),
|
|
mime.as_deref().unwrap_or(""),
|
|
&raw,
|
|
)?;
|
|
}
|
|
|
|
// Last, because adding an attachment touches the note: the source's own times are
|
|
// what this note should carry, not the moment it was imported.
|
|
let trashed_at = spec
|
|
.trashed
|
|
.then(|| Utc::now().to_rfc3339_opts(SecondsFormat::Millis, true));
|
|
conn.execute(
|
|
"UPDATE notes SET pinned = ?1, archived = ?2, remind_at = ?3, recurrence = ?4,
|
|
trashed = ?5, trashed_at = ?6,
|
|
created_at = COALESCE(?7, created_at), updated_at = COALESCE(?8, updated_at)
|
|
WHERE id = ?9",
|
|
params![
|
|
spec.pinned,
|
|
spec.archived,
|
|
spec.remind_at,
|
|
spec.recurrence,
|
|
spec.trashed,
|
|
trashed_at,
|
|
spec.created_at,
|
|
spec.updated_at,
|
|
note.id
|
|
],
|
|
)
|
|
.map_err(err)?;
|
|
Ok(true)
|
|
}
|
|
|
|
// ---- small readers --------------------------------------------------------------
|
|
|
|
fn str_of(m: &Map<String, Value>, key: &str) -> Option<String> {
|
|
m.get(key).and_then(Value::as_str).map(str::to_string)
|
|
}
|
|
|
|
fn bool_of(m: &Map<String, Value>, key: &str) -> bool {
|
|
m.get(key).and_then(Value::as_bool).unwrap_or(false)
|
|
}
|
|
|
|
fn objects(v: Option<&Value>) -> impl Iterator<Item = &Map<String, Value>> {
|
|
v.and_then(Value::as_array)
|
|
.into_iter()
|
|
.flatten()
|
|
.filter_map(Value::as_object)
|
|
}
|
|
|
|
fn strings(v: Option<&Value>) -> Vec<String> {
|
|
v.and_then(Value::as_array)
|
|
.into_iter()
|
|
.flatten()
|
|
.filter_map(Value::as_str)
|
|
.map(str::to_string)
|
|
.collect()
|
|
}
|
|
|
|
/// Any RFC 3339 instant, in the store's own form (UTC, milliseconds, `Z`). Anything
|
|
/// else is treated as absent, as the server's `parse_dt` does.
|
|
fn instant(raw: &str) -> Option<String> {
|
|
DateTime::parse_from_rfc3339(raw).ok().map(|t| {
|
|
t.with_timezone(&Utc)
|
|
.to_rfc3339_opts(SecondsFormat::Millis, true)
|
|
})
|
|
}
|
|
|
|
/// Keep's timestamps: integer microseconds since the epoch.
|
|
fn usec_instant(v: &Value) -> Option<String> {
|
|
let usec = v.as_i64().or_else(|| v.as_str()?.parse().ok())?;
|
|
DateTime::from_timestamp_micros(usec).map(|t| t.to_rfc3339_opts(SecondsFormat::Millis, true))
|
|
}
|
|
|
|
/// The image types the server infers from a name when the source gave no mime.
|
|
fn mime_from_name(name: &str) -> Option<&'static str> {
|
|
let ext = name.rsplit_once('.')?.1.to_ascii_lowercase();
|
|
match ext.as_str() {
|
|
"png" => Some("image/png"),
|
|
"jpg" | "jpeg" => Some("image/jpeg"),
|
|
"gif" => Some("image/gif"),
|
|
"webp" => Some("image/webp"),
|
|
_ => None,
|
|
}
|
|
}
|
|
|
|
fn basename(path: &str) -> &str {
|
|
path.rsplit('/').next().unwrap_or(path)
|
|
}
|
|
|
|
fn dirname(path: &str) -> &str {
|
|
path.rsplit_once('/').map(|(d, _)| d).unwrap_or("")
|
|
}
|
|
|
|
#[cfg(test)]
|
|
mod tests {
|
|
use super::*;
|
|
use crate::local::schema;
|
|
|
|
fn store() -> (Connection, BlobStore, std::path::PathBuf) {
|
|
let conn = Connection::open_in_memory().unwrap();
|
|
schema::migrate(&conn).unwrap();
|
|
let dir = std::env::temp_dir().join(format!("inkwell-portable-{}", uuid::Uuid::new_v4()));
|
|
let blobs = BlobStore::new(dir.clone()).unwrap();
|
|
(conn, blobs, dir)
|
|
}
|
|
|
|
fn all(conn: &Connection) -> Vec<Note> {
|
|
let q = super::super::models::ListQuery {
|
|
view: "notes".into(),
|
|
label_id: None,
|
|
facets: None,
|
|
sort: None,
|
|
};
|
|
store::list_notes(conn, &q).unwrap()
|
|
}
|
|
|
|
fn fixture() -> Value {
|
|
serde_json::from_str(include_str!("../../testdata/portable.json"))
|
|
.expect("portable.json parses")
|
|
}
|
|
|
|
/// The fixture's expected spec against ours, field by field, timestamps as instants.
|
|
fn assert_spec(got: &ImportSpec, want: &Value, about: &str) {
|
|
let s = |k: &str| want.get(k).and_then(Value::as_str).map(str::to_string);
|
|
let t = |k: &str| s(k).map(|v| DateTime::parse_from_rfc3339(&v).unwrap());
|
|
let ours = |v: &Option<String>| {
|
|
v.as_deref()
|
|
.map(|x| DateTime::parse_from_rfc3339(x).unwrap())
|
|
};
|
|
assert_eq!(got.title, s("title"), "{about}: title");
|
|
assert_eq!(Some(got.body.clone()), s("body"), "{about}: body");
|
|
assert_eq!(
|
|
Some(got.pinned),
|
|
want["pinned"].as_bool(),
|
|
"{about}: pinned"
|
|
);
|
|
assert_eq!(
|
|
Some(got.archived),
|
|
want["archived"].as_bool(),
|
|
"{about}: archived"
|
|
);
|
|
assert_eq!(
|
|
Some(got.trashed),
|
|
want["trashed"].as_bool(),
|
|
"{about}: trashed"
|
|
);
|
|
assert_eq!(got.labels, strings(want.get("labels")), "{about}: labels");
|
|
let items: Vec<(String, bool)> = objects(want.get("items"))
|
|
.map(|i| (str_of(i, "text").unwrap(), bool_of(i, "checked")))
|
|
.collect();
|
|
assert_eq!(got.items, items, "{about}: items");
|
|
let atts: Vec<(String, Option<String>)> = objects(want.get("attachments"))
|
|
.map(|a| (str_of(a, "file").unwrap(), str_of(a, "mime")))
|
|
.collect();
|
|
assert_eq!(got.attachments, atts, "{about}: attachments");
|
|
assert_eq!(
|
|
ours(&got.created_at),
|
|
t("created_at"),
|
|
"{about}: created_at"
|
|
);
|
|
assert_eq!(
|
|
ours(&got.updated_at),
|
|
t("updated_at"),
|
|
"{about}: updated_at"
|
|
);
|
|
if want.get("remind_at").is_some() {
|
|
assert_eq!(ours(&got.remind_at), t("remind_at"), "{about}: remind_at");
|
|
}
|
|
if want.get("recurrence").is_some() {
|
|
assert_eq!(got.recurrence, s("recurrence"), "{about}: recurrence");
|
|
}
|
|
}
|
|
|
|
#[test]
|
|
fn keep_notes_read_as_the_fixture_says() {
|
|
let cases = fixture()["keep"].as_array().unwrap().clone();
|
|
assert!(!cases.is_empty());
|
|
for case in &cases {
|
|
let note = case["note"].as_object().unwrap();
|
|
let got = keep_spec(note, case["dir"].as_str().unwrap());
|
|
assert_spec(&got, &case["spec"], case["about"].as_str().unwrap());
|
|
}
|
|
}
|
|
|
|
#[test]
|
|
fn native_notes_read_as_the_fixture_says() {
|
|
let cases = fixture()["native"].as_array().unwrap().clone();
|
|
assert!(!cases.is_empty());
|
|
for case in &cases {
|
|
let got = native_spec(case["note"].as_object().unwrap());
|
|
assert_spec(&got, &case["spec"], case["about"].as_str().unwrap());
|
|
}
|
|
}
|
|
|
|
fn zip_of(files: &[(&str, Vec<u8>)]) -> Vec<u8> {
|
|
let mut zip = ZipWriter::new(Cursor::new(Vec::new()));
|
|
let opts = SimpleFileOptions::default().compression_method(CompressionMethod::Deflated);
|
|
for (name, bytes) in files {
|
|
write_entry(&mut zip, name, bytes, opts).unwrap();
|
|
}
|
|
zip.finish().unwrap().into_inner()
|
|
}
|
|
|
|
fn sorted_keys(v: &Value) -> Vec<String> {
|
|
let mut keys: Vec<String> = v.as_object().unwrap().keys().cloned().collect();
|
|
keys.sort();
|
|
keys
|
|
}
|
|
|
|
#[test]
|
|
fn an_export_writes_the_fixture_keys_and_imports_back_whole() {
|
|
let (conn, blobs, dir) = store();
|
|
let note = store::create_note(
|
|
&conn,
|
|
&NoteCreateInput {
|
|
body: "Trip #travel\n\n- [ ] passport".into(),
|
|
items: None,
|
|
},
|
|
)
|
|
.unwrap();
|
|
store::update_note(&conn, ¬e.id, &json!({ "pinned": true, "recurrence": "weekly", "remind_at": "2026-11-01T09:00:00.000Z" })).unwrap();
|
|
store::add_attachment(
|
|
&conn,
|
|
&blobs,
|
|
¬e.id,
|
|
"ticket.pdf",
|
|
"application/pdf",
|
|
b"%PDF-1",
|
|
)
|
|
.unwrap();
|
|
let trashed = store::create_note(
|
|
&conn,
|
|
&NoteCreateInput {
|
|
body: "gone".into(),
|
|
items: None,
|
|
},
|
|
)
|
|
.unwrap();
|
|
store::trash(&conn, &trashed.id).unwrap();
|
|
|
|
let archive = export_zip(&conn, &blobs).unwrap();
|
|
|
|
// The shape, against the keys the server's export is held to as well.
|
|
let keys = &fixture()["export"];
|
|
let mut zip = ZipArchive::new(Cursor::new(archive.as_slice())).unwrap();
|
|
let mut raw = String::new();
|
|
zip.by_name("notes.json")
|
|
.unwrap()
|
|
.read_to_string(&mut raw)
|
|
.unwrap();
|
|
let doc: Value = serde_json::from_str(&raw).unwrap();
|
|
let want = |k: &str| {
|
|
let mut v = strings(keys.get(k));
|
|
v.sort();
|
|
v
|
|
};
|
|
assert_eq!(sorted_keys(&doc), want("document"));
|
|
assert_eq!(doc["app"], "inkwell");
|
|
let notes = doc["notes"].as_array().unwrap();
|
|
assert_eq!(notes.len(), 1, "the trashed note is not exported");
|
|
assert_eq!(sorted_keys(¬es[0]), want("note"));
|
|
assert_eq!(sorted_keys(&doc["labels"][0]), want("label"));
|
|
let att = ¬es[0]["attachments"][0];
|
|
assert_eq!(sorted_keys(att), want("attachment"));
|
|
let mut bytes = Vec::new();
|
|
zip.by_name(att["file"].as_str().unwrap())
|
|
.unwrap()
|
|
.read_to_end(&mut bytes)
|
|
.unwrap();
|
|
assert_eq!(bytes, b"%PDF-1");
|
|
assert!(zip
|
|
.file_names()
|
|
.any(|n| n.starts_with("notes/trip-travel-") && n.ends_with(".md")));
|
|
|
|
// And back into an empty store, whole.
|
|
let (fresh, fresh_blobs, fresh_dir) = store();
|
|
let summary = import_zip(&fresh, &fresh_blobs, &archive).unwrap();
|
|
assert_eq!(
|
|
summary,
|
|
ImportSummary {
|
|
source: "inkwell".into(),
|
|
imported: 1,
|
|
skipped: 0
|
|
}
|
|
);
|
|
let back = all(&fresh);
|
|
assert_eq!(back.len(), 1);
|
|
let back = &back[0];
|
|
let original = store::get_note(&conn, ¬e.id).unwrap();
|
|
assert_eq!(back.body, original.body);
|
|
assert!(back.pinned);
|
|
assert_eq!(back.recurrence.as_deref(), Some("weekly"));
|
|
assert_eq!(back.remind_at, original.remind_at);
|
|
assert_eq!(
|
|
back.created_at, original.created_at,
|
|
"the source's own time, not the import's"
|
|
);
|
|
assert_eq!(
|
|
back.labels
|
|
.iter()
|
|
.map(|l| l.name.as_str())
|
|
.collect::<Vec<_>>(),
|
|
["travel"]
|
|
);
|
|
assert_eq!(back.attachments.len(), 1);
|
|
assert_eq!(back.attachments[0].filename.as_deref(), Some("ticket.pdf"));
|
|
let sha = back.attachments[0].sha256.clone().unwrap();
|
|
assert_eq!(fresh_blobs.read(&sha).unwrap(), b"%PDF-1");
|
|
|
|
std::fs::remove_dir_all(dir).ok();
|
|
std::fs::remove_dir_all(fresh_dir).ok();
|
|
}
|
|
|
|
#[test]
|
|
fn a_keep_takeout_imports_including_a_photo_only_note() {
|
|
let (conn, blobs, dir) = store();
|
|
let list = serde_json::to_vec(&fixture()["keep"][0]["note"]).unwrap();
|
|
let photo_only =
|
|
br#"{"textContent": "", "isPinned": false, "attachments": [{"filePath": "p.png"}]}"#;
|
|
let archive = zip_of(&[
|
|
("Takeout/Keep/Groceries.json", list),
|
|
("Takeout/Keep/photo.jpg", b"jpeg bytes".to_vec()),
|
|
("Takeout/Keep/scan.png", b"png bytes".to_vec()),
|
|
("Takeout/Keep/Photo.json", photo_only.to_vec()),
|
|
("Takeout/Keep/p.png", b"p".to_vec()),
|
|
(
|
|
"Takeout/Keep/empty.json",
|
|
br#"{"textContent": " "}"#.to_vec(),
|
|
),
|
|
(
|
|
"Takeout/Keep/unrelated.json",
|
|
br#"{"hello": "world"}"#.to_vec(),
|
|
),
|
|
]);
|
|
let summary = import_zip(&conn, &blobs, &archive).unwrap();
|
|
assert_eq!(
|
|
summary,
|
|
ImportSummary {
|
|
source: "keep".into(),
|
|
imported: 2,
|
|
skipped: 1
|
|
}
|
|
);
|
|
|
|
let notes = all(&conn);
|
|
let groceries = notes
|
|
.iter()
|
|
.find(|n| n.body.starts_with("Groceries"))
|
|
.expect("the list note");
|
|
assert!(groceries.body.contains("- [x] Eggs"));
|
|
assert!(groceries.pinned);
|
|
assert_eq!(groceries.attachments.len(), 2);
|
|
assert_eq!(
|
|
groceries.created_at.as_deref(),
|
|
Some("2020-09-13T12:26:40.000Z")
|
|
);
|
|
let photo = notes
|
|
.iter()
|
|
.find(|n| n.body.is_empty())
|
|
.expect("the photo-only note");
|
|
assert_eq!(photo.attachments[0].mime, "image/png");
|
|
|
|
std::fs::remove_dir_all(dir).ok();
|
|
}
|
|
|
|
#[test]
|
|
fn what_is_not_an_archive_or_holds_no_notes_is_refused() {
|
|
let (conn, blobs, dir) = store();
|
|
assert!(import_zip(&conn, &blobs, b"not a zip")
|
|
.unwrap_err()
|
|
.contains("valid .zip"));
|
|
let other = zip_of(&[(
|
|
"notes.json",
|
|
br#"{"app": "someone-else", "notes": [{"body": "x"}]}"#.to_vec(),
|
|
)]);
|
|
assert!(import_zip(&conn, &blobs, &other)
|
|
.unwrap_err()
|
|
.contains("no importable notes"));
|
|
assert!(all(&conn).is_empty());
|
|
std::fs::remove_dir_all(dir).ok();
|
|
}
|
|
|
|
#[test]
|
|
fn slugs_match_the_servers() {
|
|
assert_eq!(slugify("Trip #travel"), "trip-travel");
|
|
assert_eq!(slugify(" Hello, World! "), "hello-world");
|
|
assert_eq!(slugify("snake_case -- dashes"), "snake-case-dashes");
|
|
assert_eq!(slugify("!!!"), "note");
|
|
assert_eq!(slugify(""), "note");
|
|
assert_eq!(slugify(&"a".repeat(80)).len(), 60);
|
|
}
|
|
}
|