Commit Graph

95 Commits

Author SHA1 Message Date
bvandeusen 1fd303abe3 fix(mcp): don't strip /mcp prefix; FastMCP's handler is mounted there
The dispatch wrapper was rewriting scope['path'] from '/mcp' to '/'
before handing off to FastMCP. But FastMCP's streamable_http_app
mounts the JSON-RPC handler at '/mcp' (its default), so the rewritten
'/' had no matching route and FastMCP returned 404. Auth middleware
was correctly firing first (a no-auth request still gets 401), the
bug was only on the post-auth path.

Symptom: `claude mcp add ...` succeeds, registration shows in
`claude mcp list`, but connection fails because the initialize
handshake returns 404 instead of an MCP capabilities response.

Fix: pass the scope through unmodified. FastMCP's own routing matches
the '/mcp' path.

Also tightened the integration test that should have caught this —
it was asserting `status != 401`, which a 404 trivially passes. Now
asserts `== 200`, the actual expected response for initialize.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-27 12:55:47 -04:00
bvandeusen 6aa84002b3 refactor(mcp): drop fable_ prefix from tool names; rebrand to Scribe
MCP clients see tools namespaced by the server's local name already
(mcp__<server>__<tool>), so the fable_ prefix on every tool name was
redundant and ate tokens in the model's tool list.

Tools renamed (34 total):
  fable_search → search
  fable_list_notes / get_note / create_note / update_note / delete_note → list_notes / ...
  fable_list_tasks / get_task / create_task / update_task / add_task_log → list_tasks / ...
  fable_list_projects / get_project / create_project / update_project → list_projects / ...
  fable_list_milestones / create_milestone / update_milestone → list_milestones / ...
  fable_list_events / create_event / get_event / update_event / delete_event → list_events / ...
  fable_list_tags → list_tags
  fable_get_recent → get_recent
  fable_list_persons / create_person / update_person → list_persons / ...
  fable_list_places / create_place / update_place → list_places / ...
  fable_list_lists / create_list / update_list → list_lists / ...

Also rebranded in MCP scope:
  FastMCP("fable", ...) → FastMCP("scribe", ...)
  auth realm "fable-mcp" → "scribe-mcp"
  ASGI scope key fable_user_id → scribe_user_id
  ContextVar label fable_mcp_user_id → scribe_mcp_user_id
  Tool docstrings "in Fable" / "Fable task" → "in Scribe" / "Scribe task"
  Server _INSTRUCTIONS prose

Deliberately kept:
  - The internal Python package name `fabledassistant` (per project naming
    convention — internal stays).
  - "Fabled Scribe" as the official product/brand name (page footer,
    smtp_from_name default).
  - References to the legacy `fable-mcp/` standalone package in docstrings
    explaining what we ported from — accurate until that directory is
    deleted in Phase 10.

Client impact: existing MCP registrations need
  claude mcp remove <name> && claude mcp add ...
once with a freshly-copied snippet from Settings → MCP Access. Claude
Code then re-discovers tools on connect — old conversations that
referenced fable_* tool names will see "tool not found" on those calls
until updated.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-27 11:48:55 -04:00
bvandeusen 52d6a8ed53 feat(embeddings): swap Ollama for fastembed (in-process ONNX)
Replaces the Ollama HTTP get_embedding with a fastembed.TextEmbedding
singleton loaded lazily on first call. Model: BAAI/bge-small-en-v1.5
(384-dim), cached to /data/fastembed-cache.

Public API unchanged:
  - get_embedding(text, model=None) — `model` now silently ignored
  - upsert_note_embedding
  - semantic_search_notes
  - backfill_note_embeddings

_cosine_similarity gains a defensive length-mismatch check so any
stale 768-dim row that survived the migration is treated as 0.0
similarity rather than crashing zip().

The Ollama client dep stays in pyproject for now (other services still
use it); Phase 7 removes it once chat/journal/curator are gone.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 21:02:30 -04:00
bvandeusen d4f3516552 feat(mcp): typed-entity tools (person/place/list)
Nine tools — list/create/update for each of person, place, list.
Get and delete reuse fable_get_note / fable_delete_note (typed
entities share the Note model).

Lists: the wrappers accept an `items: list[str]` for ergonomics and
translate to the {text, checked} dict shape that
services/knowledge.py and KnowledgeView.vue expect. items=[] clears;
items=None leaves unchanged.

Updates do an explicit get → merge → update round trip so updating
one typed field doesn't clobber the others stored alongside it in
entity_meta (which is a single JSONB column).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 20:53:36 -04:00
bvandeusen 6961144c3a feat(mcp): fable_list_tags + fable_get_recent
Two cross-type bootstrap tools:

- fable_list_tags: tag vocabulary with usage counts, top-N by count.
  Aggregation in Python (not SQL UNNEST) — trivial perf cost at
  personal scale, much easier to test.

- fable_get_recent: most-recently-touched items across notes, tasks,
  projects, events. Useful for Claude to ask 'what was I working on
  recently' at the start of a conversation.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 20:51:59 -04:00
bvandeusen 9a76c4718b feat(mcp): event CRUD tools
Five new tools (events weren't in fable-mcp before). Split
start_date + start_time inputs combine into a naive datetime that
services/events.py interprets in the user's local timezone.

Sentinels for update:
  - empty strings → leave unchanged
  - duration_minutes=-1 → leave unchanged
  - duration_minutes=0 → set to point event (NULL duration)
  - start_date/start_time must BOTH be set to move the event

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 20:50:54 -04:00
bvandeusen 4d6bae77b4 feat(mcp): project + milestone CRUD tools
Seven tools matching existing fable-mcp contracts:
  - fable_list/get/create/update_project (no delete; archive via status)
  - fable_list/create/update_milestone (no get; no delete)

LLM-era similarity-check / 'confirmed' guard for create_project is
NOT replicated — Claude doesn't need it. The service's auto-summary
regeneration side effect (services.projects.update_project) stays
for now; gets removed in Phase 7 along with all other LLM code.

Notable sentinels:
  - update_milestone: order_index=-1 means "leave unchanged" (0 is valid)
  - create_milestone: description="" becomes None at the service layer

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 20:24:07 -04:00
bvandeusen d086c9b606 feat(mcp): task CRUD tools + add_task_log
Five tools wrapping services/notes.py with is_task=True (tasks are
notes with non-null status) plus services/task_logs.create_log for
add_task_log. Matches existing fable-mcp contracts. No delete_task —
preserves existing surface; cancel by updating status to "cancelled".

fable_get_task enriches with parent_title (extra service call when
parent_id is set), matching the existing route's behavior.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 20:22:30 -04:00
bvandeusen b026421985 feat(mcp): note CRUD tools (list/get/create/update/delete)
Five tools wrapping services/notes.py with is_task=False. Signatures
mirror the existing fable-mcp note tool contracts so Claude usage is
unchanged.

Key behavior the tests pin down:
  - list_notes repackages (rows, total) tuple into {notes, total}
  - tag=""/search_text="" are "no filter" sentinels
  - update_note ONLY sends non-default fields to the service (the
    main risk: a default empty string overwriting real data)
  - tags=[] is an explicit clear; tags=None is "leave unchanged"
  - project_id=0 on create => orphan; on update => leave unchanged
    (preserved limitation from existing fable-mcp)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 20:20:27 -04:00
bvandeusen fd0431dfb6 feat(mcp): tools/ package + fable_search
Establishes the tool pattern: each tool module exposes register(mcp),
register_all() aggregates them, build_mcp_server() calls register_all.

fable_search mirrors the existing fable-mcp contract (q/content_type/limit
in; {results, total} out) but calls services.embeddings.semantic_search_notes
directly instead of going over HTTP. User comes from mcp.current_user_id().

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 20:18:36 -04:00
bvandeusen 3579db2f06 feat(mcp): per-request user_id contextvar for tool handlers
Adds mcp._context.current_user_id() backed by a ContextVar. The ASGI
auth middleware sets it before dispatching to FastMCP and resets it
on the way out, so tool handlers can read the acting user without
re-parsing the request scope.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 20:17:04 -04:00
bvandeusen 3cc5c7dcab test(mcp): drop the bypass test (covered implicitly)
Driving Quart's full request pipeline via a hand-rolled ASGI scope
(no lifespan startup, no hypercorn-provided state) doesn't produce
a response. The 3 remaining tests cover the actual MCP middleware
behavior. The bypass property is implicit — if the middleware ate
non-/mcp requests, every existing /api/* test would fail.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 19:58:45 -04:00
bvandeusen 38265906f1 test(mcp): drive ASGI app directly, skip Quart test_client
Quart's test_client expects its request pipeline to populate
app._preserved_context. Our /mcp middleware deliberately bypasses
that pipeline (forwarding straight to FastMCP), so test_client's
teardown blew up with AttributeError. The middleware is correct;
the test harness was wrong.

Build raw ASGI scope/receive/send and call app.asgi_app directly —
which is what production hypercorn does anyway.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 19:54:46 -04:00
bvandeusen 94f7a6de37 feat(mcp): mount /mcp endpoint with bearer-token auth
Wires FastMCP's streamable-HTTP ASGI sub-app into the Quart app via
asgi_app replacement. Requests under /mcp are stripped, auth-checked
against api_keys, and forwarded to FastMCP with fable_user_id set on
the ASGI scope. All other paths pass through to the original Quart
dispatch unchanged.

Tests cover the three auth paths (no header, invalid token, valid
token) plus a regression check that non-/mcp paths bypass the MCP
dispatch.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 19:15:42 -04:00
bvandeusen caa504913f feat(mcp): bearer-token auth resolver
Thin parser over the existing api_keys lookup. Strips the Bearer
prefix, validates the token via services/api_keys.lookup_key (which
already filters revoked keys and updates last_used_at), and returns
the user_id for the in-flight MCP request.

Tests follow the existing mock-async_session pattern in
test_api_keys.py rather than introducing a real DB fixture.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 19:13:31 -04:00
bvandeusen 7d71f126a2 fix(tests): relax voice ID regex test — don't assert HF casing convention
The voice_library regex's purpose is to prevent path traversal and
filter structurally-malformed IDs, not to enforce the HF catalog's
lowercase-language convention. Asserting that EN_US-amy-medium is
rejected was a category error — uppercase variants pass the regex
but would 404 at install time against HF, which is a harmless dead
end, not a security gap. Comment in the test now explains the scope.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-22 10:13:08 -04:00
bvandeusen 39ab5d69a9 feat(voice): admin UI to browse + install piper voices from HuggingFace
Building on the kokoro→piper swap (B1), this adds the admin-side
voice management story so additional voices can be installed without
rebuilding the image. The bundled two voices stay as immediate defaults;
everything else is opt-in via a one-click install from the catalog.

Backend (services/voice_library.py):
- fetch_catalog() pulls voices.json from the piper-voices HF repo with
  a 24h in-memory TTL. Manual refresh available via ?refresh=1 on the
  library endpoint.
- shape_catalog_for_ui() projects the raw HF dict (~250 voices, lots of
  nesting) into UI-friendly cards: id, name, language, country, quality,
  size, install state. Sorted by language_code then name for stable
  display. Install state distinguishes bundled (read-only) from user
  (admin-installed, can be removed).
- install_voice() downloads .onnx + .onnx.json into /data/voices with
  atomic .tmp → rename so a failed partial download can't leave a
  corrupt model around. Idempotent — re-installing an already-present
  voice is a no-op.
- uninstall_voice() removes /data voices; bundled /opt voices raise
  PermissionError (403 at the route layer).
- Strict voice-id regex prevents path traversal in install/uninstall.

Routes (admin-only, since these write to shared /data and affect all
users on the instance):
- GET    /api/voice/voices/library
- POST   /api/voice/voices/install
- DELETE /api/voice/voices/<voice_id>

Frontend:
- New "Voice Library" section in Settings → Voice, visible only to
  admin users. Collapsed by default; expand to load the catalog
  on-demand (doesn't hammer HF for non-admins).
- Free-text filter across id, language code, language name, country,
  and dataset name. Refresh button forces a catalog re-fetch.
- Per-voice row shows id, language/country/quality/speaker count, size,
  and either an Install button, a Remove button (user voices), or a
  "bundled" badge (read-only voices in /opt/piper-voices).
- Installs and uninstalls refresh both the library list AND the active
  voice picker so the new voice is immediately selectable.
- VoiceLibraryEntry exported from api/client.ts; new client helpers
  getVoiceLibrary/installVoice/uninstallVoice.

Tests:
- Pure-transformation unit tests for shape_catalog_for_ui,
  _resolve_file_urls, and the voice-id regex (path-traversal coverage).
- DB/network paths (fetch_catalog, install_voice) need a real
  environment — left to CI integration tests or device verification.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-22 08:18:22 -04:00
bvandeusen bf7a29e8a0 feat(llm): per-turn tool-call telemetry (generation_tool_log)
Adds an empirical surface for evaluating model swaps. One row per
assistant turn captures: model, think_enabled, tools_available,
tools_attempted, tools_succeeded, tools_failed (with error details
as JSONB). Without this, judging whether a new model "actually fires
record_moment when it should" relies on anecdote across user-reported
sessions. With it, the data is queryable directly.

Pieces:
- Migration 0046: generation_tool_log table with user_created and
  per-conversation indexes.
- Model: SQLAlchemy GenerationToolLog with to_dict() for plain-dict
  consumption outside session scope.
- Service: log_tool_outcomes() normalizes the in-app tool-call shape
  (function/result/status) into the split buckets and persists. It
  catches its own exceptions — telemetry failure must NEVER affect
  the user-facing generation flow. recent_logs() helper for read.
- Integration in run_generation: called once per turn right after
  log_generation, fire-and-forget.
- Tests: pure-normalization unit tests using a stub session — no DB
  needed in CI. Cover the success/error split, the empty-tool-calls
  case, the exception-swallowing contract, and the success=False
  edge case where status incorrectly says "success".

No UI for the telemetry yet — internal infrastructure (the operator
is the consumer, not the journal user), which the FabledRulebook
"no UI no ship" explicitly excepts. Query via psql or extend the
Fable MCP later if direct shell access gets tiresome.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-21 22:04:09 -04:00
bvandeusen 5d2d27c499 fix(journal): anti-hallucination hardening + message_count fix
Prep prose (services/journal_prep.py):
- Emit explicit "WEATHER: none available — do NOT mention weather"
  absent-marker so a small model can't invent partly-cloudy/temperature
  prose when both configured locations have empty addresses.
- Replace negative-only system rule with positive-anchored guidance
  forbidding weather/temp/precip mentions unless a numeric WEATHER
  section is present; also bans echoing parenthetical labels verbatim.
- Reword overdue header to "(past their due date, still open — backlog,
  not today's work)" and render lines as "was due <date>, N day(s)
  overdue" with correct singular/plural. Supersedes the wording noted
  in Fable task #159.
- Deterministic fabricated-weather reconciler: low-false-positive regex
  detects fabricated weather phrasing; on trip with an empty section,
  regenerate once with a corrective. Persistent fabrication logs ERROR
  rather than mangling prose.

Journal route (routes/journal.py):
- Override message_count with len(messages) in _day_payload. The chat
  path already does this; the journal path was hitting the
  Conversation.to_dict() fallback to 0 because messages aren't
  eager-loaded on that instance.

Tests:
- tests/test_journal_message_count.py — pins the model-level trap and
  the override contract (3 cases).
- tests/test_journal_prep_hardening.py — 11 cases covering the
  fabricated-weather reconciler and absent-marker rendering.
- tests/test_journal_prep_filtering.py — updated one stale assertion.

Tracks Fable task #171.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-20 18:54:35 -04:00
bvandeusen 37c704e875 feat(versions): auto-pin scan promotes stable versions
_promote_stable_versions_for_note is the pure-function core: walks
versions chronologically and pins any with a >= AUTO_PIN_STABILITY_DAYS
(2-day) gap to the next version (or to now, for the latest). Auto-
generated label describes the stability window.

_scan_one_note loads versions for one note, runs the promotion, commits
mutations to the attached rows, then calls prune_auto_pins to cap the
auto bucket. scan_user_for_auto_pins fans out across the user's notes;
scan_all_users_for_auto_pins is the top-level entrypoint for the cron.
Per-note and per-user errors are caught and logged.
2026-05-13 13:57:56 -04:00
bvandeusen bb6249e00e feat(versions): prune_auto_pins FIFO-trims auto-pinned bucket
Auto-pinned versions live in their own bucket with MAX_AUTO_PINS=25 cap.
The scan job calls this after each note's promotions complete; the
oldest auto-pinned rows are dropped past the cap. Manual pins and
rolling rows are untouched.
2026-05-13 13:57:09 -04:00
bvandeusen 925a53e0f7 feat(versions): pin_version and unpin_version services
pin_version sets pin_kind='manual' and pin_label on the target row.
Accepts already-pinned rows (promotes auto→manual, updates label).
Labels are capped at PIN_LABEL_MAX_LEN=500 chars; longer values raise
ValueError before any DB access.

unpin_version clears both fields, downgrading the row to rolling. Does
NOT delete — if the row is past the rolling FIFO depth, the next
autosave's prune will drop it.
2026-05-13 13:56:21 -04:00
bvandeusen b65d736869 feat(versions): rolling-cap prune ignores pinned versions
The DELETE inside create_version now filters pin_kind IS NULL so pinned
rows (auto or manual) aren't counted toward MAX_VERSIONS=50 and aren't
candidates for deletion. Pinned versions live indefinitely regardless
of how heavy rolling autosave traffic gets on the same note.
2026-05-13 13:55:44 -04:00
bvandeusen 90aa1f2fdb fix(tests): preserves_body test \$Note stub needs project_id
The knowledge-note return path in create_note_tool reads note.project_id;
the SimpleNamespace fake didn't define it, so the tool crashed with
AttributeError instead of returning. The task-branch test already
included project_id; mirror that here.
2026-05-13 12:30:09 -04:00
bvandeusen 103db883ad feat(tools): tasks accept description; reject body writes via tools
create_note tool:
- New 'description' parameter accepted and forwarded to the service.
- When status is set (creating a task), 'body' is dropped before the
  service call. Task bodies are owned by the consolidation pipeline.

update_note tool:
- New 'description' parameter; routed through update_fields.
- When the resolved target has is_task=True and 'body' is in the
  arguments, the call errors with a message nudging toward log_work or
  description. Knowledge notes are unaffected.

HTTP routes (POST/PATCH/PUT /api/notes) accept body freely — the
restriction is only at the LLM tool layer.
2026-05-13 12:15:09 -04:00
bvandeusen 5fa203019a feat(consolidation): trigger from log_work and status terminal transitions
log_work tool now invokes maybe_consolidate(reason='log_added') after a
successful create_log. The gate inside the consolidation service handles
threshold + setting checks.

update_note service snapshots old_status before mutation and fires
maybe_consolidate(reason='task_closed') when the status transitions into
'done' or 'cancelled'. Re-saving an already-terminal status doesn't
retrigger — only transitions count.
2026-05-13 12:13:31 -04:00
bvandeusen bda6e6c80f feat(consolidation): full consolidate_task with background model
consolidate_task reads the task title, description (read-only context),
and chronological work logs; builds a prompt via _build_consolidation_prompt;
calls generate_completion with the user's background_model setting; on a
non-empty result, writes back to Note.body, stamps consolidated_at, and
re-runs the embedding pipeline.

Errors are caught and logged. LLM failures leave body untouched so the
next trigger retries cleanly. Per-task asyncio lock prevents simultaneous
passes for the same task.
2026-05-13 12:12:37 -04:00
bvandeusen 5419330633 feat(consolidation): debounced gate for task body consolidation
New services/consolidation.py module with maybe_consolidate() — the
debounced trigger gate. Two reasons:

- log_added: gated by DEFAULT_LOG_THRESHOLD (3) counted since the task's
  consolidated_at timestamp.
- task_closed: bypasses the count gate; fires whenever status flips to
  done/cancelled.

Both reasons gated by the auto_consolidate_tasks user setting (default
on). Per-task asyncio.Lock prevents two simultaneous passes for the same
task. consolidate_task is a stub here — full implementation in the next
commit.
2026-05-13 12:11:41 -04:00
bvandeusen 362ead7f0d feat(notes): accept and return description field through service and routes
create_note service accepts a new description kwarg and forwards it to the
Note constructor. PUT/PATCH/POST routes include description in the field
whitelist. update_note already passed **fields through setattr, so the new
column is reachable without touching that signature.
2026-05-13 12:11:03 -04:00
bvandeusen a551f52682 fix(tools): score_project_match strips 'project' filler + uses title for SequenceMatcher
CI surfaced three issues:
- 'famous supply project' didn't substring-match 'Famous-Supply Work topics'
  because the trailing filler word 'project' blocked the substring tier.
  Strip {project, projects} from the query before the substring check.
- SequenceMatcher fallback against `combined` (title + description +
  summary) diluted ratios to ~0.5 for plausible matches. Use title
  directly; the 0.70 tier already handles description/summary mentions.
- Test patches used patch.object on a consumer module where
  list_projects is imported locally — patch the source module instead.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-12 20:30:37 -04:00
bvandeusen 9a96fdb3c0 refactor(tools): resolve_project uses shared score_project_match helper
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-12 16:43:05 -04:00
bvandeusen 460959f0d4 refactor(tools): search_projects uses shared score_project_match helper
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-12 16:42:34 -04:00
bvandeusen 0dbbb98cf5 feat(tools): score_project_match helper with substring-first tiering
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-12 16:42:01 -04:00
bvandeusen 42c11dedae feat(notes): _strip_type_nouns helper for search query sanitization
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-12 16:40:54 -04:00
bvandeusen 0fbb1fbd92 fix(tools): search_projects returns success:True so result isn't mislabeled error
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-12 16:40:27 -04:00
bvandeusen fc6ebf81eb feat(journal): closeout catch-up on startup when slot already passed
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-12 14:36:29 -04:00
bvandeusen b88d5ee6b3 feat(journal): register per-user closeout job at day_rollover_hour
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-12 14:35:57 -04:00
bvandeusen 020bd6614b test(journal): cover closeout skip paths (no conv / prep-only / sentinel)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-12 14:34:58 -04:00
bvandeusen 4403026797 feat(journal): run_for_user orchestrates closeout extraction
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-12 14:34:42 -04:00
bvandeusen 552943d6c0 feat(journal): closeout system prompt with structured-field guard
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-12 14:33:57 -04:00
bvandeusen 2576be9e49 feat(journal): _build_transcript caps content and message window
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-12 14:33:27 -04:00
bvandeusen e17fc088b2 feat(journal): journal_closeout._filter_messages excludes daily_prep
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-12 14:33:03 -04:00
bvandeusen 84640a0dc4 fix(events-tools): refuse ambiguous query, require event_id to disambiguate (#161)
Root cause of the 2026-04-29 dentist-appointment incident: the model
called update_event(query="Appointment") when two events had
"Appointment" in their titles. find_events_by_query returned both,
upcoming-first ordered by start_dt — matches[0] was id=2 (a stale
pre-existing event with garbage end_dt), not id=15 (the one the user
just created via the journal flow). update_event_tool silently took
matches[0] and mutated the wrong event.

Fix: a new resolver helper `_resolve_event_for_action` funnels both
update_event_tool and delete_event_tool through one disambiguation
path. Lookup precedence:
  - `event_id` → exact get_event lookup, no query at all
  - `query` matching exactly one event → proceed
  - `query` matching zero → return success=False, "no event found"
  - `query` matching 2+ events → return success=False with a
    `candidates` array of {id, title, start_dt, location} so the
    model can pick one and call again with `event_id`

The candidates list is capped at 8 to keep the model's context tight.
The error message names the count and the next-step ("pass event_id
or refine the query") so the model can self-correct in one turn.

For delete_event, the disambiguation is even more important — the
silent-matches[0] path would have deleted the wrong event outright
rather than just mutating it. The tool description leans into that:
"Deleting the wrong event is a costly user error; never guess."

Tool surface change: `query` and `event_id` are now both optional;
the tool errors clearly when neither is supplied. The model already
knows id values from prior tool results (returned in `data.id`),
which is the natural feeder for the disambiguation flow.

5 new tests in test_calendar_tool_tz.py cover:
- ambiguous query → success=False with candidate list, no mutation
- event_id supplied → bypasses query lookup entirely
- non-existent event_id → clear "no event found" error
- neither identifier → "query or event_id required" error
- same disambiguation enforced for delete_event_tool

46 calendar/events tests pass; ruff clean.

Closes Fable #161.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-04-29 14:29:13 -04:00
bvandeusen 4c58603009 feat(events): replace end_dt column with duration_minutes (#160)
Structural fix for the "end before start" bug class observed on prod
2026-04-29. Bad data became inexpressible at the schema level instead
of getting trapped in defensive read-path filters.

The hotfix that landed earlier today (94b169f) is reverted by the
preceding revert commit; this commit supersedes it cleanly with a
proper data-model change.

## Schema (migration 0043)

- Add `duration_minutes INTEGER NULLABLE` column on `events`.
- CHECK constraint: ``duration_minutes IS NULL OR duration_minutes >= 0``.
- Backfill from existing `end_dt`:
    - end_dt valid (end > start) → duration_minutes = total minutes
    - end_dt == start → duration_minutes = 0 (zero-duration point)
    - end_dt NULL or end_dt < start → duration_minutes = NULL
      (the corrupt prod row collapses cleanly to a point event)
- Drop the `end_dt` column. The wire format is preserved — `to_dict()`
  emits `end_dt` as a derived `start_dt + duration_minutes`. Existing
  API consumers (Flutter app, web frontend, CalDAV sync) keep
  receiving the same response shape; they just no longer have a way
  to PUT a stored `end_dt` that disagrees with `start_dt`.

## Service layer

- `Event.end_dt` becomes a `@property`. Setting it would require a
  setter we deliberately don't define — writes always go through
  `duration_minutes`.
- `_normalize_duration` is the single source-of-truth for input
  reduction. Accepts (start, end_dt, duration_minutes), returns the
  canonical `duration_minutes`, raises `ValueError` for negative
  durations, end-before-start, or end/duration disagreement.
- `create_event` and `update_event` accept either `end_dt` or
  `duration_minutes` for ergonomic compat; both convert via
  `_normalize_duration`. Update validates the post-update state when
  the patch includes either.
- `list_events` filter is simpler now: a coarse SQL prefilter
  (`start_dt <= date_to`) plus Python-side refinement using the
  derived `end_dt`. Avoids Postgres-specific interval arithmetic in
  the WHERE clause; refinement runs over a per-user result set so
  there's no scan-cost concern at personal scale.
- Recurring-event expansion uses `event.duration_minutes` directly
  instead of computing `end - start`. No more negative-timedelta
  hazard.

## CalDAV sync (incoming + outgoing)

- `caldav_sync.py` (pull) and `calendar_sync.py` (Radicale upsert)
  both convert iCal `DTEND` → `duration_minutes` on the way in.
  Outbound iCal still emits `DTEND` as `start_dt + duration_minutes`
  via the model's derived property. iCal interop is unchanged.

## Behavioral upgrade for `update_event`

Pure end_dt model: moving start past the existing end_dt would either
silently corrupt or hard-reject. Duration model: the duration is
preserved by default, so moving start slides the effective end
forward — which is what users mean when they "move" an event.
Explicit clear is still possible via `end_dt=None`.

## Tests

`tests/test_events_service.py`:
- 6 new `_normalize_duration` unit tests (sugar conversion, zero
  duration valid as point event, end-before-start rejected, negative
  duration rejected, inconsistent end+duration rejected, none → None)
- New behavioral test: `update_event` preserves duration when only
  start_dt changes (sliding semantics)
- New: clearing `end_dt=None` on update collapses to point event
- New: list_events surfaces a point event in the upcoming window
- New: list_events excludes a timed event whose effective end has
  already passed
- Existing mock-event helper updated to use `duration_minutes`
  instead of stored `end_dt`.

44 event-related tests pass; ruff clean.

## Out of scope (separate task)

Fable #161 — `find_events_by_query` returning multiple matches and
silently picking matches[0]. The exact root cause of how event id=2
got mutated in the first place; orthogonal to the storage model.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-04-29 14:19:44 -04:00
bvandeusen b7e7073425 Revert "fix(events): tolerate corrupt end_dt + reject end<=start at write time"
This reverts commit 94b169f31c.
2026-04-29 14:18:01 -04:00
bvandeusen 94b169f31c fix(events): tolerate corrupt end_dt + reject end<=start at write time
A prod event surfaced today with `start_dt=2026-05-01T12:00Z` and
`end_dt=2026-03-30T12:00Z` — end was 32 days BEFORE start, almost
certainly from an earlier tool-call mishap (Fable #161). The
list_events filter trusted the bogus end_dt and excluded the event
from every read path that hit the upcoming window, even though
start_dt was correctly in range. The event stayed visible in the
calendar grid (different range) but vanished from "Upcoming",
search, briefings, and journal prep events list.

This is the hotfix half of the response. The structural follow-up is
Fable #160 — replace end_dt with a duration column so invalid state
becomes inexpressible.

## A. Filter robustness in list_events

Treat `end_dt <= start_dt` as if no end_dt exists. The filter now
splits into two branches:
- valid duration: end_dt IS NOT NULL AND end_dt > start_dt AND
  end_dt >= date_from
- no/invalid duration: (end_dt IS NULL OR end_dt <= start_dt) AND
  start_dt >= date_from

Same change applied to the recurring-event expansion's `duration`
calculation, which was producing negative timedeltas for corrupted
rows and computing nonsensical occurrence end times.

## B. Write-side validation in create/update

`create_event` and `update_event` now raise ValueError when the
resulting state would have end_dt <= start_dt. Update validates
against the *post-update* state, not just the field being changed —
so pushing start_dt past an existing end_dt also fails loudly. Bad
data shouldn't be persistable from any write path.

Surfaced cleanly:
- Calendar tool wrappers (create_event_tool / update_event_tool)
  catch ValueError and return `{success: false, error: ...}`, which
  the model can read and self-correct.
- Route handlers (POST /api/events, PATCH /api/events/<id>) catch
  and return HTTP 400 with the validator's message instead of
  letting it bubble to a 500.

4 new tests in test_events_service.py:
- create rejects end before start
- create rejects equal start/end (zero duration)
- update validates the post-update state (start pushed past existing end)
- list_events surfaces an event whose end_dt is before its start_dt

34 event-related tests pass; ruff clean.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-04-29 13:48:28 -04:00
bvandeusen 9f8b451d15 fix(journal-prep): bucket tasks + drop non-proximate events (#159)
Two filtering issues that made the daily prep noisy and trained the
user to ignore it.

## Tasks: bucket into due-today / upcoming / overdue

The prep was calling `list_notes(due_before=day_date)` and labeling the
result as "tasks due today". That filter is strictly less-than, so it
returned only OVERDUE tasks (a single 68-day-stale task in this user's
case), while the prompt still framed them as fresh today's work. Each
day of the prep treated the same overdue task as new — the user
learned to ignore the line entirely.

`gather_daily_sections` now runs three queries:
- `tasks_due_today` — `due_after=day_date AND due_before=day_date+1`
- `tasks_upcoming` — next 7 days, exclusive of today
- `tasks_overdue` — strictly before today
Overdue entries carry a `days_overdue` count. `_render_sections_for_prompt`
emits three labeled headers ("TASKS DUE TODAY", "UPCOMING TASKS",
"OVERDUE TASKS (still on the list, not currently due)"). The system
prompt has a new TASK BUCKETS rule telling the model: don't call
overdue items "due today"; surface them with their staleness duration
("still on the list 68 days") and frame as a backlog reminder rather
than today's work.

Backwards-compat: `sections["tasks"]` still exists, now as the union
of all three buckets — strictly more useful than the prior overdue-
only behavior any frontend consumer was getting before.

## Events: tz-aware window + proximity filter

The user's "Birthday — 2026-09-29 (FREQ=YEARLY)" event was surfacing
in every daily prep, 5 months out. Root cause: `gather_daily_sections`
built `day_start`/`day_end` as NAIVE datetimes; `list_events` then
called `rrulestr(...).between(naive_from, naive_to)` against an
aware `dtstart`, which throws TypeError, hits the `except Exception`
fallback, and appends the canonical event row — regardless of whether
today is anywhere near a recurrence.

Fix:
1. Construct the day window as TZ-aware in the user's local timezone
   and convert to UTC before the query. RRULE expansion now runs
   correctly.
2. Defense-in-depth `_filter_proximate_events` drops events whose
   start_dt is more than 7 days from `day_date` (in the user's local
   TZ — not UTC, so a Friday 23:00 NY event isn't misclassified as
   Saturday). If list_events ever leaks a far-future row again, the
   prep doesn't surface it.

10 new tests in `tests/test_journal_prep_filtering.py` cover task
bucketing (overdue marker, due-today no-marker, no-due-date), the
proximity filter (the 4/29 reproducer, in-window keeps, local-vs-UTC
boundary, unparseable dates kept rather than suppressed), and the
rendering (overdue staleness shown, due-today doesn't repeat the date,
correct section ordering).

53 tests pass across journal_prep + journal_search + record_moment +
calendar_tool + events. Ruff clean.

Closes Fable task #159.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-04-29 09:31:12 -04:00
bvandeusen 4f18023284 fix(journal): server-side guards on record_moment links + place names (#158)
Belt-and-suspenders to the prompt-layer changes in 6c309f1. Even when
the model emits bogus task or place links, the server now refuses to
persist them.

## Task auto-linking guard

Reproducer (2026-04-27): a moment about restaging Docker on the swarm
ended up with `task_ids: [2]` (Weston's ADHD Evaluation) — the only
task in that day's prep. The model picked it up as filler.

`_filter_task_ids_by_keyword_overlap` now runs after id resolution: it
fetches each linked task's title, tokenizes both content and title
through `_content_keywords` (lowercased, stopwords stripped, <3-char
tokens dropped), and drops any link whose title shares no meaningful
keyword with the moment content. The drop is logged at INFO so we can
observe how often it fires post-deploy.

The guard runs against the merged id list, so it covers both the
preferred `task_titles` resolution path and the discouraged explicit
`task_ids` path.

## Place placeholder guard

Reproducer (2026-04-27): `place_names=["work"]` got passed to
`record_moment`. "work" / "home" / "office" aren't places — they're
role-labels for already-known geocoded locations.

`_filter_placeholder_places` drops a small set of generic single-word
labels before name resolution. Real user-named places that happen to
be one word (e.g. "Akron") pass through.

## Tests

9 new unit tests in `tests/test_record_moment_guards.py` cover:
  - keyword tokenization & stopword stripping
  - placeholder place filtering (generic, case-insensitive, real-place
    pass-through)
  - keyword-overlap filtering (the exact 4/27 reproducer, the genuine-
    reference case, mixed/partial relevance, empty input)

13 tests pass; ruff clean.

Closes Fable task #158.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-04-29 09:25:16 -04:00
bvandeusen 03d725ea3e fix(calendar-tool): anchor today's weekday in prompts + verify expected_weekday on create/update
A user asked Fable to schedule "this Friday at 8am" on Wednesday 4/29
2026. The model picked 4/30 (Thursday) and confidently labeled it
"Friday." The TZ pipeline did everything correctly given the model's
date — the bug was upstream: the model was guessing weekdays from ISO
dates without an anchor, and the calendar tools had no way to verify.

Three layered fixes:

1. **System prompts now name the weekday alongside the ISO date.**
   Both the journal-conversation prompt and the general chat prompt
   used to say "Today is 2026-04-29 (America/New_York)." They now say
   "Today is Wednesday, 2026-04-29 (...)." LLMs are unreliable at
   deriving weekday names from ISO dates; supplying the name removes
   the guess.

2. **`expected_weekday` parameter on create_event / update_event.**
   When the model passes `expected_weekday="friday"`, the backend
   computes the resolved start_date's weekday in the user's local
   timezone and rejects mismatches with a self-correcting error
   ("Date 2026-04-30 falls on Thursday, not Friday. Recompute..."),
   without creating the event. The check is local-aware: a Friday
   23:00 event in Tokyo crosses midnight UTC but the local view
   stays Friday, and the validator respects that.

3. **Tool descriptions instruct echo-and-confirm.** create_event and
   update_event descriptions now tell the model: when the user names
   a weekday, state the resolved date in the reply BEFORE calling
   the tool, and pass `expected_weekday`. Costs nothing in code,
   reinforces the validator.

6 new tests — match success, mismatch rejection (with create/update
not invoked), omitted-param backcompat, invalid weekday name, local-
not-UTC weekday computation, and the update_event variant. All 18
calendar-tool tests + 33 event-related tests pass; ruff clean.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-04-29 08:43:32 -04:00
bvandeusen 611c940527 fix(calendar-tool): split start/end into date+time to make event creation TZ-durable
A user reported "next Friday at 8am" landing on the wrong day. The
current `start` parameter accepts a combined ISO datetime string — when
the model emits something like `"2026-05-01T00:00:00Z"`, the parser
correctly honors the UTC tag and stores `2026-05-01 00:00 UTC`, which
displays as `2026-04-30 19:00` for a UTC-5 user. The bug isn't in our
parser; it's that we let the model TZ-tag the calendar day at all.

The fix moves the foot-gun: `create_event` and `update_event` now
prefer split fields (`start_date` + `start_time`, plus end variants).
A `YYYY-MM-DD` string carries no TZ metadata for a model to mis-tag,
and the backend builds the local datetime explicitly via
`datetime.combine(date, time, tzinfo=user_tz).astimezone(UTC)`. Strict
regex validation rejects anything with a TZ suffix on either field.

The legacy combined `start` / `end` fields are kept as a fallback so
saved tool-call payloads in conversation history still replay; new
calls are steered toward the split shape via the tool description.

7 new regression tests cover Eastern, Pacific, Tokyo (positive offset),
all-day inference, strict-shape rejection on both fields, backcompat
with the legacy `start` field, and the same fix for `update_event`.
27 of the event-related tests pass; ruff clean.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-04-29 08:16:25 -04:00