Commit Graph

57 Commits

Author SHA1 Message Date
bvandeusen 94f7a6de37 feat(mcp): mount /mcp endpoint with bearer-token auth
Wires FastMCP's streamable-HTTP ASGI sub-app into the Quart app via
asgi_app replacement. Requests under /mcp are stripped, auth-checked
against api_keys, and forwarded to FastMCP with fable_user_id set on
the ASGI scope. All other paths pass through to the original Quart
dispatch unchanged.

Tests cover the three auth paths (no header, invalid token, valid
token) plus a regression check that non-/mcp paths bypass the MCP
dispatch.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 19:15:42 -04:00
bvandeusen eb02603092 feat: diagnostic instrumentation for crash investigation
Recurring app/db crashes with no clear cause in existing logs.
Adds three crash-class indicators with minimal overhead (~1 log
line/min, 0.1ms work per heartbeat).

services/diagnostics.py:

1. **Heartbeat** every 60s logs a snapshot:
   - RSS memory (from /proc/self/status — no deps).
   - asyncio task count.
   - DB pool: size / checked_in / checked_out / overflow.
   - Curator busy state (from is_curator_running()).
   - Uptime.

   A sudden silence in heartbeats bounds the crash time to within
   60s. The last snapshot before silence usually rules in or out:
   memory growth -> OOM, pool exhaustion -> connection leak, hung
   curator -> stuck async task.

2. **Signal handler** for SIGTERM/SIGINT logs the signal name +
   final snapshot before letting Hypercorn handle the actual
   shutdown. Distinguishes 'orderly shutdown via signal X' from
   'silent log gap then container exit code 137' (SIGKILL / OOM-kill
   are uncatchable; their absence in our log IS the diagnostic).

3. **Asyncio exception hook** logs full tracebacks for unhandled
   task exceptions with the task/coro name. Default behaviour
   swallows these silently — exactly the pattern that locked us
   out of chat at 409 for an hour back on 2026-05-22 before we
   added the guard around run_generation.

app.py wires start_diagnostics() into before_serving and
stop_diagnostics() into after_serving. stop_diagnostics emits one
final snapshot so the silence that follows is intentional, not a
crash.

How to use the new logs to diagnose:
- App restarts with 'received SIGTERM' in the last lines:
  Orderly shutdown (docker stop / swarm restart / manual). Look
  upstream for who issued it.
- App restarts with no shutdown line, last heartbeat 30+s before:
  Likely SIGKILL — OOM-kill or container resource limit. Check
  'docker ps -a' for exit code 137, or 'dmesg | grep -i kill' on host.
- App restarts with no shutdown line, heartbeat showed climbing
  RSS: Memory leak. Snapshot the last heartbeat's MB value vs
  earlier — if it doubled over hours, OOM is the cause.
- App restarts, db_pool checked_out kept growing: Connection leak.
  Look for code paths that open async_session() but never exit
  the 'async with' block.
- App seemed alive but stopped responding to requests, heartbeats
  continued: Curator hung holding _CURATOR_RUN_LOCK. Check
  curator_busy=true across multiple heartbeats — if stuck >5min,
  the Ollama call hung. Restart Ollama or the Scribe stack.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-25 00:31:05 -04:00
bvandeusen 49325816a3 fix(journal): chat-only system prompt; don't pre-warm OLLAMA_MODEL
Two architectural bugs in the conversation+curator rollout that
explain the no-response chat in dev:

1. Journal system prompt still instructed tool calls.
   JOURNAL_CALIBRATION instructed the model to CALL record_moment,
   search_notes, save_person, etc. — but the chat surface ships tools=[]
   per the new architecture. The model received contradictory orders
   ('use these tools' + 'you have no tools') and produced either empty
   output or tool-call-shaped text that gets stripped to empty content,
   surfacing as status=error or stuck status=generating messages.
   Replaced with a chat-only calibration: ~25 lines focused on tone,
   length, anti-coaching, and the load-bearing rule 'never claim to
   have done anything for the user' (the curator handles capture
   silently and separately). JOURNAL_PERSONA also rewritten to drop
   the 'use tools to act on their behalf' line.

2. Pre-warm warmed Config.OLLAMA_MODEL ahead of user's real choice.
   _pull_model(Config.OLLAMA_MODEL, warm=True) at boot pushed the
   system default (qwen3:latest) into VRAM before _warm_user_models()
   ran for each user's actual default_model setting. On a single-GPU
   setup the second warm could swap the first out — so the user's
   chat model wasn't necessarily resident when their first message
   landed. Now we just pull the supporting models without warming
   them; only user-configured chat models get warm.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-22 16:42:33 -04:00
bvandeusen 83f1676d72 feat(journal): auto-scheduler for curator (Phase 2)
The curator now runs automatically every 15 minutes against any
journal conversation that has user messages newer than its last
curator run. Manual triggers from Phase 1b still work and now also
stamp the timestamp so the scheduler doesn't double-process.

Migration 0048:
- conversations.last_curator_run_at (timestamptz, nullable).
- Partial index ix_conversations_journal_last_curator on the column
  filtered to conversation_type='journal'. The scheduler's candidate
  query is "journal AND (NULL OR stale)" so an index narrowed to
  journal rows is the right shape — index size stays small even on
  instances with many non-journal conversations.

models/conversation.py:
- New `last_curator_run_at` column on Conversation. DateTime imported.

services/curator_scheduler.py (new):
- IntervalTrigger every 15 min via BackgroundScheduler (same pattern
  as journal_scheduler.py).
- _candidate_conversations(): SELECT journal conversations where the
  newest user message is newer than last_curator_run_at (or NULL).
  Capped at 20 per sweep so a backlog after downtime doesn't stall
  the scheduler.
- _sweep() processes candidates sequentially under an asyncio.Lock
  so overlapping ticks can't double-fire on the same conversation.
  Failed runs leave the timestamp alone — natural retry on next sweep.
- start_/stop_curator_scheduler() wired into app.py boot/shutdown.

routes/journal.py:
- Manual /api/journal/curator/run/<conv_id> stamps last_curator_run_at
  on success. Errors don't stamp so the scheduler retries.

What's still pending:
- Phase 3: feedback loop (curator summary into chat context). Currently
  the curator's summary lives in the run result but doesn't reach the
  chat model.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-22 10:07:12 -04:00
bvandeusen b1226d4e16 feat(versions): daily 03:00 UTC auto-pin scan scheduler
BackgroundScheduler with a single CronTrigger fires scan_all_users_for
_auto_pins via asyncio.run_coroutine_threadsafe (mirrors the journal-
scheduler pattern). Wired into app startup/shutdown alongside the other
schedulers.
2026-05-13 13:58:34 -04:00
bvandeusen dbd9f00061 refactor: hard-cut RSS infrastructure (scope C)
Removes the entire RSS feature surface — feeds, items, embeddings, reactions,
discussion-note flow, briefing news context, settings, env-vars, and DB
tables. Keeps the URL-generic article-reader (the read_article LLM tool)
under a clean module so the LLM can still fetch arbitrary article content
from URLs the user provides.

Backend:
- New services/article_fetcher.py — single source of trafilatura URL→text
- New services/tools/article.py — read_article tool (was nested under tools/rss)
- Delete services/rss.py, rss_classifier.py, rss_filtering.py, article_context.py
- Delete services/tools/rss.py
- Delete models/rss_feed.py (RssFeed, RssItem), models/rss_item_embedding.py
- services/embeddings.py: drop upsert/semantic_search/backfill RSS helpers
- services/llm.py: remove _build_briefing_article_context, briefing-conv branch,
  ARTICLE_DISCUSS_SEED skip-RAG branch; drop get_rss_items / add_rss_feed from
  the actions list
- services/generation_task.py: drop _maybe_save_article_discussion_note + caller
- routes/chat.py: drop /api/chat/from-article/<id> endpoint
- routes/journal.py: re-import via web.py refactor (article_fetcher path)
- services/tools/__init__.py: register `article`, drop `rss`
- services/tools/_registry.py: drop the requires=='rss' check
- app.py: drop backfill_rss_item_embeddings + backfill_rss_article_content tasks
- config.py: prose-only edit (no env var change — RSS env vars were never first-class)

Frontend:
- stores/settings.ts: drop rssEnabled
- SettingsView.vue: drop the RSS-classification mention
- api/client.ts: drop openArticleInChat (the from-article endpoint is gone)

Tests:
- Delete tests/test_rss_service.py, test_news_api.py, test_article_reading.py

Migration:
- 0042_drop_rss: DROP TABLE rss_item_embeddings, rss_item_reactions, rss_items,
  rss_feeds; DELETE settings rows for rss_enabled / briefing_*_topics

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-04-26 12:33:30 -04:00
bvandeusen ce76a003f7 feat(journal): /api/journal/* routes blueprint + cosine helper unit tests
Endpoints:
- GET/PUT /api/journal/config — per-user journal config
- GET /api/journal/today — today's journal conversation, generates daily prep on demand
- GET /api/journal/day/<iso_date> — past day's journal
- GET /api/journal/days — list of dates with journal content
- POST /api/journal/trigger-prep — manual regeneration of prep
- GET /api/journal/moments — list/search moments with filters
- PATCH /api/journal/moments/<id> — edit content/tags/pinned + junctions
- DELETE /api/journal/moments/<id>

Blueprint registered in app.py.
tests/test_journal_search.py — cosine helper coverage.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-04-25 22:40:56 -04:00
bvandeusen d9ab538ef4 feat(journal): backend services (moments, search, prep, scheduler, pipeline)
- services/moments.py — CRUD with embedding sync + entity-link helpers
- services/journal_search.py — three-mode search (temporal / entity / semantic)
  with notes-RAG isolation; cosine-similarity helper unit-tested
- services/journal_prep.py — gathers tasks/events/weather/news/projects/
  recent-moments/open-threads into a structured prep block
- services/journal_scheduler.py — per-user APScheduler cron for daily prep,
  follows the BackgroundScheduler + threadsafe-async pattern from event_scheduler
- services/journal_pipeline.py — system prompt (persona + calibration rules)
  with last-48h ambient moments injection
- app.py: wire journal scheduler start/stop hooks
- routes/settings.py: re-add live-reschedule on timezone change (now via
  journal_scheduler.update_user_schedule)

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-04-25 22:37:39 -04:00
bvandeusen 7602bf2293 feat(briefing): hard-cut tear-down
Backend:
- Delete briefing services (pipeline, scheduler, conversations, profile, tools)
- Delete routes/briefing.py + remove blueprint registration
- Move _get_temp_unit into services/weather.get_temp_unit (reads top-level temp_unit setting)
- Rename briefing_preferences.py → rss_filtering.py (functions are RSS-specific)
- Strip briefing scheduler hooks from app.py
- Strip briefing scheduler call from routes/settings.py
- Update test imports (test_rss_service, test_tz_helpers)

Frontend:
- Delete BriefingView, BriefingSetupWizard, BriefingToolStatusRow
- Strip /briefing route + nav links (AppHeader, KnowledgeView)
- Strip Settings → Briefing tab + state + functions + imports
- Strip briefing-intermediate handling from ChatMessage
- Hide /news route + nav links (NewsView depended on briefing endpoints; orphaned in tree)
- Drop unused useSettingsStore from AppHeader

The Android BriefingScreen lives in a separate repo and is not touched here.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-04-25 22:33:37 -04:00
bvandeusen b743754ec2 fix(csp): allow Google Fonts for Fraunces typeface
Add fonts.googleapis.com to style-src and fonts.gstatic.com to
font-src in Content-Security-Policy header.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-04-16 20:45:22 -04:00
bvandeusen 6cf880506d fix(voice): allow WASM compilation in CSP for Silero VAD
Add 'wasm-unsafe-eval' to script-src and blob: to worker-src in
Content-Security-Policy header. Required by onnxruntime-web to compile
the Silero VAD ONNX model. Also surface VAD init errors as a toast
instead of silent console log.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-04-16 20:10:29 -04:00
bvandeusen 0becc1439b fix(llm): correct context sizing, honor think requests, broaden delete
Three related fixes uncovered while benchmarking qwen3:14b against 8b:

- pick_num_ctx was only counting message content, missing the ~15K
  tokens of tool schemas. num_ctx=8192 was being selected while actual
  prompt_tokens hit 14K+, causing silent prompt truncation on every
  tool-using request. Now includes json.dumps(tools) in the estimate.
  KV cache priming in app.py and routes/settings.py also fetches tools
  so the primed num_ctx matches what real chat requests will use.

- _should_think's heuristic classifier was overriding explicit
  think=true requests from the frontend toggle and MCP, gating on
  message length and regex patterns. Now a pass-through — the caller
  is the source of truth. quick_capture hardcodes think=False since
  it's a fast classification path that was relying on the old gating.

- delete_note description only mentioned "note or task", so the model
  refused to call it for entries created by save_person / save_place /
  create_list. Description now explicitly lists all five note_types it
  handles.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-04-12 15:32:52 -04:00
Bryan Van Deusen 3f3156db07 feat(ollama): configurable per-model keep_alive durations
Replace the hardcoded "2h" keep_alive everywhere with a helper that
returns OLLAMA_KEEP_ALIVE_MAIN (default 30m) for the interactive model
and OLLAMA_KEEP_ALIVE_BACKGROUND (default 10m) for the background
model. Lets the main model release VRAM during long idle periods
while keeping it warm enough for bursty chat use, and stops the
sporadic background model from camping VRAM it rarely needs.

Seven call sites updated to route through llm.keep_alive_for(model):
the streaming helpers, generate_completion, the two startup warmers,
the settings KV-cache primer, and the chat warmer endpoint.

Override via env vars: OLLAMA_KEEP_ALIVE_MAIN, OLLAMA_KEEP_ALIVE_BACKGROUND.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-10 14:13:32 -04:00
bvandeusen 94d21c4512 fix(settings): audit pass — model auto-pull on startup, background_model empty-string bug, base URL validation
- Startup now pulls Config.OLLAMA_MODEL (system default chat model) — previously only
  embedding and background models were pulled; the primary chat model was skipped
- _warm_user_models expanded to also pull user-configured default_model and
  background_model overrides that are missing from Ollama, rather than logging and
  skipping them; pulls run before warm/KV-cache priming
- Add background_model to _MODEL_KEYS in settings route so clearing the dropdown
  deletes the DB row instead of saving "", which caused Ollama failures in tag
  suggestions, title generation, project summaries, and RSS classification
- Add http/https scheme validation to PUT /api/admin/base-url matching the CalDAV
  route pattern; a bad value no longer silently breaks invite/password-reset links
- Update admin voice config description: "Reload models" button exists to avoid
  a server restart, so the old "restart required" text was misleading

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-04 14:08:14 -04:00
bvandeusen edfed6b5bb feat(events): recurring expansion, CalDAV pull sync, past search, reminders
- RRULE expansion: list_events now expands recurring events into
  individual occurrences within the query window using python-dateutil
- CalDAV pull sync: new caldav_sync.py + POST /api/events/sync route;
  imports remote events into the internal store by caldav_uid
- Past event search: search_events accepts include_past=true to search
  historical events; exposed in the LLM tool definition
- Internal reminders: migration 0037 adds reminder_minutes +
  reminder_sent_at columns; event_scheduler.py checks every 5 min and
  fires push notifications; CalDAV sync job runs hourly
- reminder_minutes now stored and returned in create/update routes + tools

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-04 12:15:37 -04:00
bvandeusen ef55bcb560 feat(llm): adaptive num_ctx tiers + fix KV cache priming num_ctx mismatch
Adds pick_num_ctx() which selects the smallest context window tier
(8192, 16384, 32768) that fits the current messages with 25% headroom,
capped at OLLAMA_NUM_CTX. Threads num_ctx through generation_task.py so
every chat request uses the computed tier rather than a fixed 16384.

Fixes a critical cache miss bug: KV cache priming in app.py and
settings.py was sending requests without num_ctx, so Ollama sized the
cache at its model default (different from the 16384 real requests used),
forcing a full model reload on the first real user message. Both priming
sites now call pick_num_ctx() and pass the matching value.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-03 11:47:39 -04:00
bvandeusen 4792e6459b perf(startup): prime Ollama KV cache with system prompt on warm-up
After loading each user's chat model into VRAM, send a minimal chat
request with the real system prompt (num_predict=1) to populate the
KV cache. The first real user message then only needs to process its
own tokens rather than the full ~5,600-token system prompt, reducing
cold-start TTFT from ~25s to <1s.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-03 10:47:59 -04:00
bvandeusen 750a91898a perf(llm): route background tasks to dedicated model to preserve KV cache
Background tasks (title generation, tag suggestions, project summaries,
RSS classification) were using qwen3:8b and wiping its KV cache after
every response, preventing prefix cache hits on subsequent user messages.

Adds OLLAMA_BACKGROUND_MODEL (default: qwen2.5:0.5b) config var and
routes all background LLM calls to it, keeping qwen3:8b's KV cache
warm between user messages for consistent sub-second TTFT.

Also adds infinite scroll to KnowledgeView (replaces load-more button)
and bakes spaCy en_core_web_sm into the Docker image to eliminate the
pip install on every startup.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-03 01:33:54 -04:00
bvandeusen 80f30b705d feat: Knowledge view + entity types (People, Places, Lists)
Data model:
- Migration 0036: adds note_type TEXT (default 'note') and metadata JSONB
  to the notes table; index on note_type
- Note model: entity_type property, note_type/metadata in to_dict()
- create_note() accepts note_type and metadata params

Backend:
- /api/knowledge — unified paginated endpoint: type/tag/sort/q filters,
  semantic search via embeddings, excludes tasks
- /api/knowledge/tags — distinct tags across knowledge objects
- New LLM tools: create_person, create_place, create_list, add_to_list,
  clear_checked_items — all wired into execute_tool()
- People and places auto-injected as compact summary into LLM system prompt

Frontend:
- KnowledgeView replaces HomeView at /; left filter panel (type+tag),
  toolbar (search, sort, graph toggle), card grid with type-aware cards
  (indigo=note, emerald=person, amber=place, sky=list), load-more pagination
- Today bar: upcoming events, overdue task count, Briefing/Chat links
- Floating mini-chat sticky to bottom: creates/continues a conversation
  inline, message history expands upward, close button ends session
- Graph panel: toggles as a 420px right panel at full viewport width
- AppHeader: Knowledge, Chat, Briefing, Calendar, Tasks, Projects
- Router: / → KnowledgeView; /knowledge redirect; HomeView import removed

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-31 18:01:03 -04:00
bvandeusen e613485474 feat: full article fetching with trafilatura + html2text cleanup
- Add trafilatura + html2text to dependencies
- Replace custom HTMLStripper with html2text for RSS feed content
- Fetch full article text via httpx + trafilatura after each new item is stored;
  falls back to RSS-provided content if fetch/extraction fails
- Raise CONTENT_MAX_CHARS from 2000 to 50000 (TEXT column, no migration needed)
- Re-embed items with full article content once enrichment completes
- Startup backfill enriches existing items with short content (<1000 chars)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-30 16:33:27 -04:00
bvandeusen a773c11aa0 feat: RSS embeddings, semantic news in chat, article-to-chat, richer briefings
- Embed RSS items at fetch time (nomic-embed-text); backfill at startup
- Semantic news search injected into chat system prompt ("Recent News You've Seen")
  when items match query above 0.55 cosine threshold (independent of note RAG)
- "Discuss in chat" button on news cards — creates a seeded conversation with
  the article title + full content, navigates directly to the new chat
- Briefing compilation now passes 500-char article excerpts (not just headlines)
  to the LLM and uses 8192 num_ctx to accommodate the larger prompt

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-30 15:12:38 -04:00
bvandeusen dba41879ed feat: structured user profile with LLM-learned preferences
Replaces the freeform briefing-profile note with a DB-backed user_profiles
table. Users can edit job/industry/expertise/response preferences/interests/
work schedule via a new Settings → Profile tab. The LLM appends nightly
observations; at 14+ entries they are auto-consolidated into a learned_summary.
Profile context is injected into both briefing and chat system prompts.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-30 14:17:30 -04:00
bvandeusen 2a8c0cfa56 fix: set keep_alive to 2h on all Ollama requests
Prevents models from sitting in VRAM indefinitely. Applies to both
streaming chat calls and the non-streaming generate_completion path,
as well as the startup warm-up request.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-30 10:21:59 -04:00
bvandeusen eaf70500b8 feat: move voice enable/model config to admin UI
Replace VOICE_ENABLED env var gate with DB-backed admin setting.

- services/voice_config.py: reads voice_enabled + voice_stt_model from
  admin user's settings row (falls back to env var defaults)
- routes/admin.py: GET/PUT /api/admin/voice for admin configuration
- routes/voice.py, services/stt.py, services/tts.py: read enabled/model
  from DB via voice_config instead of Config directly
- app.py: always schedule model loaders at startup; they self-gate on
  the DB setting so no conditional needed at the call site
- SettingsView.vue: Voice section in Admin → Config tab (enable toggle +
  STT model dropdown); user Voice tab now points to admin panel when disabled

No env var required to test — enable via Settings → Admin → Config → Voice.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-29 20:22:18 -04:00
bvandeusen 6f84d90dff feat: voice S2S — faster-whisper STT, Kokoro TTS, PTT overlay
Implements full speech-to-speech pipeline (all 4 phases):

Backend (Phase 1):
- services/stt.py: lazy WhisperModel singleton, run_in_executor transcription
- services/tts.py: lazy KPipeline singleton, WAV synthesis at 24kHz/16-bit
- routes/voice.py: /api/voice/status, /voices, /transcribe, /synthesise
- config.py: VOICE_ENABLED, STT_BACKEND, STT_MODEL, TTS_BACKEND env vars
- app.py: load STT/TTS models at startup when VOICE_ENABLED=true
- llm.py: voice_mode + voice_speech_style params inject speak-naturally prefix
- generation_task.py: voice_mode passed through from chat route
- chat.py: "voice" conversation type allowed + excluded from retention cleanup
- pyproject.toml + Dockerfile: faster-whisper, kokoro, soundfile dependencies

Frontend (Phases 2–4):
- composables/useVoiceRecorder.ts: MediaRecorder PTT wrapper
- composables/useVoiceAudio.ts: AudioContext WAV playback wrapper
- BriefingView.vue: Listen button (TTS read-aloud), auto-TTS mode, mic PTT
- VoiceOverlay.vue: global floating PTT button; creates/reuses voice conv;
  full record→transcribe→stream→TTS flow; Space bar hold-to-talk via App.vue
- SettingsView.vue: Voice tab (status badge, speech style, voice/speed)
- App.vue: mounts VoiceOverlay; Space keydown/keyup fires voice:ptt-toggle
- api/client.ts: getVoiceStatus, getVoiceList, transcribeAudio, synthesiseSpeech

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-29 20:03:38 -04:00
bvandeusen 00643c778e security: fix 10 vulnerabilities from security audit
- SSRF: block private/internal URLs in image cache fetch
- SSRF: block private/internal URLs in RSS feed fetch (scheme guard)
- SSRF: block private/internal URLs in CalDAV URL setting
- Auth: require login for GET /api/images/<id> (was unauthenticated)
- Auth: restrict Ollama model pull/delete to admin users only
- Info disclosure: remove email from /api/users/search response
- OAuth: skip email-based account linking when email_verified is false
- Config: raise hard error on default SECRET_KEY when SECURE_COOKIES=true
- Rate limit: document proxy header requirement; add startup warning
- XSS: remove src/alt from global DOMPurify ADD_ATTR allowlist

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-29 00:37:13 -04:00
bvandeusen ebc79b34f9 feat(rag): RAG scoping and context isolation controls
- Migration 0030: add conversations.rag_project_id (NULL=orphan-only,
  -1=all notes, positive=project), projects.auto_summary and
  projects.summary_updated_at
- Three-value scope semantics thread from build_context() → semantic
  search and keyword fallback via orphan_only + effective_project_id
- Project summarization background job (generate_project_summary,
  backfill_project_summaries) called via Ollama; triggered on project
  update and note saves (debounced 1h); runs at startup
- New LLM tools: search_projects (SequenceMatcher scoring on
  title+description+auto_summary) and set_rag_scope (persists to DB,
  workspace-guarded, emits new_rag_scope in SSE done event)
- execute_tool() accepts conv_id + workspace_project_id; generation_task
  passes both and captures scope changes for SSE done enrichment
- Frontend: Conversation type gets rag_project_id; chat store adds
  ragProjectId computed + updateRagScope(); SSE done handler syncs scope
- ChatView: replace sidebar ProjectSelector with a scope chip pill above
  the input bar, animated dropdown, pulse on model-driven scope change

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-25 17:44:39 -04:00
bvandeusen 8d330afc6d feat(calendar): events REST API blueprint and registration
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-25 12:53:52 -04:00
bvandeusen 553c38200a feat(fable-mcp): build wheel in Docker image, serve download, add admin log tool, and Settings install UI
- Dockerfile: build fable-mcp wheel into /app/dist/ during image build
- routes/fable_mcp_dist.py: GET /api/fable-mcp/info + /download endpoints
- app.py: register fable_mcp_dist_bp
- fable_mcp/tools/admin.py: get_app_logs() hitting /api/admin/logs
- fable_mcp/server.py: fable_get_app_logs MCP tool
- SettingsView: "Fable MCP" section in API Keys tab with download button and install instructions
- client.ts: getFableMcpInfo() helper
- ci.yml: add fable-mcp/** to trigger paths

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-24 17:26:05 -04:00
bvandeusen 9cd0de3883 fix(briefing): deadlock in scheduler startup
start_briefing_scheduler was called from before_serving (event loop thread)
and used run_coroutine_threadsafe(...).result() which blocks the calling
thread waiting for the coroutine to complete — but since the calling thread
IS the event loop, the coroutine could never run, causing a 10s timeout and
zero jobs scheduled.

Fix: make start_briefing_scheduler async and await _get_briefing_enabled_users()
directly. Also use asyncio.create_task for the catch-up rather than
run_coroutine_threadsafe. The background thread jobs (_run_user_slot_sync)
continue to use run_coroutine_threadsafe correctly since they run on the
APScheduler thread, not the event loop thread.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-24 02:18:30 -04:00
bvandeusen a2ba90160c feat: kanban status buttons, task back-nav, RSS UI, weather search, briefing fixes
Project view:
- Add inline status advance buttons on kanban task cards (todo→in_progress,
  in_progress→done); buttons reveal on hover, stop link navigation

Task viewer:
- Back button navigates to task's project instead of /tasks when project_id set
- Esc key navigates to project (or /tasks); blurs focused element first

Quick capture:
- Use user's configured model instead of hardcoded Config.OLLAMA_MODEL
- Remove create_project from classifier prompt (tool not offered, caused
  task-shaped inputs to silently fall through to note fallback)

Briefing scheduler:
- Fix get_event_loop() → get_running_loop() so background thread uses the
  correct hypercorn event loop (jobs were scheduling but never executing)
- Suppress bare greeting when both LLM synthesis lanes return empty

RSS feed UI (SettingsView):
- Show last-fetched age, category badge, and feed URL per row
- Category input field when adding a feed
- Refresh all button: fetches latest items, reloads list, toasts with count
- Enter key submits add-feed form; better empty-state hint with example feeds

Weather tool:
- Accept any city/region name in addition to 'home'/'work'/'all'
- Geocodes via Nominatim + fetches live from Open-Meteo for arbitrary queries

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-24 00:42:01 -04:00
bvandeusen 47b4af281b feat: API key routes, search endpoint, conversation type wiring
- GET/POST/DELETE /api/api-keys blueprint registered
- GET /api/search?q=&content_type=&limit= semantic search endpoint
- create_conversation gains conversation_type param (default "chat")
- cleanup_old_conversations excludes mcp type from retention sweep
- POST /api/chat/conversations accepts conversation_type body field

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-23 21:01:04 -04:00
bvandeusen d131621de6 feat: APScheduler briefing slots, catch-up logic, profile close-out
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-11 20:07:16 -04:00
bvandeusen ae3dff0952 feat: briefing API routes — feeds CRUD, weather, config
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-11 19:46:04 -04:00
bvandeusen c7ef709633 Multi-user sharing, groups, and in-app notifications
Backend:
- Alembic migration 0025: groups, group_memberships, project_shares,
  note_shares, notifications tables
- models: Group, GroupMembership, ProjectShare, NoteShare, Notification
- services/access.py: permission resolution (viewer/editor/admin/owner)
- services/groups.py + routes/groups.py: full group CRUD + membership
- services/sharing.py + routes/shares.py: project/note sharing API
- services/notifications.py: in-app notification create/list/mark-read
- routes/in_app_notifications.py: GET/POST notification endpoints
- routes/users.py: user search endpoint
- services/projects.py + services/notes.py: *_for_user variants
- routes updated to use *_for_user on get/list; adds permission field
- app.py: register all new blueprints

Frontend:
- api/client.ts: ShareEntry, GroupEntry, NotificationEntry types + helpers
- stores/notifications.ts: Pinia store with polling
- NotificationBell.vue: bell icon + badge + dropdown toggle + 60s poll
- NotificationsPanel.vue: unread notification list with mark-all-read
- ShareDialog.vue: teleport modal for sharing projects/notes with users/groups
- SharedWithMeView.vue: /shared route listing shared projects and notes
- AppHeader: NotificationBell, Shared nav link
- ProjectView, NoteViewerView, TaskViewerView: Share button + ShareDialog
- SettingsView: Groups admin tab with create/delete groups + member mgmt
- router: /shared route

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-11 15:37:00 -04:00
bvandeusen bd9ba9b526 Fix scanner probes returning 200 via SPA catch-all
The 404 handler was unconditionally serving index.html (200) for all
non-API, non-static paths, including scanner probes for .php, .asp, .cgi
etc. Added _SPA_EXTENSIONS set so paths with unknown extensions get a
real 404 instead of a misleading 200.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-09 18:49:25 -04:00
bvandeusen ef141f07f8 UI polish pass: word count, slash commands, task lists, graph peek, bulk delete, export
- WordCount component (toggle words/chars vs read time, persisted mode)
- TipTap: TaskList/TaskItem extensions, slash command menu (H1-H3, lists, code, quote, task)
- Markdown serializer: task list → `- [ ]` / `- [x]` roundtrip
- GraphView: slide-in peek panel for note/task nodes (body, tags, linked nodes); tag nodes still navigate
- ChatView: bulk-select conversations with two-click confirm delete + chat retention policy (default 90d)
- NoteEditorView: pill tab bar with animated edit/preview toggle + WordCount in toolbar
- WorkspaceNoteEditor: inline search with tag matching, inline new-note creation, WordCount
- WorkspaceTaskPanel: task body rendered in slide-over + Edit link
- Settings: data export (Markdown ZIP / JSON) via GET /api/export
- Accessibility: skip-to-content link, aria-labels on all icon-only buttons
- Fix: export route used non-existent fabledassistant.database — corrected to async_session()
- Fix: ToolCallRecord.status type now includes "running" (was causing TS build error)
- Dockerfile: upgrade npm to latest before install to suppress major-version notice
- npm audit fix: patched minimatch (ReDoS) and rollup (path traversal) — 0 vulnerabilities

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-07 23:15:01 -05:00
bvandeusen 320e879788 Workspace polish: milestone selector, streaming fix, push fix, log quieting
- WorkspaceTaskPanel: add milestone <select> in task detail (PATCH /api/notes/:id),
  replace delete ✕ with trash can SVG icon
- WorkspaceView: scroll to bottom when streaming ends so final message is visible
  without a page refresh
- ToolCallCard: fix search_notes result count (was reading data.total; tool returns
  data.count), so results no longer show "0 found"
- push.py: switch from deprecated WebPusher().send(vapid_private_key=...) to
  webpush() function (pywebpush 2.x API compatibility)
- app.py: downgrade /api/health, /api/chat/status, and static asset requests from
  INFO to DEBUG in after_request logger to reduce log noise at default log level

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-07 12:58:09 -05:00
bvandeusen 48f070f773 Project-aware assist, link suggestions, project-scoped RAG, semantic search tool, SSE race fix
- Writing assistant: inject project notes as context (definition-tagged first), wikilink suggestions
- Link suggestions: server-side endpoint finds unlinked term occurrences, NoteEditorView sidebar panel
- Project-scoped RAG: ChatView ProjectSelector filters semantic+keyword search to selected project
- Semantic search tool: LLM search_notes upgraded to hybrid semantic (0.40 threshold) + keyword merge
- SSE race condition fix: drain remaining events after stream loop exits in chat.py and notes.py
- RAG_AUTO_SNIPPET raised 800→4000; sidebar include uses full note body; MAX_BODY_CHARS 8000→24000
- Enter-to-submit on writing assistant instruction textareas (note and task editors)
- DiffView: equal-line collapsing with 3-line context around changes

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-06 14:02:54 -05:00
bvandeusen 9bf047ec45 Task work log, inline writing assistant, task editor sidebar layout
Backend:
- Migration 0021: task_logs table (FK → notes + users, CASCADE, indexed)
- models/task_log.py: SQLAlchemy model with to_dict()
- services/task_logs.py: CRUD with ownership checks, _UNSET sentinel for optional duration clear
- routes/task_logs.py: GET/POST/PATCH/DELETE /api/tasks/<id>/logs
- services/tools.py: log_work LLM tool (resolves task by title, creates log entry)
- services/generation_task.py: retry assist generation up to 3× on HTTP 500

Frontend:
- types/task.ts: TaskLog interface
- TaskLogSection.vue: chronological work log with date+time timestamps, duration badge, inline edit, autofocus
- InlineAssistPanel.vue: streaming preview + diff review rendered inline in editor column
- useAssist.ts: removed chatStore.chatReady gate; toast notifications for errors
- NoteEditorView.vue + TaskEditorView.vue: inline assist panel, aside restricted to idle state
- TaskEditorView.vue: two-column layout (editor+log left, metadata sidebar right), body defaults to Preview, sidebarOpen accordion for mobile
- editor-shared.css: .assist-active-hint style

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-05 13:05:26 -05:00
bvandeusen 47c9cb39a2 Auto-generate VAPID keys at startup; no manual configuration needed
On first boot, ensure_vapid_keys() generates a fresh EC P-256 key pair
and saves it to /data/vapid_keys.json (inside the app_data volume) so
it survives container restarts. Subsequent boots load from that file.

VAPID_PRIVATE_KEY / VAPID_PUBLIC_KEY env vars still take precedence for
deployments that prefer to manage keys externally.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-02 23:07:30 -05:00
bvandeusen 7d532b82a6 Fix startup warm-up to use user model settings instead of env var
Startup no longer auto-warms Config.OLLAMA_MODEL. Instead it queries
all distinct default_model values from user settings, cross-references
with Ollama's installed models, and warms only the intersection.

Models that users have selected but not yet installed are skipped with
an info log — they are never auto-pulled. The embedding model pull
behaviour is unchanged.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-02 22:58:20 -05:00
bvandeusen 012eb1d46b Add Projects, Milestones, RAG auto-inject, push notifications, PWA, tag normalisation
## Projects & Milestones (Phases A + G)
- New models: Project, Milestone (Project → Milestone → Task hierarchy)
- notes table: project_id + milestone_id FKs; parent_id FK constraint activated
- Migrations: 0017 (projects), 0018 (push_subscriptions), 0019 (events), 0020 (milestones)
- Services: projects.py, milestones.py (CRUD + progress tracking)
- Routes: /api/projects + /api/projects/<id>/milestones
- LLM tools: create/list/get/update project; create/list milestone; project + milestone + parent_task params on note/task tools
- Frontend: ProjectListView (stacked milestone bars), ProjectView (milestone-grouped kanban), ProjectSelector, MilestoneSelector, NoteEditorView + TaskEditorView updated

## RAG Auto-injection (Phase B)
- Notes ≥0.60 cosine similarity auto-injected into system prompt (max 3, 800 chars each)
- excluded_note_ids param; ChatView "Auto-included" sidebar section

## Summarisation improvements (Phase C)
- Threshold 20→30, keep-recent 6→8, max_tokens 200→400
- Two-pass summarisation for histories >50 messages

## Browser push notifications (Phase E)
- PushSubscription model + migration; pywebpush dependency
- /api/push routes; VAPID config; fire-and-forget on generation complete
- Frontend: sw.js, push store, Settings toggle

## PWA manifest (Phase F)
- manifest.json, Apple meta tags, service worker registration in main.ts

## Tag normalisation
- All tags lowercased + deduplicated at backend (create_note/update_note) and frontend (TagInput sanitize)
- Note/Task types gain project_id + milestone_id fields; store signatures updated

## CalDAV
- Radicale embedded server reverted; back to user-configured external CalDAV

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-02 20:52:21 -05:00
bvandeusen 3d7be5888e Remove intent model entirely; quick-capture uses primary model
The separate intent model (OLLAMA_INTENT_MODEL / qwen2.5:7b) is removed
from every part of the system. All classification now uses the primary model.

Changes:
- config.py: remove OLLAMA_INTENT_MODEL
- intent.py: remove classify_intent() and all supporting infrastructure
  (_SYSTEM_PROMPT_TEMPLATE, _RESEARCH_PREFIX, _PRIOR_WORK_REFS); file now
  only contains the quick-capture classifier
- quick_capture.py: classify_capture_intent() now called with Config.OLLAMA_MODEL
- generation_task.py: remove intent_model_setting DB lookup and get_setting import;
  history summarization and research pipeline use the primary model directly
- research.py: remove intent_model parameter from run_research_pipeline() and
  _generate_sub_queries(); both use the model param throughout
- routes/settings.py: remove intent_model from model-key validation and response
- app.py: remove intent model pre-warming at startup
- SettingsView.vue: remove Intent Model selector and related refs/state

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-02 18:41:49 -05:00
bvandeusen efe0a15b8c Add image search with local cache (Phase 23)
Images found via SearXNG are fetched server-side, stored on disk, and
served from /api/images/<id> — the user's browser never contacts the
original image host.  Original URLs are preserved for citation.

New files:
- alembic/versions/0016_add_image_cache.py  — image_cache table
- src/fabledassistant/models/image_cache.py — SQLAlchemy model
- src/fabledassistant/services/images.py    — fetch/store/serve logic
- src/fabledassistant/routes/images.py      — GET /api/images/<id>

Modified:
- config.py: IMAGE_CACHE_DIR (/data/images), IMAGE_MAX_BYTES (5 MB)
- research.py: _search_searxng_images() — SearXNG categories=images
- tools.py: _IMAGE_TOOLS def + search_images branch in execute_tool
- intent.py: search_images routing rule (explicit visual language only)
- app.py: register images_bp
- docker-compose.yml: image_cache named volume mounted at /data/images
- ToolCallCard.vue: "image_search" label

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-01 12:55:10 -05:00
bvandeusen b079c583df Add /api/quick-capture endpoint for mobile/external clients
Single POST that classifies natural-language text and creates the
appropriate item (note, task, event, or todo) in one synchronous
request — no SSE, no conversation context needed.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-28 21:56:16 -05:00
bvandeusen b23e78b0ac Warm both chat and intent models into VRAM on startup
ensure_model only downloads a model if missing; it does not load it
into VRAM. The frontend warm call only covers the chat model (and only
after a user opens the dashboard). This left qwen2.5:1.5b (intent) cold,
causing simultaneous cold-load 500s when the first chat arrived.

Now both Config.OLLAMA_MODEL and Config.OLLAMA_INTENT_MODEL are warmed
at startup (after ensuring they're installed) via a fire-and-forget
/api/generate call with keep_alive=30m. The embedding model is still
pulled but not warmed (it's loaded on demand during backfill).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-25 23:18:07 -05:00
bvandeusen 667ccd70cd Security audit, bug fixes, auto-save, and writing assistant improvements
Backend security & correctness:
- Add rate_limit.py: sliding-window rate limiter (asyncio) applied to login,
  register, forgot/reset password endpoints (10/60s or 5/300s per IP)
- app.py: add security headers in after_request (X-Frame-Options, CSP,
  X-Content-Type-Options, Referrer-Policy) using setdefault to preserve SSE headers
- auth.py: refactor duplicate login_required/admin_required into shared _check_auth()
- config.py: add TRUST_PROXY_HEADERS for proxy-aware client IP resolution
- routes/auth.py: rate limiting, _client_ip() helper, cleaned-up reset_password route
- routes/chat.py, notes.py, tasks.py: int() DoS fix on last_event_id; limit capped
  at 500; date.fromisoformat() wrapped in try/except → 400 on invalid dates
- services/auth.py: fix Setting.user_id update bug (filter on NULL not user.id);
  reset_password_with_token returns int|None (user_id) instead of bool
- services/backup.py: add _security_notice to full backup JSON export
- services/assist.py: system prompt explicitly preserves markdown list structure
  and nested indented sub-items

Infrastructure:
- docker-compose.yml: add healthcheck on app service (/api/health, 10s interval)
- .dockerignore: prevent secrets/node_modules/__pycache__/.env.* leaking into build

Frontend bug fixes:
- TaskCard.vue, TaskViewerView.vue: fix isOverdue() timezone bug (ISO string compare)
- useAssist.ts: accept() now resets state to idle when document changed since proposal
- stores/chat.ts: fix memory leak in _pollUntilLoaded() (try/catch around fetchStatus)
- TiptapEditor.vue: selection offset uses closest-match strategy (not first-match)
- utils/markdown.ts: explicit DOMPurify config with FORCE_BODY; remove as const
  (DOMPurify expects mutable string[])

New features:
- Auto-save (5-minute interval) in NoteEditorView and TaskEditorView — only when
  editing an existing dirty record; silent on error, shows "Auto-saved" toast
- sectionParser.ts: top-level bullet/numbered list items are now individual sections
  in the AI Assist panel (previously treated as one undifferentiated block)
- editor-shared.css: extracted ~500 lines of CSS duplicated between both editors;
  includes .inline-assist-btn at global scope (required for teleported elements)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-19 20:08:44 -05:00
bvandeusen d6f4a6dbb6 Add semantic note search (nomic-embed-text) and per-conversation note cache
- New NoteEmbedding model + migration 0014 stores float embeddings (JSONB)
- services/embeddings.py: get_embedding, upsert_note_embedding,
  semantic_search_notes (cosine similarity), backfill_note_embeddings
- build_context() now tries semantic search first, falls back to keyword search;
  accepts cached_note_ids to reuse last-turn notes and stabilise the system
  prompt prefix for Ollama's KV cache
- generation_buffer.py: per-conversation note ID cache (get/set/clear)
- generation_task.py: passes cached IDs into build_context, updates cache
  after each turn, and invalidates it after create_note/update_note/create_task
- app.py: pulls nomic-embed-text at startup and launches a background backfill
  to embed all existing notes (30 s delay so Ollama has time to load the model)
- routes/notes.py + services/tools.py: fire-and-forget embedding update on
  every note create or update via the API or LLM tool calls

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-18 21:44:58 -05:00
bvandeusen 504091bfcf Pull intent model on startup alongside main model
If OLLAMA_INTENT_MODEL is configured and differs from OLLAMA_MODEL,
both are pulled concurrently as fire-and-forget tasks at startup.
Deduplicates via a set so pulling the same model twice is never attempted.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-18 19:37:36 -05:00