Phase 22b: Parallel research fetching, streaming synthesis, intent optimizations
research.py:
- Parallelize all 5 SearXNG queries concurrently (200ms stagger via asyncio.gather)
- Parallelize all URL fetches in parallel (asyncio.gather) — up to 15 URLs at once
instead of sequential fetches; biggest performance win (was O(n) × 15s, now ~15s flat)
- _synthesize_note accepts buf: when provided uses stream_chat (num_ctx=16384,
num_predict=8192) to emit tokens into the chat buffer in real time so users see
the note being written; falls back to generate_completion when buf=None
- Added \n\n---\n\n separator before "Research complete!" to cleanly mark boundary
after streamed synthesis content
intent.py:
- classify_intent passes num_ctx=4096 to generate_completion — reduces VRAM pressure
and prefill time for the intent model call on every single request
generation_task.py:
- _INTENT_TRIGGER_WORDS frozenset (~50 action/object/date words) + _should_skip_intent()
skips intent classification for short messages (≤10 words) with no trigger words;
saves 400-800ms model call for conversational replies ("thanks", "okay", etc.)
- Added \n\n---\n\n separator before research "done" text in research_topic branch
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
+36
-7
@@ -12,7 +12,7 @@
|
||||
> Include file-level details in the commit body when the change is non-trivial.
|
||||
|
||||
## Last Updated
|
||||
2026-02-26 — Phase 21: Intent-first pipeline, visible acknowledgment, KV-stable system prompt
|
||||
2026-02-27 — Phase 22: SearXNG web research pipeline + performance improvements (parallel fetching, streaming synthesis, intent skip heuristic, intent num_ctx)
|
||||
|
||||
## Project Overview
|
||||
Fabled Assistant is a self-hosted note-taking and task-tracking application with
|
||||
@@ -260,7 +260,7 @@ fabledassistant/
|
||||
│ ├── __init__.py
|
||||
│ ├── app.py # Quart app factory: SPA via 404 handler, JSON 404/500 for API, request logging, security headers (after_request)
|
||||
│ ├── auth.py # Auth decorators: login_required, admin_required, get_current_user_id — shared _check_auth() helper
|
||||
│ ├── config.py # Config from env vars + Docker secrets file support (_read_secret) + SECURE_COOKIES + TRUST_PROXY_HEADERS + OLLAMA_NUM_CTX (KV cache window, default 8192) + OIDC_ISSUER/CLIENT_ID/CLIENT_SECRET/SCOPES + LOCAL_AUTH_ENABLED + oidc_enabled() classmethod
|
||||
│ ├── config.py # Config from env vars + Docker secrets file support (_read_secret) + SECURE_COOKIES + TRUST_PROXY_HEADERS + OLLAMA_NUM_CTX (KV cache window, default 8192) + OIDC_ISSUER/CLIENT_ID/CLIENT_SECRET/SCOPES + LOCAL_AUTH_ENABLED + oidc_enabled() classmethod + SEARXNG_URL + searxng_enabled() classmethod
|
||||
│ ├── rate_limit.py # In-memory sliding-window rate limiter (asyncio.Lock + defaultdict); is_rate_limited(key, max, window)
|
||||
│ ├── models/
|
||||
│ │ ├── __init__.py # async_session factory, Base, imports all models
|
||||
@@ -284,12 +284,13 @@ fabledassistant/
|
||||
│ │ ├── oauth.py # OIDC/OAuth2 service: get_oidc_config (discovery, cached), build_auth_url (PKCE), exchange_code, get_userinfo, find_or_create_oauth_user (sub lookup → email auto-link → create)
|
||||
│ │ ├── backup.py # Backup/restore: export_full_backup, export_user_backup, restore_full_backup
|
||||
│ │ ├── notes.py # CRUD with user_id isolation, is_task filter, convert, backlinks, search_notes_for_context
|
||||
│ │ ├── llm.py # Ollama interaction: build_context with user_id, streaming (stream_chat + stream_chat_with_tools), ChatChunk dataclass, URL fetching; uses Config.OLLAMA_NUM_CTX for KV cache window
|
||||
│ │ ├── llm.py # Ollama interaction: build_context with user_id, streaming (stream_chat + stream_chat_with_tools), ChatChunk dataclass, URL fetching; generate_completion accepts num_ctx kwarg; uses Config.OLLAMA_NUM_CTX for KV cache window
|
||||
│ │ ├── chat.py # Conversation CRUD with user_id isolation, add_message, save/summarize as note (LLM-titled, chat-tagged)
|
||||
│ │ ├── generation_buffer.py # In-memory SSE event buffer with cancel_event, reconnect support, auto-cleanup; supports chat (int keys) and assist (string keys)
|
||||
│ │ ├── generation_task.py # Background asyncio tasks: run_generation (chat, DB flush, titles, intent-first pipeline + tool loop) + run_assist_generation (lightweight, no DB)
|
||||
│ │ ├── intent.py # Intent routing: classify_intent() makes fast non-streaming LLM call; IntentResult has ack field (one-sentence acknowledgment streamed as TTFT)
|
||||
│ │ ├── tools.py # LLM tool definitions (create/delete note+task, update_note w/tag management, get_note, list_notes, search_notes w/type filter, list_tasks, full CalDAV suite incl. search_todos) + execute_tool dispatcher
|
||||
│ │ ├── generation_task.py # Background asyncio tasks: run_generation (chat, DB flush, titles, intent-first pipeline + tool loop) + run_assist_generation (lightweight, no DB); _INTENT_TRIGGER_WORDS + _should_skip_intent() for skipping intent on conversational messages
|
||||
│ │ ├── intent.py # Intent routing: classify_intent() makes fast non-streaming LLM call (num_ctx=4096); IntentResult has ack field (one-sentence acknowledgment streamed as TTFT)
|
||||
│ │ ├── tools.py # LLM tool definitions (create/delete note+task, update_note w/tag management, get_note, list_notes, search_notes w/type filter, list_tasks, full CalDAV suite incl. search_todos, search_web, research_topic when SearXNG enabled) + execute_tool dispatcher
|
||||
│ │ ├── research.py # SearXNG research pipeline: parallel query/fetch, streaming synthesis; run_research_pipeline() → Note; constants SEARXNG_QUERIES=5, PAGES_PER_QUERY=3, MAX_SYNTHESIS_SOURCES=12, CHARS_PER_SOURCE=2000
|
||||
│ │ ├── tag_suggestions.py # LLM-powered tag suggestions: suggest_tags() builds prompt with existing tags, calls generate_completion, parses JSON response
|
||||
│ │ ├── caldav.py # CalDAV integration: full event lifecycle (create/list/search/update/delete), todos (create/list/search/update/complete/delete), list_calendars, timezone (ZoneInfo), reminders (VALARM), attendees, multi-calendar search
|
||||
│ │ ├── settings.py # Settings CRUD with user_id isolation: get_setting, set_setting, set_settings_batch, get_all_settings
|
||||
@@ -651,6 +652,35 @@ When adding a new migration, follow these conventions:
|
||||
"browse/list notes", tag management via update_note (tag_mode add/remove), search_todos.
|
||||
`generate_completion` (used by intent classifier) retries on HTTP 500 (3 attempts, 3s/6s delays)
|
||||
to handle cold model loading without failing intent classification.
|
||||
- **Intent performance optimizations:**
|
||||
- **Intent skip heuristic:** `_should_skip_intent(msg)` in `generation_task.py` skips the intent
|
||||
model call entirely for short messages (≤10 words) that contain none of the `_INTENT_TRIGGER_WORDS`
|
||||
frozenset (~50 action/object/date words). Saves 400–800ms for conversational replies like "thanks",
|
||||
"can you explain that?", "okay" without risking missed tool calls on longer or action-verb messages.
|
||||
- **Intent `num_ctx=4096`:** Intent classification calls use a 4k context window (override) instead
|
||||
of the default, reducing VRAM pressure and prefill time on every request.
|
||||
- **Web research pipeline (Phase 22):** `research.py` implements a full autonomous research pipeline
|
||||
triggered by "research X and make a note" (intent routes to `research_topic` tool) or via the 🔍
|
||||
Research button in ChatView (sends "Research: {topic}" message).
|
||||
Pipeline stages:
|
||||
1. Intent model generates 5 focused sub-queries as a JSON array (falls back to `[topic]` on parse failure)
|
||||
2. All 5 SearXNG queries execute in parallel with 200ms stagger (avoids hammering rate limiter)
|
||||
3. All unique URLs (up to 15) fetched in parallel via `asyncio.gather`; duplicates deduplicated by URL
|
||||
4. Failed fetches filtered; up to 12 sources passed to synthesis LLM
|
||||
5. Synthesis uses `stream_chat` with `num_ctx=16384`, `num_predict=8192` — tokens stream into
|
||||
the chat buffer in real time (user sees the note being written); minimum 2500 words, 6+ topic-appropriate
|
||||
sections (topic determines section structure, not hardcoded), detailed prose, `## Sources` section
|
||||
6. Note created with `tags=["research"]`; "Research complete!" separator appended to chat
|
||||
Also includes `search_web` tool for lightweight single-query searches (no note created; results
|
||||
returned to LLM for conversational answer). Both tools only added to tool list when `SEARXNG_URL` is set.
|
||||
SearXNG `429` handling: 3-attempt retry with exponential backoff per query.
|
||||
Sub-query JSON parsed with `json.JSONDecoder().raw_decode()` to handle trailing model text.
|
||||
Frontend: `ToolCallCard.vue` handles `web_search` (external links), `research_pending`, `research_note` types.
|
||||
Settings: `GET /api/settings/search?q=` proxy + Search Test section in SettingsView.
|
||||
Settings UI: 2-column grid layout (paired simple cards, `full-width` for complex sections).
|
||||
Docker: `OLLAMA_NUM_PARALLEL=2` to prevent research and live chat queuing each other.
|
||||
SearXNG setup: add app server IP to `botdetection.ip_lists.pass_ip` in SearXNG `settings.yml`
|
||||
to bypass rate limiter for trusted backend requests while keeping it active for public users.
|
||||
- **CalDAV calendar integration:** Per-user CalDAV settings (URL, username, password, calendar name, timezone).
|
||||
LLM tools: `create_event` (all_day, recurrence, timezone, reminder_minutes, attendees, calendar_name),
|
||||
`list_events`, `search_events`, `update_event`, `delete_event`, `list_calendars`,
|
||||
@@ -744,7 +774,6 @@ When adding a new migration, follow these conventions:
|
||||
- Calendar/timeline view for tasks
|
||||
- Import/export (Markdown files, JSON)
|
||||
- Email integration (read/send emails from chat via IMAP/SMTP tools)
|
||||
- Web search support (LLM tool to search the web and summarize results)
|
||||
- Session invalidation on user deletion
|
||||
- **Note relation map (Obsidian-style graph):** Interactive force-directed graph visualizing connections between
|
||||
notes via wikilinks (`[[Title]]`) and shared tags. Nodes = notes/tasks; edges = wikilink references or
|
||||
|
||||
Reference in New Issue
Block a user