Phase 22b: Parallel research fetching, streaming synthesis, intent optimizations

research.py:
- Parallelize all 5 SearXNG queries concurrently (200ms stagger via asyncio.gather)
- Parallelize all URL fetches in parallel (asyncio.gather) — up to 15 URLs at once
  instead of sequential fetches; biggest performance win (was O(n) × 15s, now ~15s flat)
- _synthesize_note accepts buf: when provided uses stream_chat (num_ctx=16384,
  num_predict=8192) to emit tokens into the chat buffer in real time so users see
  the note being written; falls back to generate_completion when buf=None
- Added \n\n---\n\n separator before "Research complete!" to cleanly mark boundary
  after streamed synthesis content

intent.py:
- classify_intent passes num_ctx=4096 to generate_completion — reduces VRAM pressure
  and prefill time for the intent model call on every single request

generation_task.py:
- _INTENT_TRIGGER_WORDS frozenset (~50 action/object/date words) + _should_skip_intent()
  skips intent classification for short messages (≤10 words) with no trigger words;
  saves 400-800ms model call for conversational replies ("thanks", "okay", etc.)
- Added \n\n---\n\n separator before research "done" text in research_topic branch

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
2026-02-27 18:24:15 -05:00
parent 590682a5d2
commit df4c52412d
4 changed files with 133 additions and 40 deletions
+1 -1
View File
@@ -150,7 +150,7 @@ async def classify_intent(
messages.append({"role": "user", "content": user_message})
try:
raw = await generate_completion(messages, model, max_tokens=350)
raw = await generate_completion(messages, model, max_tokens=350, num_ctx=4096)
except Exception:
logger.warning("Intent classification LLM call failed", exc_info=True)
return IntentResult()