Reduce perceived latency: move context build into task, title fire-and-forget, think:False on aux calls

- build_context() moved from route handler into run_generation() background task.
  The 202 response now returns immediately; client connects to SSE before
  note search / URL fetch begins, so 'Building context...' status is visible.
- _generate_title() runs in a fire-and-forget asyncio.create_task() after the
  'done' SSE event fires. Users see their response complete 2–5s sooner on new
  conversations; title appears later in the sidebar without blocking the stream.
- generate_completion() now sets think:False and accepts a max_tokens limit.
  Intent classifier passes max_tokens=200 (JSON only), title generator passes
  max_tokens=30 (short title), eliminating qwen3 thinking-mode overhead on these
  auxiliary calls.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
2026-02-18 18:50:37 -05:00
parent 765e99bb24
commit 92bf2768b6
4 changed files with 49 additions and 38 deletions
+1 -1
View File
@@ -136,7 +136,7 @@ async def classify_intent(
messages.append({"role": "user", "content": user_message})
try:
raw = await generate_completion(messages, model)
raw = await generate_completion(messages, model, max_tokens=200)
except Exception:
logger.warning("Intent classification LLM call failed", exc_info=True)
return IntentResult()