Reduce perceived latency: move context build into task, title fire-and-forget, think:False on aux calls
- build_context() moved from route handler into run_generation() background task. The 202 response now returns immediately; client connects to SSE before note search / URL fetch begins, so 'Building context...' status is visible. - _generate_title() runs in a fire-and-forget asyncio.create_task() after the 'done' SSE event fires. Users see their response complete 2–5s sooner on new conversations; title appears later in the sidebar without blocking the stream. - generate_completion() now sets think:False and accepts a max_tokens limit. Intent classifier passes max_tokens=200 (JSON only), title generator passes max_tokens=30 (short title), eliminating qwen3 thinking-mode overhead on these auxiliary calls. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
@@ -136,7 +136,7 @@ async def classify_intent(
|
||||
messages.append({"role": "user", "content": user_message})
|
||||
|
||||
try:
|
||||
raw = await generate_completion(messages, model)
|
||||
raw = await generate_completion(messages, model, max_tokens=200)
|
||||
except Exception:
|
||||
logger.warning("Intent classification LLM call failed", exc_info=True)
|
||||
return IntentResult()
|
||||
|
||||
Reference in New Issue
Block a user