Reduce perceived latency: move context build into task, title fire-and-forget, think:False on aux calls
- build_context() moved from route handler into run_generation() background task. The 202 response now returns immediately; client connects to SSE before note search / URL fetch begins, so 'Building context...' status is visible. - _generate_title() runs in a fire-and-forget asyncio.create_task() after the 'done' SSE event fires. Users see their response complete 2–5s sooner on new conversations; title appears later in the sidebar without blocking the stream. - generate_completion() now sets think:False and accepts a max_tokens limit. Intent classifier passes max_tokens=200 (JSON only), title generator passes max_tokens=30 (short title), eliminating qwen3 thinking-mode overhead on these auxiliary calls. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
@@ -173,12 +173,18 @@ async def stream_chat_with_tools(
|
||||
break
|
||||
|
||||
|
||||
async def generate_completion(messages: list[dict], model: str) -> str:
|
||||
async def generate_completion(messages: list[dict], model: str, max_tokens: int = 4096) -> str:
|
||||
"""Non-streaming chat completion, returns full response text."""
|
||||
async with httpx.AsyncClient(timeout=httpx.Timeout(1800.0, connect=30.0, read=300.0)) as client:
|
||||
resp = await client.post(
|
||||
f"{Config.OLLAMA_URL}/api/chat",
|
||||
json={"model": model, "messages": messages, "stream": False},
|
||||
json={
|
||||
"model": model,
|
||||
"messages": messages,
|
||||
"stream": False,
|
||||
"think": False,
|
||||
"options": {"num_predict": max_tokens},
|
||||
},
|
||||
)
|
||||
resp.raise_for_status()
|
||||
data = resp.json()
|
||||
|
||||
Reference in New Issue
Block a user