Reduce perceived latency: move context build into task, title fire-and-forget, think:False on aux calls

- build_context() moved from route handler into run_generation() background task.
  The 202 response now returns immediately; client connects to SSE before
  note search / URL fetch begins, so 'Building context...' status is visible.
- _generate_title() runs in a fire-and-forget asyncio.create_task() after the
  'done' SSE event fires. Users see their response complete 2–5s sooner on new
  conversations; title appears later in the sidebar without blocking the stream.
- generate_completion() now sets think:False and accepts a max_tokens limit.
  Intent classifier passes max_tokens=200 (JSON only), title generator passes
  max_tokens=30 (short title), eliminating qwen3 thinking-mode overhead on these
  auxiliary calls.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
2026-02-18 18:50:37 -05:00
parent 765e99bb24
commit 92bf2768b6
4 changed files with 49 additions and 38 deletions
+8 -2
View File
@@ -173,12 +173,18 @@ async def stream_chat_with_tools(
break
async def generate_completion(messages: list[dict], model: str) -> str:
async def generate_completion(messages: list[dict], model: str, max_tokens: int = 4096) -> str:
"""Non-streaming chat completion, returns full response text."""
async with httpx.AsyncClient(timeout=httpx.Timeout(1800.0, connect=30.0, read=300.0)) as client:
resp = await client.post(
f"{Config.OLLAMA_URL}/api/chat",
json={"model": model, "messages": messages, "stream": False},
json={
"model": model,
"messages": messages,
"stream": False,
"think": False,
"options": {"num_predict": max_tokens},
},
)
resp.raise_for_status()
data = resp.json()