GPU support, parallel intent+context, and increased context window
Docker Compose: - Enable Ollama GPU passthrough (nvidia, count: all) in both dev and prod files - Add OLLAMA_FLASH_ATTENTION=1 (faster attention on GPU in both files) - Add OLLAMA_MAX_LOADED_MODELS=2 and OLLAMA_KEEP_ALIVE=30m to prod (was already in dev) - Remove 8G memory limit from prod Ollama service (CPU-bound constraint, no longer valid) llm.py: - Increase num_ctx 16384 → 32768 in stream_chat and stream_chat_with_tools (GPU VRAM allows it) - Increase num_predict cap 4096 → 8192 for tool-augmented responses generation_task.py: - Parallelize build_context, get_tools_for_user, and get_setting all from the start - As soon as tools list is ready (fast DB call), launch classify_intent as an asyncio.Task - Await build_context and classify_intent together via asyncio.gather - Intent result is pre-computed before the generation loop; loop just reads pre_intent on round 0 - intent_ms timing now reflects wall-clock time from intent start to completion Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
@@ -80,7 +80,7 @@ async def stream_chat(
|
||||
options: dict | None = None,
|
||||
) -> AsyncGenerator[str, None]:
|
||||
"""Stream chat completion from Ollama, yielding content chunks."""
|
||||
merged_options = {"num_ctx": 16384}
|
||||
merged_options = {"num_ctx": 32768}
|
||||
if options:
|
||||
merged_options.update(options)
|
||||
payload: dict = {"model": model, "messages": messages, "stream": True, "options": merged_options}
|
||||
@@ -121,10 +121,9 @@ async def stream_chat_with_tools(
|
||||
ChatChunk(type="tool_calls") is yielded. Always ends with
|
||||
ChatChunk(type="done").
|
||||
"""
|
||||
options: dict = {"num_ctx": 16384}
|
||||
# Disable thinking mode for models like qwen3 — it interferes with tool calling
|
||||
options: dict = {"num_ctx": 32768}
|
||||
if tools:
|
||||
options["num_predict"] = 4096
|
||||
options["num_predict"] = 8192
|
||||
payload: dict = {
|
||||
"model": model,
|
||||
"messages": messages,
|
||||
|
||||
Reference in New Issue
Block a user