GPU support, parallel intent+context, and increased context window

Docker Compose:
- Enable Ollama GPU passthrough (nvidia, count: all) in both dev and prod files
- Add OLLAMA_FLASH_ATTENTION=1 (faster attention on GPU in both files)
- Add OLLAMA_MAX_LOADED_MODELS=2 and OLLAMA_KEEP_ALIVE=30m to prod (was already in dev)
- Remove 8G memory limit from prod Ollama service (CPU-bound constraint, no longer valid)

llm.py:
- Increase num_ctx 16384 → 32768 in stream_chat and stream_chat_with_tools (GPU VRAM allows it)
- Increase num_predict cap 4096 → 8192 for tool-augmented responses

generation_task.py:
- Parallelize build_context, get_tools_for_user, and get_setting all from the start
- As soon as tools list is ready (fast DB call), launch classify_intent as an asyncio.Task
- Await build_context and classify_intent together via asyncio.gather
- Intent result is pre-computed before the generation loop; loop just reads pre_intent on round 0
- intent_ms timing now reflects wall-clock time from intent start to completion

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
2026-02-18 19:29:31 -05:00
parent 7ff60eb8a6
commit 931a059e9f
4 changed files with 58 additions and 50 deletions
+3 -4
View File
@@ -80,7 +80,7 @@ async def stream_chat(
options: dict | None = None,
) -> AsyncGenerator[str, None]:
"""Stream chat completion from Ollama, yielding content chunks."""
merged_options = {"num_ctx": 16384}
merged_options = {"num_ctx": 32768}
if options:
merged_options.update(options)
payload: dict = {"model": model, "messages": messages, "stream": True, "options": merged_options}
@@ -121,10 +121,9 @@ async def stream_chat_with_tools(
ChatChunk(type="tool_calls") is yielded. Always ends with
ChatChunk(type="done").
"""
options: dict = {"num_ctx": 16384}
# Disable thinking mode for models like qwen3 — it interferes with tool calling
options: dict = {"num_ctx": 32768}
if tools:
options["num_predict"] = 4096
options["num_predict"] = 8192
payload: dict = {
"model": model,
"messages": messages,