GPU support, parallel intent+context, and increased context window
Docker Compose: - Enable Ollama GPU passthrough (nvidia, count: all) in both dev and prod files - Add OLLAMA_FLASH_ATTENTION=1 (faster attention on GPU in both files) - Add OLLAMA_MAX_LOADED_MODELS=2 and OLLAMA_KEEP_ALIVE=30m to prod (was already in dev) - Remove 8G memory limit from prod Ollama service (CPU-bound constraint, no longer valid) llm.py: - Increase num_ctx 16384 → 32768 in stream_chat and stream_chat_with_tools (GPU VRAM allows it) - Increase num_predict cap 4096 → 8192 for tool-augmented responses generation_task.py: - Parallelize build_context, get_tools_for_user, and get_setting all from the start - As soon as tools list is ready (fast DB call), launch classify_intent as an asyncio.Task - Await build_context and classify_intent together via asyncio.gather - Intent result is pre-computed before the generation loop; loop just reads pre_intent on round 0 - intent_ms timing now reflects wall-clock time from intent start to completion Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
+8
-9
@@ -35,15 +35,14 @@ services:
|
||||
environment:
|
||||
OLLAMA_MAX_LOADED_MODELS: "2"
|
||||
OLLAMA_KEEP_ALIVE: "30m"
|
||||
# To enable GPU support, uncomment the deploy section below
|
||||
# (requires nvidia-container-toolkit)
|
||||
# deploy:
|
||||
# resources:
|
||||
# reservations:
|
||||
# devices:
|
||||
# - driver: nvidia
|
||||
# count: all
|
||||
# capabilities: [gpu]
|
||||
OLLAMA_FLASH_ATTENTION: "1"
|
||||
deploy:
|
||||
resources:
|
||||
reservations:
|
||||
devices:
|
||||
- driver: nvidia
|
||||
count: all
|
||||
capabilities: [gpu]
|
||||
|
||||
volumes:
|
||||
pgdata:
|
||||
|
||||
Reference in New Issue
Block a user