perf(llm): route background tasks to dedicated model to preserve KV cache
Background tasks (title generation, tag suggestions, project summaries, RSS classification) were using qwen3:8b and wiping its KV cache after every response, preventing prefix cache hits on subsequent user messages. Adds OLLAMA_BACKGROUND_MODEL (default: qwen2.5:0.5b) config var and routes all background LLM calls to it, keeping qwen3:8b's KV cache warm between user messages for consistent sub-second TTFT. Also adds infinite scroll to KnowledgeView (replaces load-more button) and bakes spaCy en_core_web_sm into the Docker image to eliminate the pip install on every startup. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
@@ -24,6 +24,10 @@ class Config:
|
||||
)
|
||||
OLLAMA_URL: str = os.environ.get("OLLAMA_URL", "http://localhost:11434")
|
||||
OLLAMA_MODEL: str = os.environ.get("OLLAMA_MODEL", "qwen3:latest")
|
||||
# Lightweight model for background tasks (title generation, tag suggestions,
|
||||
# project summaries, RSS classification). Using a separate model keeps the
|
||||
# main model's KV cache intact between user messages, enabling prefix cache hits.
|
||||
OLLAMA_BACKGROUND_MODEL: str = os.environ.get("OLLAMA_BACKGROUND_MODEL", "qwen2.5:0.5b")
|
||||
# KV cache context window for generation. Keep this as small as practical —
|
||||
# a larger context forces more KV cache into CPU RAM, drastically slowing prefill.
|
||||
# 16384 covers ~30+ message conversations with our system prompt comfortably.
|
||||
|
||||
Reference in New Issue
Block a user