perf(llm): route background tasks to dedicated model to preserve KV cache
Background tasks (title generation, tag suggestions, project summaries, RSS classification) were using qwen3:8b and wiping its KV cache after every response, preventing prefix cache hits on subsequent user messages. Adds OLLAMA_BACKGROUND_MODEL (default: qwen2.5:0.5b) config var and routes all background LLM calls to it, keeping qwen3:8b's KV cache warm between user messages for consistent sub-second TTFT. Also adds infinite scroll to KnowledgeView (replaces load-more button) and bakes spaCy en_core_web_sm into the Docker image to eliminate the pip install on every startup. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
@@ -122,7 +122,7 @@ async def generate_project_summary(user_id: int, project_id: int) -> None:
|
||||
from fabledassistant.services.llm import generate_completion
|
||||
from fabledassistant.config import Config
|
||||
messages = [{"role": "user", "content": prompt}]
|
||||
summary = await generate_completion(messages, model=Config.OLLAMA_MODEL, max_tokens=400)
|
||||
summary = await generate_completion(messages, model=Config.OLLAMA_BACKGROUND_MODEL, max_tokens=400)
|
||||
if not summary:
|
||||
return
|
||||
|
||||
|
||||
Reference in New Issue
Block a user