perf(llm): route background tasks to dedicated model to preserve KV cache

Background tasks (title generation, tag suggestions, project summaries,
RSS classification) were using qwen3:8b and wiping its KV cache after
every response, preventing prefix cache hits on subsequent user messages.

Adds OLLAMA_BACKGROUND_MODEL (default: qwen2.5:0.5b) config var and
routes all background LLM calls to it, keeping qwen3:8b's KV cache
warm between user messages for consistent sub-second TTFT.

Also adds infinite scroll to KnowledgeView (replaces load-more button)
and bakes spaCy en_core_web_sm into the Docker image to eliminate the
pip install on every startup.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
2026-04-03 01:33:54 -04:00
parent 888b736ecd
commit 750a91898a
9 changed files with 53 additions and 27 deletions
+1 -1
View File
@@ -238,7 +238,7 @@ async def save_response_as_note(user_id: int, message_id: int) -> dict:
},
{"role": "user", "content": msg.content[:2000]},
]
title = await generate_completion(prompt_messages, model)
title = await generate_completion(prompt_messages, Config.OLLAMA_BACKGROUND_MODEL)
title = title.strip().strip('"\'').strip()[:100]
except Exception:
logger.warning("Failed to generate note title, using fallback", exc_info=True)