perf(llm): route background tasks to dedicated model to preserve KV cache
Background tasks (title generation, tag suggestions, project summaries, RSS classification) were using qwen3:8b and wiping its KV cache after every response, preventing prefix cache hits on subsequent user messages. Adds OLLAMA_BACKGROUND_MODEL (default: qwen2.5:0.5b) config var and routes all background LLM calls to it, keeping qwen3:8b's KV cache warm between user messages for consistent sub-second TTFT. Also adds infinite scroll to KnowledgeView (replaces load-more button) and bakes spaCy en_core_web_sm into the Docker image to eliminate the pip install on every startup. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
+2
-1
@@ -19,7 +19,8 @@ RUN --mount=type=cache,target=/root/.cache/pip \
|
||||
# Install CPU-only torch first so pip doesn't pull full CUDA wheels (~2 GB) for kokoro/transformers.
|
||||
RUN --mount=type=cache,target=/root/.cache/pip \
|
||||
pip install torch --index-url https://download.pytorch.org/whl/cpu \
|
||||
&& pip install faster-whisper kokoro soundfile
|
||||
&& pip install faster-whisper kokoro soundfile \
|
||||
&& python -m spacy download en_core_web_sm
|
||||
|
||||
# Build the fable-mcp wheel so it can be served for download
|
||||
COPY fable-mcp/ fable-mcp/
|
||||
|
||||
Reference in New Issue
Block a user