feat(ollama): configurable per-model keep_alive durations
Replace the hardcoded "2h" keep_alive everywhere with a helper that returns OLLAMA_KEEP_ALIVE_MAIN (default 30m) for the interactive model and OLLAMA_KEEP_ALIVE_BACKGROUND (default 10m) for the background model. Lets the main model release VRAM during long idle periods while keeping it warm enough for bursty chat use, and stops the sporadic background model from camping VRAM it rarely needs. Seven call sites updated to route through llm.keep_alive_for(model): the streaming helpers, generate_completion, the two startup warmers, the settings KV-cache primer, and the chat warmer endpoint. Override via env vars: OLLAMA_KEEP_ALIVE_MAIN, OLLAMA_KEEP_ALIVE_BACKGROUND. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -28,6 +28,12 @@ class Config:
|
||||
# project summaries, RSS classification). Using a separate model keeps the
|
||||
# main model's KV cache intact between user messages, enabling prefix cache hits.
|
||||
OLLAMA_BACKGROUND_MODEL: str = os.environ.get("OLLAMA_BACKGROUND_MODEL", "qwen2.5:3b")
|
||||
# Ollama keep_alive — how long a model stays resident in VRAM after its last
|
||||
# request. Main model gets a longer window since it's used interactively;
|
||||
# the background model is called sporadically and doesn't need to camp VRAM.
|
||||
# Format matches Ollama's duration strings: "30m", "10m", "1h", "0s", "-1" (forever).
|
||||
OLLAMA_KEEP_ALIVE_MAIN: str = os.environ.get("OLLAMA_KEEP_ALIVE_MAIN", "30m")
|
||||
OLLAMA_KEEP_ALIVE_BACKGROUND: str = os.environ.get("OLLAMA_KEEP_ALIVE_BACKGROUND", "10m")
|
||||
# KV cache context window for generation. Keep this as small as practical —
|
||||
# a larger context forces more KV cache into CPU RAM, drastically slowing prefill.
|
||||
# 16384 covers ~30+ message conversations with our system prompt comfortably.
|
||||
|
||||
Reference in New Issue
Block a user