feat(llm): adaptive num_ctx tiers + fix KV cache priming num_ctx mismatch

Adds pick_num_ctx() which selects the smallest context window tier
(8192, 16384, 32768) that fits the current messages with 25% headroom,
capped at OLLAMA_NUM_CTX. Threads num_ctx through generation_task.py so
every chat request uses the computed tier rather than a fixed 16384.

Fixes a critical cache miss bug: KV cache priming in app.py and
settings.py was sending requests without num_ctx, so Ollama sized the
cache at its model default (different from the 16384 real requests used),
forcing a full model reload on the first real user message. Both priming
sites now call pick_num_ctx() and pass the matching value.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
2026-04-03 11:47:39 -04:00
parent a6888953dc
commit ef55bcb560
4 changed files with 48 additions and 13 deletions
+6 -4
View File
@@ -183,16 +183,18 @@ def create_app() -> Quart:
"""Send a minimal chat request to prime Ollama's KV cache with the user's system prompt.
This ensures the next real user message only needs to process its own tokens
rather than the full ~5,600-token system prompt, cutting TTFT from ~25s to <1s.
rather than the full ~4,650-token system prompt, cutting TTFT from ~25s to <1s.
The num_ctx must match what real requests will use so Ollama doesn't reload.
"""
try:
from fabledassistant.services.llm import build_context
from fabledassistant.services.llm import build_context, pick_num_ctx
messages, _ = await build_context(
user_id=user_id,
history=[],
current_note_id=None,
user_message=" ",
)
num_ctx = pick_num_ctx(messages)
async with httpx.AsyncClient(timeout=120.0) as client:
await client.post(
f"{Config.OLLAMA_URL}/api/chat",
@@ -200,11 +202,11 @@ def create_app() -> Quart:
"model": model,
"messages": messages,
"stream": False,
"options": {"num_predict": 1},
"options": {"num_predict": 1, "num_ctx": num_ctx},
"keep_alive": "2h",
},
)
logger.info("Primed KV cache for user %d with model '%s'", user_id, model)
logger.info("Primed KV cache for user %d with model '%s' (num_ctx=%d)", user_id, model, num_ctx)
except Exception:
logger.warning("Failed to prime KV cache for user %d", user_id, exc_info=True)