fix(llm): correct context sizing, honor think requests, broaden delete

Three related fixes uncovered while benchmarking qwen3:14b against 8b:

- pick_num_ctx was only counting message content, missing the ~15K
  tokens of tool schemas. num_ctx=8192 was being selected while actual
  prompt_tokens hit 14K+, causing silent prompt truncation on every
  tool-using request. Now includes json.dumps(tools) in the estimate.
  KV cache priming in app.py and routes/settings.py also fetches tools
  so the primed num_ctx matches what real chat requests will use.

- _should_think's heuristic classifier was overriding explicit
  think=true requests from the frontend toggle and MCP, gating on
  message length and regex patterns. Now a pass-through — the caller
  is the source of truth. quick_capture hardcodes think=False since
  it's a fast classification path that was relying on the old gating.

- delete_note description only mentioned "note or task", so the model
  refused to call it for entries created by save_person / save_place /
  create_list. Description now explicitly lists all five note_types it
  handles.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
2026-04-12 15:32:52 -04:00
parent a6fe1c0d7c
commit 0becc1439b
6 changed files with 38 additions and 61 deletions
+6 -1
View File
@@ -16,6 +16,7 @@ async def _prime_kv_cache_bg(user_id: int, model: str) -> None:
"""Fire-and-forget: prime Ollama's KV cache with the user's system prompt."""
import httpx
from fabledassistant.services.llm import build_context, pick_num_ctx
from fabledassistant.services.tools import get_tools_for_user
try:
messages, _ = await build_context(
user_id=user_id,
@@ -23,7 +24,11 @@ async def _prime_kv_cache_bg(user_id: int, model: str) -> None:
current_note_id=None,
user_message=" ",
)
num_ctx = pick_num_ctx(messages)
# Size the prime to match what real chat requests will use, including
# tool schemas — otherwise Ollama reloads the model on the first real
# request and throws away the cache we just built.
tools = await get_tools_for_user(user_id)
num_ctx = pick_num_ctx(messages, tools=tools)
from fabledassistant.services.llm import keep_alive_for
async with httpx.AsyncClient(timeout=120.0) as client:
await client.post(