fix(llm): correct context sizing, honor think requests, broaden delete
Three related fixes uncovered while benchmarking qwen3:14b against 8b: - pick_num_ctx was only counting message content, missing the ~15K tokens of tool schemas. num_ctx=8192 was being selected while actual prompt_tokens hit 14K+, causing silent prompt truncation on every tool-using request. Now includes json.dumps(tools) in the estimate. KV cache priming in app.py and routes/settings.py also fetches tools so the primed num_ctx matches what real chat requests will use. - _should_think's heuristic classifier was overriding explicit think=true requests from the frontend toggle and MCP, gating on message length and regex patterns. Now a pass-through — the caller is the source of truth. quick_capture hardcodes think=False since it's a fast classification path that was relying on the old gating. - delete_note description only mentioned "note or task", so the model refused to call it for entries created by save_person / save_place / create_list. Description now explicitly lists all five note_types it handles. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
@@ -38,12 +38,20 @@ def keep_alive_for(model: str) -> str:
|
||||
return Config.OLLAMA_KEEP_ALIVE_MAIN
|
||||
|
||||
|
||||
def pick_num_ctx(messages: list[dict]) -> int:
|
||||
"""Return the smallest context tier that fits *messages* with 25% headroom.
|
||||
def pick_num_ctx(messages: list[dict], tools: list[dict] | None = None) -> int:
|
||||
"""Return the smallest context tier that fits *messages* + *tools* with 25% headroom.
|
||||
|
||||
The ``tools`` JSON schemas are a large, often-overlooked chunk of the prompt.
|
||||
With ~40 tools in the registry the schemas alone can be 6-10K tokens — enough
|
||||
that omitting them from the estimate causes silent prompt truncation.
|
||||
|
||||
Stays at or below Config.OLLAMA_NUM_CTX (the configured ceiling).
|
||||
"""
|
||||
total_chars = sum(len(m.get("content") or "") for m in messages)
|
||||
if tools:
|
||||
# Serialize the same way Ollama will see them. json.dumps gives us a
|
||||
# faithful char count for the schema payload without any guesswork.
|
||||
total_chars += len(json.dumps(tools))
|
||||
estimated_tokens = int(total_chars / 3.5)
|
||||
needed = int(estimated_tokens * 1.25) + 256 # 25% headroom + output buffer
|
||||
cap = Config.OLLAMA_NUM_CTX
|
||||
|
||||
Reference in New Issue
Block a user