fix(llm): correct context sizing, honor think requests, broaden delete
Three related fixes uncovered while benchmarking qwen3:14b against 8b: - pick_num_ctx was only counting message content, missing the ~15K tokens of tool schemas. num_ctx=8192 was being selected while actual prompt_tokens hit 14K+, causing silent prompt truncation on every tool-using request. Now includes json.dumps(tools) in the estimate. KV cache priming in app.py and routes/settings.py also fetches tools so the primed num_ctx matches what real chat requests will use. - _should_think's heuristic classifier was overriding explicit think=true requests from the frontend toggle and MCP, gating on message length and regex patterns. Now a pass-through — the caller is the source of truth. quick_capture hardcodes think=False since it's a fast classification path that was relying on the old gating. - delete_note description only mentioned "note or task", so the model refused to call it for entries created by save_person / save_place / create_list. Description now explicitly lists all five note_types it handles. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
@@ -11,7 +11,6 @@ from quart import Blueprint, jsonify, request
|
||||
|
||||
from fabledassistant.auth import get_current_user_id, login_required
|
||||
from fabledassistant.config import Config
|
||||
from fabledassistant.services.generation_task import _should_think
|
||||
from fabledassistant.services.llm import stream_chat_with_tools
|
||||
from fabledassistant.services.tools import execute_tool, get_tools_for_user
|
||||
|
||||
@@ -53,11 +52,10 @@ async def quick_capture_route():
|
||||
{"role": "user", "content": text},
|
||||
]
|
||||
|
||||
think = _should_think(text, think_requested=True)
|
||||
|
||||
# Quick capture is a fast classification path — never think.
|
||||
tool_calls: list[dict] = []
|
||||
try:
|
||||
async for chunk in stream_chat_with_tools(messages, model, tools=capture_tools, think=think, num_ctx=4096):
|
||||
async for chunk in stream_chat_with_tools(messages, model, tools=capture_tools, think=False, num_ctx=4096):
|
||||
if chunk.type == "tool_calls" and chunk.tool_calls:
|
||||
tool_calls = chunk.tool_calls
|
||||
except Exception:
|
||||
|
||||
@@ -16,6 +16,7 @@ async def _prime_kv_cache_bg(user_id: int, model: str) -> None:
|
||||
"""Fire-and-forget: prime Ollama's KV cache with the user's system prompt."""
|
||||
import httpx
|
||||
from fabledassistant.services.llm import build_context, pick_num_ctx
|
||||
from fabledassistant.services.tools import get_tools_for_user
|
||||
try:
|
||||
messages, _ = await build_context(
|
||||
user_id=user_id,
|
||||
@@ -23,7 +24,11 @@ async def _prime_kv_cache_bg(user_id: int, model: str) -> None:
|
||||
current_note_id=None,
|
||||
user_message=" ",
|
||||
)
|
||||
num_ctx = pick_num_ctx(messages)
|
||||
# Size the prime to match what real chat requests will use, including
|
||||
# tool schemas — otherwise Ollama reloads the model on the first real
|
||||
# request and throws away the cache we just built.
|
||||
tools = await get_tools_for_user(user_id)
|
||||
num_ctx = pick_num_ctx(messages, tools=tools)
|
||||
from fabledassistant.services.llm import keep_alive_for
|
||||
async with httpx.AsyncClient(timeout=120.0) as client:
|
||||
await client.post(
|
||||
|
||||
Reference in New Issue
Block a user