fix(chat): surface silent generations instead of empty bubbles

Qwen3:14b sometimes burns output tokens on tool-calling attempts whose
emission doesn't parse into any field we read — eval_count > 0 but no
thinking/content/tool_calls ever stream to the caller. Generation
completes "successfully," the user sees an empty assistant bubble, and
no error is logged. Seen in conv 220 today.

Two safety rails:

- stream_chat_with_tools now tracks whether it yielded anything; when
  Ollama's done frame reports eval_count > 0 with zero yields, log a
  warning including the last ~5 raw frames so the next occurrence leaves
  breadcrumbs for diagnosis.

- run_generation checks the same post-condition after the tool loop
  exits and, if content is empty with no tool calls but output_tokens
  > 0, substitutes a visible fallback message and streams it as a chunk
  so the user gets something readable instead of a blank bubble.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
2026-04-15 17:35:30 -04:00
parent 4228e9a384
commit ba0cb07c91
2 changed files with 43 additions and 1 deletions
+20 -1
View File
@@ -265,20 +265,31 @@ async def stream_chat_with_tools(
) as resp:
await _raise_ollama_error(resp, model)
accumulated_tool_calls: list[dict] = []
# Silent-generation diagnostic: if Ollama reports non-zero eval_count
# but we never yielded any thinking/content/tool_calls, something
# in the frames isn't landing in a field we read. Capture the last
# few frames so we can see what Ollama actually sent.
yielded_anything = False
recent_frames: list[str] = []
async for line in resp.aiter_lines():
if not line.strip():
continue
if len(recent_frames) >= 5:
recent_frames.pop(0)
recent_frames.append(line[:500])
data = json.loads(line)
msg = data.get("message", {})
# Thinking chunks (qwen3 chain-of-thought, only when think=True)
thinking = msg.get("thinking", "")
if thinking:
yielded_anything = True
yield ChatChunk(type="thinking", content=thinking)
# Content chunks
chunk = msg.get("content", "")
if chunk:
yielded_anything = True
yield ChatChunk(type="content", content=chunk)
# Collect tool calls from any message (some models
@@ -294,13 +305,21 @@ async def stream_chat_with_tools(
len(accumulated_tool_calls),
json.dumps(accumulated_tool_calls)[:500],
)
yielded_anything = True
yield ChatChunk(type="tool_calls", tool_calls=accumulated_tool_calls)
else:
logger.debug("Ollama done with no tool calls")
eval_count = data.get("eval_count") or 0
if not yielded_anything and eval_count > 0:
logger.warning(
"Ollama silent generation: model=%s eval_count=%d but no "
"thinking/content/tool_calls were yielded. Last frames: %s",
model, eval_count, recent_frames,
)
yield ChatChunk(
type="done",
prompt_tokens=data.get("prompt_eval_count"),
output_tokens=data.get("eval_count"),
output_tokens=eval_count,
)
break