When the intent router detects a tool call, the acknowledgment sentence
and the tool now execute concurrently via asyncio.gather. The acknowledgment
uses the small intent model (already in VRAM) with max_tokens=40, so it
completes in ~200-400ms — the user sees text almost immediately instead of
staring at a status label for the full main-model TTFT (~22s).
The acknowledgment text is:
- Streamed to the client as a chunk event (clears the status spinner)
- Included in the assistant message for round 1 so the main LLM continues
coherently from where the acknowledgment left off
- Recorded in TTFT timing (acknowledgment counts as first token)
Varied phrasing is enforced in the system prompt so responses feel natural
rather than formulaic.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
bvandeusen
1b63371bb3