Files
FabledScribe/src/fabledassistant
bvandeusen 98d3fca277 Implement optimistic streaming to eliminate intent classification latency
Start the main LLM stream immediately after build_context finishes instead
of waiting for intent classification to complete. Race the two concurrently:

- Intent wins before first token → cancel stream, execute tool (tool path
  unchanged: confirmation, acknowledgment, multi-round loop all preserved)
- First token wins → discard intent, user sees output immediately

For pure chat messages (no tool needed, the common case) this eliminates
the full intent classification RTT from TTFT. For tool calls, intent
typically wins the race since it finishes before the main model produces
its first token, so tool behaviour is unchanged in practice.

Also extracts _drain_queue() as a module-level async generator helper.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-25 23:03:30 -05:00
..