Phase 21: Intent-first pipeline, visible ack, KV-stable system prompt
Pipeline changes (generation_task.py, intent.py): - Remove optimistic streaming queue/race (_drain_queue deleted) - Remove _generate_acknowledgment — ack now embedded in intent JSON - Round 0: await intent (~400ms), stream ack immediately as TTFT, then execute tool sequentially; chat-only streams directly - IntentResult.ack: one-sentence acknowledgment, intent max_tokens 200→350 - _parse_intent extracts and trims ack field KV cache stability (llm.py, generation_buffer.py, generation_task.py): - build_context: replace cached_note_ids with include_note_ids - Auto-found notes populate context_meta["auto_notes"] for sidebar but are NOT injected into system prompt (--- Related Notes --- removed) - Explicitly included notes injected as --- Included Notes --- - _conv_note_cache dict + get/set/clear functions removed from generation_buffer.py - All clear_conv_note_cache() calls removed Cold model retry (llm.py): - generate_completion (used by classify_intent) retries on HTTP 500: 3 attempts with 3s/6s delays — prevents intent failure during cold load API + frontend (routes/chat.py, stores/chat.ts, views/ChatView.vue, components/ChatPanel.vue): - exclude_note_ids → include_note_ids throughout - ChatView sidebar: Suggested (auto-found, + to include) + In Context (× to remove) - ChatPanel: remove exclude button from context pills; no IDs passed to sendMessage Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
@@ -29,8 +29,6 @@ const noteSearchResults = ref<{ id: number; title: string }[]>([]);
|
||||
const noteSearchLoading = ref(false);
|
||||
let noteSearchTimer: ReturnType<typeof setTimeout> | null = null;
|
||||
|
||||
// Exclude tracking
|
||||
const excludedNoteIds = ref<Set<number>>(new Set());
|
||||
|
||||
const streamingRendered = computed(() => {
|
||||
if (!store.streamingContent) return "";
|
||||
@@ -65,11 +63,7 @@ async function sendMessage() {
|
||||
messageInput.value = "";
|
||||
resetTextareaHeight();
|
||||
scrollToBottom();
|
||||
await store.sendMessage(
|
||||
content,
|
||||
contextNoteId,
|
||||
excludedNoteIds.value.size ? [...excludedNoteIds.value] : undefined,
|
||||
);
|
||||
await store.sendMessage(content, contextNoteId);
|
||||
scrollToBottom();
|
||||
}
|
||||
|
||||
@@ -151,10 +145,6 @@ function removeAttachedNote() {
|
||||
function promoteAutoNote(note: { id: number; title: string }) {
|
||||
attachedNote.value = note;
|
||||
}
|
||||
|
||||
function excludeAutoNote(noteId: number) {
|
||||
excludedNoteIds.value = new Set([...excludedNoteIds.value, noteId]);
|
||||
}
|
||||
</script>
|
||||
|
||||
<template>
|
||||
@@ -198,11 +188,6 @@ function excludeAutoNote(noteId: number) {
|
||||
@click="promoteAutoNote(note)"
|
||||
title="Attach for next message"
|
||||
>+</button>
|
||||
<button
|
||||
class="context-pill-btn exclude"
|
||||
@click="excludeAutoNote(note.id)"
|
||||
title="Exclude from auto-search"
|
||||
>×</button>
|
||||
</span>
|
||||
</div>
|
||||
|
||||
@@ -463,9 +448,6 @@ function excludeAutoNote(noteId: number) {
|
||||
.context-pill-btn.promote:hover {
|
||||
color: var(--color-primary);
|
||||
}
|
||||
.context-pill-btn.exclude:hover {
|
||||
color: var(--color-danger, #e74c3c);
|
||||
}
|
||||
|
||||
/* Input wrapper */
|
||||
.panel-input-wrapper {
|
||||
|
||||
@@ -119,7 +119,7 @@ export const useChatStore = defineStore("chat", () => {
|
||||
async function sendMessage(
|
||||
content: string,
|
||||
contextNoteId?: number | null,
|
||||
excludeNoteIds?: number[],
|
||||
includeNoteIds?: number[],
|
||||
think = false,
|
||||
contextNoteTitle?: string,
|
||||
) {
|
||||
@@ -167,7 +167,7 @@ export const useChatStore = defineStore("chat", () => {
|
||||
{
|
||||
content,
|
||||
context_note_id: contextNoteId,
|
||||
exclude_note_ids: excludeNoteIds?.length ? excludeNoteIds : undefined,
|
||||
include_note_ids: includeNoteIds?.length ? includeNoteIds : undefined,
|
||||
think,
|
||||
},
|
||||
);
|
||||
|
||||
@@ -30,11 +30,12 @@ const noteSearchResults = ref<{ id: number; title: string }[]>([]);
|
||||
const noteSearchLoading = ref(false);
|
||||
let noteSearchTimer: ReturnType<typeof setTimeout> | null = null;
|
||||
|
||||
// Exclude tracking (session-scoped per conversation)
|
||||
const excludedNoteIds = ref<Set<number>>(new Set());
|
||||
// Explicitly included notes (user clicked "+ include" in sidebar)
|
||||
const includedNoteIds = ref<Set<number>>(new Set());
|
||||
const includedNotes = ref<{ id: number; title: string }[]>([]);
|
||||
|
||||
// Persistent context notes — populated as assistant responds, cleared on conv change
|
||||
const contextNotes = ref<{ id: number; title: string }[]>([]);
|
||||
// Suggested notes — auto-found by search, not yet included
|
||||
const suggestedNotes = ref<{ id: number; title: string }[]>([]);
|
||||
|
||||
let prevConvId: number | null = null;
|
||||
|
||||
@@ -91,8 +92,9 @@ watch(convId, async (newId) => {
|
||||
}
|
||||
prevConvId = newId ?? null;
|
||||
|
||||
excludedNoteIds.value = new Set();
|
||||
contextNotes.value = [];
|
||||
includedNoteIds.value = new Set();
|
||||
includedNotes.value = [];
|
||||
suggestedNotes.value = [];
|
||||
attachedNote.value = null;
|
||||
store.lastContextMeta = null;
|
||||
if (newId) {
|
||||
@@ -117,12 +119,11 @@ watch(
|
||||
() => store.lastContextMeta?.auto_notes,
|
||||
(newNotes) => {
|
||||
if (!newNotes) return;
|
||||
const excluded = excludedNoteIds.value;
|
||||
const existing = new Set(contextNotes.value.map((n) => n.id));
|
||||
const alreadyIncluded = includedNoteIds.value;
|
||||
const alreadySuggested = new Set(suggestedNotes.value.map((n) => n.id));
|
||||
for (const note of newNotes) {
|
||||
if (!excluded.has(note.id) && !existing.has(note.id)) {
|
||||
contextNotes.value.push(note);
|
||||
existing.add(note.id);
|
||||
if (!alreadyIncluded.has(note.id) && !alreadySuggested.has(note.id)) {
|
||||
suggestedNotes.value.push(note);
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -180,7 +181,7 @@ async function sendMessage() {
|
||||
await store.sendMessage(
|
||||
content,
|
||||
contextNoteId,
|
||||
excludedNoteIds.value.size ? [...excludedNoteIds.value] : undefined,
|
||||
includedNoteIds.value.size ? [...includedNoteIds.value] : undefined,
|
||||
true, // enable thinking in the full chat view
|
||||
contextNoteTitle,
|
||||
);
|
||||
@@ -282,9 +283,20 @@ function removeAttachedNote() {
|
||||
attachedNote.value = null;
|
||||
}
|
||||
|
||||
function excludeAutoNote(noteId: number) {
|
||||
excludedNoteIds.value = new Set([...excludedNoteIds.value, noteId]);
|
||||
contextNotes.value = contextNotes.value.filter((n) => n.id !== noteId);
|
||||
function includeNote(note: { id: number; title: string }) {
|
||||
if (includedNoteIds.value.has(note.id)) return;
|
||||
includedNoteIds.value = new Set([...includedNoteIds.value, note.id]);
|
||||
includedNotes.value.push(note);
|
||||
suggestedNotes.value = suggestedNotes.value.filter((n) => n.id !== note.id);
|
||||
}
|
||||
|
||||
function removeIncludedNote(noteId: number) {
|
||||
includedNoteIds.value = new Set([...includedNoteIds.value].filter((id) => id !== noteId));
|
||||
const removed = includedNotes.value.find((n) => n.id === noteId);
|
||||
includedNotes.value = includedNotes.value.filter((n) => n.id !== noteId);
|
||||
if (removed && !suggestedNotes.value.some((n) => n.id === noteId)) {
|
||||
suggestedNotes.value.push(removed);
|
||||
}
|
||||
}
|
||||
|
||||
// Keyboard shortcuts
|
||||
@@ -420,26 +432,40 @@ onUnmounted(() => {
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<!-- Persistent context sidebar -->
|
||||
<aside v-if="contextNotes.length || attachedNote" class="context-sidebar">
|
||||
<div class="context-sidebar-header">In Context</div>
|
||||
<!-- Context sidebar -->
|
||||
<aside v-if="includedNotes.length || suggestedNotes.length || attachedNote" class="context-sidebar">
|
||||
<!-- IN CONTEXT section -->
|
||||
<template v-if="attachedNote || includedNotes.length">
|
||||
<div class="context-sidebar-header">In Context</div>
|
||||
|
||||
<!-- Manually attached note — pinned until sent -->
|
||||
<div v-if="attachedNote" class="context-note context-note-pinned">
|
||||
<span class="context-note-icon" title="Attached for this message">📌</span>
|
||||
<router-link :to="`/notes/${attachedNote.id}`" class="context-note-name">
|
||||
{{ attachedNote.title }}
|
||||
</router-link>
|
||||
<button class="context-note-remove" @click="removeAttachedNote" title="Remove">×</button>
|
||||
</div>
|
||||
<!-- Manually attached note — pinned until sent -->
|
||||
<div v-if="attachedNote" class="context-note context-note-pinned">
|
||||
<span class="context-note-icon" title="Attached for this message">📌</span>
|
||||
<router-link :to="`/notes/${attachedNote.id}`" class="context-note-name">
|
||||
{{ attachedNote.title }}
|
||||
</router-link>
|
||||
<button class="context-note-remove" @click="removeAttachedNote" title="Remove">×</button>
|
||||
</div>
|
||||
|
||||
<!-- Auto-found notes — persist across turns -->
|
||||
<div v-for="note in contextNotes" :key="note.id" class="context-note">
|
||||
<router-link :to="`/notes/${note.id}`" class="context-note-name">
|
||||
{{ note.title }}
|
||||
</router-link>
|
||||
<button class="context-note-remove" @click="excludeAutoNote(note.id)" title="Remove from context">×</button>
|
||||
</div>
|
||||
<!-- Explicitly included notes -->
|
||||
<div v-for="note in includedNotes" :key="note.id" class="context-note context-note-included">
|
||||
<router-link :to="`/notes/${note.id}`" class="context-note-name">
|
||||
{{ note.title }}
|
||||
</router-link>
|
||||
<button class="context-note-remove" @click="removeIncludedNote(note.id)" title="Remove from context">×</button>
|
||||
</div>
|
||||
</template>
|
||||
|
||||
<!-- SUGGESTED section -->
|
||||
<template v-if="suggestedNotes.length">
|
||||
<div class="context-sidebar-header" :class="{ 'context-sidebar-header-gap': attachedNote || includedNotes.length }">Suggested</div>
|
||||
<div v-for="note in suggestedNotes" :key="note.id" class="context-note context-note-suggested">
|
||||
<router-link :to="`/notes/${note.id}`" class="context-note-name">
|
||||
{{ note.title }}
|
||||
</router-link>
|
||||
<button class="context-note-add" @click="includeNote(note)" title="Add to context">+</button>
|
||||
</div>
|
||||
</template>
|
||||
</aside>
|
||||
</div>
|
||||
|
||||
@@ -729,6 +755,15 @@ onUnmounted(() => {
|
||||
.context-note-name:hover {
|
||||
color: var(--color-primary);
|
||||
}
|
||||
.context-sidebar-header-gap {
|
||||
margin-top: 0.6rem;
|
||||
}
|
||||
.context-note-included {
|
||||
border-color: var(--color-primary);
|
||||
}
|
||||
.context-note-suggested {
|
||||
opacity: 0.85;
|
||||
}
|
||||
.context-note-remove {
|
||||
background: none;
|
||||
border: none;
|
||||
@@ -742,6 +777,20 @@ onUnmounted(() => {
|
||||
.context-note-remove:hover {
|
||||
color: var(--color-danger, #e74c3c);
|
||||
}
|
||||
.context-note-add {
|
||||
background: none;
|
||||
border: none;
|
||||
cursor: pointer;
|
||||
color: var(--color-primary);
|
||||
font-size: 1.1rem;
|
||||
font-weight: 700;
|
||||
line-height: 1;
|
||||
padding: 0 0.1rem;
|
||||
flex-shrink: 0;
|
||||
}
|
||||
.context-note-add:hover {
|
||||
opacity: 0.75;
|
||||
}
|
||||
.messages-inner {
|
||||
margin-top: auto;
|
||||
}
|
||||
|
||||
@@ -113,7 +113,7 @@ async def send_message_route(conv_id: int):
|
||||
if not content:
|
||||
return jsonify({"error": "content is required"}), 400
|
||||
context_note_id = data.get("context_note_id")
|
||||
exclude_note_ids = data.get("exclude_note_ids") or []
|
||||
include_note_ids = data.get("include_note_ids") or []
|
||||
think = bool(data.get("think", False))
|
||||
|
||||
# Reject if generation already running for this conversation
|
||||
@@ -142,7 +142,7 @@ async def send_message_route(conv_id: int):
|
||||
buf, history, model,
|
||||
uid, conv_id, conv.title, content,
|
||||
context_note_id=context_note_id,
|
||||
exclude_note_ids=exclude_note_ids,
|
||||
include_note_ids=include_note_ids,
|
||||
think=think,
|
||||
))
|
||||
|
||||
|
||||
@@ -75,26 +75,6 @@ class GenerationBuffer:
|
||||
# Module-level singleton registry
|
||||
_buffers: dict[int | str, GenerationBuffer] = {}
|
||||
|
||||
# Per-conversation note context cache — maps conv_id → sorted list of note IDs.
|
||||
# Stores the note IDs that were last included in the system prompt so that
|
||||
# subsequent turns in the same conversation can reuse them, stabilizing the
|
||||
# system prompt prefix and improving Ollama's KV cache hit rate.
|
||||
_conv_note_cache: dict[int, list[int]] = {}
|
||||
|
||||
|
||||
def get_conv_note_cache(conv_id: int) -> list[int]:
|
||||
"""Return cached note IDs for a conversation (empty list if none)."""
|
||||
return list(_conv_note_cache.get(conv_id, []))
|
||||
|
||||
|
||||
def set_conv_note_cache(conv_id: int, note_ids: list[int]) -> None:
|
||||
"""Store note IDs to reuse on the next turn of this conversation."""
|
||||
_conv_note_cache[conv_id] = list(note_ids)
|
||||
|
||||
|
||||
def clear_conv_note_cache(conv_id: int) -> None:
|
||||
"""Invalidate the note cache for a conversation (e.g. after a note write)."""
|
||||
_conv_note_cache.pop(conv_id, None)
|
||||
_cleanup_task: asyncio.Task | None = None
|
||||
_GRACE_PERIOD = 60.0 # seconds to keep completed buffers
|
||||
|
||||
|
||||
@@ -21,9 +21,6 @@ from fabledassistant.models.conversation import Message
|
||||
from fabledassistant.services.generation_buffer import (
|
||||
GenerationBuffer,
|
||||
GenerationState,
|
||||
clear_conv_note_cache,
|
||||
get_conv_note_cache,
|
||||
set_conv_note_cache,
|
||||
)
|
||||
from fabledassistant.services.llm import ChatChunk, build_context, generate_completion, stream_chat, stream_chat_with_tools, summarize_history_for_context
|
||||
from fabledassistant.services.chat import update_conversation_title
|
||||
@@ -97,39 +94,6 @@ _TOOL_ACTIONS: dict[str, str] = {
|
||||
}
|
||||
|
||||
|
||||
async def _generate_acknowledgment(user_content: str, tool_name: str, model: str) -> str:
|
||||
"""Generate a brief conversational acknowledgment that runs in parallel with tool execution.
|
||||
|
||||
Uses the intent model (small, fast, already in VRAM) so the sentence is ready
|
||||
within ~200-400ms. Returned string includes a trailing double-newline so the
|
||||
main LLM response starts on a new paragraph.
|
||||
"""
|
||||
action = _TOOL_ACTIONS.get(tool_name, "work on that")
|
||||
messages = [
|
||||
{
|
||||
"role": "system",
|
||||
"content": (
|
||||
"You are a helpful assistant. Write ONE short, natural sentence acknowledging "
|
||||
"what you are about to do. Vary your phrasing — do not always start with "
|
||||
"'Let me'. Be warm and conversational. Do not answer the question yet. "
|
||||
"Output only the sentence, nothing else."
|
||||
),
|
||||
},
|
||||
{
|
||||
"role": "user",
|
||||
"content": f"User said: {user_content}\nYou are about to: {action}",
|
||||
},
|
||||
]
|
||||
try:
|
||||
ack = await generate_completion(messages, model, max_tokens=40)
|
||||
ack = ack.strip()
|
||||
if ack:
|
||||
return ack + "\n\n"
|
||||
except Exception:
|
||||
logger.warning("Failed to generate acknowledgment", exc_info=True)
|
||||
return ""
|
||||
|
||||
|
||||
async def _generate_title(messages: list[dict], model: str) -> str:
|
||||
"""Ask the LLM for a concise conversation title."""
|
||||
# Build conversation text like summarize_conversation_as_note
|
||||
@@ -175,26 +139,6 @@ async def _update_message(
|
||||
await session.commit()
|
||||
|
||||
|
||||
async def _drain_queue(
|
||||
prefetched: list[ChatChunk],
|
||||
queue: asyncio.Queue,
|
||||
) -> AsyncGenerator[ChatChunk, None]:
|
||||
"""Yield pre-fetched chunks then drain remaining chunks from the queue.
|
||||
|
||||
A None sentinel in the queue signals the stream is finished.
|
||||
A BaseException in the queue is re-raised so callers see the error.
|
||||
"""
|
||||
for chunk in prefetched:
|
||||
yield chunk
|
||||
while True:
|
||||
item = await queue.get()
|
||||
if item is None:
|
||||
break
|
||||
if isinstance(item, BaseException):
|
||||
raise item
|
||||
yield item
|
||||
|
||||
|
||||
async def _stream_with_retry(
|
||||
messages: list[dict],
|
||||
model: str,
|
||||
@@ -239,7 +183,7 @@ async def run_generation(
|
||||
conv_title: str,
|
||||
user_content: str,
|
||||
context_note_id: int | None = None,
|
||||
exclude_note_ids: list[int] | None = None,
|
||||
include_note_ids: list[int] | None = None,
|
||||
think: bool = False,
|
||||
) -> None:
|
||||
"""Stream LLM response into buffer with periodic DB flushes."""
|
||||
@@ -266,17 +210,13 @@ async def run_generation(
|
||||
buf.append_event("status", {"status": "Summarizing conversation history..."})
|
||||
history_to_use, history_summary = await summarize_history_for_context(history, intent_model)
|
||||
|
||||
# Phase 3: Build context. Start intent classification concurrently when tools
|
||||
# are available so it runs in parallel with the embedding/DB work in build_context.
|
||||
# We only block on context (need messages to stream) — intent result is consumed
|
||||
# later via a race with the first streaming token.
|
||||
cached_note_ids = get_conv_note_cache(conv_id) or None
|
||||
|
||||
# Phase 3: Build context and start intent classification in parallel.
|
||||
# We block on context (need messages to stream) — intent is consumed
|
||||
# after context is ready, at the start of round 0.
|
||||
context_task = asyncio.create_task(build_context(
|
||||
user_id, history_to_use, context_note_id, user_content,
|
||||
exclude_note_ids=exclude_note_ids,
|
||||
history_summary=history_summary,
|
||||
cached_note_ids=cached_note_ids,
|
||||
include_note_ids=include_note_ids,
|
||||
))
|
||||
|
||||
intent_task: asyncio.Task[IntentResult] | None = None
|
||||
@@ -292,11 +232,6 @@ async def run_generation(
|
||||
|
||||
messages, context_meta = await context_task
|
||||
|
||||
# Update the note cache with whatever notes ended up in context.
|
||||
new_note_ids = context_meta.get("auto_note_ids") or []
|
||||
if new_note_ids:
|
||||
set_conv_note_cache(conv_id, new_note_ids)
|
||||
|
||||
# Emit context event
|
||||
buf.append_event("context", {"context": context_meta})
|
||||
|
||||
@@ -319,89 +254,27 @@ async def run_generation(
|
||||
round_tool_calls: list[dict] = []
|
||||
logger.info("Generation round %d started for conv %d (model=%s)", _round, conv_id, model)
|
||||
|
||||
# --- Round 0 with tools: optimistic streaming ---
|
||||
# Start the main stream immediately (into a queue) while the intent
|
||||
# classifier finishes in the background. Race the two:
|
||||
# • Intent wins before first token → check for tool call, cancel stream if needed
|
||||
# • First token wins → discard intent, stream has already started
|
||||
# --- Round 0 with tools: intent-first pipeline ---
|
||||
# Wait for intent result, then act immediately. The ack sentence
|
||||
# (embedded in the intent JSON) is streamed at TTFT (~400ms), then
|
||||
# the tool runs while the user is reading it.
|
||||
if _round == 0 and tools and intent_task is not None:
|
||||
stream_queue: asyncio.Queue[ChatChunk | BaseException | None] = asyncio.Queue(maxsize=256)
|
||||
intent = await intent_task
|
||||
timing["intent_ms"] = int((time.monotonic() - t_intent) * 1000)
|
||||
|
||||
async def _fill_queue() -> None:
|
||||
# Retry on Ollama 500 (model cold-loading race) up to 2 times.
|
||||
last_exc: BaseException | None = None
|
||||
for attempt in range(3):
|
||||
if attempt > 0:
|
||||
wait = 3.0 * attempt
|
||||
logger.warning(
|
||||
"Ollama stream 500 (attempt %d/3), retrying in %.0fs", attempt, wait
|
||||
)
|
||||
await asyncio.sleep(wait)
|
||||
try:
|
||||
async for c in stream_chat_with_tools(messages, model, tools=tools, think=think):
|
||||
await stream_queue.put(c)
|
||||
last_exc = None
|
||||
break
|
||||
except httpx.HTTPStatusError as exc:
|
||||
last_exc = exc
|
||||
if exc.response.status_code != 500:
|
||||
break # non-500 errors are not retryable
|
||||
except BaseException as exc:
|
||||
last_exc = exc
|
||||
break
|
||||
if last_exc is not None:
|
||||
await stream_queue.put(last_exc)
|
||||
await stream_queue.put(None)
|
||||
|
||||
stream_fill_task = asyncio.create_task(_fill_queue())
|
||||
buf.append_event("status", {"status": "Generating response..."})
|
||||
|
||||
queue_peek = asyncio.create_task(stream_queue.get())
|
||||
race_done, _ = await asyncio.wait(
|
||||
{intent_task, queue_peek},
|
||||
return_when=asyncio.FIRST_COMPLETED,
|
||||
)
|
||||
|
||||
intent = IntentResult()
|
||||
prefetched: list[ChatChunk] = []
|
||||
use_tool_path = False
|
||||
|
||||
if intent_task in race_done and queue_peek not in race_done:
|
||||
# Intent finished before the first streaming token arrived.
|
||||
timing["intent_ms"] = int((time.monotonic() - t_intent) * 1000)
|
||||
intent = intent_task.result()
|
||||
|
||||
if intent.should_execute:
|
||||
# Cancel the optimistic stream — we'll execute a tool instead.
|
||||
stream_fill_task.cancel()
|
||||
queue_peek.cancel()
|
||||
use_tool_path = True
|
||||
else:
|
||||
# No tool needed — collect the first chunk and keep streaming.
|
||||
first = await queue_peek
|
||||
if isinstance(first, BaseException):
|
||||
raise first
|
||||
if first is not None:
|
||||
prefetched.append(first)
|
||||
else:
|
||||
# Stream produced a token before intent finished (or simultaneously).
|
||||
# Discard intent — the response is already on its way.
|
||||
if not intent_task.done():
|
||||
intent_task.cancel()
|
||||
else:
|
||||
timing["intent_ms"] = int((time.monotonic() - t_intent) * 1000)
|
||||
|
||||
first = queue_peek.result() if queue_peek in race_done else await queue_peek
|
||||
if isinstance(first, BaseException):
|
||||
raise first
|
||||
if first is not None:
|
||||
prefetched.append(first)
|
||||
|
||||
if use_tool_path:
|
||||
# === Tool path (intent won the race) ===
|
||||
if intent.should_execute:
|
||||
tool_name = intent.tool_name
|
||||
confirmed = True
|
||||
|
||||
# Stream ack immediately — this becomes TTFT
|
||||
ack_text = (intent.ack or "").strip()
|
||||
if ack_text:
|
||||
ack_with_newline = ack_text + "\n\n"
|
||||
buf.append_event("chunk", {"chunk": ack_with_newline})
|
||||
buf.content_so_far += ack_with_newline
|
||||
if timing["ttft_ms"] is None:
|
||||
timing["ttft_ms"] = int((time.monotonic() - t_start) * 1000)
|
||||
|
||||
confirmed = True
|
||||
if tool_name in _WRITE_TOOLS:
|
||||
loop = asyncio.get_running_loop()
|
||||
confirm_future: asyncio.Future = loop.create_future()
|
||||
@@ -449,24 +322,11 @@ async def run_generation(
|
||||
|
||||
if confirmed:
|
||||
buf.append_event("status", {"status": f"{_TOOL_LABELS.get(tool_name, 'Working')}..."})
|
||||
|
||||
t_tool = time.monotonic()
|
||||
result, ack_text = await asyncio.gather(
|
||||
execute_tool(user_id, tool_name, intent.arguments),
|
||||
_generate_acknowledgment(user_content, tool_name, intent_model),
|
||||
)
|
||||
result = await execute_tool(user_id, tool_name, intent.arguments)
|
||||
timing["tools"].append({"name": tool_name, "ms": int((time.monotonic() - t_tool) * 1000)})
|
||||
logger.info("Intent-routed tool %s result: success=%s", tool_name, result.get("success"))
|
||||
|
||||
if ack_text:
|
||||
buf.append_event("chunk", {"chunk": ack_text})
|
||||
buf.content_so_far += ack_text
|
||||
if timing["ttft_ms"] is None:
|
||||
timing["ttft_ms"] = int((time.monotonic() - t_start) * 1000)
|
||||
|
||||
if result.get("success") and tool_name in {"create_task", "create_note", "update_note", "delete_note", "delete_task"}:
|
||||
clear_conv_note_cache(conv_id)
|
||||
|
||||
tool_record = {
|
||||
"function": tool_name,
|
||||
"arguments": intent.arguments,
|
||||
@@ -478,35 +338,23 @@ async def run_generation(
|
||||
|
||||
messages.append({
|
||||
"role": "assistant",
|
||||
"content": ack_text,
|
||||
"content": buf.content_so_far,
|
||||
"tool_calls": [
|
||||
{"function": {"name": tool_name, "arguments": intent.arguments}}
|
||||
],
|
||||
})
|
||||
messages.append({
|
||||
"role": "tool",
|
||||
"content": json.dumps(result),
|
||||
})
|
||||
continue # Round 1: stream the response incorporating tool result
|
||||
messages.append({"role": "tool", "content": json.dumps(result)})
|
||||
continue # Round 1: stream response with tool result
|
||||
|
||||
# Declined write tool — fall through to stream a fresh response.
|
||||
# Declined write tool — fall through to fresh stream.
|
||||
if cancelled:
|
||||
break
|
||||
|
||||
# === Stream path for round 0 ===
|
||||
# Either intent said no-tool, or a write tool was declined.
|
||||
# For no-tool: drain the already-started queue (prefetched + remaining).
|
||||
# For declined: start a fresh stream (queue was cancelled).
|
||||
# No tool (or declined write tool): stream directly, no queue.
|
||||
buf.append_event("status", {"status": "Generating response..."})
|
||||
t_stream = time.monotonic()
|
||||
|
||||
if use_tool_path:
|
||||
# Declined write tool — the optimistic stream was cancelled, start fresh.
|
||||
stream_source: AsyncGenerator = _stream_with_retry(messages, model, tools, think)
|
||||
else:
|
||||
stream_source = _drain_queue(prefetched, stream_queue)
|
||||
|
||||
async for chunk in stream_source:
|
||||
async for chunk in _stream_with_retry(messages, model, tools, think):
|
||||
if buf.cancel_event.is_set():
|
||||
cancelled = True
|
||||
break
|
||||
@@ -541,9 +389,6 @@ async def run_generation(
|
||||
timing["tools"].append({"name": tool_name, "ms": int((time.monotonic() - t_tool) * 1000)})
|
||||
logger.info("Tool %s result: success=%s", tool_name, result.get("success"))
|
||||
|
||||
if result.get("success") and tool_name in {"create_task", "create_note", "update_note", "delete_note", "delete_task"}:
|
||||
clear_conv_note_cache(conv_id)
|
||||
|
||||
tool_record = {
|
||||
"function": tool_name,
|
||||
"arguments": arguments,
|
||||
@@ -618,9 +463,6 @@ async def run_generation(
|
||||
timing["tools"].append({"name": tool_name, "ms": int((time.monotonic() - t_tool) * 1000)})
|
||||
logger.info("Tool %s result: success=%s", tool_name, result.get("success"))
|
||||
|
||||
if result.get("success") and tool_name in {"create_task", "create_note", "update_note", "delete_note", "delete_task"}:
|
||||
clear_conv_note_cache(conv_id)
|
||||
|
||||
tool_record = {
|
||||
"function": tool_name,
|
||||
"arguments": arguments,
|
||||
|
||||
@@ -22,6 +22,7 @@ class IntentResult:
|
||||
tool_name: str | None = None # None = no tool, just chat
|
||||
arguments: dict = field(default_factory=dict)
|
||||
confidence: str = "high" # "high", "medium", or "low"
|
||||
ack: str | None = None # One-sentence acknowledgment to stream immediately
|
||||
|
||||
@property
|
||||
def should_execute(self) -> bool:
|
||||
@@ -61,8 +62,8 @@ Available tools:
|
||||
{tool_summary}
|
||||
|
||||
Respond with ONLY a JSON object, no other text:
|
||||
- If a tool should be called: {{"tool": "tool_name", "arguments": {{...}}, "confidence": "high"|"medium"|"low"}}
|
||||
- If it's general chat: {{"tool": null, "confidence": "high"}}
|
||||
- If a tool should be called: {{"tool": "tool_name", "arguments": {{...}}, "confidence": "high"|"medium"|"low", "ack": "One short sentence describing what you're about to do."}}
|
||||
- If it's general chat: {{"tool": null, "confidence": "high", "ack": null}}
|
||||
|
||||
Confidence levels:
|
||||
- "high": the intent is clear and all required arguments are unambiguous
|
||||
@@ -99,6 +100,7 @@ Rules:
|
||||
- "read", "open", "show me", "what does X say", "display", "pull up" a specific note → use get_note with query=<note name>.
|
||||
- "list my notes", "show notes", "recent notes", "browse notes", "notes tagged X" → use list_notes (with optional q or tags).
|
||||
- "tag X with Y", "add tag Y to X", "untag Y from X", "remove tag Y from X" → use update_note with tags=[Y] and tag_mode="add" or "remove".
|
||||
- "ack": one short, natural sentence confirming the action (tool path only). Vary phrasing — do not always start with "Let me". Omit (null) for chat-only responses.
|
||||
- Do NOT wrap the JSON in markdown code fences."""
|
||||
|
||||
|
||||
@@ -142,7 +144,7 @@ async def classify_intent(
|
||||
messages.append({"role": "user", "content": user_message})
|
||||
|
||||
try:
|
||||
raw = await generate_completion(messages, model, max_tokens=200)
|
||||
raw = await generate_completion(messages, model, max_tokens=350)
|
||||
except Exception:
|
||||
logger.warning("Intent classification LLM call failed", exc_info=True)
|
||||
return IntentResult()
|
||||
@@ -192,11 +194,15 @@ def _parse_intent(raw: str, tools: list[dict]) -> IntentResult:
|
||||
if not isinstance(arguments, dict):
|
||||
arguments = {}
|
||||
|
||||
ack = parsed.get("ack") or None
|
||||
if ack is not None:
|
||||
ack = ack.strip() or None
|
||||
|
||||
logger.info(
|
||||
"Intent classified: tool=%s, confidence=%s, args=%s",
|
||||
tool_name, confidence, json.dumps(arguments)[:200],
|
||||
)
|
||||
return IntentResult(tool_name=tool_name, arguments=arguments, confidence=confidence)
|
||||
return IntentResult(tool_name=tool_name, arguments=arguments, confidence=confidence, ack=ack)
|
||||
|
||||
|
||||
def _try_json(text: str) -> dict | list | None:
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
import asyncio
|
||||
import json
|
||||
import logging
|
||||
import re
|
||||
@@ -178,21 +179,41 @@ async def stream_chat_with_tools(
|
||||
|
||||
|
||||
async def generate_completion(messages: list[dict], model: str, max_tokens: int = 4096) -> str:
|
||||
"""Non-streaming chat completion, returns full response text."""
|
||||
async with httpx.AsyncClient(timeout=httpx.Timeout(1800.0, connect=30.0, read=300.0)) as client:
|
||||
resp = await client.post(
|
||||
f"{Config.OLLAMA_URL}/api/chat",
|
||||
json={
|
||||
"model": model,
|
||||
"messages": messages,
|
||||
"stream": False,
|
||||
"think": False,
|
||||
"options": {"num_predict": max_tokens},
|
||||
},
|
||||
)
|
||||
resp.raise_for_status()
|
||||
data = resp.json()
|
||||
return data.get("message", {}).get("content", "")
|
||||
"""Non-streaming chat completion, returns full response text.
|
||||
|
||||
Retries up to 2 times on Ollama 500 errors (cold model loading race).
|
||||
"""
|
||||
last_exc: Exception | None = None
|
||||
for attempt in range(3):
|
||||
if attempt > 0:
|
||||
delay = 3.0 * attempt
|
||||
logger.warning(
|
||||
"generate_completion 500 (attempt %d/3), retrying in %.0fs", attempt, delay
|
||||
)
|
||||
await asyncio.sleep(delay)
|
||||
try:
|
||||
async with httpx.AsyncClient(timeout=httpx.Timeout(1800.0, connect=30.0, read=300.0)) as client:
|
||||
resp = await client.post(
|
||||
f"{Config.OLLAMA_URL}/api/chat",
|
||||
json={
|
||||
"model": model,
|
||||
"messages": messages,
|
||||
"stream": False,
|
||||
"think": False,
|
||||
"options": {"num_predict": max_tokens},
|
||||
},
|
||||
)
|
||||
resp.raise_for_status()
|
||||
data = resp.json()
|
||||
return data.get("message", {}).get("content", "")
|
||||
except httpx.HTTPStatusError as exc:
|
||||
last_exc = exc
|
||||
if exc.response.status_code != 500:
|
||||
break
|
||||
except Exception as exc:
|
||||
last_exc = exc
|
||||
break
|
||||
raise last_exc
|
||||
|
||||
|
||||
async def fetch_url_content(url: str) -> str:
|
||||
@@ -307,7 +328,7 @@ async def build_context(
|
||||
user_message: str,
|
||||
exclude_note_ids: list[int] | None = None,
|
||||
history_summary: str | None = None,
|
||||
cached_note_ids: list[int] | None = None,
|
||||
include_note_ids: list[int] | None = None,
|
||||
) -> tuple[list[dict], dict]:
|
||||
"""Build messages array for Ollama with system prompt and context.
|
||||
|
||||
@@ -376,64 +397,58 @@ async def build_context(
|
||||
f"--- End Note ---"
|
||||
)
|
||||
|
||||
# Find related notes to inject into context.
|
||||
# Priority: (1) use cached note IDs from a previous turn in this conversation
|
||||
# (2) try semantic search via nomic-embed-text
|
||||
# (3) fall back to keyword search
|
||||
# The cache stabilises the system prompt prefix so Ollama's KV cache stays warm.
|
||||
# Search for related notes to populate sidebar candidates.
|
||||
# Results are NOT injected into the system prompt automatically — this keeps
|
||||
# the system prompt prefix stable so Ollama's KV cache can reuse prefill state.
|
||||
# Users can explicitly include notes via the sidebar (include_note_ids).
|
||||
search_exclude = set(exclude_set)
|
||||
if current_note_id:
|
||||
search_exclude.add(current_note_id)
|
||||
|
||||
found_notes = []
|
||||
if cached_note_ids:
|
||||
# Load the same notes as last turn — keeps system prompt prefix identical.
|
||||
try:
|
||||
from fabledassistant.services.notes import get_note as _get_note
|
||||
for nid in cached_note_ids:
|
||||
if nid not in search_exclude:
|
||||
n = await _get_note(user_id, nid)
|
||||
if n:
|
||||
found_notes.append(n)
|
||||
except Exception:
|
||||
logger.warning("Failed to load cached notes for context", exc_info=True)
|
||||
found_notes = []
|
||||
# Try semantic search first; fall back to keyword search on failure / no results.
|
||||
try:
|
||||
from fabledassistant.services.embeddings import semantic_search_notes
|
||||
found_notes = await semantic_search_notes(
|
||||
user_id, user_message, exclude_ids=search_exclude or None, limit=3
|
||||
)
|
||||
except Exception:
|
||||
logger.warning("Semantic note search failed, falling back to keyword search", exc_info=True)
|
||||
|
||||
if not found_notes:
|
||||
# Try semantic search first; fall back to keyword search on failure / no results.
|
||||
try:
|
||||
from fabledassistant.services.embeddings import semantic_search_notes
|
||||
found_notes = await semantic_search_notes(
|
||||
user_id, user_message, exclude_ids=search_exclude or None, limit=3
|
||||
)
|
||||
except Exception:
|
||||
logger.warning("Semantic note search failed, falling back to keyword search", exc_info=True)
|
||||
keywords = _extract_keywords(user_message)
|
||||
if keywords:
|
||||
try:
|
||||
found_notes = await search_notes_for_context(
|
||||
user_id, keywords, exclude_ids=search_exclude or None, limit=3
|
||||
)
|
||||
except Exception:
|
||||
logger.warning("Failed to search notes for context", exc_info=True)
|
||||
|
||||
if not found_notes:
|
||||
keywords = _extract_keywords(user_message)
|
||||
if keywords:
|
||||
try:
|
||||
found_notes = await search_notes_for_context(
|
||||
user_id, keywords, exclude_ids=search_exclude or None, limit=3
|
||||
)
|
||||
except Exception:
|
||||
logger.warning("Failed to search notes for context", exc_info=True)
|
||||
|
||||
if found_notes:
|
||||
snippets: list[str] = []
|
||||
for n in found_notes:
|
||||
body_preview = n.body[:2000] if n.body else ""
|
||||
snippets.append(f"- {n.title}: {body_preview}")
|
||||
context_meta["auto_notes"].append({"id": n.id, "title": n.title})
|
||||
system_parts.append(
|
||||
"\n\n--- Related Notes ---\n"
|
||||
+ "\n".join(snippets)
|
||||
+ "\n--- End Related Notes ---"
|
||||
)
|
||||
|
||||
# Expose note IDs so the caller can update the per-conversation cache.
|
||||
# Populate sidebar candidates (never auto-injected).
|
||||
for n in found_notes:
|
||||
context_meta["auto_notes"].append({"id": n.id, "title": n.title})
|
||||
context_meta["auto_note_ids"] = [n.id for n in found_notes]
|
||||
|
||||
# Inject explicitly included notes (user opted in via sidebar click).
|
||||
if include_note_ids:
|
||||
from fabledassistant.services.notes import get_note as _get_note
|
||||
included_snippets: list[str] = []
|
||||
for nid in include_note_ids:
|
||||
try:
|
||||
n = await _get_note(user_id, nid)
|
||||
if n:
|
||||
body_preview = n.body[:2000] if n.body else ""
|
||||
included_snippets.append(f"- {n.title}: {body_preview}")
|
||||
except Exception:
|
||||
logger.warning("Failed to load included note %d for context", nid, exc_info=True)
|
||||
if included_snippets:
|
||||
system_parts.append(
|
||||
"\n\n--- Included Notes ---\n"
|
||||
+ "\n".join(included_snippets)
|
||||
+ "\n--- End Included Notes ---"
|
||||
)
|
||||
|
||||
# Fetch URL content from user message
|
||||
urls = _find_urls(user_message)
|
||||
for url in urls[:2]: # Limit to 2 URLs
|
||||
|
||||
+32
-22
@@ -12,7 +12,7 @@
|
||||
> Include file-level details in the commit body when the change is non-trivial.
|
||||
|
||||
## Last Updated
|
||||
2026-02-26 — Phase 20: Dedicated tag field (chip input), tags no longer extracted from body
|
||||
2026-02-26 — Phase 21: Intent-first pipeline, visible acknowledgment, KV-stable system prompt
|
||||
|
||||
## Project Overview
|
||||
Fabled Assistant is a self-hosted note-taking and task-tracking application with
|
||||
@@ -75,11 +75,15 @@ for AI-assisted features.
|
||||
reconnection support. Frontend uses `fetch()` + `ReadableStream`. Simpler than
|
||||
WebSockets; Quart supports async generators natively for SSE.
|
||||
- **Context building server-side:** Backend fetches URL content and searches notes —
|
||||
frontend just sends the message text + optional note ID + optional exclude list.
|
||||
frontend sends the message text + optional context note ID + optional `include_note_ids`.
|
||||
Keyword extraction uses simple word splitting with stopword filtering (no embeddings).
|
||||
`build_context()` returns `(messages, context_meta)` tuple; metadata includes
|
||||
auto-found note IDs/titles sent to frontend via SSE `context` event before streaming.
|
||||
Multi-word search splits terms into per-word ILIKE with AND logic (not adjacent match).
|
||||
Auto-found notes populate the sidebar but are **not** injected into the system prompt
|
||||
automatically — users must click `+` in the sidebar to include them (stable system
|
||||
prompt prefix enables Ollama KV cache reuse). Explicitly included notes appear as
|
||||
`--- Included Notes ---` in the system prompt.
|
||||
- **Reverse proxy required for production:** The app does not terminate TLS. A
|
||||
reverse proxy (Nginx/Traefik/Caddy) must sit in front of port 5000. Do not
|
||||
expose the app directly to the internet.
|
||||
@@ -283,8 +287,8 @@ fabledassistant/
|
||||
│ │ ├── llm.py # Ollama interaction: build_context with user_id, streaming (stream_chat + stream_chat_with_tools), ChatChunk dataclass, URL fetching; uses Config.OLLAMA_NUM_CTX for KV cache window
|
||||
│ │ ├── chat.py # Conversation CRUD with user_id isolation, add_message, save/summarize as note (LLM-titled, chat-tagged)
|
||||
│ │ ├── generation_buffer.py # In-memory SSE event buffer with cancel_event, reconnect support, auto-cleanup; supports chat (int keys) and assist (string keys)
|
||||
│ │ ├── generation_task.py # Background asyncio tasks: run_generation (chat, DB flush, titles, optimistic streaming + intent racing + tool loop) + run_assist_generation (lightweight, no DB); _drain_queue() async generator helper
|
||||
│ │ ├── intent.py # Intent routing: classify_intent() makes fast non-streaming LLM call to detect tool intent before streaming
|
||||
│ │ ├── generation_task.py # Background asyncio tasks: run_generation (chat, DB flush, titles, intent-first pipeline + tool loop) + run_assist_generation (lightweight, no DB)
|
||||
│ │ ├── intent.py # Intent routing: classify_intent() makes fast non-streaming LLM call; IntentResult has ack field (one-sentence acknowledgment streamed as TTFT)
|
||||
│ │ ├── tools.py # LLM tool definitions (create/delete note+task, update_note w/tag management, get_note, list_notes, search_notes w/type filter, list_tasks, full CalDAV suite incl. search_todos) + execute_tool dispatcher
|
||||
│ │ ├── tag_suggestions.py # LLM-powered tag suggestions: suggest_tags() builds prompt with existing tags, calls generate_completion, parses JSON response
|
||||
│ │ ├── caldav.py # CalDAV integration: full event lifecycle (create/list/search/update/delete), todos (create/list/search/update/complete/delete), list_calendars, timezone (ZoneInfo), reminders (VALARM), attendees, multi-calendar search
|
||||
@@ -317,7 +321,7 @@ fabledassistant/
|
||||
│ │ ├── auth.ts # Auth state: user, isAuthenticated, isAdmin, oauthEnabled, localAuthEnabled, login/register/logout/checkAuth/checkHasUsers
|
||||
│ │ ├── notes.ts # CRUD + tag filter, resolveTitle, convertToTask, convertToNote, fetchBacklinks, fetchAllTags (with toast errors)
|
||||
│ │ ├── tasks.ts # CRUD + status/priority filter, patchStatus (with toast errors)
|
||||
│ │ ├── chat.ts # Conversation CRUD, sendMessage (SSE streaming), status polling (memory-leak-safe _pollUntilLoaded), running models, model warming, updateConversationModel (with toast errors)
|
||||
│ │ ├── chat.ts # Conversation CRUD, sendMessage (SSE streaming, includeNoteIds param), status polling (memory-leak-safe _pollUntilLoaded), running models, model warming, updateConversationModel (with toast errors)
|
||||
│ │ ├── settings.ts # App settings: assistantName, defaultModel, installedModels, defaultChatModel, defaultIntentModel, pullModel, deleteModel (with toast errors)
|
||||
│ │ └── toast.ts # Toast notification state (success/error/warning), 4s auto-dismiss, dismiss(id)
|
||||
│ ├── types/
|
||||
@@ -342,7 +346,7 @@ fabledassistant/
|
||||
│ │ ├── RegisterView.vue # Register form with password confirmation; shows "closed" message when registration disabled
|
||||
│ │ ├── RegisterInviteView.vue # Invitation-based registration: validates token, creates account with pre-set email
|
||||
│ │ ├── UserManagementView.vue # Admin user management: registration toggle, invitations (send/revoke), user list with delete
|
||||
│ │ ├── ChatView.vue # Dedicated /chat page: responsive sidebar (overlay on mobile), bubble messages, note picker, persistent context sidebar (right panel, hidden mobile), model selector in header
|
||||
│ │ ├── ChatView.vue # Dedicated /chat page: responsive sidebar (overlay on mobile), bubble messages, note picker, context sidebar with “In Context” (user-included, ×) + “Suggested” (auto-found, +), model selector in header
|
||||
│ │ ├── HomeView.vue # Chat-first dashboard: quick actions + chat widget (top, full-width), inline response panel, two-column grid (3fr tasks / 2fr notes); task sections: Overdue, Due Today, Due This Week, High Priority, In Progress, Other (capped 10, due-dated first); 8 recent notes; model warming on mount
|
||||
│ │ ├── SettingsView.vue # Settings page: assistant name, chat/intent model dropdowns (populated from installed models), email change (with password confirmation for local-auth users), change password, notifications, CalDAV, SMTP (admin), base URL (admin), data export/restore (admin)
|
||||
│ │ ├── NotesListView.vue # Note list: search, sort, tag filter pills, pagination
|
||||
@@ -354,7 +358,7 @@ fabledassistant/
|
||||
│ ├── components/
|
||||
│ │ ├── LogsView.vue # Admin log viewer: stats summary, category/search/date filters, paginated table with IP column + expandable detail rows (expands on ip_address or details)
|
||||
│ │ ├── AppHeader.vue # Nav bar: brand, nav links (incl. admin Logs), status indicator, theme toggle, user info + logout, hamburger menu (mobile)
|
||||
│ │ ├── ChatPanel.vue # Slide-out chat panel (right side overlay, receives contextNoteId prop), bubble-style messages, floating dark input, note picker, context pills with promote/exclude
|
||||
│ │ ├── ChatPanel.vue # Slide-out chat panel (right side overlay, receives contextNoteId prop), bubble-style messages, floating dark input, note picker, context pills with promote (+) only (no exclude)
|
||||
│ │ ├── ModelSelector.vue # Model dropdown (v-model pattern): fetches installed + running models, hot/cold indicators
|
||||
│ │ ├── DashboardChatInput.vue # Inline chat bar: ModelSelector + note picker + textarea + send button; emits submit event
|
||||
│ │ ├── ToolCallCard.vue # Compact card for tool call results (created/deleted task/note, note content, notes list, search results, CalDAV events/todos, errors) + suggested tag pills with apply-on-click
|
||||
@@ -432,7 +436,7 @@ fabledassistant/
|
||||
| GET | `/api/chat/conversations/:id` | Get conversation with all messages |
|
||||
| DELETE | `/api/chat/conversations/:id` | Delete conversation (cascades to messages) |
|
||||
| PATCH | `/api/chat/conversations/:id` | Update conversation title or model (body: `{title?, model?}`) |
|
||||
| POST | `/api/chat/conversations/:id/messages` | Start generation: save user message, launch background task, return 202 (body: `{content, context_note_id?, exclude_note_ids?}`) |
|
||||
| POST | `/api/chat/conversations/:id/messages` | Start generation: save user message, launch background task, return 202 (body: `{content, context_note_id?, include_note_ids?}`) |
|
||||
| GET | `/api/chat/conversations/:id/generation/stream` | SSE endpoint tailing generation buffer; supports `Last-Event-ID` reconnection; emits `context`, `chunk`, `done`, `error` events |
|
||||
| POST | `/api/chat/conversations/:id/generation/cancel` | Cancel active generation (sets cancel_event, saves partial content) |
|
||||
| POST | `/api/chat/messages/:id/save-as-note` | Save assistant message as a new note (LLM-generated title, tagged `chat`) |
|
||||
@@ -565,9 +569,12 @@ When adding a new migration, follow these conventions:
|
||||
- Background generation with `GenerationBuffer` (in-memory SSE fan-out, `Last-Event-ID` reconnect, 60s cleanup)
|
||||
- Stop generation with partial content preservation
|
||||
- Note-aware context building: current note + keyword search for related notes + URL fetching
|
||||
- **Persistent context sidebar:** Right panel in ChatView accumulates auto-found notes across turns.
|
||||
Manually attached note appears with 📌 pin, clears after send. × excludes from future auto-search.
|
||||
Hidden on mobile (≤768px). Replaces old ephemeral context pills in the message stream.
|
||||
- **Context sidebar (Phase 21):** Right panel in ChatView shows two sections: **Suggested** (auto-found
|
||||
by semantic/keyword search — click `+` to include) and **In Context** (explicitly included by user —
|
||||
click `×` to remove). Removing an included note moves it back to Suggested. Manually attached note
|
||||
appears with 📌 pin in the In Context section, clears after send. Hidden on mobile (≤768px).
|
||||
Auto-found notes are shown as sidebar candidates only — they are NOT injected into the system prompt
|
||||
automatically, keeping the system prompt prefix stable for Ollama KV cache reuse.
|
||||
- Note picker (paperclip) in chat input; attached note title passed to store for optimistic render.
|
||||
`context_note_title` synthesised server-side at conversation load via batch `get_notes_by_ids()`;
|
||||
message badge shows note title instead of "Note #N".
|
||||
@@ -614,21 +621,22 @@ When adding a new migration, follow these conventions:
|
||||
(`search_todos` keyword-filters the todo list — companion to `list_todos`)
|
||||
- **Streaming status transparency:** The backend emits `status` SSE events at each pipeline stage
|
||||
so the user always sees what's happening instead of a blank progress dot. Stages:
|
||||
(1) `"Generating response..."` immediately after context is built (no blocking wait for intent);
|
||||
(2) human-readable tool label (e.g. `"Creating calendar event..."`) before each tool executes
|
||||
(both intent-routed and native); (3) `"Composing response..."` before tool follow-up rounds.
|
||||
(1) intent ack text streamed as a `chunk` event (tool responses — TTFT ~400ms) or
|
||||
`"Generating response..."` status event (chat-only responses);
|
||||
(2) human-readable tool label (e.g. `"Creating calendar event..."`) before each tool executes;
|
||||
(3) `"Composing response..."` before tool follow-up rounds.
|
||||
Frontend: `chat.ts` stores `streamingStatus` ref, cleared on first content chunk or on done/error.
|
||||
`ChatView.vue` shows a pulsing dot + italic label above the content while status is active, then
|
||||
falls back to the blinking cursor when content streams in. `HomeView.vue` dashboard panel shows
|
||||
the status label in place of `...` before any content arrives.
|
||||
- **Intent routing (optimistic streaming):** On the first round, the backend races intent
|
||||
classification against the start of the LLM stream using `asyncio.wait(FIRST_COMPLETED)`.
|
||||
The LLM stream is immediately started into an `asyncio.Queue` while `classify_intent()` runs
|
||||
concurrently. If the stream produces its first token before classification completes, the intent
|
||||
task is cancelled and the user sees tokens immediately (zero blocking for pure chat). If
|
||||
classification finishes first and detects a tool call, the stream is cancelled and the tool
|
||||
executes directly — bypassing the model's native (sometimes unreliable) tool calling API. Falls
|
||||
through to normal streaming when no tool is detected or classification fails.
|
||||
- **Intent routing (intent-first pipeline, Phase 21):** On the first round, `build_context()` and
|
||||
`classify_intent()` run concurrently. Once intent returns (~400ms), the pipeline immediately acts:
|
||||
if a tool is detected, the intent's one-sentence `ack` field is streamed as the first chunk
|
||||
(becoming TTFT), the tool executes, then the main model generates a follow-up response with the
|
||||
tool result. For chat-only responses, the model streams directly with no ack prefix.
|
||||
No optimistic streaming queue or race — eliminates wasted GPU prefill when intent won the race.
|
||||
`IntentResult.ack` (one-sentence acknowledgment) is embedded in the intent JSON output, so no
|
||||
additional LLM call is needed for acknowledgment. Intent model `max_tokens` 350 (was 200).
|
||||
Dedicated intent model configurable via `OLLAMA_INTENT_MODEL` env var (default `qwen2.5:1.5b`)
|
||||
or per-user `intent_model` setting — smaller/faster model for routing. Main model default
|
||||
is `qwen3:latest` (configurable via `OLLAMA_MODEL` env var or per-user `default_model` setting
|
||||
@@ -641,6 +649,8 @@ When adding a new migration, follow these conventions:
|
||||
search_events), update_note vs create_note disambiguation, reminder_minutes conversion,
|
||||
delete_note vs delete_task disambiguation, get_note for "read/show me this note", list_notes for
|
||||
"browse/list notes", tag management via update_note (tag_mode add/remove), search_todos.
|
||||
`generate_completion` (used by intent classifier) retries on HTTP 500 (3 attempts, 3s/6s delays)
|
||||
to handle cold model loading without failing intent classification.
|
||||
- **CalDAV calendar integration:** Per-user CalDAV settings (URL, username, password, calendar name, timezone).
|
||||
LLM tools: `create_event` (all_day, recurrence, timezone, reminder_minutes, attendees, calendar_name),
|
||||
`list_events`, `search_events`, `update_event`, `delete_event`, `list_calendars`,
|
||||
|
||||
Reference in New Issue
Block a user