Add LLM tool calling for creating tasks, notes, and searching from chat

Ollama tool/function calling integration allows the LLM to create tasks,
create notes, and search existing notes on behalf of the user during chat.
Multi-round tool loop (max 5 rounds) lets the model execute tools then
produce a natural language response. Tool results are persisted in a new
JSONB column on messages and rendered as compact cards with linked titles.

- Migration 0013: add tool_calls JSONB column to messages
- New services/tools.py: tool definitions + execute_tool dispatcher
- llm.py: ChatChunk dataclass, stream_chat_with_tools(), date in system prompt
- generation_task.py: multi-round tool call loop with SSE tool_call events
- Frontend: ToolCallRecord type, streamingToolCalls in store, ToolCallCard
  component, rendering in ChatMessage and ChatView streaming bubble

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
2026-02-14 23:34:36 -05:00
parent f089b16080
commit 8996b45e50
11 changed files with 522 additions and 29 deletions
@@ -0,0 +1,17 @@
"""Add tool_calls JSONB column to messages."""
from alembic import op
revision = "0013"
down_revision = "0012"
def upgrade() -> None:
op.execute("""
ALTER TABLE messages
ADD COLUMN IF NOT EXISTS tool_calls JSONB
""")
def downgrade() -> None:
op.drop_column("messages", "tool_calls")
+14
View File
@@ -2,6 +2,7 @@
import { computed } from "vue"; import { computed } from "vue";
import { renderMarkdown } from "@/utils/markdown"; import { renderMarkdown } from "@/utils/markdown";
import { useSettingsStore } from "@/stores/settings"; import { useSettingsStore } from "@/stores/settings";
import ToolCallCard from "@/components/ToolCallCard.vue";
import type { Message } from "@/types/chat"; import type { Message } from "@/types/chat";
const settingsStore = useSettingsStore(); const settingsStore = useSettingsStore();
@@ -48,6 +49,13 @@ const roleLabel = computed(() => {
</div> </div>
</div> </div>
<div class="message-content prose" v-html="rendered"></div> <div class="message-content prose" v-html="rendered"></div>
<div v-if="message.tool_calls?.length" class="tool-calls">
<ToolCallCard
v-for="(tc, i) in message.tool_calls"
:key="i"
:tool-call="tc"
/>
</div>
<div <div
v-if="message.context_note_id" v-if="message.context_note_id"
class="context-badge" class="context-badge"
@@ -157,6 +165,12 @@ const roleLabel = computed(() => {
color: #fff; color: #fff;
border-color: var(--color-primary); border-color: var(--color-primary);
} }
.tool-calls {
display: flex;
flex-wrap: wrap;
gap: 0.3rem;
margin-top: 0.4rem;
}
.context-badge { .context-badge {
margin-top: 0.4rem; margin-top: 0.4rem;
font-size: 0.75rem; font-size: 0.75rem;
+124
View File
@@ -0,0 +1,124 @@
<script setup lang="ts">
import { computed } from "vue";
import type { ToolCallRecord } from "@/types/chat";
const props = defineProps<{
toolCall: ToolCallRecord;
}>();
const label = computed(() => {
if (!props.toolCall.result.success) return "Error";
switch (props.toolCall.result.type) {
case "task":
return "Created task";
case "note":
return "Created note";
case "search":
return "Searched notes";
default:
return "Tool call";
}
});
const title = computed(() => {
const data = props.toolCall.result.data;
if (!data) return null;
if (typeof data.title === "string") return data.title;
return null;
});
const linkTo = computed(() => {
const data = props.toolCall.result.data;
if (!data || typeof data.id !== "number") return null;
return `/notes/${data.id}`;
});
const searchResults = computed(() => {
const data = props.toolCall.result.data;
if (!data || props.toolCall.result.type !== "search") return null;
const results = data.results as Array<{ id: number; title: string; type: string }> | undefined;
return results && results.length > 0 ? results : null;
});
</script>
<template>
<div class="tool-call-card" :class="{ error: toolCall.status === 'error' }">
<span class="tool-label">{{ label }}</span>
<template v-if="toolCall.status === 'error'">
<span class="tool-error">{{ toolCall.result.error }}</span>
</template>
<template v-else-if="searchResults">
<span class="tool-search-info">{{ (toolCall.result.data?.total as number) ?? 0 }} found</span>
<div class="tool-search-results">
<router-link
v-for="r in searchResults"
:key="r.id"
:to="`/notes/${r.id}`"
class="tool-search-item"
>
{{ r.title || "Untitled" }}
</router-link>
</div>
</template>
<template v-else-if="linkTo">
<router-link :to="linkTo" class="tool-link">{{ title }}</router-link>
</template>
</div>
</template>
<style scoped>
.tool-call-card {
display: inline-flex;
flex-wrap: wrap;
align-items: center;
gap: 0.35rem;
background: var(--color-bg-secondary);
border: 1px solid var(--color-border);
border-radius: 12px;
padding: 0.3rem 0.6rem;
font-size: 0.8rem;
margin-top: 0.4rem;
}
.tool-call-card.error {
border-color: var(--color-danger, #e74c3c);
}
.tool-label {
font-weight: 600;
color: var(--color-text-muted);
text-transform: uppercase;
letter-spacing: 0.03em;
font-size: 0.7rem;
}
.tool-link {
color: var(--color-primary);
text-decoration: none;
}
.tool-link:hover {
text-decoration: underline;
}
.tool-error {
color: var(--color-danger, #e74c3c);
}
.tool-search-info {
color: var(--color-text-muted);
}
.tool-search-results {
display: flex;
flex-wrap: wrap;
gap: 0.25rem;
width: 100%;
}
.tool-search-item {
color: var(--color-primary);
text-decoration: none;
font-size: 0.8rem;
}
.tool-search-item:hover {
text-decoration: underline;
}
.tool-search-item:not(:last-child)::after {
content: ",";
color: var(--color-text-muted);
margin-right: 0.1rem;
}
</style>
+15
View File
@@ -17,6 +17,7 @@ import type {
OllamaStatus, OllamaStatus,
RunningModel, RunningModel,
SendMessageResponse, SendMessageResponse,
ToolCallRecord,
} from "@/types/chat"; } from "@/types/chat";
export const useChatStore = defineStore("chat", () => { export const useChatStore = defineStore("chat", () => {
@@ -26,6 +27,7 @@ export const useChatStore = defineStore("chat", () => {
const loading = ref(false); const loading = ref(false);
const streaming = ref(false); const streaming = ref(false);
const streamingContent = ref(""); const streamingContent = ref("");
const streamingToolCalls = ref<ToolCallRecord[]>([]);
const lastContextMeta = ref<ContextMeta | null>(null); const lastContextMeta = ref<ContextMeta | null>(null);
const models = ref<OllamaModel[]>([]); const models = ref<OllamaModel[]>([]);
const runningModels = ref<RunningModel[]>([]); const runningModels = ref<RunningModel[]>([]);
@@ -187,6 +189,12 @@ export const useChatStore = defineStore("chat", () => {
case "chunk": case "chunk":
streamingContent.value += event.data.chunk as string; streamingContent.value += event.data.chunk as string;
break; break;
case "tool_call":
streamingToolCalls.value = [
...streamingToolCalls.value,
event.data.tool_call as ToolCallRecord,
];
break;
case "done": case "done":
gotDone = true; gotDone = true;
{ {
@@ -196,12 +204,16 @@ export const useChatStore = defineStore("chat", () => {
role: "assistant", role: "assistant",
content: streamingContent.value, content: streamingContent.value,
context_note_id: null, context_note_id: null,
tool_calls: streamingToolCalls.value.length
? [...streamingToolCalls.value]
: null,
created_at: new Date().toISOString(), created_at: new Date().toISOString(),
}; };
if (currentConversation.value?.id === convId) { if (currentConversation.value?.id === convId) {
currentConversation.value.messages.push(assistantMsg); currentConversation.value.messages.push(assistantMsg);
} }
streamingContent.value = ""; streamingContent.value = "";
streamingToolCalls.value = [];
streaming.value = false; streaming.value = false;
// Update conversation in list // Update conversation in list
@@ -216,6 +228,7 @@ export const useChatStore = defineStore("chat", () => {
gotDone = true; gotDone = true;
streaming.value = false; streaming.value = false;
streamingContent.value = ""; streamingContent.value = "";
streamingToolCalls.value = [];
useToastStore().show( useToastStore().show(
"Chat error: " + (event.data.error as string), "Chat error: " + (event.data.error as string),
"error", "error",
@@ -252,6 +265,7 @@ export const useChatStore = defineStore("chat", () => {
streaming.value = false; streaming.value = false;
streamingContent.value = ""; streamingContent.value = "";
streamingToolCalls.value = [];
} }
async function cancelGeneration() { async function cancelGeneration() {
@@ -357,6 +371,7 @@ export const useChatStore = defineStore("chat", () => {
loading, loading,
streaming, streaming,
streamingContent, streamingContent,
streamingToolCalls,
lastContextMeta, lastContextMeta,
models, models,
runningModels, runningModels,
+8
View File
@@ -1,3 +1,10 @@
export interface ToolCallRecord {
function: string;
arguments: Record<string, unknown>;
result: { success: boolean; type?: string; data?: Record<string, unknown>; error?: string };
status: "success" | "error";
}
export interface Message { export interface Message {
id: number; id: number;
conversation_id: number; conversation_id: number;
@@ -5,6 +12,7 @@ export interface Message {
content: string; content: string;
status?: "complete" | "generating" | "error"; status?: "complete" | "generating" | "error";
context_note_id: number | null; context_note_id: number | null;
tool_calls?: ToolCallRecord[] | null;
created_at: string; created_at: string;
} }
+14
View File
@@ -7,6 +7,7 @@ import { apiGet } from "@/api/client";
import { renderMarkdown } from "@/utils/markdown"; import { renderMarkdown } from "@/utils/markdown";
import ChatMessage from "@/components/ChatMessage.vue"; import ChatMessage from "@/components/ChatMessage.vue";
import ModelSelector from "@/components/ModelSelector.vue"; import ModelSelector from "@/components/ModelSelector.vue";
import ToolCallCard from "@/components/ToolCallCard.vue";
import type { Note } from "@/types/note"; import type { Note } from "@/types/note";
const route = useRoute(); const route = useRoute();
@@ -400,6 +401,13 @@ function excludeAutoNote(noteId: number) {
<div class="message-header"> <div class="message-header">
<span class="role-label">{{ settingsStore.assistantName }}</span> <span class="role-label">{{ settingsStore.assistantName }}</span>
</div> </div>
<div v-if="store.streamingToolCalls.length" class="streaming-tool-calls">
<ToolCallCard
v-for="(tc, i) in store.streamingToolCalls"
:key="i"
:tool-call="tc"
/>
</div>
<div class="message-content prose" v-html="streamingRendered"></div> <div class="message-content prose" v-html="streamingRendered"></div>
<span class="typing-indicator"></span> <span class="typing-indicator"></span>
</div> </div>
@@ -690,6 +698,12 @@ function excludeAutoNote(noteId: number) {
word-break: break-word; word-break: break-word;
} }
.streaming-tool-calls {
display: flex;
flex-wrap: wrap;
gap: 0.3rem;
margin-bottom: 0.4rem;
}
.typing-indicator { .typing-indicator {
display: inline-block; display: inline-block;
width: 6px; width: 6px;
@@ -2,6 +2,7 @@ from datetime import datetime, timezone
from sqlalchemy import DateTime, ForeignKey, Index, Integer, Text from sqlalchemy import DateTime, ForeignKey, Index, Integer, Text
from sqlalchemy import inspect from sqlalchemy import inspect
from sqlalchemy.dialects.postgresql import JSONB
from sqlalchemy.orm import Mapped, mapped_column, relationship from sqlalchemy.orm import Mapped, mapped_column, relationship
from fabledassistant.models import Base from fabledassistant.models import Base
@@ -65,6 +66,7 @@ class Message(Base):
context_note_id: Mapped[int | None] = mapped_column( context_note_id: Mapped[int | None] = mapped_column(
Integer, ForeignKey("notes.id", ondelete="SET NULL"), nullable=True Integer, ForeignKey("notes.id", ondelete="SET NULL"), nullable=True
) )
tool_calls: Mapped[list | None] = mapped_column(JSONB, nullable=True)
created_at: Mapped[datetime] = mapped_column( created_at: Mapped[datetime] = mapped_column(
DateTime(timezone=True), default=lambda: datetime.now(timezone.utc) DateTime(timezone=True), default=lambda: datetime.now(timezone.utc)
) )
@@ -83,5 +85,6 @@ class Message(Base):
"content": self.content, "content": self.content,
"status": self.status, "status": self.status,
"context_note_id": self.context_note_id, "context_note_id": self.context_note_id,
"tool_calls": self.tool_calls,
"created_at": self.created_at.isoformat(), "created_at": self.created_at.isoformat(),
} }
+89 -18
View File
@@ -4,6 +4,7 @@ Streams from Ollama into a GenerationBuffer, periodically flushing to DB.
Runs independently of any HTTP connection. Runs independently of any HTTP connection.
""" """
import json
import logging import logging
import time import time
@@ -12,8 +13,9 @@ from sqlalchemy import update
from fabledassistant.models import async_session from fabledassistant.models import async_session
from fabledassistant.models.conversation import Message from fabledassistant.models.conversation import Message
from fabledassistant.services.generation_buffer import GenerationBuffer, GenerationState from fabledassistant.services.generation_buffer import GenerationBuffer, GenerationState
from fabledassistant.services.llm import generate_completion, stream_chat from fabledassistant.services.llm import ChatChunk, generate_completion, stream_chat, stream_chat_with_tools
from fabledassistant.services.chat import update_conversation_title from fabledassistant.services.chat import update_conversation_title
from fabledassistant.services.tools import TOOL_DEFINITIONS, execute_tool
logger = logging.getLogger(__name__) logger = logging.getLogger(__name__)
@@ -47,12 +49,20 @@ async def _generate_title(messages: list[dict], model: str) -> str:
return title[:100] if title else "" return title[:100] if title else ""
async def _update_message(message_id: int, content: str, status: str) -> None: async def _update_message(
message_id: int,
content: str,
status: str,
tool_calls: list[dict] | None = None,
) -> None:
values: dict = {"content": content, "status": status}
if tool_calls is not None:
values["tool_calls"] = tool_calls
async with async_session() as session: async with async_session() as session:
await session.execute( await session.execute(
update(Message) update(Message)
.where(Message.id == message_id) .where(Message.id == message_id)
.values(content=content, status=status) .values(**values)
) )
await session.commit() await session.commit()
@@ -68,33 +78,94 @@ async def run_generation(
user_content: str, user_content: str,
) -> None: ) -> None:
"""Stream LLM response into buffer with periodic DB flushes.""" """Stream LLM response into buffer with periodic DB flushes."""
MAX_TOOL_ROUNDS = 5
msg_id = buf.assistant_message_id msg_id = buf.assistant_message_id
# Emit context event # Emit context event
buf.append_event("context", {"context": context_meta}) buf.append_event("context", {"context": context_meta})
last_flush = time.monotonic() last_flush = time.monotonic()
all_tool_calls: list[dict] = []
try: try:
cancelled = False cancelled = False
async for chunk in stream_chat(messages, model):
if buf.cancel_event.is_set(): for _round in range(MAX_TOOL_ROUNDS + 1):
cancelled = True round_tool_calls: list[dict] = []
async for chunk in stream_chat_with_tools(messages, model, tools=TOOL_DEFINITIONS):
if buf.cancel_event.is_set():
cancelled = True
break
if chunk.type == "content":
buf.content_so_far += chunk.content
buf.append_event("chunk", {"chunk": chunk.content})
# Periodic DB flush
now = time.monotonic()
if now - last_flush >= DB_FLUSH_INTERVAL:
try:
await _update_message(msg_id, buf.content_so_far, "generating")
except Exception:
logger.warning("Failed periodic flush for message %d", msg_id, exc_info=True)
last_flush = now
elif chunk.type == "tool_calls" and chunk.tool_calls:
# Process each tool call
for tc in chunk.tool_calls:
fn = tc.get("function", {})
tool_name = fn.get("name", "")
arguments = fn.get("arguments", {})
result = await execute_tool(user_id, tool_name, arguments)
tool_record = {
"function": tool_name,
"arguments": arguments,
"result": result,
"status": "success" if result.get("success") else "error",
}
round_tool_calls.append(tool_record)
all_tool_calls.append(tool_record)
# Emit tool_call SSE event
buf.append_event("tool_call", {"tool_call": tool_record})
if cancelled:
break break
buf.content_so_far += chunk
buf.append_event("chunk", {"chunk": chunk})
# Periodic DB flush # If no tool calls this round, the LLM gave its final text response
now = time.monotonic() if not round_tool_calls:
if now - last_flush >= DB_FLUSH_INTERVAL: break
try:
await _update_message(msg_id, buf.content_so_far, "generating")
except Exception:
logger.warning("Failed periodic flush for message %d", msg_id, exc_info=True)
last_flush = now
# Final save (partial content on cancel is still valid) # Append assistant tool_call message and tool results to conversation
await _update_message(msg_id, buf.content_so_far, "complete") # for the next round
messages.append({
"role": "assistant",
"content": buf.content_so_far,
"tool_calls": [
{"function": {"name": tc["function"], "arguments": tc["arguments"]}}
for tc in round_tool_calls
],
})
for tc in round_tool_calls:
messages.append({
"role": "tool",
"content": json.dumps(tc["result"]),
})
# Reset content for the next round (LLM will produce a new response)
buf.content_so_far = ""
# Final save
await _update_message(
msg_id,
buf.content_so_far,
"complete",
tool_calls=all_tool_calls if all_tool_calls else None,
)
# Count non-system messages to decide on title generation # Count non-system messages to decide on title generation
non_system = [m for m in messages if m["role"] != "system"] non_system = [m for m in messages if m["role"] != "system"]
+58 -1
View File
@@ -2,6 +2,8 @@ import json
import logging import logging
import re import re
from collections.abc import AsyncGenerator from collections.abc import AsyncGenerator
from dataclasses import dataclass, field
from typing import Literal
import httpx import httpx
@@ -98,6 +100,55 @@ async def stream_chat(
break break
@dataclass
class ChatChunk:
"""A chunk yielded by stream_chat_with_tools."""
type: Literal["content", "tool_calls", "done"]
content: str = ""
tool_calls: list[dict] | None = None
async def stream_chat_with_tools(
messages: list[dict],
model: str,
tools: list[dict] | None = None,
) -> AsyncGenerator[ChatChunk, None]:
"""Stream chat completion from Ollama with tool support.
Yields ChatChunk objects. If the model returns tool_calls, a
ChatChunk(type="tool_calls") is yielded. Always ends with
ChatChunk(type="done").
"""
payload: dict = {"model": model, "messages": messages, "stream": True}
if tools:
payload["tools"] = tools
async with httpx.AsyncClient(timeout=httpx.Timeout(1800.0, connect=30.0, read=300.0)) as client:
async with client.stream(
"POST",
f"{Config.OLLAMA_URL}/api/chat",
json=payload,
) as resp:
resp.raise_for_status()
async for line in resp.aiter_lines():
if not line.strip():
continue
data = json.loads(line)
msg = data.get("message", {})
# Content chunks
chunk = msg.get("content", "")
if chunk:
yield ChatChunk(type="content", content=chunk)
# Check for tool calls on done
if data.get("done"):
tool_calls = msg.get("tool_calls")
if tool_calls:
yield ChatChunk(type="tool_calls", tool_calls=tool_calls)
yield ChatChunk(type="done")
break
async def generate_completion(messages: list[dict], model: str) -> str: async def generate_completion(messages: list[dict], model: str) -> str:
"""Non-streaming chat completion, returns full response text.""" """Non-streaming chat completion, returns full response text."""
async with httpx.AsyncClient(timeout=httpx.Timeout(1800.0, connect=30.0, read=300.0)) as client: async with httpx.AsyncClient(timeout=httpx.Timeout(1800.0, connect=30.0, read=300.0)) as client:
@@ -166,11 +217,17 @@ async def build_context(
which notes were included as context. which notes were included as context.
""" """
exclude_set = set(exclude_note_ids or []) exclude_set = set(exclude_note_ids or [])
from datetime import date as date_type
assistant_name = await get_setting(user_id, "assistant_name", "Fable") assistant_name = await get_setting(user_id, "assistant_name", "Fable")
today = date_type.today().isoformat()
system_parts = [ system_parts = [
f"You are a helpful assistant named {assistant_name}, integrated into a note-taking and task-tracking app called Fabled Assistant. " f"You are a helpful assistant named {assistant_name}, integrated into a note-taking and task-tracking app called Fabled Assistant. "
"Help users with their notes, tasks, and general questions. " "Help users with their notes, tasks, and general questions. "
"When note context is provided, use it to give relevant answers." "When note context is provided, use it to give relevant answers. "
f"Today's date is {today}. "
"You have tools available to create tasks, create notes, and search the user's notes. "
"Use them when the user asks you to create, add, or find tasks and notes. "
"For relative dates like 'Friday' or 'next week', resolve them to YYYY-MM-DD format."
] ]
context_meta: dict = { context_meta: dict = {
+159
View File
@@ -0,0 +1,159 @@
"""Tool definitions and executor for LLM tool calling."""
import logging
from datetime import date, datetime
from fabledassistant.services.notes import create_note, list_notes
logger = logging.getLogger(__name__)
TOOL_DEFINITIONS = [
{
"type": "function",
"function": {
"name": "create_task",
"description": "Create a new task for the user. Use this when the user asks you to add a task, todo, or reminder.",
"parameters": {
"type": "object",
"properties": {
"title": {
"type": "string",
"description": "The task title",
},
"body": {
"type": "string",
"description": "Optional task description or details",
},
"due_date": {
"type": "string",
"description": "Optional due date in YYYY-MM-DD format",
},
"priority": {
"type": "string",
"enum": ["none", "low", "medium", "high"],
"description": "Task priority level",
},
},
"required": ["title"],
},
},
},
{
"type": "function",
"function": {
"name": "create_note",
"description": "Create a new note for the user. Use this when the user asks you to write down, save, or record something as a note.",
"parameters": {
"type": "object",
"properties": {
"title": {
"type": "string",
"description": "The note title",
},
"body": {
"type": "string",
"description": "The note content in markdown",
},
},
"required": ["title"],
},
},
},
{
"type": "function",
"function": {
"name": "search_notes",
"description": "Search the user's notes and tasks. Use this when the user asks about their existing notes, tasks, or wants to find something they've written.",
"parameters": {
"type": "object",
"properties": {
"query": {
"type": "string",
"description": "Search query to find notes/tasks",
},
},
"required": ["query"],
},
},
},
]
def _parse_due_date(value: str | None) -> date | None:
"""Parse a due date string, returning None on failure."""
if not value:
return None
try:
return datetime.strptime(value, "%Y-%m-%d").date()
except (ValueError, TypeError):
logger.warning("Invalid due_date format: %s", value)
return None
async def execute_tool(user_id: int, tool_name: str, arguments: dict) -> dict:
"""Execute a tool call and return the result."""
try:
if tool_name == "create_task":
note = await create_note(
user_id=user_id,
title=arguments.get("title", "Untitled Task"),
body=arguments.get("body", ""),
status="todo",
priority=arguments.get("priority", "none"),
due_date=_parse_due_date(arguments.get("due_date")),
)
return {
"success": True,
"type": "task",
"data": {
"id": note.id,
"title": note.title,
"status": note.status,
"priority": note.priority,
"due_date": str(note.due_date) if note.due_date else None,
},
}
elif tool_name == "create_note":
note = await create_note(
user_id=user_id,
title=arguments.get("title", "Untitled Note"),
body=arguments.get("body", ""),
)
return {
"success": True,
"type": "note",
"data": {
"id": note.id,
"title": note.title,
},
}
elif tool_name == "search_notes":
query = arguments.get("query", "")
notes, total = await list_notes(user_id=user_id, q=query, limit=5)
results = []
for n in notes:
results.append({
"id": n.id,
"title": n.title,
"type": "task" if n.status is not None else "note",
"status": n.status,
"preview": (n.body[:200] + "...") if n.body and len(n.body) > 200 else (n.body or ""),
})
return {
"success": True,
"type": "search",
"data": {
"query": query,
"total": total,
"results": results,
},
}
else:
return {"success": False, "error": f"Unknown tool: {tool_name}"}
except Exception as e:
logger.exception("Tool execution failed: %s", tool_name)
return {"success": False, "error": str(e)}
+21 -10
View File
@@ -12,7 +12,7 @@
> Include file-level details in the commit body when the change is non-trivial. > Include file-level details in the commit body when the change is non-trivial.
## Last Updated ## Last Updated
2026-02-14 — Phase 6.1: Model status colors, timeout tuning, SSE error feedback 2026-02-14 — Phase 7: LLM tool calling (create tasks, create notes, search notes from chat)
## Project Overview ## Project Overview
Fabled Assistant is a self-hosted note-taking and task-tracking application with Fabled Assistant is a self-hosted note-taking and task-tracking application with
@@ -202,10 +202,11 @@ for AI-assisted features.
### Messages ### Messages
- `id` (int PK), `conversation_id` (FK to conversations, CASCADE), `role` (str: system/user/assistant), - `id` (int PK), `conversation_id` (FK to conversations, CASCADE), `role` (str: system/user/assistant),
`content` (str), `status` (str, default `'complete'``complete`/`generating`/`error`), `content` (str), `status` (str, default `'complete'``complete`/`generating`/`error`),
`context_note_id` (nullable FK to notes, SET NULL), `created_at` `context_note_id` (nullable FK to notes, SET NULL), `tool_calls` (nullable JSONB), `created_at`
- Index on `conversation_id` - Index on `conversation_id`
- `context_note_id` tracks which note was attached as context when message was sent - `context_note_id` tracks which note was attached as context when message was sent
- `to_dict()` returns: `id`, `conversation_id`, `role`, `content`, `status`, `context_note_id`, `created_at` - `tool_calls` stores an array of tool call records `[{function, arguments, result, status}]` for assistant messages that used tools
- `to_dict()` returns: `id`, `conversation_id`, `role`, `content`, `status`, `context_note_id`, `tool_calls`, `created_at`
## Project Structure (Current) ## Project Structure (Current)
``` ```
@@ -230,7 +231,8 @@ fabledassistant/
│ ├── 0009_add_message_status.py # Add status column to messages table (default 'complete') │ ├── 0009_add_message_status.py # Add status column to messages table (default 'complete')
│ ├── 0010_add_app_logs_table.py # App logs table for audit, usage, and error logging │ ├── 0010_add_app_logs_table.py # App logs table for audit, usage, and error logging
│ ├── 0011_add_password_reset_tokens.py # Password reset tokens table │ ├── 0011_add_password_reset_tokens.py # Password reset tokens table
── 0012_add_invitation_tokens.py # Invitation tokens table ── 0012_add_invitation_tokens.py # Invitation tokens table
│ └── 0013_add_tool_calls_to_messages.py # Add tool_calls JSONB column to messages
├── src/ ├── src/
│ └── fabledassistant/ │ └── fabledassistant/
│ ├── __init__.py │ ├── __init__.py
@@ -258,10 +260,11 @@ fabledassistant/
│ │ ├── auth.py # Auth business logic: hash_password, verify_password, create_user, authenticate, get_user_by_id, is_registration_open, list_users, delete_user, set_registration_open, password reset tokens, invitation tokens (create, validate, register_with_invitation, list_pending, revoke) │ │ ├── auth.py # Auth business logic: hash_password, verify_password, create_user, authenticate, get_user_by_id, is_registration_open, list_users, delete_user, set_registration_open, password reset tokens, invitation tokens (create, validate, register_with_invitation, list_pending, revoke)
│ │ ├── backup.py # Backup/restore: export_full_backup, export_user_backup, restore_full_backup │ │ ├── backup.py # Backup/restore: export_full_backup, export_user_backup, restore_full_backup
│ │ ├── notes.py # CRUD with user_id isolation, is_task filter, convert, backlinks, search_notes_for_context │ │ ├── notes.py # CRUD with user_id isolation, is_task filter, convert, backlinks, search_notes_for_context
│ │ ├── llm.py # Ollama interaction: build_context with user_id, streaming, URL fetching │ │ ├── llm.py # Ollama interaction: build_context with user_id, streaming (stream_chat + stream_chat_with_tools), ChatChunk dataclass, URL fetching
│ │ ├── chat.py # Conversation CRUD with user_id isolation, add_message, save/summarize as note (LLM-titled, chat-tagged) │ │ ├── chat.py # Conversation CRUD with user_id isolation, add_message, save/summarize as note (LLM-titled, chat-tagged)
│ │ ├── generation_buffer.py # In-memory SSE event buffer with cancel_event, reconnect support, auto-cleanup; supports chat (int keys) and assist (string keys) │ │ ├── generation_buffer.py # In-memory SSE event buffer with cancel_event, reconnect support, auto-cleanup; supports chat (int keys) and assist (string keys)
│ │ ├── generation_task.py # Background asyncio tasks: run_generation (chat, DB flush, titles) + run_assist_generation (lightweight, no DB) │ │ ├── generation_task.py # Background asyncio tasks: run_generation (chat, DB flush, titles, tool loop) + run_assist_generation (lightweight, no DB)
│ │ ├── tools.py # LLM tool definitions (create_task, create_note, search_notes) + execute_tool dispatcher
│ │ ├── settings.py # Settings CRUD with user_id isolation: get_setting, set_setting, set_settings_batch, get_all_settings │ │ ├── settings.py # Settings CRUD with user_id isolation: get_setting, set_setting, set_settings_batch, get_all_settings
│ │ ├── logging.py # App logging: log_audit, log_usage, log_error, get_logs, get_log_stats, delete_old_logs, start_log_retention_loop │ │ ├── logging.py # App logging: log_audit, log_usage, log_error, get_logs, get_log_stats, delete_old_logs, start_log_retention_loop
│ │ ├── email.py # SMTP email service: get_smtp_config, is_smtp_configured, send_email, send_test_email │ │ ├── email.py # SMTP email service: get_smtp_config, is_smtp_configured, send_email, send_test_email
@@ -295,7 +298,7 @@ fabledassistant/
│ ├── types/ │ ├── types/
│ │ ├── auth.ts # User interface, AuthStatus interface │ │ ├── auth.ts # User interface, AuthStatus interface
│ │ ├── note.ts # Note interface (with status, priority, due_date, is_task) + TaskStatus, TaskPriority types + NoteListResponse │ │ ├── note.ts # Note interface (with status, priority, due_date, is_task) + TaskStatus, TaskPriority types + NoteListResponse
│ │ ├── chat.ts # Message, Conversation, ConversationDetail, ContextMeta, OllamaModel, RunningModel, OllamaStatus interfaces │ │ ├── chat.ts # ToolCallRecord, Message, Conversation, ConversationDetail, ContextMeta, OllamaModel, RunningModel, OllamaStatus interfaces
│ │ ├── settings.ts # AppSettings interface, ModelInfo interface (name, description, size, bestFor, category) │ │ ├── settings.ts # AppSettings interface, ModelInfo interface (name, description, size, bestFor, category)
│ │ └── task.ts # Task = re-export of Note; TaskListResponse │ │ └── task.ts # Task = re-export of Note; TaskListResponse
│ ├── extensions/ │ ├── extensions/
@@ -328,7 +331,8 @@ fabledassistant/
│ │ ├── ChatPanel.vue # Slide-out chat panel (right side overlay, receives contextNoteId prop), bubble-style messages, floating dark input, note picker, context pills with promote/exclude │ │ ├── ChatPanel.vue # Slide-out chat panel (right side overlay, receives contextNoteId prop), bubble-style messages, floating dark input, note picker, context pills with promote/exclude
│ │ ├── ModelSelector.vue # Model dropdown (v-model pattern): fetches installed + running models, hot/cold indicators │ │ ├── ModelSelector.vue # Model dropdown (v-model pattern): fetches installed + running models, hot/cold indicators
│ │ ├── DashboardChatInput.vue # Inline chat bar: ModelSelector + note picker + textarea + send button; emits submit event │ │ ├── DashboardChatInput.vue # Inline chat bar: ModelSelector + note picker + textarea + send button; emits submit event
│ │ ├── ChatMessage.vue # Message bubble: markdown rendering, configurable assistant name label, "Save as Note" action on assistant messages │ │ ├── ToolCallCard.vue # Compact card for tool call results (created task/note link, search results, errors)
│ │ ├── ChatMessage.vue # Message bubble: markdown rendering, tool call cards, configurable assistant name label, "Save as Note" action on assistant messages
│ │ ├── NoteCard.vue # Card with rendered markdown preview (v-html), TagPill, tag-click emit, hover edit button │ │ ├── NoteCard.vue # Card with rendered markdown preview (v-html), TagPill, tag-click emit, hover edit button
│ │ ├── TaskCard.vue # Card with rendered preview (body not description), StatusBadge (clickable), PriorityBadge, due date, tags, hover edit button │ │ ├── TaskCard.vue # Card with rendered preview (body not description), StatusBadge (clickable), PriorityBadge, due date, tags, hover edit button
│ │ ├── StatusBadge.vue # Color-coded status badge, optional clickable cycling │ │ ├── StatusBadge.vue # Color-coded status badge, optional clickable cycling
@@ -422,7 +426,7 @@ container startup.
### Migration Chain ### Migration Chain
``` ```
0001_create_notes_table.py → 0002_create_tasks_table.py → 0003_task_note_companion.py → 0004_merge_tasks_into_notes.py → 0005_add_chat_tables.py → 0006_add_settings_table.py → 0007_add_title_and_updated_at_indexes.py → 0008_add_users_and_user_id.py → 0009_add_message_status.py → 0010_add_app_logs_table.py → 0011_add_password_reset_tokens.py → 0012_add_invitation_tokens.py 0001_create_notes_table.py → 0002_create_tasks_table.py → 0003_task_note_companion.py → 0004_merge_tasks_into_notes.py → 0005_add_chat_tables.py → 0006_add_settings_table.py → 0007_add_title_and_updated_at_indexes.py → 0008_add_users_and_user_id.py → 0009_add_message_status.py → 0010_add_app_logs_table.py → 0011_add_password_reset_tokens.py → 0012_add_invitation_tokens.py → 0013_add_tool_calls_to_messages.py
``` ```
### How Migrations Run ### How Migrations Run
@@ -531,6 +535,13 @@ When adding a new migration, follow these conventions:
- Ollama configured with `OLLAMA_MAX_LOADED_MODELS=2` and `OLLAMA_KEEP_ALIVE=30m` - Ollama configured with `OLLAMA_MAX_LOADED_MODELS=2` and `OLLAMA_KEEP_ALIVE=30m`
- Timeout tuning: connect timeout 30s (cold model loading), warm timeout 300s, pull timeout 1800s - Timeout tuning: connect timeout 30s (cold model loading), warm timeout 300s, pull timeout 1800s
- SSE reconnection failure shows error toast instead of silently recovering - SSE reconnection failure shows error toast instead of silently recovering
- **LLM tool calling:** Models with tool support can create tasks, create notes, and search notes
on behalf of the user during chat. Multi-round tool loop (max 5 rounds) allows the LLM to
execute tools and then produce a natural language response incorporating results. Tool call
results are persisted in message `tool_calls` JSONB column and rendered as compact ToolCallCard
components (linked titles for created items, search result lists). SSE emits `tool_call` events
for real-time rendering during streaming. System prompt includes today's date for relative date
resolution. Graceful degradation: models without tool support respond normally.
### Authentication & User Management ### Authentication & User Management
- Session cookie auth with bcrypt, first-user-is-admin, orphaned data claiming - Session cookie auth with bcrypt, first-user-is-admin, orphaned data claiming
@@ -571,7 +582,7 @@ When adding a new migration, follow these conventions:
- Multi-stage Dockerfile: Node build → Python runtime, auto-migration on startup - Multi-stage Dockerfile: Node build → Python runtime, auto-migration on startup
- Docker Compose (dev) + Docker Swarm production stack with secrets, overlay networks, health checks - Docker Compose (dev) + Docker Swarm production stack with secrets, overlay networks, health checks
- Quart serves Vue SPA from static + REST API under `/api/`; 404 handler for SPA routing - Quart serves Vue SPA from static + REST API under `/api/`; 404 handler for SPA routing
- Alembic migrations: 12 migrations, all raw SQL with idempotency guards (IF NOT EXISTS, DO $$ BEGIN...EXCEPTION) - Alembic migrations: 13 migrations, all raw SQL with idempotency guards (IF NOT EXISTS, DO $$ BEGIN...EXCEPTION)
- Config from env vars + Docker secrets file support (`_read_secret`) - Config from env vars + Docker secrets file support (`_read_secret`)
## Development Workflow ## Development Workflow