diff --git a/docs/2026-04-10-agentic-briefing-design.md b/docs/2026-04-10-agentic-briefing-design.md new file mode 100644 index 0000000..2614389 --- /dev/null +++ b/docs/2026-04-10-agentic-briefing-design.md @@ -0,0 +1,216 @@ +# Agentic Briefing — Design Spec + +**Date:** 2026-04-10 +**Status:** Proposed +**Author:** bvandeusen + Claude + +--- + +## Problem + +The current briefing pipeline hallucinates calendar events, tasks, and news items that do not exist in the user's actual data. Observed examples from production: + +- A morning briefing asserting "your dentist appointment is still in progress" when no such event existed +- An 8am check-in mentioning "a quick meeting at 10:30 AM" with no backing calendar entry +- A midday check-in inventing "a team huddle at 2:30 PM" and "the Q2 budget draft due by Friday" +- The model calling `search_images` in response to a clarifying question about a fabricated meeting + +These are not bugs in data retrieval — the data gathering code (`_gather_internal`, `_gather_external`) returns correct values. They are a structural consequence of how the briefing context is assembled. + +## Root cause — the "no receipt" problem + +The current `run_compilation` pipeline does this: + +1. Python code gathers data (tasks, events, weather, news) from the database +2. Python formats the data into a structured text blob as the user-role message: `TODAY'S EVENTS: ...`, `DUE TODAY: ...`, etc. +3. The LLM is called **once** with `[system_prompt, user_prompt]` and produces prose +4. Only the prose reply is written to the conversation. **The underlying data is never persisted in the conversation history.** + +When the user later chats in the briefing conversation, the chat endpoint loads the full conversation history — which contains the model's *prose* from earlier but not the data that prose was derived from. The model's own prior output becomes the only source of "truth" available for follow-up questions. If that prose asserted a fact (real or hallucinated), the model has no way to distinguish it from ground truth when generating the next reply, and it will double down. + +Compounding factors: + +- **Empty sections are silently omitted.** If `calendar_events` is empty, the user prompt contains no `TODAY'S EVENTS:` line at all. The model has no explicit "zero events" signal — combined with an imperative system prompt ("note calendar events and tasks"), it interprets the silence as "I should mention some" and fabricates. +- **Scheduled slot injections append synthetic turns.** `run_slot_injection` writes a fake `[Midday briefing update]` user message and the assistant reply into the persistent conversation. By evening, the chat history contains three separate briefings, each potentially with errors, all treated as equal-weight context on follow-up. +- **The `search_images` tool is available during briefing chat**, with a negative-instruction description ("Not for factual questions"). Small and mid-sized models frequently ignore negative guidance in tool descriptions and call the tool anyway. + +## Solution — agentic briefing (the receipt model) + +Replace the one-shot synthesis with a tool-call loop. The briefing is no longer "a text blob synthesized from pre-gathered data." It is **a scheduled agent run**: the LLM is given a system prompt, a curated set of read-only data tools, and a trigger ("generate the morning briefing"). The model must call tools to see what exists. Every tool call and tool result becomes part of the conversation history, where it lives as a permanent, structured receipt. + +### Why this fixes the hallucination + +When the model calls `list_events(today) → []`, that empty array is now a persistent message in the conversation. On a follow-up question like "what meeting?", the chat endpoint loads the conversation and the model sees its own tool result from the morning showing no events. Answering "you had a meeting at 10:30" would require the model to directly contradict a tool result sitting two messages back — something LLMs are much more reliable at avoiding than contradicting their own prior prose. + +In short: **the model cannot fabricate what has a visible receipt proving it does not exist.** + +### What the model sees — before vs after + +**Before** — the model sees one structured blob and produces prose. The blob is discarded. + +``` +[system] You are a personal assistant... weave together what matters... +[user] Date: 2026-04-10 + WEATHER: Home — partly cloudy, 8–16°C + DUE TODAY: Fix briefing hallucinations + OVERDUE (2 tasks): ... +[assistant] Morning! Looks like a quiet day with two overdue items... +``` + +**After** — the model is given tools and iterates to ground truth. + +``` +[system] You are the user's personal assistant giving their morning briefing. + Use tools to see what's relevant. Only mention things you learned from + tool calls. If a tool returns nothing, say so honestly. +[user] Generate the morning briefing. +[assistant] tool_call: list_tasks(filter="due_today") +[tool] [{"id": 81, "title": "Fix briefing hallucinations", ...}] +[assistant] tool_call: list_tasks(filter="overdue") +[tool] [{"id": 42, ...}, {"id": 43, ...}] +[assistant] tool_call: list_events(today) +[tool] [] +[assistant] tool_call: get_weather(location="home") +[tool] {"description": "partly cloudy", "temp_min": 8, "temp_max": 16} +[assistant] tool_call: get_rss_items(max=8) +[tool] [...] +[assistant] Morning! Partly cloudy today, 8 to 16 — nothing on the calendar + so it's a clean run at the desk. Two things to keep in mind... +``` + +The conversation now contains verifiable receipts: `list_events(today)` returned `[]`, and that result sits in context forever (until pruned). Follow-up questions operate against those receipts. + +## Architecture + +### Existing infrastructure (reused, not rebuilt) + +The codebase already has the agentic primitives — they're used for regular chat: + +- **`llm.py::stream_chat_with_tools`** — the streaming tool-use loop that talks to Ollama with a `tools` parameter and yields tool-call chunks +- **`generation_task.py::_stream_with_retry`** — wraps `stream_chat_with_tools` with retry-on-500 behavior for cold-model races +- **`tools.py`** — defines 40+ tool schemas and an execution dispatcher + +The briefing bypasses all of this and calls a one-shot `_llm_synthesise` helper. The refactor is mainly "route briefings through the same pipeline regular chat already uses." + +### New modules + +**`briefing_tools.py`** — a small wrapper exposing a curated read-only subset of `tools.py` for briefing runs. This is an **explicit allowlist**, not a blocklist, so newly-added tools must be opted in: + +| Tool | Purpose | +|---|---| +| `list_tasks` (with filter args) | See what's actionable today, overdue, high priority | +| `list_events` (today / upcoming) | Know what's on the calendar | +| `get_weather` | Current/forecast weather | +| `get_rss_items` | Pull news themes filtered by user preferences | +| `list_projects` | Understand project context | +| `search_projects` | Surface active project summaries | +| `list_notes` (recent) | Capture follow-ups from yesterday | + +**Explicitly omitted** from the briefing tool set: + +- `search_images`, `search_web`, `research_topic`, `read_article` — external search is not a briefing concern, and `search_images` is the source of the "Peter Kyle Science Secretary" image-search bug +- All `create_*`, `update_*`, `delete_*` — briefings are read-only; a scheduled background job must not decide to mutate the user's data on its own +- `set_rag_scope`, `calculate` — not relevant to briefing content + +### New briefing function + +**`briefing_pipeline.py::run_agentic_briefing(user_id, slot, model, conversation_id)`** replaces `run_compilation`'s body (and eventually `run_slot_injection`). Internally: + +1. Build a slot-specific system prompt (see below) +2. Load the curated briefing tools from `briefing_tools.py` +3. Seed messages with `[system, user]` where the user message is a simple trigger like `"Generate the morning briefing."` +4. Enter a tool-call loop (max 8 iterations): + - Call `stream_chat_with_tools` + - If the model returns `tool_calls`, execute them via the existing dispatcher, append tool results, continue + - If the model returns a final assistant message with no pending tool calls, break +5. Return `(final_prose, full_message_list, metadata)` + +The full message list is important: it's written to the conversation along with the final prose, so the tool-call receipts become part of the persistent record. + +### Slot-specific system prompts + +Compilation (full morning briefing): + +``` +You are the user's personal assistant giving their full morning briefing. +Use the tools available to see what's actually relevant today — tasks due, +overdue items, events on the calendar, weather, news themes, project state +— and weave it into a warm, natural-sounding summary. + +Rules: +- Call tools to see the data. Never assert facts you didn't learn from a tool. +- If a tool returns nothing (no events today, no overdue tasks), say so + honestly. Don't fabricate items to fill space. +- Write flowing prose. No markdown, no headers, no bullet points. +- Aim for 6–10 sentences. Skip topics that have nothing interesting. +- Close on one or two concrete, actionable suggestions. + +User profile (for tone and preferences): +{profile_body} +``` + +Check-ins (midday, afternoon): + +``` +You are the user's personal assistant giving a brief {slot} check-in. +Use tools to see what's changed since this morning. Focus on progress +and what's still unaddressed. + +Rules: +- Call tools to see current state. Never assert facts without tool results. +- If nothing meaningful has changed, say so briefly — don't invent progress. +- 3–5 sentences, natural prose, no markdown. +``` + +### Conversation hygiene — removing the fake user messages + +The current scheduler appends two fake messages on every slot injection: + +```python +await post_message(conv.id, "user", f"[{slot.title()} briefing update]") +await post_message(conv.id, "assistant", text) +``` + +Under the agentic model, we drop the fake `"user"` message entirely. Slot updates become plain assistant messages tagged with `metadata.briefing_slot = "midday"`. The chat endpoint's message loader is updated to filter these when building the LLM context on follow-ups, so a user chatting in the briefing conversation doesn't see three earlier briefings smashed into their history. The UI continues to show them as visible timeline entries. + +(This is the Option A filter from the earlier discussion — a small, surgical change compared to migrating briefings to a separate sidecar table. Sidecar storage remains a possible future step.) + +### Ollama setup compatibility + +The existing Ollama deployment uses non-parallel mode across two GPUs, with a concern about context duplication between cards. This is the correct setup for agentic briefings: + +- Each tool-call iteration shares the same KV cache on the same GPU, so appending tool results is cheap +- There is no race where a second iteration could land on a different card with a different cache state +- The trade-off is serialization: a briefing in progress will block a concurrent user chat request until it finishes, but scheduled briefings are rare enough (4× per day) that this is acceptable + +If GPU contention becomes a problem later, the right lever is pinning specific models to specific GPUs (e.g., background tasks on GPU 1, interactive models on GPU 0) — not enabling parallelism. + +## Cost & trade-offs + +- **Latency:** a briefing now makes 5–7 inference calls (one per tool-call decision plus the final prose) instead of 1. On a local Ollama with a 7B model, expect 15–40s per briefing vs the current 5–10s. Acceptable for a background job; if it becomes painful at the user-facing check-in slots, investigate letting the model batch independent tool calls in a single turn (Ollama supports this). +- **Model requirements:** the default model must be reliable at tool calling. `qwen2.5:7b`, `llama3.1:8b`, `mistral-small-3:24b`, and similar handle it well. Models in the ≤3B class typically fail — they either emit no tool calls and return empty prose, or hallucinate invalid tool arguments. If a user's `default_model` is too small, agentic mode should fall back to legacy mode with a log warning. +- **Context growth:** tool results bloat the conversation. At 8 tool calls per compilation × 4 slots per day, a daily briefing conversation can reach 20-30 KB of JSON-heavy history. Fine for a day; aged briefings should be archived/rolled up after 7 days. +- **Tool subset drift:** someone adds a new mutating tool to `tools.py` and forgets to update the briefing allowlist. Mitigated by the allowlist model — the default for new tools is "not exposed to briefings." +- **Infinite loop safety:** a buggy model could tool-call forever. Hard cap at 8 iterations, log a warning if hit, return whatever prose was last produced (or a fallback message). + +## Migration path + +Ship incrementally, each step independently reversible: + +**PR 1 — Agentic compilation behind a feature flag.** Add `briefing_tools.py`, add `run_agentic_briefing`, add a per-user setting `briefing_mode: "legacy" | "agentic"` (default legacy). Route only the 4am compilation through the new path when the flag is set. Keep slot-injection on the legacy path. Enable the flag on the author's account first, validate output quality over several days, then flip the default for all users. No DB migration required — the setting lives in the existing `settings` table. + +**PR 2 — Agentic slot injections + conversation hygiene.** Migrate midday/afternoon check-ins to the same pipeline. Remove the fake `[Midday briefing update]` user-role message; slot updates become plain assistant messages tagged with `metadata.briefing_slot`. Add a chat-message-loader filter that excludes slot-tagged messages from the LLM context on follow-ups (they remain visible in the UI). + +**PR 3 — UI polish.** Collapsed tool-call status row in the briefing card ("✓ checked calendar · ✓ looked at tasks · ✓ pulled weather"), expanding to show tool results on click. Tool-result cards (weather, news, task list) rendered inline where useful. + +**PR 4 (optional) — Sidecar storage for briefing snapshots.** If the chat-filter approach in PR 2 feels too hacky, migrate briefings out of the conversations table and into a dedicated `briefing_snapshots` table. Frontend renders them as pinned timeline cards separate from chat. Larger refactor; defer until PR 1–3 prove the approach works. + +## Secondary win — tightening the main chat system prompt + +The regular chat is already agentic (it uses `stream_chat_with_tools`), but its system prompt does not explicitly require the model to ground factual claims in tool results. The prompt discipline introduced for briefings — *"Never assert facts you didn't learn from a tool. If a tool returns nothing, say so honestly."* — is worth applying to the main chat system prompt in a follow-up PR. The mechanism already works; only the framing needs tightening. + +## Out of scope + +- The Android Flutter client's failure to render `search_images` tool-result cards. This is a separate rendering gap in the mobile app and does not affect the server-side fix. Tracked separately. +- Re-evaluating the Ollama parallelism setting. The current non-parallel config is correct for this work. +- Replacing the background model for title/summary/observation extraction. That model's role is unrelated to briefings. diff --git a/src/fabledassistant/services/briefing_pipeline.py b/src/fabledassistant/services/briefing_pipeline.py index 6bf64b6..662fab1 100644 --- a/src/fabledassistant/services/briefing_pipeline.py +++ b/src/fabledassistant/services/briefing_pipeline.py @@ -455,7 +455,199 @@ async def _gather_external(user_id: int) -> dict: } -# ── LLM synthesis ───────────────────────────────────────────────────────────── +# ── Agentic briefing (tool-use loop) ────────────────────────────────────────── + +_BRIEFING_AGENT_MAX_ROUNDS = 8 +_BRIEFING_AGENT_NUM_CTX = 8192 + + +def _agentic_system_prompt(profile_body: str, slot: str) -> str: + """System prompt for the agentic briefing path. + + Pushes the model to ground every factual claim in a tool result and + to be honest when tools return nothing, rather than fabricating + content to fill the narrative. + """ + if slot == "compilation": + base = ( + "You are the user's personal assistant giving their full morning briefing. " + "Use the tools available to see what's actually relevant today — tasks due, " + "overdue items, events on the calendar, weather, news themes, project state — " + "and weave it into a warm, natural-sounding summary.\n\n" + "Rules:\n" + "- Call tools to see the data. Never assert facts you didn't learn from a tool.\n" + "- If a tool returns nothing (no events today, no overdue tasks), say so honestly. " + "Don't fabricate items to fill space.\n" + "- Write flowing prose. No markdown, no headers, no bullet points.\n" + "- Aim for 6 to 10 sentences. Skip topics that have nothing interesting.\n" + "- Close on one or two concrete, actionable suggestions.\n\n" + ) + elif slot == "weekly_review": + base = ( + "You are the user's personal assistant delivering a weekly review. " + "Use the tools available to see what was accomplished this week, what's still " + "overdue, how many notes were captured, and what's coming up in the next seven days. " + "Write a reflective recap that celebrates real progress and gently flags what's stuck.\n\n" + "Rules:\n" + "- Call tools to see the data. Never assert facts you didn't learn from a tool.\n" + "- If a category is empty, say so honestly rather than inventing items.\n" + "- Write flowing prose. No markdown, no bullet points.\n" + "- Aim for 5 to 8 sentences. Reflective and encouraging tone.\n\n" + ) + else: # morning, midday, afternoon check-ins + base = ( + f"You are the user's personal assistant giving a brief {slot} check-in. " + "Use tools to see what's changed since this morning. Focus on progress and " + "what's still unaddressed.\n\n" + "Rules:\n" + "- Call tools to see current state. Never assert facts without tool results.\n" + "- If nothing meaningful has changed, say so briefly — don't invent progress.\n" + "- 3 to 5 sentences, natural prose, no markdown.\n\n" + ) + + if profile_body: + base += f"User profile (tone and preferences):\n{profile_body}\n" + return base + + +def _agentic_user_trigger(slot: str, date_str: str) -> str: + """Seed user-role message that kicks off the agentic run.""" + labels = { + "compilation": "morning briefing", + "morning": "morning check-in", + "midday": "midday check-in", + "afternoon": "afternoon wrap-up", + "weekly_review": "weekly review", + } + label = labels.get(slot, f"{slot} briefing") + return f"Generate my {label} for {date_str}." + + +async def run_agentic_briefing( + user_id: int, + slot: str, + model: str, + conv_id: int | None = None, +) -> tuple[str, list[dict]]: + """ + Run the agentic briefing loop for a user and slot. + + Uses the chat pipeline's tool-use loop with a curated read-only tool + subset and a slot-specific system prompt. Every fact the model states + is either derived from a tool result visible in the returned message + list or it's the model hallucinating — so follow-up chat in the same + conversation can hold the model to what the tool results actually show. + + Returns ``(final_prose, message_list)`` where ``message_list`` is the + full sequence including system, user trigger, tool calls, and tool + results. Callers are expected to persist those intermediate turns + alongside the final prose so the receipts remain in conversation + history on follow-up. + + If the loop fails or the model returns empty prose, returns + ``("", [])`` and the caller should fall back to the legacy path. + """ + from fabledassistant.services.llm import stream_chat_with_tools, ChatChunk # noqa: F401 + from fabledassistant.services.tools import execute_tool + from fabledassistant.services.briefing_tools import get_briefing_tools + from fabledassistant.services.user_profile import build_profile_context + from datetime import date as _date + + profile_context = await build_profile_context(user_id) + tools = await get_briefing_tools(user_id) + + if not tools: + logger.warning( + "Agentic briefing for user %d slot %s: no tools available — aborting", + user_id, slot, + ) + return "", [] + + date_str = _date.today().isoformat() + messages: list[dict] = [ + {"role": "system", "content": _agentic_system_prompt(profile_context, slot)}, + {"role": "user", "content": _agentic_user_trigger(slot, date_str)}, + ] + + final_text = "" + for round_idx in range(_BRIEFING_AGENT_MAX_ROUNDS): + accumulated_content = "" + accumulated_tool_calls: list[dict] = [] + + try: + async for chunk in stream_chat_with_tools( + messages, model, tools=tools, think=False, + num_ctx=_BRIEFING_AGENT_NUM_CTX, + ): + if chunk.type == "content" and chunk.content: + accumulated_content += chunk.content + elif chunk.type == "tool_calls" and chunk.tool_calls: + accumulated_tool_calls.extend(chunk.tool_calls) + except Exception: + logger.warning( + "Agentic briefing stream failed (user %d, slot %s, round %d)", + user_id, slot, round_idx, exc_info=True, + ) + return "", [] + + # Append the assistant turn (content + any tool calls) to history + assistant_msg: dict = {"role": "assistant", "content": accumulated_content} + if accumulated_tool_calls: + assistant_msg["tool_calls"] = accumulated_tool_calls + messages.append(assistant_msg) + + # No tool calls → the model is done + if not accumulated_tool_calls: + final_text = accumulated_content.strip() + break + + # Execute each tool call and append results as tool-role messages + for tc in accumulated_tool_calls: + fn = tc.get("function") or {} + tool_name = fn.get("name", "") + arguments = fn.get("arguments") or {} + if isinstance(arguments, str): + try: + import json as _json + arguments = _json.loads(arguments) + except Exception: + arguments = {} + + try: + result = await execute_tool(user_id, tool_name, arguments, conv_id=conv_id) + except Exception as exc: + logger.warning( + "Tool %s failed during agentic briefing: %s", tool_name, exc, + ) + result = {"success": False, "error": str(exc)} + + # Serialize the result compactly for the model's context + import json as _json + try: + result_str = _json.dumps(result, default=str)[:4000] + except Exception: + result_str = str(result)[:4000] + + messages.append({ + "role": "tool", + "content": result_str, + "tool_name": tool_name, + }) + else: + logger.warning( + "Agentic briefing hit max rounds (%d) for user %d slot %s — using last content", + _BRIEFING_AGENT_MAX_ROUNDS, user_id, slot, + ) + # Walk back to find the last assistant message with non-empty content + for m in reversed(messages): + if m.get("role") == "assistant" and m.get("content"): + final_text = m["content"].strip() + break + + return final_text, messages + + +# ── Legacy one-shot synthesis ───────────────────────────────────────────────── async def _llm_synthesise(system_prompt: str, user_prompt: str, model: str, num_ctx: int = 4096) -> str: """Single non-streaming LLM call. Returns the assistant's response text.""" @@ -812,17 +1004,34 @@ async def run_compilation( "topic_scores": topic_scores, } - briefing_text = await _llm_synthesise( - _unified_system_prompt(profile_context, slot), - _unified_user_prompt(internal_data_filtered, external_data_filtered, slot, temp_unit), - model, - num_ctx=8192, - ) + briefing_mode = await get_setting(user_id, "briefing_mode", "legacy") + agentic_messages: list[dict] = [] + briefing_text = "" + + if briefing_mode == "agentic": + briefing_text, agentic_messages = await run_agentic_briefing( + user_id, slot, model, conv_id=None, + ) + if not briefing_text: + logger.warning( + "Agentic briefing returned empty for user %d slot %s — falling back to legacy path", + user_id, slot, + ) + + if not briefing_text: + briefing_text = await _llm_synthesise( + _unified_system_prompt(profile_context, slot), + _unified_user_prompt(internal_data_filtered, external_data_filtered, slot, temp_unit), + model, + num_ctx=8192, + ) # ── Post-processing ───────────────────────────────────────────────────────── await upsert_task_snapshots(user_id, all_tasks) metadata: dict = {"rss_item_ids": rss_item_ids, "rss_items": rss_items_meta, "weather": weather_card} + if agentic_messages: + metadata["agentic_messages"] = agentic_messages if not briefing_text: logger.warning("Briefing compilation produced no content for user %d slot %s", user_id, slot) diff --git a/src/fabledassistant/services/briefing_tools.py b/src/fabledassistant/services/briefing_tools.py new file mode 100644 index 0000000..295cad2 --- /dev/null +++ b/src/fabledassistant/services/briefing_tools.py @@ -0,0 +1,62 @@ +""" +Curated read-only tool subset for agentic briefings. + +The main chat pipeline exposes 40+ tools via ``tools.get_tools_for_user``, +including mutating tools (``create_task``, ``delete_note``) and +external-search tools (``search_images``, ``search_web``). Neither is +appropriate for a scheduled background job that generates briefings — +briefings are read-only and should not reach out to the internet on the +user's behalf, and leaving high-noise tools in the list increases the +chance of spurious calls (e.g. ``search_images`` firing on "what +meeting?"). + +This module maintains an explicit allowlist. New tools added to +``tools.py`` are not automatically exposed to briefings — they must be +opted in by name here. +""" + +import logging + +from fabledassistant.services.tools import get_tools_for_user + +logger = logging.getLogger(__name__) + +# Explicit allowlist — tools a briefing is permitted to call. Read-only only. +# Any tool not listed here is invisible to the briefing model. +BRIEFING_TOOL_NAMES: frozenset[str] = frozenset({ + # Tasks — what's actionable today, overdue, or high priority + "list_tasks", + # Calendar — internal event store and CalDAV (if configured) + "list_events", + "search_events", + # Weather — today's forecast for the user's configured locations + "get_weather", + # News — RSS items filtered by user preferences + "get_rss_items", + # Projects — context for prioritization and narrative continuity + "list_projects", + "search_projects", + "get_project", + # Notes — surface recent captures for "pick up where you left off" + "list_notes", + "get_note", +}) + + +async def get_briefing_tools(user_id: int) -> list[dict]: + """Return the tool schemas a briefing run is permitted to call. + + Builds the user's full tool list (so user-specific gating such as + CalDAV availability still applies) and filters it down to the + briefing allowlist. + """ + all_tools = await get_tools_for_user(user_id) + filtered = [ + t for t in all_tools + if t.get("function", {}).get("name") in BRIEFING_TOOL_NAMES + ] + logger.debug( + "Briefing tools for user %d: %d of %d selected", + user_id, len(filtered), len(all_tools), + ) + return filtered