FabledScribe/docs/2026-04-10-agentic-briefing-design.md

Agentic Briefing — Design Spec

Date: 2026-04-10 Status: Proposed Author: bvandeusen + Claude

---

Problem

The current briefing pipeline hallucinates calendar events, tasks, and news items that do not exist in the user's actual data. Observed examples from production:

• A morning briefing asserting "your dentist appointment is still in progress" when no such event existed
• An 8am check-in mentioning "a quick meeting at 10:30 AM" with no backing calendar entry
• A midday check-in inventing "a team huddle at 2:30 PM" and "the Q2 budget draft due by Friday"
• The model calling search_images in response to a clarifying question about a fabricated meeting

These are not bugs in data retrieval — the data gathering code (_gather_internal, _gather_external) returns correct values. They are a structural consequence of how the briefing context is assembled.

Root cause — the "no receipt" problem

The current run_compilation pipeline does this:

1. Python code gathers data (tasks, events, weather, news) from the database
2. Python formats the data into a structured text blob as the user-role message: TODAY'S EVENTS: ..., DUE TODAY: ..., etc.
3. The LLM is called once with [system_prompt, user_prompt] and produces prose
4. Only the prose reply is written to the conversation. The underlying data is never persisted in the conversation history.

When the user later chats in the briefing conversation, the chat endpoint loads the full conversation history — which contains the model's prose from earlier but not the data that prose was derived from. The model's own prior output becomes the only source of "truth" available for follow-up questions. If that prose asserted a fact (real or hallucinated), the model has no way to distinguish it from ground truth when generating the next reply, and it will double down.

Compounding factors:

• Empty sections are silently omitted. If calendar_events is empty, the user prompt contains no TODAY'S EVENTS: line at all. The model has no explicit "zero events" signal — combined with an imperative system prompt ("note calendar events and tasks"), it interprets the silence as "I should mention some" and fabricates.
• Scheduled slot injections append synthetic turns. run_slot_injection writes a fake [Midday briefing update] user message and the assistant reply into the persistent conversation. By evening, the chat history contains three separate briefings, each potentially with errors, all treated as equal-weight context on follow-up.
• The search_images tool is available during briefing chat, with a negative-instruction description ("Not for factual questions"). Small and mid-sized models frequently ignore negative guidance in tool descriptions and call the tool anyway.

Solution — agentic briefing (the receipt model)

Replace the one-shot synthesis with a tool-call loop. The briefing is no longer "a text blob synthesized from pre-gathered data." It is a scheduled agent run: the LLM is given a system prompt, a curated set of read-only data tools, and a trigger ("generate the morning briefing"). The model must call tools to see what exists. Every tool call and tool result becomes part of the conversation history, where it lives as a permanent, structured receipt.

Why this fixes the hallucination

When the model calls list_events(today) → [], that empty array is now a persistent message in the conversation. On a follow-up question like "what meeting?", the chat endpoint loads the conversation and the model sees its own tool result from the morning showing no events. Answering "you had a meeting at 10:30" would require the model to directly contradict a tool result sitting two messages back — something LLMs are much more reliable at avoiding than contradicting their own prior prose.

In short: the model cannot fabricate what has a visible receipt proving it does not exist.

What the model sees — before vs after

Before — the model sees one structured blob and produces prose. The blob is discarded.

[system] You are a personal assistant... weave together what matters...
[user]   Date: 2026-04-10
         WEATHER: Home — partly cloudy, 8–16°C
         DUE TODAY: Fix briefing hallucinations
         OVERDUE (2 tasks): ...
[assistant] Morning! Looks like a quiet day with two overdue items...

After — the model is given tools and iterates to ground truth.

[system] You are the user's personal assistant giving their morning briefing.
         Use tools to see what's relevant. Only mention things you learned from
         tool calls. If a tool returns nothing, say so honestly.
[user]   Generate the morning briefing.
[assistant] tool_call: list_tasks(filter="due_today")
[tool]   [{"id": 81, "title": "Fix briefing hallucinations", ...}]
[assistant] tool_call: list_tasks(filter="overdue")
[tool]   [{"id": 42, ...}, {"id": 43, ...}]
[assistant] tool_call: list_events(today)
[tool]   []
[assistant] tool_call: get_weather(location="home")
[tool]   {"description": "partly cloudy", "temp_min": 8, "temp_max": 16}
[assistant] tool_call: get_rss_items(max=8)
[tool]   [...]
[assistant] Morning! Partly cloudy today, 8 to 16 — nothing on the calendar
         so it's a clean run at the desk. Two things to keep in mind...

The conversation now contains verifiable receipts: list_events(today) returned [], and that result sits in context forever (until pruned). Follow-up questions operate against those receipts.

Architecture

Existing infrastructure (reused, not rebuilt)

The codebase already has the agentic primitives — they're used for regular chat:

• llm.py::stream_chat_with_tools — the streaming tool-use loop that talks to Ollama with a tools parameter and yields tool-call chunks
• generation_task.py::_stream_with_retry — wraps stream_chat_with_tools with retry-on-500 behavior for cold-model races
• tools.py — defines 40+ tool schemas and an execution dispatcher

The briefing bypasses all of this and calls a one-shot _llm_synthesise helper. The refactor is mainly "route briefings through the same pipeline regular chat already uses."

New modules

briefing_tools.py — a small wrapper exposing a curated read-only subset of tools.py for briefing runs. This is an explicit allowlist, not a blocklist, so newly-added tools must be opted in:

Tool	Purpose
list_tasks (with filter args)	See what's actionable today, overdue, high priority
list_events (today / upcoming)	Know what's on the calendar
get_weather	Current/forecast weather
get_rss_items	Pull news themes filtered by user preferences
list_projects	Understand project context
search_projects	Surface active project summaries
list_notes (recent)	Capture follow-ups from yesterday

Explicitly omitted from the briefing tool set:

• search_images, search_web, research_topic, read_article — external search is not a briefing concern, and search_images is the source of the "Peter Kyle Science Secretary" image-search bug
• All create_*, update_*, delete_* — briefings are read-only; a scheduled background job must not decide to mutate the user's data on its own
• set_rag_scope, calculate — not relevant to briefing content

New briefing function

briefing_pipeline.py::run_agentic_briefing(user_id, slot, model, conversation_id) replaces run_compilation's body (and eventually run_slot_injection). Internally:

1. Build a slot-specific system prompt (see below)
2. Load the curated briefing tools from briefing_tools.py
3. Seed messages with [system, user] where the user message is a simple trigger like "Generate the morning briefing."
4. Enter a tool-call loop (max 8 iterations):
   • Call stream_chat_with_tools
   • If the model returns tool_calls, execute them via the existing dispatcher, append tool results, continue
   • If the model returns a final assistant message with no pending tool calls, break
5. Return (final_prose, full_message_list, metadata)

The full message list is important: it's written to the conversation along with the final prose, so the tool-call receipts become part of the persistent record.

Slot-specific system prompts

Compilation (full morning briefing):

You are the user's personal assistant giving their full morning briefing.
Use the tools available to see what's actually relevant today — tasks due,
overdue items, events on the calendar, weather, news themes, project state
— and weave it into a warm, natural-sounding summary.

Rules:
- Call tools to see the data. Never assert facts you didn't learn from a tool.
- If a tool returns nothing (no events today, no overdue tasks), say so
  honestly. Don't fabricate items to fill space.
- Write flowing prose. No markdown, no headers, no bullet points.
- Aim for 6–10 sentences. Skip topics that have nothing interesting.
- Close on one or two concrete, actionable suggestions.

User profile (for tone and preferences):
{profile_body}

Check-ins (midday, afternoon):

You are the user's personal assistant giving a brief {slot} check-in.
Use tools to see what's changed since this morning. Focus on progress
and what's still unaddressed.

Rules:
- Call tools to see current state. Never assert facts without tool results.
- If nothing meaningful has changed, say so briefly — don't invent progress.
- 3–5 sentences, natural prose, no markdown.

Conversation hygiene — removing the fake user messages

The current scheduler appends two fake messages on every slot injection:

await post_message(conv.id, "user", f"[{slot.title()} briefing update]")
await post_message(conv.id, "assistant", text)

Under the agentic model, we drop the fake "user" message entirely. Slot updates become plain assistant messages tagged with metadata.briefing_slot = "midday". The chat endpoint's message loader is updated to filter these when building the LLM context on follow-ups, so a user chatting in the briefing conversation doesn't see three earlier briefings smashed into their history. The UI continues to show them as visible timeline entries.

(This is the Option A filter from the earlier discussion — a small, surgical change compared to migrating briefings to a separate sidecar table. Sidecar storage remains a possible future step.)

Ollama setup compatibility

The existing Ollama deployment uses non-parallel mode across two GPUs, with a concern about context duplication between cards. This is the correct setup for agentic briefings:

• Each tool-call iteration shares the same KV cache on the same GPU, so appending tool results is cheap
• There is no race where a second iteration could land on a different card with a different cache state
• The trade-off is serialization: a briefing in progress will block a concurrent user chat request until it finishes, but scheduled briefings are rare enough (4× per day) that this is acceptable

If GPU contention becomes a problem later, the right lever is pinning specific models to specific GPUs (e.g., background tasks on GPU 1, interactive models on GPU 0) — not enabling parallelism.

Cost & trade-offs

• Latency: a briefing now makes 5–7 inference calls (one per tool-call decision plus the final prose) instead of 1. On a local Ollama with a 7B model, expect 15–40s per briefing vs the current 5–10s. Acceptable for a background job; if it becomes painful at the user-facing check-in slots, investigate letting the model batch independent tool calls in a single turn (Ollama supports this).
• Model requirements: the default model must be reliable at tool calling. qwen2.5:7b, llama3.1:8b, mistral-small-3:24b, and similar handle it well. Models in the ≤3B class typically fail — they either emit no tool calls and return empty prose, or hallucinate invalid tool arguments. If a user's default_model is too small, agentic mode should fall back to legacy mode with a log warning.
• Context growth: tool results bloat the conversation. At 8 tool calls per compilation × 4 slots per day, a daily briefing conversation can reach 20-30 KB of JSON-heavy history. Fine for a day; aged briefings should be archived/rolled up after 7 days.
• Tool subset drift: someone adds a new mutating tool to tools.py and forgets to update the briefing allowlist. Mitigated by the allowlist model — the default for new tools is "not exposed to briefings."
• Infinite loop safety: a buggy model could tool-call forever. Hard cap at 8 iterations, log a warning if hit, return whatever prose was last produced (or a fallback message).

Migration path

Ship incrementally, each step independently reversible:

PR 1 — Agentic compilation behind a feature flag. Add briefing_tools.py, add run_agentic_briefing, add a per-user setting briefing_mode: "legacy" | "agentic" (default legacy). Route only the 4am compilation through the new path when the flag is set. Keep slot-injection on the legacy path. Enable the flag on the author's account first, validate output quality over several days, then flip the default for all users. No DB migration required — the setting lives in the existing settings table.

PR 2 — Agentic slot injections + conversation hygiene. Migrate midday/afternoon check-ins to the same pipeline. Remove the fake [Midday briefing update] user-role message; slot updates become plain assistant messages tagged with metadata.briefing_slot. Add a chat-message-loader filter that excludes slot-tagged messages from the LLM context on follow-ups (they remain visible in the UI).

PR 3 — UI polish. Collapsed tool-call status row in the briefing card ("✓ checked calendar · ✓ looked at tasks · ✓ pulled weather"), expanding to show tool results on click. Tool-result cards (weather, news, task list) rendered inline where useful.

PR 4 (optional) — Sidecar storage for briefing snapshots. If the chat-filter approach in PR 2 feels too hacky, migrate briefings out of the conversations table and into a dedicated briefing_snapshots table. Frontend renders them as pinned timeline cards separate from chat. Larger refactor; defer until PR 1–3 prove the approach works.

Secondary win — tightening the main chat system prompt

The regular chat is already agentic (it uses stream_chat_with_tools), but its system prompt does not explicitly require the model to ground factual claims in tool results. The prompt discipline introduced for briefings — "Never assert facts you didn't learn from a tool. If a tool returns nothing, say so honestly." — is worth applying to the main chat system prompt in a follow-up PR. The mechanism already works; only the framing needs tightening.

Out of scope

• The Android Flutter client's failure to render search_images tool-result cards. This is a separate rendering gap in the mobile app and does not affect the server-side fix. Tracked separately.
• Re-evaluating the Ollama parallelism setting. The current non-parallel config is correct for this work.
• Replacing the background model for title/summary/observation extraction. That model's role is unrelated to briefings.