FabledScribe/docs/superpowers/specs/2026-04-06-research-multi-note-design.md

Research Pipeline — Multi-Note Redesign

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Replace the single monolithic research note with a set of focused, topic-driven notes plus an index note that links them — making research output browsable, TTS-friendly, and well-organized.

Architecture: Two new LLM calls (outline generation + N parallel section syntheses) replace the single large synthesis call. Public API unchanged — callers receive the index note. Fallback to single-note behavior on any outline failure.

Tech Stack: Python/Quart backend, existing research.py service, asyncio.gather for parallelism.

---

Problem

The current pipeline synthesizes one note with a minimum of 2500 words and 6 sections. This creates:

• Notes too large to read or listen to comfortably
• No way to navigate directly to a specific sub-topic
• TTS failures on long prose (8000-char route limit, unbounded sentence buffers)

---

Pipeline Flow

Public signature unchanged:

async def run_research_pipeline(
    topic: str,
    user_id: int,
    model: str,
    buf=None,
    project_id: int | None = None,
) -> Note:  # returns the index note

Execution order:

1. Generate sub-queries           (unchanged)
2. Search + fetch sources         (unchanged)
3. Generate topic outline         (NEW — one LLM call → 3–7 section dicts)
4. Synthesize each section note   (NEW — parallelized via asyncio.gather)
5. Create all section notes in DB (sequential, tagged ["research"], same project_id)
6. Create index note              (NEW — links all sections)
7. Return index note

Status messages via buf.append_event("status", ...):

• "Generating outline…"
• "Writing: [Section Title]…" (one per section, emitted before synthesis starts)
• "Saving [N] notes…"

No note content is streamed into chat. After the tool call resolves, the LLM writes a brief conversational summary citing the index note title and section count.

---

Outline Generation

New function: _generate_outline(topic, sources, model) -> list[dict]

Sends all fetched sources to the model with a prompt requesting a JSON array:

[
  {"title": "Quantum Entanglement: Mechanisms", "focus": "How entanglement works at the physical level"},
  {"title": "Quantum Computing Hardware", "focus": "Ion traps, superconducting qubits, photonic approaches"}
]

Prompt requirements:

• Produce 3–7 sections covering distinct aspects of the topic
• Titles must work as standalone note titles (no "Overview" or "Introduction" generics)
• No overlap between sections
• focus is one sentence describing what this section should specifically cover

Guardrails:

• Fewer than 3 sections parsed → fall back to single-note synthesis
• JSON parse failure → fall back to single-note synthesis
• More than 8 sections → truncate to 8

Model params: max_tokens=400, num_ctx=16384 (outline is short)

---

Section Synthesis

New function: _synthesize_section(section_title, section_focus, sources, model) -> tuple[str, str]

Returns (title, body_markdown).

All sections receive all fetched sources. The section_focus field in the prompt directs the model to draw only what's relevant to that section's scope.

Prompt requirements:

• 300–600 words of substantive prose
• Do NOT include a # Title heading (title is set separately)
• End with a brief ## Sources list of relevant URLs from the provided sources
• Focus strictly on section_focus — ignore source material outside that scope

Model params: num_predict=2048, num_ctx=16384 (reduced from 8192 — sufficient for 600 words, prevents rambling)

Parallelism: All section synthesis calls run via asyncio.gather. Wall-clock time stays close to a single synthesis call despite producing N notes.

---

Note Creation and Index Note

Section notes:

• Tags: ["research"]
• project_id: same as passed to pipeline (or None)
• Title: from outline title field
• Created sequentially (avoids DB contention)

Index note:

• Tags: ["research", "research-index"]
• project_id: same as section notes
• Title: "Research: [topic]"
• Created last (after all section notes exist)

Index note body format:

Research overview for **[topic]** — [YYYY-MM-DD]

Generated from [N] web sources across [M] sections.

## Sections

- **[Section 1 Title]** — [focus sentence]
- **[Section 2 Title]** — [focus sentence]
...

*Search for any section title to read it.*

The index note is what run_research_pipeline returns. The existing research_topic tool handler uses note.id and note.title — both remain valid with the index note.

---

Error Handling

Scenario	Behaviour
Outline generation raises	Fall back to single-note synthesis (current behaviour)
Outline JSON unparseable	Fall back to single-note synthesis
Outline returns < 3 sections	Fall back to single-note synthesis
Outline returns > 8 sections	Truncate to 8, continue
A section synthesis raises	Log warning, skip that section; continue with remaining
All section syntheses fail	Fall back to single-note synthesis
A section note DB save fails	Log warning, skip from index; index note still created
No sources fetched	Raise ValueError as today — unchanged

The fallback in every case is the current single-note pipeline. Research never silently produces nothing.

---

What Is NOT Changing

• Public function signature of run_research_pipeline
• Sub-query generation (_generate_sub_queries)
• SearXNG search and URL fetching
• _search_searxng, _search_searxng_images, fetch_url_content
• The research_topic tool definition and handler in tools.py
• The quick_capture research path
• Any frontend component