# Research Pipeline — Multi-Note Redesign

> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.

**Goal:** Replace the single monolithic research note with a set of focused, topic-driven notes plus an index note that links them — making research output browsable, TTS-friendly, and well-organized.

**Architecture:** Two new LLM calls (outline generation + N parallel section syntheses) replace the single large synthesis call. Public API unchanged — callers receive the index note. Fallback to single-note behavior on any outline failure.

**Tech Stack:** Python/Quart backend, existing `research.py` service, asyncio.gather for parallelism.

---

## Problem

The current pipeline synthesizes one note with a minimum of 2500 words and 6 sections. This creates:
- Notes too large to read or listen to comfortably
- No way to navigate directly to a specific sub-topic
- TTS failures on long prose (8000-char route limit, unbounded sentence buffers)

---

## Pipeline Flow

Public signature unchanged:
```python
async def run_research_pipeline(
    topic: str,
    user_id: int,
    model: str,
    buf=None,
    project_id: int | None = None,
) -> Note:  # returns the index note
```

Execution order:

```
1. Generate sub-queries           (unchanged)
2. Search + fetch sources         (unchanged)
3. Generate topic outline         (NEW — one LLM call → 3–7 section dicts)
4. Synthesize each section note   (NEW — parallelized via asyncio.gather)
5. Create all section notes in DB (sequential, tagged ["research"], same project_id)
6. Create index note              (NEW — links all sections)
7. Return index note
```

Status messages via `buf.append_event("status", ...)`:
- `"Generating outline…"`
- `"Writing: [Section Title]…"` (one per section, emitted before synthesis starts)
- `"Saving [N] notes…"`

No note content is streamed into chat. After the tool call resolves, the LLM writes a brief conversational summary citing the index note title and section count.

---

## Outline Generation

New function: `_generate_outline(topic, sources, model) -> list[dict]`

Sends all fetched sources to the model with a prompt requesting a JSON array:

```json
[
  {"title": "Quantum Entanglement: Mechanisms", "focus": "How entanglement works at the physical level"},
  {"title": "Quantum Computing Hardware", "focus": "Ion traps, superconducting qubits, photonic approaches"}
]
```

**Prompt requirements:**
- Produce 3–7 sections covering distinct aspects of the topic
- Titles must work as standalone note titles (no "Overview" or "Introduction" generics)
- No overlap between sections
- `focus` is one sentence describing what this section should specifically cover

**Guardrails:**
- Fewer than 3 sections parsed → fall back to single-note synthesis
- JSON parse failure → fall back to single-note synthesis
- More than 8 sections → truncate to 8

**Model params:** `max_tokens=400, num_ctx=16384` (outline is short)

---

## Section Synthesis

New function: `_synthesize_section(section_title, section_focus, sources, model) -> tuple[str, str]`

Returns `(title, body_markdown)`.

All sections receive all fetched sources. The `section_focus` field in the prompt directs the model to draw only what's relevant to that section's scope.

**Prompt requirements:**
- 300–600 words of substantive prose
- Do NOT include a `# Title` heading (title is set separately)
- End with a brief `## Sources` list of relevant URLs from the provided sources
- Focus strictly on `section_focus` — ignore source material outside that scope

**Model params:** `num_predict=2048, num_ctx=16384` (reduced from 8192 — sufficient for 600 words, prevents rambling)

**Parallelism:** All section synthesis calls run via `asyncio.gather`. Wall-clock time stays close to a single synthesis call despite producing N notes.

---

## Note Creation and Index Note

**Section notes:**
- Tags: `["research"]`
- `project_id`: same as passed to pipeline (or None)
- Title: from outline `title` field
- Created sequentially (avoids DB contention)

**Index note:**
- Tags: `["research", "research-index"]`
- `project_id`: same as section notes
- Title: `"Research: [topic]"`
- Created last (after all section notes exist)

**Index note body format:**
```markdown
Research overview for **[topic]** — [YYYY-MM-DD]

Generated from [N] web sources across [M] sections.

## Sections

- **[Section 1 Title]** — [focus sentence]
- **[Section 2 Title]** — [focus sentence]
...

*Search for any section title to read it.*
```

The index note is what `run_research_pipeline` returns. The existing `research_topic` tool handler uses `note.id` and `note.title` — both remain valid with the index note.

---

## Error Handling

| Scenario | Behaviour |
|---|---|
| Outline generation raises | Fall back to single-note synthesis (current behaviour) |
| Outline JSON unparseable | Fall back to single-note synthesis |
| Outline returns < 3 sections | Fall back to single-note synthesis |
| Outline returns > 8 sections | Truncate to 8, continue |
| A section synthesis raises | Log warning, skip that section; continue with remaining |
| All section syntheses fail | Fall back to single-note synthesis |
| A section note DB save fails | Log warning, skip from index; index note still created |
| No sources fetched | Raise `ValueError` as today — unchanged |

The fallback in every case is the current single-note pipeline. Research never silently produces nothing.

---

## What Is NOT Changing

- Public function signature of `run_research_pipeline`
- Sub-query generation (`_generate_sub_queries`)
- SearXNG search and URL fetching
- `_search_searxng`, `_search_searxng_images`, `fetch_url_content`
- The `research_topic` tool definition and handler in `tools.py`
- The `quick_capture` research path
- Any frontend component