163 lines
5.9 KiB
Markdown
163 lines
5.9 KiB
Markdown
# Research Pipeline — Multi-Note Redesign
|
||
|
||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||
|
||
**Goal:** Replace the single monolithic research note with a set of focused, topic-driven notes plus an index note that links them — making research output browsable, TTS-friendly, and well-organized.
|
||
|
||
**Architecture:** Two new LLM calls (outline generation + N parallel section syntheses) replace the single large synthesis call. Public API unchanged — callers receive the index note. Fallback to single-note behavior on any outline failure.
|
||
|
||
**Tech Stack:** Python/Quart backend, existing `research.py` service, asyncio.gather for parallelism.
|
||
|
||
---
|
||
|
||
## Problem
|
||
|
||
The current pipeline synthesizes one note with a minimum of 2500 words and 6 sections. This creates:
|
||
- Notes too large to read or listen to comfortably
|
||
- No way to navigate directly to a specific sub-topic
|
||
- TTS failures on long prose (8000-char route limit, unbounded sentence buffers)
|
||
|
||
---
|
||
|
||
## Pipeline Flow
|
||
|
||
Public signature unchanged:
|
||
```python
|
||
async def run_research_pipeline(
|
||
topic: str,
|
||
user_id: int,
|
||
model: str,
|
||
buf=None,
|
||
project_id: int | None = None,
|
||
) -> Note: # returns the index note
|
||
```
|
||
|
||
Execution order:
|
||
|
||
```
|
||
1. Generate sub-queries (unchanged)
|
||
2. Search + fetch sources (unchanged)
|
||
3. Generate topic outline (NEW — one LLM call → 3–7 section dicts)
|
||
4. Synthesize each section note (NEW — parallelized via asyncio.gather)
|
||
5. Create all section notes in DB (sequential, tagged ["research"], same project_id)
|
||
6. Create index note (NEW — links all sections)
|
||
7. Return index note
|
||
```
|
||
|
||
Status messages via `buf.append_event("status", ...)`:
|
||
- `"Generating outline…"`
|
||
- `"Writing: [Section Title]…"` (one per section, emitted before synthesis starts)
|
||
- `"Saving [N] notes…"`
|
||
|
||
No note content is streamed into chat. After the tool call resolves, the LLM writes a brief conversational summary citing the index note title and section count.
|
||
|
||
---
|
||
|
||
## Outline Generation
|
||
|
||
New function: `_generate_outline(topic, sources, model) -> list[dict]`
|
||
|
||
Sends all fetched sources to the model with a prompt requesting a JSON array:
|
||
|
||
```json
|
||
[
|
||
{"title": "Quantum Entanglement: Mechanisms", "focus": "How entanglement works at the physical level"},
|
||
{"title": "Quantum Computing Hardware", "focus": "Ion traps, superconducting qubits, photonic approaches"}
|
||
]
|
||
```
|
||
|
||
**Prompt requirements:**
|
||
- Produce 3–7 sections covering distinct aspects of the topic
|
||
- Titles must work as standalone note titles (no "Overview" or "Introduction" generics)
|
||
- No overlap between sections
|
||
- `focus` is one sentence describing what this section should specifically cover
|
||
|
||
**Guardrails:**
|
||
- Fewer than 3 sections parsed → fall back to single-note synthesis
|
||
- JSON parse failure → fall back to single-note synthesis
|
||
- More than 8 sections → truncate to 8
|
||
|
||
**Model params:** `max_tokens=400, num_ctx=16384` (outline is short)
|
||
|
||
---
|
||
|
||
## Section Synthesis
|
||
|
||
New function: `_synthesize_section(section_title, section_focus, sources, model) -> tuple[str, str]`
|
||
|
||
Returns `(title, body_markdown)`.
|
||
|
||
All sections receive all fetched sources. The `section_focus` field in the prompt directs the model to draw only what's relevant to that section's scope.
|
||
|
||
**Prompt requirements:**
|
||
- 300–600 words of substantive prose
|
||
- Do NOT include a `# Title` heading (title is set separately)
|
||
- End with a brief `## Sources` list of relevant URLs from the provided sources
|
||
- Focus strictly on `section_focus` — ignore source material outside that scope
|
||
|
||
**Model params:** `num_predict=2048, num_ctx=16384` (reduced from 8192 — sufficient for 600 words, prevents rambling)
|
||
|
||
**Parallelism:** All section synthesis calls run via `asyncio.gather`. Wall-clock time stays close to a single synthesis call despite producing N notes.
|
||
|
||
---
|
||
|
||
## Note Creation and Index Note
|
||
|
||
**Section notes:**
|
||
- Tags: `["research"]`
|
||
- `project_id`: same as passed to pipeline (or None)
|
||
- Title: from outline `title` field
|
||
- Created sequentially (avoids DB contention)
|
||
|
||
**Index note:**
|
||
- Tags: `["research", "research-index"]`
|
||
- `project_id`: same as section notes
|
||
- Title: `"Research: [topic]"`
|
||
- Created last (after all section notes exist)
|
||
|
||
**Index note body format:**
|
||
```markdown
|
||
Research overview for **[topic]** — [YYYY-MM-DD]
|
||
|
||
Generated from [N] web sources across [M] sections.
|
||
|
||
## Sections
|
||
|
||
- **[Section 1 Title]** — [focus sentence]
|
||
- **[Section 2 Title]** — [focus sentence]
|
||
...
|
||
|
||
*Search for any section title to read it.*
|
||
```
|
||
|
||
The index note is what `run_research_pipeline` returns. The existing `research_topic` tool handler uses `note.id` and `note.title` — both remain valid with the index note.
|
||
|
||
---
|
||
|
||
## Error Handling
|
||
|
||
| Scenario | Behaviour |
|
||
|---|---|
|
||
| Outline generation raises | Fall back to single-note synthesis (current behaviour) |
|
||
| Outline JSON unparseable | Fall back to single-note synthesis |
|
||
| Outline returns < 3 sections | Fall back to single-note synthesis |
|
||
| Outline returns > 8 sections | Truncate to 8, continue |
|
||
| A section synthesis raises | Log warning, skip that section; continue with remaining |
|
||
| All section syntheses fail | Fall back to single-note synthesis |
|
||
| A section note DB save fails | Log warning, skip from index; index note still created |
|
||
| No sources fetched | Raise `ValueError` as today — unchanged |
|
||
|
||
The fallback in every case is the current single-note pipeline. Research never silently produces nothing.
|
||
|
||
---
|
||
|
||
## What Is NOT Changing
|
||
|
||
- Public function signature of `run_research_pipeline`
|
||
- Sub-query generation (`_generate_sub_queries`)
|
||
- SearXNG search and URL fetching
|
||
- `_search_searxng`, `_search_searxng_images`, `fetch_url_content`
|
||
- The `research_topic` tool definition and handler in `tools.py`
|
||
- The `quick_capture` research path
|
||
- Any frontend component
|