docs: add research multi-note redesign spec
This commit is contained in:
@@ -0,0 +1,162 @@
|
||||
# Research Pipeline — Multi-Note Redesign
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||
|
||||
**Goal:** Replace the single monolithic research note with a set of focused, topic-driven notes plus an index note that links them — making research output browsable, TTS-friendly, and well-organized.
|
||||
|
||||
**Architecture:** Two new LLM calls (outline generation + N parallel section syntheses) replace the single large synthesis call. Public API unchanged — callers receive the index note. Fallback to single-note behavior on any outline failure.
|
||||
|
||||
**Tech Stack:** Python/Quart backend, existing `research.py` service, asyncio.gather for parallelism.
|
||||
|
||||
---
|
||||
|
||||
## Problem
|
||||
|
||||
The current pipeline synthesizes one note with a minimum of 2500 words and 6 sections. This creates:
|
||||
- Notes too large to read or listen to comfortably
|
||||
- No way to navigate directly to a specific sub-topic
|
||||
- TTS failures on long prose (8000-char route limit, unbounded sentence buffers)
|
||||
|
||||
---
|
||||
|
||||
## Pipeline Flow
|
||||
|
||||
Public signature unchanged:
|
||||
```python
|
||||
async def run_research_pipeline(
|
||||
topic: str,
|
||||
user_id: int,
|
||||
model: str,
|
||||
buf=None,
|
||||
project_id: int | None = None,
|
||||
) -> Note: # returns the index note
|
||||
```
|
||||
|
||||
Execution order:
|
||||
|
||||
```
|
||||
1. Generate sub-queries (unchanged)
|
||||
2. Search + fetch sources (unchanged)
|
||||
3. Generate topic outline (NEW — one LLM call → 3–7 section dicts)
|
||||
4. Synthesize each section note (NEW — parallelized via asyncio.gather)
|
||||
5. Create all section notes in DB (sequential, tagged ["research"], same project_id)
|
||||
6. Create index note (NEW — links all sections)
|
||||
7. Return index note
|
||||
```
|
||||
|
||||
Status messages via `buf.append_event("status", ...)`:
|
||||
- `"Generating outline…"`
|
||||
- `"Writing: [Section Title]…"` (one per section, emitted before synthesis starts)
|
||||
- `"Saving [N] notes…"`
|
||||
|
||||
No note content is streamed into chat. After the tool call resolves, the LLM writes a brief conversational summary citing the index note title and section count.
|
||||
|
||||
---
|
||||
|
||||
## Outline Generation
|
||||
|
||||
New function: `_generate_outline(topic, sources, model) -> list[dict]`
|
||||
|
||||
Sends all fetched sources to the model with a prompt requesting a JSON array:
|
||||
|
||||
```json
|
||||
[
|
||||
{"title": "Quantum Entanglement: Mechanisms", "focus": "How entanglement works at the physical level"},
|
||||
{"title": "Quantum Computing Hardware", "focus": "Ion traps, superconducting qubits, photonic approaches"}
|
||||
]
|
||||
```
|
||||
|
||||
**Prompt requirements:**
|
||||
- Produce 3–7 sections covering distinct aspects of the topic
|
||||
- Titles must work as standalone note titles (no "Overview" or "Introduction" generics)
|
||||
- No overlap between sections
|
||||
- `focus` is one sentence describing what this section should specifically cover
|
||||
|
||||
**Guardrails:**
|
||||
- Fewer than 3 sections parsed → fall back to single-note synthesis
|
||||
- JSON parse failure → fall back to single-note synthesis
|
||||
- More than 8 sections → truncate to 8
|
||||
|
||||
**Model params:** `max_tokens=400, num_ctx=16384` (outline is short)
|
||||
|
||||
---
|
||||
|
||||
## Section Synthesis
|
||||
|
||||
New function: `_synthesize_section(section_title, section_focus, sources, model) -> tuple[str, str]`
|
||||
|
||||
Returns `(title, body_markdown)`.
|
||||
|
||||
All sections receive all fetched sources. The `section_focus` field in the prompt directs the model to draw only what's relevant to that section's scope.
|
||||
|
||||
**Prompt requirements:**
|
||||
- 300–600 words of substantive prose
|
||||
- Do NOT include a `# Title` heading (title is set separately)
|
||||
- End with a brief `## Sources` list of relevant URLs from the provided sources
|
||||
- Focus strictly on `section_focus` — ignore source material outside that scope
|
||||
|
||||
**Model params:** `num_predict=2048, num_ctx=16384` (reduced from 8192 — sufficient for 600 words, prevents rambling)
|
||||
|
||||
**Parallelism:** All section synthesis calls run via `asyncio.gather`. Wall-clock time stays close to a single synthesis call despite producing N notes.
|
||||
|
||||
---
|
||||
|
||||
## Note Creation and Index Note
|
||||
|
||||
**Section notes:**
|
||||
- Tags: `["research"]`
|
||||
- `project_id`: same as passed to pipeline (or None)
|
||||
- Title: from outline `title` field
|
||||
- Created sequentially (avoids DB contention)
|
||||
|
||||
**Index note:**
|
||||
- Tags: `["research", "research-index"]`
|
||||
- `project_id`: same as section notes
|
||||
- Title: `"Research: [topic]"`
|
||||
- Created last (after all section notes exist)
|
||||
|
||||
**Index note body format:**
|
||||
```markdown
|
||||
Research overview for **[topic]** — [YYYY-MM-DD]
|
||||
|
||||
Generated from [N] web sources across [M] sections.
|
||||
|
||||
## Sections
|
||||
|
||||
- **[Section 1 Title]** — [focus sentence]
|
||||
- **[Section 2 Title]** — [focus sentence]
|
||||
...
|
||||
|
||||
*Search for any section title to read it.*
|
||||
```
|
||||
|
||||
The index note is what `run_research_pipeline` returns. The existing `research_topic` tool handler uses `note.id` and `note.title` — both remain valid with the index note.
|
||||
|
||||
---
|
||||
|
||||
## Error Handling
|
||||
|
||||
| Scenario | Behaviour |
|
||||
|---|---|
|
||||
| Outline generation raises | Fall back to single-note synthesis (current behaviour) |
|
||||
| Outline JSON unparseable | Fall back to single-note synthesis |
|
||||
| Outline returns < 3 sections | Fall back to single-note synthesis |
|
||||
| Outline returns > 8 sections | Truncate to 8, continue |
|
||||
| A section synthesis raises | Log warning, skip that section; continue with remaining |
|
||||
| All section syntheses fail | Fall back to single-note synthesis |
|
||||
| A section note DB save fails | Log warning, skip from index; index note still created |
|
||||
| No sources fetched | Raise `ValueError` as today — unchanged |
|
||||
|
||||
The fallback in every case is the current single-note pipeline. Research never silently produces nothing.
|
||||
|
||||
---
|
||||
|
||||
## What Is NOT Changing
|
||||
|
||||
- Public function signature of `run_research_pipeline`
|
||||
- Sub-query generation (`_generate_sub_queries`)
|
||||
- SearXNG search and URL fetching
|
||||
- `_search_searxng`, `_search_searxng_images`, `fetch_url_content`
|
||||
- The `research_topic` tool definition and handler in `tools.py`
|
||||
- The `quick_capture` research path
|
||||
- Any frontend component
|
||||
Reference in New Issue
Block a user