docs: add unified lookup & Wikipedia implementation plan
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,782 @@
|
||||
# Unified Lookup Tool & Wikipedia Integration — Implementation Plan
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||
|
||||
**Goal:** Replace `search_web` with a unified `lookup` tool (Wikipedia-first, SearXNG fallback) and add Wikipedia as a source in the research pipeline.
|
||||
|
||||
**Architecture:** New `wikipedia.py` service with `wiki_summary` and `wiki_search`. `lookup` tool in `web.py` replaces `search_web`. Research pipeline in `research.py` gains Wikipedia sources alongside SearXNG. All `search_web` references across the codebase are updated.
|
||||
|
||||
**Tech Stack:** Python 3.12, httpx, pytest, asyncio
|
||||
|
||||
---
|
||||
|
||||
### Task 1: Wikipedia Service Module
|
||||
|
||||
**Files:**
|
||||
- Create: `src/fabledassistant/services/wikipedia.py`
|
||||
- Create: `tests/test_wikipedia.py`
|
||||
|
||||
- [ ] **Step 1: Write failing tests for `wiki_summary`**
|
||||
|
||||
```python
|
||||
# tests/test_wikipedia.py
|
||||
import pytest
|
||||
from unittest.mock import AsyncMock, patch, MagicMock
|
||||
import httpx
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_wiki_summary_returns_extract():
|
||||
from fabledassistant.services.wikipedia import wiki_summary
|
||||
|
||||
mock_response = MagicMock()
|
||||
mock_response.status_code = 200
|
||||
mock_response.json.return_value = {
|
||||
"type": "standard",
|
||||
"title": "Python (programming language)",
|
||||
"extract": "Python is a high-level programming language.",
|
||||
"content_urls": {
|
||||
"desktop": {"page": "https://en.wikipedia.org/wiki/Python_(programming_language)"}
|
||||
},
|
||||
}
|
||||
mock_response.raise_for_status = MagicMock()
|
||||
|
||||
with patch("fabledassistant.services.wikipedia.httpx.AsyncClient") as mock_client_cls:
|
||||
mock_client = AsyncMock()
|
||||
mock_client.__aenter__ = AsyncMock(return_value=mock_client)
|
||||
mock_client.__aexit__ = AsyncMock(return_value=False)
|
||||
mock_client.get = AsyncMock(return_value=mock_response)
|
||||
mock_client_cls.return_value = mock_client
|
||||
|
||||
result = await wiki_summary("Python programming language")
|
||||
|
||||
assert result is not None
|
||||
assert result["title"] == "Python (programming language)"
|
||||
assert "high-level" in result["extract"]
|
||||
assert "wikipedia.org" in result["url"]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_wiki_summary_returns_none_on_404():
|
||||
from fabledassistant.services.wikipedia import wiki_summary
|
||||
|
||||
with patch("fabledassistant.services.wikipedia.httpx.AsyncClient") as mock_client_cls:
|
||||
mock_client = AsyncMock()
|
||||
mock_client.__aenter__ = AsyncMock(return_value=mock_client)
|
||||
mock_client.__aexit__ = AsyncMock(return_value=False)
|
||||
mock_client.get = AsyncMock(side_effect=httpx.HTTPStatusError(
|
||||
"Not Found", request=MagicMock(), response=MagicMock(status_code=404)
|
||||
))
|
||||
mock_client_cls.return_value = mock_client
|
||||
|
||||
result = await wiki_summary("xyznonexistenttopic123")
|
||||
|
||||
assert result is None
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_wiki_summary_returns_none_on_disambiguation():
|
||||
from fabledassistant.services.wikipedia import wiki_summary
|
||||
|
||||
mock_response = MagicMock()
|
||||
mock_response.status_code = 200
|
||||
mock_response.json.return_value = {
|
||||
"type": "disambiguation",
|
||||
"title": "Python",
|
||||
"extract": "Python may refer to...",
|
||||
}
|
||||
mock_response.raise_for_status = MagicMock()
|
||||
|
||||
with patch("fabledassistant.services.wikipedia.httpx.AsyncClient") as mock_client_cls:
|
||||
mock_client = AsyncMock()
|
||||
mock_client.__aenter__ = AsyncMock(return_value=mock_client)
|
||||
mock_client.__aexit__ = AsyncMock(return_value=False)
|
||||
mock_client.get = AsyncMock(return_value=mock_response)
|
||||
mock_client_cls.return_value = mock_client
|
||||
|
||||
result = await wiki_summary("Python")
|
||||
|
||||
assert result is None
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run tests to verify they fail**
|
||||
|
||||
Run: `uv run pytest tests/test_wikipedia.py -v`
|
||||
Expected: FAIL with `ModuleNotFoundError: No module named 'fabledassistant.services.wikipedia'`
|
||||
|
||||
- [ ] **Step 3: Implement `wiki_summary`**
|
||||
|
||||
```python
|
||||
# src/fabledassistant/services/wikipedia.py
|
||||
"""Wikipedia API: lightweight topic lookups and article search."""
|
||||
|
||||
import logging
|
||||
from urllib.parse import quote as url_quote
|
||||
|
||||
import httpx
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
_SUMMARY_URL = "https://en.wikipedia.org/api/rest_v1/page/summary"
|
||||
_SEARCH_URL = "https://en.wikipedia.org/w/api.php"
|
||||
_TIMEOUT = 5.0
|
||||
_USER_AGENT = "FabledAssistant/1.0 (https://fabledsword.com)"
|
||||
|
||||
|
||||
async def wiki_summary(query: str) -> dict | None:
|
||||
"""Look up a topic by title via the Wikipedia REST summary endpoint.
|
||||
|
||||
Returns {"title", "extract", "url"} on hit, None on miss.
|
||||
"""
|
||||
encoded = url_quote(query.replace(" ", "_"), safe="")
|
||||
try:
|
||||
async with httpx.AsyncClient(
|
||||
timeout=_TIMEOUT, headers={"User-Agent": _USER_AGENT}
|
||||
) as client:
|
||||
resp = await client.get(f"{_SUMMARY_URL}/{encoded}", follow_redirects=True)
|
||||
resp.raise_for_status()
|
||||
data = resp.json()
|
||||
except Exception:
|
||||
logger.debug("Wikipedia summary lookup failed for %r", query, exc_info=True)
|
||||
return None
|
||||
|
||||
if data.get("type") == "disambiguation":
|
||||
return None
|
||||
|
||||
extract = data.get("extract", "").strip()
|
||||
if not extract:
|
||||
return None
|
||||
|
||||
url = (
|
||||
data.get("content_urls", {}).get("desktop", {}).get("page")
|
||||
or f"https://en.wikipedia.org/wiki/{encoded}"
|
||||
)
|
||||
return {"title": data.get("title", query), "extract": extract, "url": url}
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Run tests to verify they pass**
|
||||
|
||||
Run: `uv run pytest tests/test_wikipedia.py -v`
|
||||
Expected: 3 passed
|
||||
|
||||
- [ ] **Step 5: Write failing tests for `wiki_search`**
|
||||
|
||||
Add to `tests/test_wikipedia.py`:
|
||||
|
||||
```python
|
||||
@pytest.mark.asyncio
|
||||
async def test_wiki_search_returns_results():
|
||||
from fabledassistant.services.wikipedia import wiki_search
|
||||
|
||||
search_response = MagicMock()
|
||||
search_response.status_code = 200
|
||||
search_response.json.return_value = {
|
||||
"query": {
|
||||
"search": [
|
||||
{"title": "QUIC"},
|
||||
{"title": "HTTP/3"},
|
||||
]
|
||||
}
|
||||
}
|
||||
search_response.raise_for_status = MagicMock()
|
||||
|
||||
summary_response = MagicMock()
|
||||
summary_response.status_code = 200
|
||||
summary_response.json.return_value = {
|
||||
"type": "standard",
|
||||
"title": "QUIC",
|
||||
"extract": "QUIC is a transport layer protocol.",
|
||||
"content_urls": {"desktop": {"page": "https://en.wikipedia.org/wiki/QUIC"}},
|
||||
}
|
||||
summary_response.raise_for_status = MagicMock()
|
||||
|
||||
with patch("fabledassistant.services.wikipedia.httpx.AsyncClient") as mock_client_cls:
|
||||
mock_client = AsyncMock()
|
||||
mock_client.__aenter__ = AsyncMock(return_value=mock_client)
|
||||
mock_client.__aexit__ = AsyncMock(return_value=False)
|
||||
mock_client.get = AsyncMock(side_effect=[search_response, summary_response, summary_response])
|
||||
mock_client_cls.return_value = mock_client
|
||||
|
||||
results = await wiki_search("QUIC protocol", limit=2)
|
||||
|
||||
assert len(results) >= 1
|
||||
assert results[0]["title"] == "QUIC"
|
||||
assert "transport" in results[0]["extract"]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_wiki_search_returns_empty_on_failure():
|
||||
from fabledassistant.services.wikipedia import wiki_search
|
||||
|
||||
with patch("fabledassistant.services.wikipedia.httpx.AsyncClient") as mock_client_cls:
|
||||
mock_client = AsyncMock()
|
||||
mock_client.__aenter__ = AsyncMock(return_value=mock_client)
|
||||
mock_client.__aexit__ = AsyncMock(return_value=False)
|
||||
mock_client.get = AsyncMock(side_effect=httpx.ConnectError("connection failed"))
|
||||
mock_client_cls.return_value = mock_client
|
||||
|
||||
results = await wiki_search("anything")
|
||||
|
||||
assert results == []
|
||||
```
|
||||
|
||||
- [ ] **Step 6: Implement `wiki_search`**
|
||||
|
||||
Add to `src/fabledassistant/services/wikipedia.py`:
|
||||
|
||||
```python
|
||||
async def wiki_search(query: str, limit: int = 3) -> list[dict]:
|
||||
"""Search Wikipedia for articles matching a query.
|
||||
|
||||
Returns [{"title", "extract", "url"}, ...] (may be empty).
|
||||
"""
|
||||
try:
|
||||
async with httpx.AsyncClient(
|
||||
timeout=_TIMEOUT, headers={"User-Agent": _USER_AGENT}
|
||||
) as client:
|
||||
resp = await client.get(_SEARCH_URL, params={
|
||||
"action": "query",
|
||||
"list": "search",
|
||||
"srsearch": query,
|
||||
"srlimit": str(limit),
|
||||
"format": "json",
|
||||
})
|
||||
resp.raise_for_status()
|
||||
hits = resp.json().get("query", {}).get("search", [])
|
||||
if not hits:
|
||||
return []
|
||||
|
||||
results: list[dict] = []
|
||||
for hit in hits:
|
||||
title = hit.get("title", "")
|
||||
if not title:
|
||||
continue
|
||||
encoded = url_quote(title.replace(" ", "_"), safe="")
|
||||
try:
|
||||
summary_resp = await client.get(
|
||||
f"{_SUMMARY_URL}/{encoded}", follow_redirects=True,
|
||||
)
|
||||
summary_resp.raise_for_status()
|
||||
data = summary_resp.json()
|
||||
except Exception:
|
||||
continue
|
||||
if data.get("type") == "disambiguation":
|
||||
continue
|
||||
extract = data.get("extract", "").strip()
|
||||
if not extract:
|
||||
continue
|
||||
url = (
|
||||
data.get("content_urls", {}).get("desktop", {}).get("page")
|
||||
or f"https://en.wikipedia.org/wiki/{encoded}"
|
||||
)
|
||||
results.append({"title": data.get("title", title), "extract": extract, "url": url})
|
||||
return results
|
||||
except Exception:
|
||||
logger.debug("Wikipedia search failed for %r", query, exc_info=True)
|
||||
return []
|
||||
```
|
||||
|
||||
- [ ] **Step 7: Run all wikipedia tests**
|
||||
|
||||
Run: `uv run pytest tests/test_wikipedia.py -v`
|
||||
Expected: 5 passed
|
||||
|
||||
- [ ] **Step 8: Commit**
|
||||
|
||||
```bash
|
||||
git add src/fabledassistant/services/wikipedia.py tests/test_wikipedia.py
|
||||
git commit -m "feat: add wikipedia service with summary lookup and search"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 2: Lookup Tool (replaces search_web)
|
||||
|
||||
**Files:**
|
||||
- Modify: `src/fabledassistant/services/tools/web.py`
|
||||
- Create: `tests/test_lookup_tool.py`
|
||||
|
||||
- [ ] **Step 1: Write failing tests for `lookup`**
|
||||
|
||||
```python
|
||||
# tests/test_lookup_tool.py
|
||||
import pytest
|
||||
from unittest.mock import AsyncMock, patch, MagicMock
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_lookup_wikipedia_hit():
|
||||
"""lookup returns wikipedia source when wiki_summary succeeds."""
|
||||
wiki_data = {
|
||||
"title": "QUIC",
|
||||
"extract": "QUIC is a transport layer protocol.",
|
||||
"url": "https://en.wikipedia.org/wiki/QUIC",
|
||||
}
|
||||
|
||||
with patch("fabledassistant.services.tools.web.wiki_summary", new_callable=AsyncMock, return_value=wiki_data):
|
||||
from fabledassistant.services.tools.web import lookup_tool
|
||||
result = await lookup_tool(user_id=1, arguments={"query": "QUIC"})
|
||||
|
||||
assert result["success"] is True
|
||||
assert result["type"] == "lookup"
|
||||
assert result["source"] == "wikipedia"
|
||||
assert result["data"]["title"] == "QUIC"
|
||||
assert "transport" in result["data"]["extract"]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_lookup_wikipedia_miss_searxng_fallback():
|
||||
"""lookup falls back to SearXNG + article fetch when Wikipedia misses."""
|
||||
searxng_results = [
|
||||
{"url": "https://example.com/quic", "title": "QUIC Explained", "snippet": "An overview..."},
|
||||
]
|
||||
|
||||
with patch("fabledassistant.services.tools.web.wiki_summary", new_callable=AsyncMock, return_value=None), \
|
||||
patch("fabledassistant.services.tools.web.Config") as mock_config, \
|
||||
patch("fabledassistant.services.tools.web._search_searxng", new_callable=AsyncMock, return_value=searxng_results), \
|
||||
patch("fabledassistant.services.tools.web._fetch_full_article", new_callable=AsyncMock, return_value="Full article about QUIC..."):
|
||||
mock_config.searxng_enabled.return_value = True
|
||||
from fabledassistant.services.tools.web import lookup_tool
|
||||
result = await lookup_tool(user_id=1, arguments={"query": "QUIC"})
|
||||
|
||||
assert result["success"] is True
|
||||
assert result["type"] == "lookup"
|
||||
assert result["source"] == "web"
|
||||
assert result["data"]["results"]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_lookup_wikipedia_miss_no_searxng():
|
||||
"""lookup returns no-results when Wikipedia misses and SearXNG is not configured."""
|
||||
with patch("fabledassistant.services.tools.web.wiki_summary", new_callable=AsyncMock, return_value=None), \
|
||||
patch("fabledassistant.services.tools.web.Config") as mock_config:
|
||||
mock_config.searxng_enabled.return_value = False
|
||||
from fabledassistant.services.tools.web import lookup_tool
|
||||
result = await lookup_tool(user_id=1, arguments={"query": "xyznonexistent"})
|
||||
|
||||
assert result["success"] is True
|
||||
assert result["source"] == "none"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_lookup_always_available():
|
||||
"""lookup tool must appear in get_tools_for_user regardless of SearXNG config."""
|
||||
with patch("fabledassistant.services.tools._registry.is_caldav_configured", new_callable=AsyncMock, return_value=False), \
|
||||
patch("fabledassistant.services.settings.get_setting", new_callable=AsyncMock, return_value="false"):
|
||||
from fabledassistant.services.tools import get_tools_for_user
|
||||
tools = await get_tools_for_user(user_id=1)
|
||||
tool_names = {t["function"]["name"] for t in tools}
|
||||
assert "lookup" in tool_names
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run tests to verify they fail**
|
||||
|
||||
Run: `uv run pytest tests/test_lookup_tool.py -v`
|
||||
Expected: FAIL (no `lookup_tool` function)
|
||||
|
||||
- [ ] **Step 3: Replace `search_web` with `lookup` in `web.py`**
|
||||
|
||||
Replace the `search_web_tool` function (lines 12–36 of `src/fabledassistant/services/tools/web.py`) with:
|
||||
|
||||
```python
|
||||
@tool(
|
||||
name="lookup",
|
||||
description=(
|
||||
"Look up a topic, concept, or factual question. Returns a concise answer from "
|
||||
"Wikipedia or web sources. Use for definitions, explanations, 'what is X', "
|
||||
"'how does Y work', current events, or version numbers. No note is saved. "
|
||||
"For comprehensive written reports saved as notes, use research_topic instead."
|
||||
),
|
||||
parameters={
|
||||
"query": {"type": "string", "description": "The topic or question to look up"},
|
||||
},
|
||||
required=["query"],
|
||||
)
|
||||
async def lookup_tool(*, user_id, arguments, **_ctx):
|
||||
from fabledassistant.config import Config
|
||||
from fabledassistant.services.wikipedia import wiki_summary
|
||||
|
||||
query = arguments.get("query", "")
|
||||
|
||||
# 1. Try Wikipedia first
|
||||
wiki = await wiki_summary(query)
|
||||
if wiki:
|
||||
return {
|
||||
"success": True,
|
||||
"type": "lookup",
|
||||
"source": "wikipedia",
|
||||
"data": wiki,
|
||||
}
|
||||
|
||||
# 2. Fall back to SearXNG + article fetch
|
||||
if Config.searxng_enabled():
|
||||
from fabledassistant.services.research import _search_searxng
|
||||
from fabledassistant.services.rss import _fetch_full_article
|
||||
|
||||
results = await _search_searxng(query)
|
||||
if results:
|
||||
articles: list[dict] = []
|
||||
for r in results[:2]:
|
||||
url = r.get("url", "")
|
||||
if not url:
|
||||
continue
|
||||
content = await _fetch_full_article(url)
|
||||
articles.append({
|
||||
"url": url,
|
||||
"title": r.get("title", url),
|
||||
"snippet": r.get("snippet", ""),
|
||||
"content": (content or "")[:4000],
|
||||
})
|
||||
if articles:
|
||||
return {
|
||||
"success": True,
|
||||
"type": "lookup",
|
||||
"source": "web",
|
||||
"data": {"query": query, "results": articles},
|
||||
}
|
||||
|
||||
# 3. No sources available
|
||||
return {
|
||||
"success": True,
|
||||
"type": "lookup",
|
||||
"source": "none",
|
||||
"data": {
|
||||
"query": query,
|
||||
"message": "No results found. You can answer from your own knowledge.",
|
||||
},
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Run lookup tests to verify they pass**
|
||||
|
||||
Run: `uv run pytest tests/test_lookup_tool.py -v`
|
||||
Expected: 4 passed
|
||||
|
||||
- [ ] **Step 5: Commit**
|
||||
|
||||
```bash
|
||||
git add src/fabledassistant/services/tools/web.py tests/test_lookup_tool.py
|
||||
git commit -m "feat: replace search_web with unified lookup tool (Wikipedia + SearXNG fallback)"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 3: Update All `search_web` References
|
||||
|
||||
**Files:**
|
||||
- Modify: `src/fabledassistant/services/tools/web.py:45` (research_topic description)
|
||||
- Modify: `src/fabledassistant/services/tools/web.py:67` (search_images description)
|
||||
- Modify: `src/fabledassistant/services/tools/rss.py:75` (read_article description)
|
||||
- Modify: `src/fabledassistant/services/generation_task.py:133` (status label map)
|
||||
- Modify: `src/fabledassistant/services/llm.py:608` (action list)
|
||||
|
||||
- [ ] **Step 1: Update `research_topic` description**
|
||||
|
||||
In `src/fabledassistant/services/tools/web.py`, change the `research_topic` description from:
|
||||
|
||||
```python
|
||||
"For a quick factual answer without saving a note, use search_web."
|
||||
```
|
||||
|
||||
to:
|
||||
|
||||
```python
|
||||
"For a quick factual answer without saving a note, use lookup."
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Update `search_images` description**
|
||||
|
||||
In `src/fabledassistant/services/tools/web.py`, change the `search_images` description from:
|
||||
|
||||
```python
|
||||
description="Search and display images inline. Use ONLY when the user explicitly asks to see, show, or find an image or photo. Not for factual questions — use search_web for those.",
|
||||
```
|
||||
|
||||
to:
|
||||
|
||||
```python
|
||||
description="Search and display images inline. Use ONLY when the user explicitly asks to see, show, or find an image or photo. Not for factual questions — use lookup for those.",
|
||||
```
|
||||
|
||||
- [ ] **Step 3: Update `read_article` description**
|
||||
|
||||
In `src/fabledassistant/services/tools/rss.py`, change:
|
||||
|
||||
```python
|
||||
"Do NOT use search_web for URLs — use this tool instead."
|
||||
```
|
||||
|
||||
to:
|
||||
|
||||
```python
|
||||
"Do NOT use lookup for URLs — use this tool instead."
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Update status label map in `generation_task.py`**
|
||||
|
||||
In `src/fabledassistant/services/generation_task.py`, line 133, change:
|
||||
|
||||
```python
|
||||
"search_web": "Searching the web",
|
||||
```
|
||||
|
||||
to:
|
||||
|
||||
```python
|
||||
"lookup": "Looking up information",
|
||||
```
|
||||
|
||||
- [ ] **Step 5: Update action list in `llm.py`**
|
||||
|
||||
In `src/fabledassistant/services/llm.py`, line 608, change:
|
||||
|
||||
```python
|
||||
actions.extend(["search_web", "research_topic", "search_images"])
|
||||
```
|
||||
|
||||
to:
|
||||
|
||||
```python
|
||||
actions.extend(["lookup", "research_topic", "search_images"])
|
||||
```
|
||||
|
||||
- [ ] **Step 6: Run full test suite to check for regressions**
|
||||
|
||||
Run: `uv run pytest tests/ -v`
|
||||
Expected: All tests pass (no test references `search_web` by name in assertions)
|
||||
|
||||
- [ ] **Step 7: Commit**
|
||||
|
||||
```bash
|
||||
git add src/fabledassistant/services/tools/web.py src/fabledassistant/services/tools/rss.py src/fabledassistant/services/generation_task.py src/fabledassistant/services/llm.py
|
||||
git commit -m "refactor: update all search_web references to lookup"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 4: Add Wikipedia Sources to Research Pipeline
|
||||
|
||||
**Files:**
|
||||
- Modify: `src/fabledassistant/services/research.py`
|
||||
- Modify: `tests/test_research_pipeline.py`
|
||||
|
||||
- [ ] **Step 1: Write failing test for Wikipedia in research pipeline**
|
||||
|
||||
Add to `tests/test_research_pipeline.py`:
|
||||
|
||||
```python
|
||||
@pytest.mark.asyncio
|
||||
async def test_pipeline_includes_wikipedia_sources():
|
||||
"""run_research_pipeline should merge Wikipedia results into the source pool."""
|
||||
from unittest.mock import MagicMock
|
||||
|
||||
wiki_results = [{"title": "Wiki Article", "extract": "Wikipedia content about the topic.", "url": "https://en.wikipedia.org/wiki/Topic"}]
|
||||
|
||||
outline = [
|
||||
{"title": "Section A", "focus": "Focus A"},
|
||||
{"title": "Section B", "focus": "Focus B"},
|
||||
]
|
||||
|
||||
note_id_counter = iter(range(30, 40))
|
||||
|
||||
def _make_note(user_id, title, body, tags, project_id=None, parent_id=None):
|
||||
n = MagicMock()
|
||||
n.id = next(note_id_counter)
|
||||
n.title = title
|
||||
return n
|
||||
|
||||
with patch("fabledassistant.services.research._generate_sub_queries", new_callable=AsyncMock, return_value=["q1"]), \
|
||||
patch("fabledassistant.services.research._search_searxng", new_callable=AsyncMock, return_value=[{"url": "http://x.com", "title": "X", "snippet": "s"}]), \
|
||||
patch("fabledassistant.services.research.wiki_search", new_callable=AsyncMock, return_value=wiki_results), \
|
||||
patch("fabledassistant.services.research.fetch_url_content", new_callable=AsyncMock, return_value="content"), \
|
||||
patch("fabledassistant.services.research._generate_outline", new_callable=AsyncMock, return_value=outline) as mock_outline, \
|
||||
patch("fabledassistant.services.research._synthesize_section", new_callable=AsyncMock, side_effect=lambda t, f, s, m: (t, f"Body for {t}")), \
|
||||
patch("fabledassistant.services.research._generate_executive_summary", new_callable=AsyncMock, return_value="Summary."), \
|
||||
patch("fabledassistant.services.research.create_note", new_callable=AsyncMock, side_effect=_make_note), \
|
||||
patch("fabledassistant.services.research.update_note", new_callable=AsyncMock):
|
||||
|
||||
from fabledassistant.services.research import run_research_pipeline
|
||||
await run_research_pipeline("test topic", user_id=1, model="test-model")
|
||||
|
||||
# The sources passed to _generate_outline should include the Wikipedia article
|
||||
sources_arg = mock_outline.call_args[0][1] # second positional arg
|
||||
source_urls = [s["url"] for s in sources_arg]
|
||||
assert "https://en.wikipedia.org/wiki/Topic" in source_urls
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run test to verify it fails**
|
||||
|
||||
Run: `uv run pytest tests/test_research_pipeline.py::test_pipeline_includes_wikipedia_sources -v`
|
||||
Expected: FAIL (no `wiki_search` import in research.py)
|
||||
|
||||
- [ ] **Step 3: Add Wikipedia sources to the research pipeline**
|
||||
|
||||
In `src/fabledassistant/services/research.py`, add the import at the top (after existing imports):
|
||||
|
||||
```python
|
||||
from fabledassistant.services.wikipedia import wiki_search
|
||||
```
|
||||
|
||||
Then modify Step 2 (the parallel search section, around lines 208–246). Replace:
|
||||
|
||||
```python
|
||||
# Step 2: Search all queries in parallel (200 ms stagger to avoid hammering SearXNG)
|
||||
async def _search_with_stagger(i: int, query: str) -> tuple[str, list[dict]]:
|
||||
if i > 0:
|
||||
await asyncio.sleep(0.2 * i)
|
||||
_status(f"Searching: {query}...")
|
||||
results = await _search_searxng(query)
|
||||
logger.info("Research: query '%s' → %d results", query, len(results))
|
||||
return query, results
|
||||
|
||||
search_results = await asyncio.gather(
|
||||
*[_search_with_stagger(i, q) for i, q in enumerate(queries)]
|
||||
)
|
||||
|
||||
# Deduplicate URLs across all queries
|
||||
seen_urls: set[str] = set()
|
||||
url_tasks: list[tuple[str, dict, str]] = [] # (url, result_dict, query)
|
||||
for query, results in search_results:
|
||||
for result in results[:PAGES_PER_QUERY]:
|
||||
url = result.get("url", "")
|
||||
if url and url not in seen_urls:
|
||||
seen_urls.add(url)
|
||||
url_tasks.append((url, result, query))
|
||||
|
||||
# Fetch all unique URLs in parallel
|
||||
async def _fetch_source(url: str, result: dict, query: str) -> dict:
|
||||
title = result.get("title", url)
|
||||
_status(f"Reading: {title[:60]}...")
|
||||
content = await fetch_url_content(url)
|
||||
return {
|
||||
"url": url,
|
||||
"title": title,
|
||||
"query": query,
|
||||
"snippet": result.get("snippet", ""),
|
||||
"content": content,
|
||||
}
|
||||
|
||||
all_sources: list[dict] = list(await asyncio.gather(
|
||||
*[_fetch_source(url, result, query) for url, result, query in url_tasks]
|
||||
))
|
||||
```
|
||||
|
||||
with:
|
||||
|
||||
```python
|
||||
# Step 2: Search all queries in parallel (SearXNG + Wikipedia)
|
||||
async def _search_with_stagger(i: int, query: str) -> tuple[str, list[dict]]:
|
||||
if i > 0:
|
||||
await asyncio.sleep(0.2 * i)
|
||||
_status(f"Searching: {query}...")
|
||||
results = await _search_searxng(query)
|
||||
logger.info("Research: query '%s' → %d results", query, len(results))
|
||||
return query, results
|
||||
|
||||
async def _wiki_for_query(query: str) -> list[dict]:
|
||||
return await wiki_search(query, limit=1)
|
||||
|
||||
searxng_task = asyncio.gather(
|
||||
*[_search_with_stagger(i, q) for i, q in enumerate(queries)]
|
||||
)
|
||||
wiki_task = asyncio.gather(
|
||||
*[_wiki_for_query(q) for q in queries]
|
||||
)
|
||||
search_results, wiki_results = await asyncio.gather(searxng_task, wiki_task)
|
||||
|
||||
# Deduplicate URLs across all queries
|
||||
seen_urls: set[str] = set()
|
||||
url_tasks: list[tuple[str, dict, str]] = [] # (url, result_dict, query)
|
||||
wiki_sources: list[dict] = [] # Wikipedia articles (already have content)
|
||||
|
||||
for query, results in search_results:
|
||||
for result in results[:PAGES_PER_QUERY]:
|
||||
url = result.get("url", "")
|
||||
if url and url not in seen_urls:
|
||||
seen_urls.add(url)
|
||||
url_tasks.append((url, result, query))
|
||||
|
||||
# Add Wikipedia results (they already have content via extract)
|
||||
for query, wiki_hits in zip(queries, wiki_results):
|
||||
for hit in wiki_hits:
|
||||
url = hit.get("url", "")
|
||||
if url and url not in seen_urls:
|
||||
seen_urls.add(url)
|
||||
wiki_sources.append({
|
||||
"url": url,
|
||||
"title": hit["title"],
|
||||
"query": query,
|
||||
"snippet": hit["extract"][:200],
|
||||
"content": hit["extract"],
|
||||
})
|
||||
|
||||
# Fetch all unique SearXNG URLs in parallel
|
||||
async def _fetch_source(url: str, result: dict, query: str) -> dict:
|
||||
title = result.get("title", url)
|
||||
_status(f"Reading: {title[:60]}...")
|
||||
content = await fetch_url_content(url)
|
||||
return {
|
||||
"url": url,
|
||||
"title": title,
|
||||
"query": query,
|
||||
"snippet": result.get("snippet", ""),
|
||||
"content": content,
|
||||
}
|
||||
|
||||
fetched_sources: list[dict] = list(await asyncio.gather(
|
||||
*[_fetch_source(url, result, query) for url, result, query in url_tasks]
|
||||
))
|
||||
|
||||
all_sources = wiki_sources + fetched_sources
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Run the new test to verify it passes**
|
||||
|
||||
Run: `uv run pytest tests/test_research_pipeline.py::test_pipeline_includes_wikipedia_sources -v`
|
||||
Expected: PASS
|
||||
|
||||
- [ ] **Step 5: Run the full research test suite for regressions**
|
||||
|
||||
Run: `uv run pytest tests/test_research_pipeline.py -v`
|
||||
Expected: All tests pass
|
||||
|
||||
- [ ] **Step 6: Run full test suite**
|
||||
|
||||
Run: `uv run pytest tests/ -v`
|
||||
Expected: All tests pass
|
||||
|
||||
- [ ] **Step 7: Commit**
|
||||
|
||||
```bash
|
||||
git add src/fabledassistant/services/research.py tests/test_research_pipeline.py
|
||||
git commit -m "feat: add Wikipedia as research pipeline source alongside SearXNG"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 5: Lint, Typecheck, Final Verification
|
||||
|
||||
**Files:**
|
||||
- All modified files
|
||||
|
||||
- [ ] **Step 1: Run ruff lint**
|
||||
|
||||
Run: `uv run ruff check src/fabledassistant/services/wikipedia.py src/fabledassistant/services/tools/web.py src/fabledassistant/services/research.py tests/test_wikipedia.py tests/test_lookup_tool.py`
|
||||
Expected: All checks passed (fix any issues if not)
|
||||
|
||||
- [ ] **Step 2: Run typecheck**
|
||||
|
||||
Run: `cd frontend && npx vue-tsc --noEmit` (frontend unchanged, but verify nothing broke)
|
||||
Expected: Clean
|
||||
|
||||
- [ ] **Step 3: Run full test suite one last time**
|
||||
|
||||
Run: `uv run pytest tests/ -v`
|
||||
Expected: All tests pass
|
||||
|
||||
- [ ] **Step 4: Commit any lint fixes if needed**
|
||||
|
||||
```bash
|
||||
git add -u
|
||||
git commit -m "fix: lint cleanup for lookup/wikipedia changes"
|
||||
```
|
||||
Reference in New Issue
Block a user