6f8c2201ed
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
783 lines
27 KiB
Markdown
783 lines
27 KiB
Markdown
# Unified Lookup Tool & Wikipedia Integration — Implementation Plan
|
||
|
||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||
|
||
**Goal:** Replace `search_web` with a unified `lookup` tool (Wikipedia-first, SearXNG fallback) and add Wikipedia as a source in the research pipeline.
|
||
|
||
**Architecture:** New `wikipedia.py` service with `wiki_summary` and `wiki_search`. `lookup` tool in `web.py` replaces `search_web`. Research pipeline in `research.py` gains Wikipedia sources alongside SearXNG. All `search_web` references across the codebase are updated.
|
||
|
||
**Tech Stack:** Python 3.12, httpx, pytest, asyncio
|
||
|
||
---
|
||
|
||
### Task 1: Wikipedia Service Module
|
||
|
||
**Files:**
|
||
- Create: `src/fabledassistant/services/wikipedia.py`
|
||
- Create: `tests/test_wikipedia.py`
|
||
|
||
- [ ] **Step 1: Write failing tests for `wiki_summary`**
|
||
|
||
```python
|
||
# tests/test_wikipedia.py
|
||
import pytest
|
||
from unittest.mock import AsyncMock, patch, MagicMock
|
||
import httpx
|
||
|
||
|
||
@pytest.mark.asyncio
|
||
async def test_wiki_summary_returns_extract():
|
||
from fabledassistant.services.wikipedia import wiki_summary
|
||
|
||
mock_response = MagicMock()
|
||
mock_response.status_code = 200
|
||
mock_response.json.return_value = {
|
||
"type": "standard",
|
||
"title": "Python (programming language)",
|
||
"extract": "Python is a high-level programming language.",
|
||
"content_urls": {
|
||
"desktop": {"page": "https://en.wikipedia.org/wiki/Python_(programming_language)"}
|
||
},
|
||
}
|
||
mock_response.raise_for_status = MagicMock()
|
||
|
||
with patch("fabledassistant.services.wikipedia.httpx.AsyncClient") as mock_client_cls:
|
||
mock_client = AsyncMock()
|
||
mock_client.__aenter__ = AsyncMock(return_value=mock_client)
|
||
mock_client.__aexit__ = AsyncMock(return_value=False)
|
||
mock_client.get = AsyncMock(return_value=mock_response)
|
||
mock_client_cls.return_value = mock_client
|
||
|
||
result = await wiki_summary("Python programming language")
|
||
|
||
assert result is not None
|
||
assert result["title"] == "Python (programming language)"
|
||
assert "high-level" in result["extract"]
|
||
assert "wikipedia.org" in result["url"]
|
||
|
||
|
||
@pytest.mark.asyncio
|
||
async def test_wiki_summary_returns_none_on_404():
|
||
from fabledassistant.services.wikipedia import wiki_summary
|
||
|
||
with patch("fabledassistant.services.wikipedia.httpx.AsyncClient") as mock_client_cls:
|
||
mock_client = AsyncMock()
|
||
mock_client.__aenter__ = AsyncMock(return_value=mock_client)
|
||
mock_client.__aexit__ = AsyncMock(return_value=False)
|
||
mock_client.get = AsyncMock(side_effect=httpx.HTTPStatusError(
|
||
"Not Found", request=MagicMock(), response=MagicMock(status_code=404)
|
||
))
|
||
mock_client_cls.return_value = mock_client
|
||
|
||
result = await wiki_summary("xyznonexistenttopic123")
|
||
|
||
assert result is None
|
||
|
||
|
||
@pytest.mark.asyncio
|
||
async def test_wiki_summary_returns_none_on_disambiguation():
|
||
from fabledassistant.services.wikipedia import wiki_summary
|
||
|
||
mock_response = MagicMock()
|
||
mock_response.status_code = 200
|
||
mock_response.json.return_value = {
|
||
"type": "disambiguation",
|
||
"title": "Python",
|
||
"extract": "Python may refer to...",
|
||
}
|
||
mock_response.raise_for_status = MagicMock()
|
||
|
||
with patch("fabledassistant.services.wikipedia.httpx.AsyncClient") as mock_client_cls:
|
||
mock_client = AsyncMock()
|
||
mock_client.__aenter__ = AsyncMock(return_value=mock_client)
|
||
mock_client.__aexit__ = AsyncMock(return_value=False)
|
||
mock_client.get = AsyncMock(return_value=mock_response)
|
||
mock_client_cls.return_value = mock_client
|
||
|
||
result = await wiki_summary("Python")
|
||
|
||
assert result is None
|
||
```
|
||
|
||
- [ ] **Step 2: Run tests to verify they fail**
|
||
|
||
Run: `uv run pytest tests/test_wikipedia.py -v`
|
||
Expected: FAIL with `ModuleNotFoundError: No module named 'fabledassistant.services.wikipedia'`
|
||
|
||
- [ ] **Step 3: Implement `wiki_summary`**
|
||
|
||
```python
|
||
# src/fabledassistant/services/wikipedia.py
|
||
"""Wikipedia API: lightweight topic lookups and article search."""
|
||
|
||
import logging
|
||
from urllib.parse import quote as url_quote
|
||
|
||
import httpx
|
||
|
||
logger = logging.getLogger(__name__)
|
||
|
||
_SUMMARY_URL = "https://en.wikipedia.org/api/rest_v1/page/summary"
|
||
_SEARCH_URL = "https://en.wikipedia.org/w/api.php"
|
||
_TIMEOUT = 5.0
|
||
_USER_AGENT = "FabledAssistant/1.0 (https://fabledsword.com)"
|
||
|
||
|
||
async def wiki_summary(query: str) -> dict | None:
|
||
"""Look up a topic by title via the Wikipedia REST summary endpoint.
|
||
|
||
Returns {"title", "extract", "url"} on hit, None on miss.
|
||
"""
|
||
encoded = url_quote(query.replace(" ", "_"), safe="")
|
||
try:
|
||
async with httpx.AsyncClient(
|
||
timeout=_TIMEOUT, headers={"User-Agent": _USER_AGENT}
|
||
) as client:
|
||
resp = await client.get(f"{_SUMMARY_URL}/{encoded}", follow_redirects=True)
|
||
resp.raise_for_status()
|
||
data = resp.json()
|
||
except Exception:
|
||
logger.debug("Wikipedia summary lookup failed for %r", query, exc_info=True)
|
||
return None
|
||
|
||
if data.get("type") == "disambiguation":
|
||
return None
|
||
|
||
extract = data.get("extract", "").strip()
|
||
if not extract:
|
||
return None
|
||
|
||
url = (
|
||
data.get("content_urls", {}).get("desktop", {}).get("page")
|
||
or f"https://en.wikipedia.org/wiki/{encoded}"
|
||
)
|
||
return {"title": data.get("title", query), "extract": extract, "url": url}
|
||
```
|
||
|
||
- [ ] **Step 4: Run tests to verify they pass**
|
||
|
||
Run: `uv run pytest tests/test_wikipedia.py -v`
|
||
Expected: 3 passed
|
||
|
||
- [ ] **Step 5: Write failing tests for `wiki_search`**
|
||
|
||
Add to `tests/test_wikipedia.py`:
|
||
|
||
```python
|
||
@pytest.mark.asyncio
|
||
async def test_wiki_search_returns_results():
|
||
from fabledassistant.services.wikipedia import wiki_search
|
||
|
||
search_response = MagicMock()
|
||
search_response.status_code = 200
|
||
search_response.json.return_value = {
|
||
"query": {
|
||
"search": [
|
||
{"title": "QUIC"},
|
||
{"title": "HTTP/3"},
|
||
]
|
||
}
|
||
}
|
||
search_response.raise_for_status = MagicMock()
|
||
|
||
summary_response = MagicMock()
|
||
summary_response.status_code = 200
|
||
summary_response.json.return_value = {
|
||
"type": "standard",
|
||
"title": "QUIC",
|
||
"extract": "QUIC is a transport layer protocol.",
|
||
"content_urls": {"desktop": {"page": "https://en.wikipedia.org/wiki/QUIC"}},
|
||
}
|
||
summary_response.raise_for_status = MagicMock()
|
||
|
||
with patch("fabledassistant.services.wikipedia.httpx.AsyncClient") as mock_client_cls:
|
||
mock_client = AsyncMock()
|
||
mock_client.__aenter__ = AsyncMock(return_value=mock_client)
|
||
mock_client.__aexit__ = AsyncMock(return_value=False)
|
||
mock_client.get = AsyncMock(side_effect=[search_response, summary_response, summary_response])
|
||
mock_client_cls.return_value = mock_client
|
||
|
||
results = await wiki_search("QUIC protocol", limit=2)
|
||
|
||
assert len(results) >= 1
|
||
assert results[0]["title"] == "QUIC"
|
||
assert "transport" in results[0]["extract"]
|
||
|
||
|
||
@pytest.mark.asyncio
|
||
async def test_wiki_search_returns_empty_on_failure():
|
||
from fabledassistant.services.wikipedia import wiki_search
|
||
|
||
with patch("fabledassistant.services.wikipedia.httpx.AsyncClient") as mock_client_cls:
|
||
mock_client = AsyncMock()
|
||
mock_client.__aenter__ = AsyncMock(return_value=mock_client)
|
||
mock_client.__aexit__ = AsyncMock(return_value=False)
|
||
mock_client.get = AsyncMock(side_effect=httpx.ConnectError("connection failed"))
|
||
mock_client_cls.return_value = mock_client
|
||
|
||
results = await wiki_search("anything")
|
||
|
||
assert results == []
|
||
```
|
||
|
||
- [ ] **Step 6: Implement `wiki_search`**
|
||
|
||
Add to `src/fabledassistant/services/wikipedia.py`:
|
||
|
||
```python
|
||
async def wiki_search(query: str, limit: int = 3) -> list[dict]:
|
||
"""Search Wikipedia for articles matching a query.
|
||
|
||
Returns [{"title", "extract", "url"}, ...] (may be empty).
|
||
"""
|
||
try:
|
||
async with httpx.AsyncClient(
|
||
timeout=_TIMEOUT, headers={"User-Agent": _USER_AGENT}
|
||
) as client:
|
||
resp = await client.get(_SEARCH_URL, params={
|
||
"action": "query",
|
||
"list": "search",
|
||
"srsearch": query,
|
||
"srlimit": str(limit),
|
||
"format": "json",
|
||
})
|
||
resp.raise_for_status()
|
||
hits = resp.json().get("query", {}).get("search", [])
|
||
if not hits:
|
||
return []
|
||
|
||
results: list[dict] = []
|
||
for hit in hits:
|
||
title = hit.get("title", "")
|
||
if not title:
|
||
continue
|
||
encoded = url_quote(title.replace(" ", "_"), safe="")
|
||
try:
|
||
summary_resp = await client.get(
|
||
f"{_SUMMARY_URL}/{encoded}", follow_redirects=True,
|
||
)
|
||
summary_resp.raise_for_status()
|
||
data = summary_resp.json()
|
||
except Exception:
|
||
continue
|
||
if data.get("type") == "disambiguation":
|
||
continue
|
||
extract = data.get("extract", "").strip()
|
||
if not extract:
|
||
continue
|
||
url = (
|
||
data.get("content_urls", {}).get("desktop", {}).get("page")
|
||
or f"https://en.wikipedia.org/wiki/{encoded}"
|
||
)
|
||
results.append({"title": data.get("title", title), "extract": extract, "url": url})
|
||
return results
|
||
except Exception:
|
||
logger.debug("Wikipedia search failed for %r", query, exc_info=True)
|
||
return []
|
||
```
|
||
|
||
- [ ] **Step 7: Run all wikipedia tests**
|
||
|
||
Run: `uv run pytest tests/test_wikipedia.py -v`
|
||
Expected: 5 passed
|
||
|
||
- [ ] **Step 8: Commit**
|
||
|
||
```bash
|
||
git add src/fabledassistant/services/wikipedia.py tests/test_wikipedia.py
|
||
git commit -m "feat: add wikipedia service with summary lookup and search"
|
||
```
|
||
|
||
---
|
||
|
||
### Task 2: Lookup Tool (replaces search_web)
|
||
|
||
**Files:**
|
||
- Modify: `src/fabledassistant/services/tools/web.py`
|
||
- Create: `tests/test_lookup_tool.py`
|
||
|
||
- [ ] **Step 1: Write failing tests for `lookup`**
|
||
|
||
```python
|
||
# tests/test_lookup_tool.py
|
||
import pytest
|
||
from unittest.mock import AsyncMock, patch, MagicMock
|
||
|
||
|
||
@pytest.mark.asyncio
|
||
async def test_lookup_wikipedia_hit():
|
||
"""lookup returns wikipedia source when wiki_summary succeeds."""
|
||
wiki_data = {
|
||
"title": "QUIC",
|
||
"extract": "QUIC is a transport layer protocol.",
|
||
"url": "https://en.wikipedia.org/wiki/QUIC",
|
||
}
|
||
|
||
with patch("fabledassistant.services.tools.web.wiki_summary", new_callable=AsyncMock, return_value=wiki_data):
|
||
from fabledassistant.services.tools.web import lookup_tool
|
||
result = await lookup_tool(user_id=1, arguments={"query": "QUIC"})
|
||
|
||
assert result["success"] is True
|
||
assert result["type"] == "lookup"
|
||
assert result["source"] == "wikipedia"
|
||
assert result["data"]["title"] == "QUIC"
|
||
assert "transport" in result["data"]["extract"]
|
||
|
||
|
||
@pytest.mark.asyncio
|
||
async def test_lookup_wikipedia_miss_searxng_fallback():
|
||
"""lookup falls back to SearXNG + article fetch when Wikipedia misses."""
|
||
searxng_results = [
|
||
{"url": "https://example.com/quic", "title": "QUIC Explained", "snippet": "An overview..."},
|
||
]
|
||
|
||
with patch("fabledassistant.services.tools.web.wiki_summary", new_callable=AsyncMock, return_value=None), \
|
||
patch("fabledassistant.services.tools.web.Config") as mock_config, \
|
||
patch("fabledassistant.services.tools.web._search_searxng", new_callable=AsyncMock, return_value=searxng_results), \
|
||
patch("fabledassistant.services.tools.web._fetch_full_article", new_callable=AsyncMock, return_value="Full article about QUIC..."):
|
||
mock_config.searxng_enabled.return_value = True
|
||
from fabledassistant.services.tools.web import lookup_tool
|
||
result = await lookup_tool(user_id=1, arguments={"query": "QUIC"})
|
||
|
||
assert result["success"] is True
|
||
assert result["type"] == "lookup"
|
||
assert result["source"] == "web"
|
||
assert result["data"]["results"]
|
||
|
||
|
||
@pytest.mark.asyncio
|
||
async def test_lookup_wikipedia_miss_no_searxng():
|
||
"""lookup returns no-results when Wikipedia misses and SearXNG is not configured."""
|
||
with patch("fabledassistant.services.tools.web.wiki_summary", new_callable=AsyncMock, return_value=None), \
|
||
patch("fabledassistant.services.tools.web.Config") as mock_config:
|
||
mock_config.searxng_enabled.return_value = False
|
||
from fabledassistant.services.tools.web import lookup_tool
|
||
result = await lookup_tool(user_id=1, arguments={"query": "xyznonexistent"})
|
||
|
||
assert result["success"] is True
|
||
assert result["source"] == "none"
|
||
|
||
|
||
@pytest.mark.asyncio
|
||
async def test_lookup_always_available():
|
||
"""lookup tool must appear in get_tools_for_user regardless of SearXNG config."""
|
||
with patch("fabledassistant.services.tools._registry.is_caldav_configured", new_callable=AsyncMock, return_value=False), \
|
||
patch("fabledassistant.services.settings.get_setting", new_callable=AsyncMock, return_value="false"):
|
||
from fabledassistant.services.tools import get_tools_for_user
|
||
tools = await get_tools_for_user(user_id=1)
|
||
tool_names = {t["function"]["name"] for t in tools}
|
||
assert "lookup" in tool_names
|
||
```
|
||
|
||
- [ ] **Step 2: Run tests to verify they fail**
|
||
|
||
Run: `uv run pytest tests/test_lookup_tool.py -v`
|
||
Expected: FAIL (no `lookup_tool` function)
|
||
|
||
- [ ] **Step 3: Replace `search_web` with `lookup` in `web.py`**
|
||
|
||
Replace the `search_web_tool` function (lines 12–36 of `src/fabledassistant/services/tools/web.py`) with:
|
||
|
||
```python
|
||
@tool(
|
||
name="lookup",
|
||
description=(
|
||
"Look up a topic, concept, or factual question. Returns a concise answer from "
|
||
"Wikipedia or web sources. Use for definitions, explanations, 'what is X', "
|
||
"'how does Y work', current events, or version numbers. No note is saved. "
|
||
"For comprehensive written reports saved as notes, use research_topic instead."
|
||
),
|
||
parameters={
|
||
"query": {"type": "string", "description": "The topic or question to look up"},
|
||
},
|
||
required=["query"],
|
||
)
|
||
async def lookup_tool(*, user_id, arguments, **_ctx):
|
||
from fabledassistant.config import Config
|
||
from fabledassistant.services.wikipedia import wiki_summary
|
||
|
||
query = arguments.get("query", "")
|
||
|
||
# 1. Try Wikipedia first
|
||
wiki = await wiki_summary(query)
|
||
if wiki:
|
||
return {
|
||
"success": True,
|
||
"type": "lookup",
|
||
"source": "wikipedia",
|
||
"data": wiki,
|
||
}
|
||
|
||
# 2. Fall back to SearXNG + article fetch
|
||
if Config.searxng_enabled():
|
||
from fabledassistant.services.research import _search_searxng
|
||
from fabledassistant.services.rss import _fetch_full_article
|
||
|
||
results = await _search_searxng(query)
|
||
if results:
|
||
articles: list[dict] = []
|
||
for r in results[:2]:
|
||
url = r.get("url", "")
|
||
if not url:
|
||
continue
|
||
content = await _fetch_full_article(url)
|
||
articles.append({
|
||
"url": url,
|
||
"title": r.get("title", url),
|
||
"snippet": r.get("snippet", ""),
|
||
"content": (content or "")[:4000],
|
||
})
|
||
if articles:
|
||
return {
|
||
"success": True,
|
||
"type": "lookup",
|
||
"source": "web",
|
||
"data": {"query": query, "results": articles},
|
||
}
|
||
|
||
# 3. No sources available
|
||
return {
|
||
"success": True,
|
||
"type": "lookup",
|
||
"source": "none",
|
||
"data": {
|
||
"query": query,
|
||
"message": "No results found. You can answer from your own knowledge.",
|
||
},
|
||
}
|
||
```
|
||
|
||
- [ ] **Step 4: Run lookup tests to verify they pass**
|
||
|
||
Run: `uv run pytest tests/test_lookup_tool.py -v`
|
||
Expected: 4 passed
|
||
|
||
- [ ] **Step 5: Commit**
|
||
|
||
```bash
|
||
git add src/fabledassistant/services/tools/web.py tests/test_lookup_tool.py
|
||
git commit -m "feat: replace search_web with unified lookup tool (Wikipedia + SearXNG fallback)"
|
||
```
|
||
|
||
---
|
||
|
||
### Task 3: Update All `search_web` References
|
||
|
||
**Files:**
|
||
- Modify: `src/fabledassistant/services/tools/web.py:45` (research_topic description)
|
||
- Modify: `src/fabledassistant/services/tools/web.py:67` (search_images description)
|
||
- Modify: `src/fabledassistant/services/tools/rss.py:75` (read_article description)
|
||
- Modify: `src/fabledassistant/services/generation_task.py:133` (status label map)
|
||
- Modify: `src/fabledassistant/services/llm.py:608` (action list)
|
||
|
||
- [ ] **Step 1: Update `research_topic` description**
|
||
|
||
In `src/fabledassistant/services/tools/web.py`, change the `research_topic` description from:
|
||
|
||
```python
|
||
"For a quick factual answer without saving a note, use search_web."
|
||
```
|
||
|
||
to:
|
||
|
||
```python
|
||
"For a quick factual answer without saving a note, use lookup."
|
||
```
|
||
|
||
- [ ] **Step 2: Update `search_images` description**
|
||
|
||
In `src/fabledassistant/services/tools/web.py`, change the `search_images` description from:
|
||
|
||
```python
|
||
description="Search and display images inline. Use ONLY when the user explicitly asks to see, show, or find an image or photo. Not for factual questions — use search_web for those.",
|
||
```
|
||
|
||
to:
|
||
|
||
```python
|
||
description="Search and display images inline. Use ONLY when the user explicitly asks to see, show, or find an image or photo. Not for factual questions — use lookup for those.",
|
||
```
|
||
|
||
- [ ] **Step 3: Update `read_article` description**
|
||
|
||
In `src/fabledassistant/services/tools/rss.py`, change:
|
||
|
||
```python
|
||
"Do NOT use search_web for URLs — use this tool instead."
|
||
```
|
||
|
||
to:
|
||
|
||
```python
|
||
"Do NOT use lookup for URLs — use this tool instead."
|
||
```
|
||
|
||
- [ ] **Step 4: Update status label map in `generation_task.py`**
|
||
|
||
In `src/fabledassistant/services/generation_task.py`, line 133, change:
|
||
|
||
```python
|
||
"search_web": "Searching the web",
|
||
```
|
||
|
||
to:
|
||
|
||
```python
|
||
"lookup": "Looking up information",
|
||
```
|
||
|
||
- [ ] **Step 5: Update action list in `llm.py`**
|
||
|
||
In `src/fabledassistant/services/llm.py`, line 608, change:
|
||
|
||
```python
|
||
actions.extend(["search_web", "research_topic", "search_images"])
|
||
```
|
||
|
||
to:
|
||
|
||
```python
|
||
actions.extend(["lookup", "research_topic", "search_images"])
|
||
```
|
||
|
||
- [ ] **Step 6: Run full test suite to check for regressions**
|
||
|
||
Run: `uv run pytest tests/ -v`
|
||
Expected: All tests pass (no test references `search_web` by name in assertions)
|
||
|
||
- [ ] **Step 7: Commit**
|
||
|
||
```bash
|
||
git add src/fabledassistant/services/tools/web.py src/fabledassistant/services/tools/rss.py src/fabledassistant/services/generation_task.py src/fabledassistant/services/llm.py
|
||
git commit -m "refactor: update all search_web references to lookup"
|
||
```
|
||
|
||
---
|
||
|
||
### Task 4: Add Wikipedia Sources to Research Pipeline
|
||
|
||
**Files:**
|
||
- Modify: `src/fabledassistant/services/research.py`
|
||
- Modify: `tests/test_research_pipeline.py`
|
||
|
||
- [ ] **Step 1: Write failing test for Wikipedia in research pipeline**
|
||
|
||
Add to `tests/test_research_pipeline.py`:
|
||
|
||
```python
|
||
@pytest.mark.asyncio
|
||
async def test_pipeline_includes_wikipedia_sources():
|
||
"""run_research_pipeline should merge Wikipedia results into the source pool."""
|
||
from unittest.mock import MagicMock
|
||
|
||
wiki_results = [{"title": "Wiki Article", "extract": "Wikipedia content about the topic.", "url": "https://en.wikipedia.org/wiki/Topic"}]
|
||
|
||
outline = [
|
||
{"title": "Section A", "focus": "Focus A"},
|
||
{"title": "Section B", "focus": "Focus B"},
|
||
]
|
||
|
||
note_id_counter = iter(range(30, 40))
|
||
|
||
def _make_note(user_id, title, body, tags, project_id=None, parent_id=None):
|
||
n = MagicMock()
|
||
n.id = next(note_id_counter)
|
||
n.title = title
|
||
return n
|
||
|
||
with patch("fabledassistant.services.research._generate_sub_queries", new_callable=AsyncMock, return_value=["q1"]), \
|
||
patch("fabledassistant.services.research._search_searxng", new_callable=AsyncMock, return_value=[{"url": "http://x.com", "title": "X", "snippet": "s"}]), \
|
||
patch("fabledassistant.services.research.wiki_search", new_callable=AsyncMock, return_value=wiki_results), \
|
||
patch("fabledassistant.services.research.fetch_url_content", new_callable=AsyncMock, return_value="content"), \
|
||
patch("fabledassistant.services.research._generate_outline", new_callable=AsyncMock, return_value=outline) as mock_outline, \
|
||
patch("fabledassistant.services.research._synthesize_section", new_callable=AsyncMock, side_effect=lambda t, f, s, m: (t, f"Body for {t}")), \
|
||
patch("fabledassistant.services.research._generate_executive_summary", new_callable=AsyncMock, return_value="Summary."), \
|
||
patch("fabledassistant.services.research.create_note", new_callable=AsyncMock, side_effect=_make_note), \
|
||
patch("fabledassistant.services.research.update_note", new_callable=AsyncMock):
|
||
|
||
from fabledassistant.services.research import run_research_pipeline
|
||
await run_research_pipeline("test topic", user_id=1, model="test-model")
|
||
|
||
# The sources passed to _generate_outline should include the Wikipedia article
|
||
sources_arg = mock_outline.call_args[0][1] # second positional arg
|
||
source_urls = [s["url"] for s in sources_arg]
|
||
assert "https://en.wikipedia.org/wiki/Topic" in source_urls
|
||
```
|
||
|
||
- [ ] **Step 2: Run test to verify it fails**
|
||
|
||
Run: `uv run pytest tests/test_research_pipeline.py::test_pipeline_includes_wikipedia_sources -v`
|
||
Expected: FAIL (no `wiki_search` import in research.py)
|
||
|
||
- [ ] **Step 3: Add Wikipedia sources to the research pipeline**
|
||
|
||
In `src/fabledassistant/services/research.py`, add the import at the top (after existing imports):
|
||
|
||
```python
|
||
from fabledassistant.services.wikipedia import wiki_search
|
||
```
|
||
|
||
Then modify Step 2 (the parallel search section, around lines 208–246). Replace:
|
||
|
||
```python
|
||
# Step 2: Search all queries in parallel (200 ms stagger to avoid hammering SearXNG)
|
||
async def _search_with_stagger(i: int, query: str) -> tuple[str, list[dict]]:
|
||
if i > 0:
|
||
await asyncio.sleep(0.2 * i)
|
||
_status(f"Searching: {query}...")
|
||
results = await _search_searxng(query)
|
||
logger.info("Research: query '%s' → %d results", query, len(results))
|
||
return query, results
|
||
|
||
search_results = await asyncio.gather(
|
||
*[_search_with_stagger(i, q) for i, q in enumerate(queries)]
|
||
)
|
||
|
||
# Deduplicate URLs across all queries
|
||
seen_urls: set[str] = set()
|
||
url_tasks: list[tuple[str, dict, str]] = [] # (url, result_dict, query)
|
||
for query, results in search_results:
|
||
for result in results[:PAGES_PER_QUERY]:
|
||
url = result.get("url", "")
|
||
if url and url not in seen_urls:
|
||
seen_urls.add(url)
|
||
url_tasks.append((url, result, query))
|
||
|
||
# Fetch all unique URLs in parallel
|
||
async def _fetch_source(url: str, result: dict, query: str) -> dict:
|
||
title = result.get("title", url)
|
||
_status(f"Reading: {title[:60]}...")
|
||
content = await fetch_url_content(url)
|
||
return {
|
||
"url": url,
|
||
"title": title,
|
||
"query": query,
|
||
"snippet": result.get("snippet", ""),
|
||
"content": content,
|
||
}
|
||
|
||
all_sources: list[dict] = list(await asyncio.gather(
|
||
*[_fetch_source(url, result, query) for url, result, query in url_tasks]
|
||
))
|
||
```
|
||
|
||
with:
|
||
|
||
```python
|
||
# Step 2: Search all queries in parallel (SearXNG + Wikipedia)
|
||
async def _search_with_stagger(i: int, query: str) -> tuple[str, list[dict]]:
|
||
if i > 0:
|
||
await asyncio.sleep(0.2 * i)
|
||
_status(f"Searching: {query}...")
|
||
results = await _search_searxng(query)
|
||
logger.info("Research: query '%s' → %d results", query, len(results))
|
||
return query, results
|
||
|
||
async def _wiki_for_query(query: str) -> list[dict]:
|
||
return await wiki_search(query, limit=1)
|
||
|
||
searxng_task = asyncio.gather(
|
||
*[_search_with_stagger(i, q) for i, q in enumerate(queries)]
|
||
)
|
||
wiki_task = asyncio.gather(
|
||
*[_wiki_for_query(q) for q in queries]
|
||
)
|
||
search_results, wiki_results = await asyncio.gather(searxng_task, wiki_task)
|
||
|
||
# Deduplicate URLs across all queries
|
||
seen_urls: set[str] = set()
|
||
url_tasks: list[tuple[str, dict, str]] = [] # (url, result_dict, query)
|
||
wiki_sources: list[dict] = [] # Wikipedia articles (already have content)
|
||
|
||
for query, results in search_results:
|
||
for result in results[:PAGES_PER_QUERY]:
|
||
url = result.get("url", "")
|
||
if url and url not in seen_urls:
|
||
seen_urls.add(url)
|
||
url_tasks.append((url, result, query))
|
||
|
||
# Add Wikipedia results (they already have content via extract)
|
||
for query, wiki_hits in zip(queries, wiki_results):
|
||
for hit in wiki_hits:
|
||
url = hit.get("url", "")
|
||
if url and url not in seen_urls:
|
||
seen_urls.add(url)
|
||
wiki_sources.append({
|
||
"url": url,
|
||
"title": hit["title"],
|
||
"query": query,
|
||
"snippet": hit["extract"][:200],
|
||
"content": hit["extract"],
|
||
})
|
||
|
||
# Fetch all unique SearXNG URLs in parallel
|
||
async def _fetch_source(url: str, result: dict, query: str) -> dict:
|
||
title = result.get("title", url)
|
||
_status(f"Reading: {title[:60]}...")
|
||
content = await fetch_url_content(url)
|
||
return {
|
||
"url": url,
|
||
"title": title,
|
||
"query": query,
|
||
"snippet": result.get("snippet", ""),
|
||
"content": content,
|
||
}
|
||
|
||
fetched_sources: list[dict] = list(await asyncio.gather(
|
||
*[_fetch_source(url, result, query) for url, result, query in url_tasks]
|
||
))
|
||
|
||
all_sources = wiki_sources + fetched_sources
|
||
```
|
||
|
||
- [ ] **Step 4: Run the new test to verify it passes**
|
||
|
||
Run: `uv run pytest tests/test_research_pipeline.py::test_pipeline_includes_wikipedia_sources -v`
|
||
Expected: PASS
|
||
|
||
- [ ] **Step 5: Run the full research test suite for regressions**
|
||
|
||
Run: `uv run pytest tests/test_research_pipeline.py -v`
|
||
Expected: All tests pass
|
||
|
||
- [ ] **Step 6: Run full test suite**
|
||
|
||
Run: `uv run pytest tests/ -v`
|
||
Expected: All tests pass
|
||
|
||
- [ ] **Step 7: Commit**
|
||
|
||
```bash
|
||
git add src/fabledassistant/services/research.py tests/test_research_pipeline.py
|
||
git commit -m "feat: add Wikipedia as research pipeline source alongside SearXNG"
|
||
```
|
||
|
||
---
|
||
|
||
### Task 5: Lint, Typecheck, Final Verification
|
||
|
||
**Files:**
|
||
- All modified files
|
||
|
||
- [ ] **Step 1: Run ruff lint**
|
||
|
||
Run: `uv run ruff check src/fabledassistant/services/wikipedia.py src/fabledassistant/services/tools/web.py src/fabledassistant/services/research.py tests/test_wikipedia.py tests/test_lookup_tool.py`
|
||
Expected: All checks passed (fix any issues if not)
|
||
|
||
- [ ] **Step 2: Run typecheck**
|
||
|
||
Run: `cd frontend && npx vue-tsc --noEmit` (frontend unchanged, but verify nothing broke)
|
||
Expected: Clean
|
||
|
||
- [ ] **Step 3: Run full test suite one last time**
|
||
|
||
Run: `uv run pytest tests/ -v`
|
||
Expected: All tests pass
|
||
|
||
- [ ] **Step 4: Commit any lint fixes if needed**
|
||
|
||
```bash
|
||
git add -u
|
||
git commit -m "fix: lint cleanup for lookup/wikipedia changes"
|
||
```
|