FabledScribe/docs/superpowers/plans/2026-04-17-unified-lookup-wikipedia.md

Unified Lookup Tool & Wikipedia Integration — Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Replace search_web with a unified lookup tool (Wikipedia-first, SearXNG fallback) and add Wikipedia as a source in the research pipeline.

Architecture: New wikipedia.py service with wiki_summary and wiki_search. lookup tool in web.py replaces search_web. Research pipeline in research.py gains Wikipedia sources alongside SearXNG. All search_web references across the codebase are updated.

Tech Stack: Python 3.12, httpx, pytest, asyncio

---

Task 1: Wikipedia Service Module

Files:

• Create: src/fabledassistant/services/wikipedia.py

• Create: tests/test_wikipedia.py

• Step 1: Write failing tests for wiki_summary

# tests/test_wikipedia.py
import pytest
from unittest.mock import AsyncMock, patch, MagicMock
import httpx


@pytest.mark.asyncio
async def test_wiki_summary_returns_extract():
    from fabledassistant.services.wikipedia import wiki_summary

    mock_response = MagicMock()
    mock_response.status_code = 200
    mock_response.json.return_value = {
        "type": "standard",
        "title": "Python (programming language)",
        "extract": "Python is a high-level programming language.",
        "content_urls": {
            "desktop": {"page": "https://en.wikipedia.org/wiki/Python_(programming_language)"}
        },
    }
    mock_response.raise_for_status = MagicMock()

    with patch("fabledassistant.services.wikipedia.httpx.AsyncClient") as mock_client_cls:
        mock_client = AsyncMock()
        mock_client.__aenter__ = AsyncMock(return_value=mock_client)
        mock_client.__aexit__ = AsyncMock(return_value=False)
        mock_client.get = AsyncMock(return_value=mock_response)
        mock_client_cls.return_value = mock_client

        result = await wiki_summary("Python programming language")

    assert result is not None
    assert result["title"] == "Python (programming language)"
    assert "high-level" in result["extract"]
    assert "wikipedia.org" in result["url"]


@pytest.mark.asyncio
async def test_wiki_summary_returns_none_on_404():
    from fabledassistant.services.wikipedia import wiki_summary

    with patch("fabledassistant.services.wikipedia.httpx.AsyncClient") as mock_client_cls:
        mock_client = AsyncMock()
        mock_client.__aenter__ = AsyncMock(return_value=mock_client)
        mock_client.__aexit__ = AsyncMock(return_value=False)
        mock_client.get = AsyncMock(side_effect=httpx.HTTPStatusError(
            "Not Found", request=MagicMock(), response=MagicMock(status_code=404)
        ))
        mock_client_cls.return_value = mock_client

        result = await wiki_summary("xyznonexistenttopic123")

    assert result is None


@pytest.mark.asyncio
async def test_wiki_summary_returns_none_on_disambiguation():
    from fabledassistant.services.wikipedia import wiki_summary

    mock_response = MagicMock()
    mock_response.status_code = 200
    mock_response.json.return_value = {
        "type": "disambiguation",
        "title": "Python",
        "extract": "Python may refer to...",
    }
    mock_response.raise_for_status = MagicMock()

    with patch("fabledassistant.services.wikipedia.httpx.AsyncClient") as mock_client_cls:
        mock_client = AsyncMock()
        mock_client.__aenter__ = AsyncMock(return_value=mock_client)
        mock_client.__aexit__ = AsyncMock(return_value=False)
        mock_client.get = AsyncMock(return_value=mock_response)
        mock_client_cls.return_value = mock_client

        result = await wiki_summary("Python")

    assert result is None

• Step 2: Run tests to verify they fail

Run: uv run pytest tests/test_wikipedia.py -v Expected: FAIL with ModuleNotFoundError: No module named 'fabledassistant.services.wikipedia'

• Step 3: Implement wiki_summary

# src/fabledassistant/services/wikipedia.py
"""Wikipedia API: lightweight topic lookups and article search."""

import logging
from urllib.parse import quote as url_quote

import httpx

logger = logging.getLogger(__name__)

_SUMMARY_URL = "https://en.wikipedia.org/api/rest_v1/page/summary"
_SEARCH_URL = "https://en.wikipedia.org/w/api.php"
_TIMEOUT = 5.0
_USER_AGENT = "FabledAssistant/1.0 (https://fabledsword.com)"


async def wiki_summary(query: str) -> dict | None:
    """Look up a topic by title via the Wikipedia REST summary endpoint.

    Returns {"title", "extract", "url"} on hit, None on miss.
    """
    encoded = url_quote(query.replace(" ", "_"), safe="")
    try:
        async with httpx.AsyncClient(
            timeout=_TIMEOUT, headers={"User-Agent": _USER_AGENT}
        ) as client:
            resp = await client.get(f"{_SUMMARY_URL}/{encoded}", follow_redirects=True)
            resp.raise_for_status()
            data = resp.json()
    except Exception:
        logger.debug("Wikipedia summary lookup failed for %r", query, exc_info=True)
        return None

    if data.get("type") == "disambiguation":
        return None

    extract = data.get("extract", "").strip()
    if not extract:
        return None

    url = (
        data.get("content_urls", {}).get("desktop", {}).get("page")
        or f"https://en.wikipedia.org/wiki/{encoded}"
    )
    return {"title": data.get("title", query), "extract": extract, "url": url}

• Step 4: Run tests to verify they pass

Run: uv run pytest tests/test_wikipedia.py -v Expected: 3 passed

• Step 5: Write failing tests for wiki_search

Add to tests/test_wikipedia.py:

@pytest.mark.asyncio
async def test_wiki_search_returns_results():
    from fabledassistant.services.wikipedia import wiki_search

    search_response = MagicMock()
    search_response.status_code = 200
    search_response.json.return_value = {
        "query": {
            "search": [
                {"title": "QUIC"},
                {"title": "HTTP/3"},
            ]
        }
    }
    search_response.raise_for_status = MagicMock()

    summary_response = MagicMock()
    summary_response.status_code = 200
    summary_response.json.return_value = {
        "type": "standard",
        "title": "QUIC",
        "extract": "QUIC is a transport layer protocol.",
        "content_urls": {"desktop": {"page": "https://en.wikipedia.org/wiki/QUIC"}},
    }
    summary_response.raise_for_status = MagicMock()

    with patch("fabledassistant.services.wikipedia.httpx.AsyncClient") as mock_client_cls:
        mock_client = AsyncMock()
        mock_client.__aenter__ = AsyncMock(return_value=mock_client)
        mock_client.__aexit__ = AsyncMock(return_value=False)
        mock_client.get = AsyncMock(side_effect=[search_response, summary_response, summary_response])
        mock_client_cls.return_value = mock_client

        results = await wiki_search("QUIC protocol", limit=2)

    assert len(results) >= 1
    assert results[0]["title"] == "QUIC"
    assert "transport" in results[0]["extract"]


@pytest.mark.asyncio
async def test_wiki_search_returns_empty_on_failure():
    from fabledassistant.services.wikipedia import wiki_search

    with patch("fabledassistant.services.wikipedia.httpx.AsyncClient") as mock_client_cls:
        mock_client = AsyncMock()
        mock_client.__aenter__ = AsyncMock(return_value=mock_client)
        mock_client.__aexit__ = AsyncMock(return_value=False)
        mock_client.get = AsyncMock(side_effect=httpx.ConnectError("connection failed"))
        mock_client_cls.return_value = mock_client

        results = await wiki_search("anything")

    assert results == []

• Step 6: Implement wiki_search

Add to src/fabledassistant/services/wikipedia.py:

async def wiki_search(query: str, limit: int = 3) -> list[dict]:
    """Search Wikipedia for articles matching a query.

    Returns [{"title", "extract", "url"}, ...] (may be empty).
    """
    try:
        async with httpx.AsyncClient(
            timeout=_TIMEOUT, headers={"User-Agent": _USER_AGENT}
        ) as client:
            resp = await client.get(_SEARCH_URL, params={
                "action": "query",
                "list": "search",
                "srsearch": query,
                "srlimit": str(limit),
                "format": "json",
            })
            resp.raise_for_status()
            hits = resp.json().get("query", {}).get("search", [])
            if not hits:
                return []

            results: list[dict] = []
            for hit in hits:
                title = hit.get("title", "")
                if not title:
                    continue
                encoded = url_quote(title.replace(" ", "_"), safe="")
                try:
                    summary_resp = await client.get(
                        f"{_SUMMARY_URL}/{encoded}", follow_redirects=True,
                    )
                    summary_resp.raise_for_status()
                    data = summary_resp.json()
                except Exception:
                    continue
                if data.get("type") == "disambiguation":
                    continue
                extract = data.get("extract", "").strip()
                if not extract:
                    continue
                url = (
                    data.get("content_urls", {}).get("desktop", {}).get("page")
                    or f"https://en.wikipedia.org/wiki/{encoded}"
                )
                results.append({"title": data.get("title", title), "extract": extract, "url": url})
            return results
    except Exception:
        logger.debug("Wikipedia search failed for %r", query, exc_info=True)
        return []

• Step 7: Run all wikipedia tests

Run: uv run pytest tests/test_wikipedia.py -v Expected: 5 passed

• Step 8: Commit

git add src/fabledassistant/services/wikipedia.py tests/test_wikipedia.py
git commit -m "feat: add wikipedia service with summary lookup and search"

---

Task 2: Lookup Tool (replaces search_web)

Files:

• Modify: src/fabledassistant/services/tools/web.py

• Create: tests/test_lookup_tool.py

• Step 1: Write failing tests for lookup

# tests/test_lookup_tool.py
import pytest
from unittest.mock import AsyncMock, patch, MagicMock


@pytest.mark.asyncio
async def test_lookup_wikipedia_hit():
    """lookup returns wikipedia source when wiki_summary succeeds."""
    wiki_data = {
        "title": "QUIC",
        "extract": "QUIC is a transport layer protocol.",
        "url": "https://en.wikipedia.org/wiki/QUIC",
    }

    with patch("fabledassistant.services.tools.web.wiki_summary", new_callable=AsyncMock, return_value=wiki_data):
        from fabledassistant.services.tools.web import lookup_tool
        result = await lookup_tool(user_id=1, arguments={"query": "QUIC"})

    assert result["success"] is True
    assert result["type"] == "lookup"
    assert result["source"] == "wikipedia"
    assert result["data"]["title"] == "QUIC"
    assert "transport" in result["data"]["extract"]


@pytest.mark.asyncio
async def test_lookup_wikipedia_miss_searxng_fallback():
    """lookup falls back to SearXNG + article fetch when Wikipedia misses."""
    searxng_results = [
        {"url": "https://example.com/quic", "title": "QUIC Explained", "snippet": "An overview..."},
    ]

    with patch("fabledassistant.services.tools.web.wiki_summary", new_callable=AsyncMock, return_value=None), \
         patch("fabledassistant.services.tools.web.Config") as mock_config, \
         patch("fabledassistant.services.tools.web._search_searxng", new_callable=AsyncMock, return_value=searxng_results), \
         patch("fabledassistant.services.tools.web._fetch_full_article", new_callable=AsyncMock, return_value="Full article about QUIC..."):
        mock_config.searxng_enabled.return_value = True
        from fabledassistant.services.tools.web import lookup_tool
        result = await lookup_tool(user_id=1, arguments={"query": "QUIC"})

    assert result["success"] is True
    assert result["type"] == "lookup"
    assert result["source"] == "web"
    assert result["data"]["results"]


@pytest.mark.asyncio
async def test_lookup_wikipedia_miss_no_searxng():
    """lookup returns no-results when Wikipedia misses and SearXNG is not configured."""
    with patch("fabledassistant.services.tools.web.wiki_summary", new_callable=AsyncMock, return_value=None), \
         patch("fabledassistant.services.tools.web.Config") as mock_config:
        mock_config.searxng_enabled.return_value = False
        from fabledassistant.services.tools.web import lookup_tool
        result = await lookup_tool(user_id=1, arguments={"query": "xyznonexistent"})

    assert result["success"] is True
    assert result["source"] == "none"


@pytest.mark.asyncio
async def test_lookup_always_available():
    """lookup tool must appear in get_tools_for_user regardless of SearXNG config."""
    with patch("fabledassistant.services.tools._registry.is_caldav_configured", new_callable=AsyncMock, return_value=False), \
         patch("fabledassistant.services.settings.get_setting", new_callable=AsyncMock, return_value="false"):
        from fabledassistant.services.tools import get_tools_for_user
        tools = await get_tools_for_user(user_id=1)
        tool_names = {t["function"]["name"] for t in tools}
        assert "lookup" in tool_names

• Step 2: Run tests to verify they fail

Run: uv run pytest tests/test_lookup_tool.py -v Expected: FAIL (no lookup_tool function)

• Step 3: Replace search_web with lookup in web.py

Replace the search_web_tool function (lines 12–36 of src/fabledassistant/services/tools/web.py) with:

@tool(
    name="lookup",
    description=(
        "Look up a topic, concept, or factual question. Returns a concise answer from "
        "Wikipedia or web sources. Use for definitions, explanations, 'what is X', "
        "'how does Y work', current events, or version numbers. No note is saved. "
        "For comprehensive written reports saved as notes, use research_topic instead."
    ),
    parameters={
        "query": {"type": "string", "description": "The topic or question to look up"},
    },
    required=["query"],
)
async def lookup_tool(*, user_id, arguments, **_ctx):
    from fabledassistant.config import Config
    from fabledassistant.services.wikipedia import wiki_summary

    query = arguments.get("query", "")

    # 1. Try Wikipedia first
    wiki = await wiki_summary(query)
    if wiki:
        return {
            "success": True,
            "type": "lookup",
            "source": "wikipedia",
            "data": wiki,
        }

    # 2. Fall back to SearXNG + article fetch
    if Config.searxng_enabled():
        from fabledassistant.services.research import _search_searxng
        from fabledassistant.services.rss import _fetch_full_article

        results = await _search_searxng(query)
        if results:
            articles: list[dict] = []
            for r in results[:2]:
                url = r.get("url", "")
                if not url:
                    continue
                content = await _fetch_full_article(url)
                articles.append({
                    "url": url,
                    "title": r.get("title", url),
                    "snippet": r.get("snippet", ""),
                    "content": (content or "")[:4000],
                })
            if articles:
                return {
                    "success": True,
                    "type": "lookup",
                    "source": "web",
                    "data": {"query": query, "results": articles},
                }

    # 3. No sources available
    return {
        "success": True,
        "type": "lookup",
        "source": "none",
        "data": {
            "query": query,
            "message": "No results found. You can answer from your own knowledge.",
        },
    }

• Step 4: Run lookup tests to verify they pass

Run: uv run pytest tests/test_lookup_tool.py -v Expected: 4 passed

• Step 5: Commit

git add src/fabledassistant/services/tools/web.py tests/test_lookup_tool.py
git commit -m "feat: replace search_web with unified lookup tool (Wikipedia + SearXNG fallback)"

---

Task 3: Update All search_web References

Files:

• Modify: src/fabledassistant/services/tools/web.py:45 (research_topic description)

• Modify: src/fabledassistant/services/tools/web.py:67 (search_images description)

• Modify: src/fabledassistant/services/tools/rss.py:75 (read_article description)

• Modify: src/fabledassistant/services/generation_task.py:133 (status label map)

• Modify: src/fabledassistant/services/llm.py:608 (action list)

• Step 1: Update research_topic description

In src/fabledassistant/services/tools/web.py, change the research_topic description from:

        "For a quick factual answer without saving a note, use search_web."

to:

        "For a quick factual answer without saving a note, use lookup."

• Step 2: Update search_images description

In src/fabledassistant/services/tools/web.py, change the search_images description from:

    description="Search and display images inline. Use ONLY when the user explicitly asks to see, show, or find an image or photo. Not for factual questions — use search_web for those.",

to:

    description="Search and display images inline. Use ONLY when the user explicitly asks to see, show, or find an image or photo. Not for factual questions — use lookup for those.",

• Step 3: Update read_article description

In src/fabledassistant/services/tools/rss.py, change:

        "Do NOT use search_web for URLs — use this tool instead."

to:

        "Do NOT use lookup for URLs — use this tool instead."

• Step 4: Update status label map in generation_task.py

In src/fabledassistant/services/generation_task.py, line 133, change:

    "search_web": "Searching the web",

to:

    "lookup": "Looking up information",

• Step 5: Update action list in llm.py

In src/fabledassistant/services/llm.py, line 608, change:

        actions.extend(["search_web", "research_topic", "search_images"])

to:

        actions.extend(["lookup", "research_topic", "search_images"])

• Step 6: Run full test suite to check for regressions

Run: uv run pytest tests/ -v Expected: All tests pass (no test references search_web by name in assertions)

• Step 7: Commit

git add src/fabledassistant/services/tools/web.py src/fabledassistant/services/tools/rss.py src/fabledassistant/services/generation_task.py src/fabledassistant/services/llm.py
git commit -m "refactor: update all search_web references to lookup"

---

Task 4: Add Wikipedia Sources to Research Pipeline

Files:

• Modify: src/fabledassistant/services/research.py

• Modify: tests/test_research_pipeline.py

• Step 1: Write failing test for Wikipedia in research pipeline

Add to tests/test_research_pipeline.py:

@pytest.mark.asyncio
async def test_pipeline_includes_wikipedia_sources():
    """run_research_pipeline should merge Wikipedia results into the source pool."""
    from unittest.mock import MagicMock

    wiki_results = [{"title": "Wiki Article", "extract": "Wikipedia content about the topic.", "url": "https://en.wikipedia.org/wiki/Topic"}]

    outline = [
        {"title": "Section A", "focus": "Focus A"},
        {"title": "Section B", "focus": "Focus B"},
    ]

    note_id_counter = iter(range(30, 40))

    def _make_note(user_id, title, body, tags, project_id=None, parent_id=None):
        n = MagicMock()
        n.id = next(note_id_counter)
        n.title = title
        return n

    with patch("fabledassistant.services.research._generate_sub_queries", new_callable=AsyncMock, return_value=["q1"]), \
         patch("fabledassistant.services.research._search_searxng", new_callable=AsyncMock, return_value=[{"url": "http://x.com", "title": "X", "snippet": "s"}]), \
         patch("fabledassistant.services.research.wiki_search", new_callable=AsyncMock, return_value=wiki_results), \
         patch("fabledassistant.services.research.fetch_url_content", new_callable=AsyncMock, return_value="content"), \
         patch("fabledassistant.services.research._generate_outline", new_callable=AsyncMock, return_value=outline) as mock_outline, \
         patch("fabledassistant.services.research._synthesize_section", new_callable=AsyncMock, side_effect=lambda t, f, s, m: (t, f"Body for {t}")), \
         patch("fabledassistant.services.research._generate_executive_summary", new_callable=AsyncMock, return_value="Summary."), \
         patch("fabledassistant.services.research.create_note", new_callable=AsyncMock, side_effect=_make_note), \
         patch("fabledassistant.services.research.update_note", new_callable=AsyncMock):

        from fabledassistant.services.research import run_research_pipeline
        await run_research_pipeline("test topic", user_id=1, model="test-model")

    # The sources passed to _generate_outline should include the Wikipedia article
    sources_arg = mock_outline.call_args[0][1]  # second positional arg
    source_urls = [s["url"] for s in sources_arg]
    assert "https://en.wikipedia.org/wiki/Topic" in source_urls

• Step 2: Run test to verify it fails

Run: uv run pytest tests/test_research_pipeline.py::test_pipeline_includes_wikipedia_sources -v Expected: FAIL (no wiki_search import in research.py)

• Step 3: Add Wikipedia sources to the research pipeline

In src/fabledassistant/services/research.py, add the import at the top (after existing imports):

from fabledassistant.services.wikipedia import wiki_search

Then modify Step 2 (the parallel search section, around lines 208–246). Replace:

    # Step 2: Search all queries in parallel (200 ms stagger to avoid hammering SearXNG)
    async def _search_with_stagger(i: int, query: str) -> tuple[str, list[dict]]:
        if i > 0:
            await asyncio.sleep(0.2 * i)
        _status(f"Searching: {query}...")
        results = await _search_searxng(query)
        logger.info("Research: query '%s' → %d results", query, len(results))
        return query, results

    search_results = await asyncio.gather(
        *[_search_with_stagger(i, q) for i, q in enumerate(queries)]
    )

    # Deduplicate URLs across all queries
    seen_urls: set[str] = set()
    url_tasks: list[tuple[str, dict, str]] = []  # (url, result_dict, query)
    for query, results in search_results:
        for result in results[:PAGES_PER_QUERY]:
            url = result.get("url", "")
            if url and url not in seen_urls:
                seen_urls.add(url)
                url_tasks.append((url, result, query))

    # Fetch all unique URLs in parallel
    async def _fetch_source(url: str, result: dict, query: str) -> dict:
        title = result.get("title", url)
        _status(f"Reading: {title[:60]}...")
        content = await fetch_url_content(url)
        return {
            "url": url,
            "title": title,
            "query": query,
            "snippet": result.get("snippet", ""),
            "content": content,
        }

    all_sources: list[dict] = list(await asyncio.gather(
        *[_fetch_source(url, result, query) for url, result, query in url_tasks]
    ))

with:

    # Step 2: Search all queries in parallel (SearXNG + Wikipedia)
    async def _search_with_stagger(i: int, query: str) -> tuple[str, list[dict]]:
        if i > 0:
            await asyncio.sleep(0.2 * i)
        _status(f"Searching: {query}...")
        results = await _search_searxng(query)
        logger.info("Research: query '%s' → %d results", query, len(results))
        return query, results

    async def _wiki_for_query(query: str) -> list[dict]:
        return await wiki_search(query, limit=1)

    searxng_task = asyncio.gather(
        *[_search_with_stagger(i, q) for i, q in enumerate(queries)]
    )
    wiki_task = asyncio.gather(
        *[_wiki_for_query(q) for q in queries]
    )
    search_results, wiki_results = await asyncio.gather(searxng_task, wiki_task)

    # Deduplicate URLs across all queries
    seen_urls: set[str] = set()
    url_tasks: list[tuple[str, dict, str]] = []  # (url, result_dict, query)
    wiki_sources: list[dict] = []  # Wikipedia articles (already have content)

    for query, results in search_results:
        for result in results[:PAGES_PER_QUERY]:
            url = result.get("url", "")
            if url and url not in seen_urls:
                seen_urls.add(url)
                url_tasks.append((url, result, query))

    # Add Wikipedia results (they already have content via extract)
    for query, wiki_hits in zip(queries, wiki_results):
        for hit in wiki_hits:
            url = hit.get("url", "")
            if url and url not in seen_urls:
                seen_urls.add(url)
                wiki_sources.append({
                    "url": url,
                    "title": hit["title"],
                    "query": query,
                    "snippet": hit["extract"][:200],
                    "content": hit["extract"],
                })

    # Fetch all unique SearXNG URLs in parallel
    async def _fetch_source(url: str, result: dict, query: str) -> dict:
        title = result.get("title", url)
        _status(f"Reading: {title[:60]}...")
        content = await fetch_url_content(url)
        return {
            "url": url,
            "title": title,
            "query": query,
            "snippet": result.get("snippet", ""),
            "content": content,
        }

    fetched_sources: list[dict] = list(await asyncio.gather(
        *[_fetch_source(url, result, query) for url, result, query in url_tasks]
    ))

    all_sources = wiki_sources + fetched_sources

• Step 4: Run the new test to verify it passes

Run: uv run pytest tests/test_research_pipeline.py::test_pipeline_includes_wikipedia_sources -v Expected: PASS

• Step 5: Run the full research test suite for regressions

Run: uv run pytest tests/test_research_pipeline.py -v Expected: All tests pass

• Step 6: Run full test suite

Run: uv run pytest tests/ -v Expected: All tests pass

• Step 7: Commit

git add src/fabledassistant/services/research.py tests/test_research_pipeline.py
git commit -m "feat: add Wikipedia as research pipeline source alongside SearXNG"

---

Task 5: Lint, Typecheck, Final Verification

Files:

• All modified files

• Step 1: Run ruff lint

Run: uv run ruff check src/fabledassistant/services/wikipedia.py src/fabledassistant/services/tools/web.py src/fabledassistant/services/research.py tests/test_wikipedia.py tests/test_lookup_tool.py Expected: All checks passed (fix any issues if not)

• Step 2: Run typecheck

Run: cd frontend && npx vue-tsc --noEmit (frontend unchanged, but verify nothing broke) Expected: Clean

• Step 3: Run full test suite one last time

Run: uv run pytest tests/ -v Expected: All tests pass

• Step 4: Commit any lint fixes if needed

git add -u
git commit -m "fix: lint cleanup for lookup/wikipedia changes"