FabledScribe/docs/superpowers/specs/2026-04-17-unified-lookup-wikipedia-design.md

Unified Lookup Tool & Wikipedia Integration

Goal

Replace the fragmented search_web tool with a single lookup tool that checks Wikipedia first and falls back to SearXNG web search. Add Wikipedia as an additional source in the research pipeline. Result: one lightweight tool for factual questions (always available, no config required), and richer research output.

Architecture

Two changes to the search/knowledge stack:

1. New lookup tool replaces search_web. Tries Wikipedia REST API summary endpoint first (~200ms, reliable, no config). Falls back to SearXNG + trafilatura article fetch when Wikipedia misses and SearXNG is configured. Always available (no requires field).

2. Wikipedia sources in research pipeline. During sub-query execution, wiki_search runs alongside _search_searxng. Wikipedia articles merge into the source pool and get deduplicated by URL.

Shared Wikipedia logic lives in a new wikipedia.py service module.

Components

src/fabledassistant/services/wikipedia.py (new)

Two async functions:

wiki_summary(query: str) -> dict | None

• Direct title lookup via https://en.wikipedia.org/api/rest_v1/page/summary/{title}
• Returns {"title": str, "extract": str, "url": str} on hit
• Returns None on 404, disambiguation pages ("type": "disambiguation"), network errors, or empty extracts
• 5-second timeout
• User-Agent: "FabledAssistant/1.0 (https://fabledsword.com)"

wiki_search(query: str, limit: int = 3) -> list[dict]

• Search via https://en.wikipedia.org/w/api.php?action=query&list=search&srsearch={query}&srlimit={limit}&format=json
• For each search result, fetch its summary via the summary endpoint to get the extract
• Returns [{"title": str, "extract": str, "url": str}, ...]
• Returns [] on any failure
• Same timeout and User-Agent as above

src/fabledassistant/services/tools/web.py (modified)

Remove: search_web_tool

Add: lookup_tool

@tool(
    name="lookup",
    description="Look up a topic, concept, or factual question. Returns a concise
                 answer from Wikipedia or web sources. Use for definitions,
                 explanations, 'what is X', 'how does Y work'. For comprehensive
                 written reports saved as notes, use research_topic instead.",
    parameters={
        "query": {"type": "string", "description": "The topic or question to look up"},
    },
    required=["query"],
)

No requires field — always available.

Logic:

1. Call wiki_summary(query)
2. If Wikipedia returns a result: return {"success": True, "type": "lookup", "source": "wikipedia", "data": {"title": ..., "extract": ..., "url": ...}}
3. If Wikipedia misses and Config.searxng_enabled():
   • Call _search_searxng(query) to get search results
   • Fetch top 1-2 result URLs via _fetch_full_article (from rss.py, trafilatura-based)
   • Return {"success": True, "type": "lookup", "source": "web", "data": {"query": ..., "results": [...], "content": ...}}
4. If Wikipedia misses and no SearXNG: return {"success": True, "type": "lookup", "source": "none", "data": {"query": ..., "message": "No results found. You can answer from your own knowledge."}}

src/fabledassistant/services/research.py (modified)

In Step 2 (parallel search):

• For each sub-query, run wiki_search(query, limit=1) concurrently with _search_searxng(query)
• Merge Wikipedia results into the per-query result list

In Step 3 (deduplication):

• When deduplicating URLs, Wikipedia URLs (wikipedia.org) are checked against SearXNG results
• If a Wikipedia article URL already appears in SearXNG results, skip the duplicate

Wikipedia article content for synthesis:

• The extract from wiki_search is used as the source content (no additional fetch needed, unlike SearXNG URLs which require fetch_url_content)
• This means Wikipedia sources are available immediately without an HTTP fetch step

Error Handling

• All Wikipedia API failures (network, timeout, malformed JSON) return None/[] silently
• lookup never raises — always returns a response the model can work with
• In the research pipeline, Wikipedia is purely additive; its failure never degrades existing SearXNG-based research
• Disambiguation pages are detected via "type": "disambiguation" in the summary response and treated as a miss

Testing

tests/test_wikipedia.py (new)

• test_wiki_summary_returns_extract — mock successful summary response, verify return shape
• test_wiki_summary_returns_none_on_404 — mock 404, verify None
• test_wiki_summary_returns_none_on_disambiguation — mock disambiguation response, verify None
• test_wiki_search_returns_results — mock search API + summary fetches, verify list
• test_wiki_search_returns_empty_on_failure — mock network error, verify []

tests/test_lookup_tool.py (new)

• test_lookup_wikipedia_hit — mock wiki_summary returning data, verify tool returns wikipedia source
• test_lookup_wikipedia_miss_searxng_fallback — mock wiki_summary returning None, SearXNG returning results + article fetch, verify web source
• test_lookup_wikipedia_miss_no_searxng — mock both missing, verify graceful "no results" response
• test_lookup_always_available — verify the tool appears in get_tools_for_user regardless of SearXNG config

tests/test_research_pipeline.py (add to existing)

• test_research_includes_wikipedia_sources — mock wiki_search alongside SearXNG, verify Wikipedia results appear in source pool

All tests mock HTTP calls — no live API hits.

What Doesn't Change

• read_article tool — stays as-is (explicit URL fetch, different purpose)
• research_topic tool definition — stays as-is (same name, description, parameters)
• generation_task.py research interception — stays as-is
• search_images tool — stays as-is
• _search_searxng and _search_searxng_images — stay as-is
• _fetch_full_article in rss.py — stays as-is, reused by lookup for SearXNG fallback