From ec49e24c8db6557ebe69a44284b9e73fc8cd31e9 Mon Sep 17 00:00:00 2001 From: Bryan Van Deusen Date: Fri, 17 Apr 2026 21:17:48 -0400 Subject: [PATCH] docs: add unified lookup tool & Wikipedia integration spec Co-Authored-By: Claude Opus 4.6 --- ...6-04-17-unified-lookup-wikipedia-design.md | 119 ++++++++++++++++++ 1 file changed, 119 insertions(+) create mode 100644 docs/superpowers/specs/2026-04-17-unified-lookup-wikipedia-design.md diff --git a/docs/superpowers/specs/2026-04-17-unified-lookup-wikipedia-design.md b/docs/superpowers/specs/2026-04-17-unified-lookup-wikipedia-design.md new file mode 100644 index 0000000..1b68d10 --- /dev/null +++ b/docs/superpowers/specs/2026-04-17-unified-lookup-wikipedia-design.md @@ -0,0 +1,119 @@ +# Unified Lookup Tool & Wikipedia Integration + +## Goal + +Replace the fragmented `search_web` tool with a single `lookup` tool that checks Wikipedia first and falls back to SearXNG web search. Add Wikipedia as an additional source in the research pipeline. Result: one lightweight tool for factual questions (always available, no config required), and richer research output. + +## Architecture + +Two changes to the search/knowledge stack: + +1. **New `lookup` tool** replaces `search_web`. Tries Wikipedia REST API summary endpoint first (~200ms, reliable, no config). Falls back to SearXNG + trafilatura article fetch when Wikipedia misses and SearXNG is configured. Always available (no `requires` field). + +2. **Wikipedia sources in research pipeline.** During sub-query execution, `wiki_search` runs alongside `_search_searxng`. Wikipedia articles merge into the source pool and get deduplicated by URL. + +Shared Wikipedia logic lives in a new `wikipedia.py` service module. + +## Components + +### `src/fabledassistant/services/wikipedia.py` (new) + +Two async functions: + +**`wiki_summary(query: str) -> dict | None`** +- Direct title lookup via `https://en.wikipedia.org/api/rest_v1/page/summary/{title}` +- Returns `{"title": str, "extract": str, "url": str}` on hit +- Returns `None` on 404, disambiguation pages (`"type": "disambiguation"`), network errors, or empty extracts +- 5-second timeout +- User-Agent: `"FabledAssistant/1.0 (https://fabledsword.com)"` + +**`wiki_search(query: str, limit: int = 3) -> list[dict]`** +- Search via `https://en.wikipedia.org/w/api.php?action=query&list=search&srsearch={query}&srlimit={limit}&format=json` +- For each search result, fetch its summary via the summary endpoint to get the extract +- Returns `[{"title": str, "extract": str, "url": str}, ...]` +- Returns `[]` on any failure +- Same timeout and User-Agent as above + +### `src/fabledassistant/services/tools/web.py` (modified) + +**Remove:** `search_web_tool` + +**Add:** `lookup_tool` + +``` +@tool( + name="lookup", + description="Look up a topic, concept, or factual question. Returns a concise + answer from Wikipedia or web sources. Use for definitions, + explanations, 'what is X', 'how does Y work'. For comprehensive + written reports saved as notes, use research_topic instead.", + parameters={ + "query": {"type": "string", "description": "The topic or question to look up"}, + }, + required=["query"], +) +``` + +No `requires` field — always available. + +**Logic:** +1. Call `wiki_summary(query)` +2. If Wikipedia returns a result: return `{"success": True, "type": "lookup", "source": "wikipedia", "data": {"title": ..., "extract": ..., "url": ...}}` +3. If Wikipedia misses and `Config.searxng_enabled()`: + - Call `_search_searxng(query)` to get search results + - Fetch top 1-2 result URLs via `_fetch_full_article` (from `rss.py`, trafilatura-based) + - Return `{"success": True, "type": "lookup", "source": "web", "data": {"query": ..., "results": [...], "content": ...}}` +4. If Wikipedia misses and no SearXNG: return `{"success": True, "type": "lookup", "source": "none", "data": {"query": ..., "message": "No results found. You can answer from your own knowledge."}}` + +### `src/fabledassistant/services/research.py` (modified) + +**In Step 2 (parallel search):** +- For each sub-query, run `wiki_search(query, limit=1)` concurrently with `_search_searxng(query)` +- Merge Wikipedia results into the per-query result list + +**In Step 3 (deduplication):** +- When deduplicating URLs, Wikipedia URLs (`wikipedia.org`) are checked against SearXNG results +- If a Wikipedia article URL already appears in SearXNG results, skip the duplicate + +**Wikipedia article content for synthesis:** +- The `extract` from `wiki_search` is used as the source content (no additional fetch needed, unlike SearXNG URLs which require `fetch_url_content`) +- This means Wikipedia sources are available immediately without an HTTP fetch step + +## Error Handling + +- All Wikipedia API failures (network, timeout, malformed JSON) return `None`/`[]` silently +- `lookup` never raises — always returns a response the model can work with +- In the research pipeline, Wikipedia is purely additive; its failure never degrades existing SearXNG-based research +- Disambiguation pages are detected via `"type": "disambiguation"` in the summary response and treated as a miss + +## Testing + +### `tests/test_wikipedia.py` (new) + +- `test_wiki_summary_returns_extract` — mock successful summary response, verify return shape +- `test_wiki_summary_returns_none_on_404` — mock 404, verify `None` +- `test_wiki_summary_returns_none_on_disambiguation` — mock disambiguation response, verify `None` +- `test_wiki_search_returns_results` — mock search API + summary fetches, verify list +- `test_wiki_search_returns_empty_on_failure` — mock network error, verify `[]` + +### `tests/test_lookup_tool.py` (new) + +- `test_lookup_wikipedia_hit` — mock `wiki_summary` returning data, verify tool returns wikipedia source +- `test_lookup_wikipedia_miss_searxng_fallback` — mock `wiki_summary` returning None, SearXNG returning results + article fetch, verify web source +- `test_lookup_wikipedia_miss_no_searxng` — mock both missing, verify graceful "no results" response +- `test_lookup_always_available` — verify the tool appears in `get_tools_for_user` regardless of SearXNG config + +### `tests/test_research_pipeline.py` (add to existing) + +- `test_research_includes_wikipedia_sources` — mock `wiki_search` alongside SearXNG, verify Wikipedia results appear in source pool + +All tests mock HTTP calls — no live API hits. + +## What Doesn't Change + +- `read_article` tool — stays as-is (explicit URL fetch, different purpose) +- `research_topic` tool definition — stays as-is (same name, description, parameters) +- `generation_task.py` research interception — stays as-is +- `search_images` tool — stays as-is +- `_search_searxng` and `_search_searxng_images` — stay as-is +- `_fetch_full_article` in `rss.py` — stays as-is, reused by `lookup` for SearXNG fallback