120 lines
5.9 KiB
Markdown
120 lines
5.9 KiB
Markdown
# Unified Lookup Tool & Wikipedia Integration
|
|
|
|
## Goal
|
|
|
|
Replace the fragmented `search_web` tool with a single `lookup` tool that checks Wikipedia first and falls back to SearXNG web search. Add Wikipedia as an additional source in the research pipeline. Result: one lightweight tool for factual questions (always available, no config required), and richer research output.
|
|
|
|
## Architecture
|
|
|
|
Two changes to the search/knowledge stack:
|
|
|
|
1. **New `lookup` tool** replaces `search_web`. Tries Wikipedia REST API summary endpoint first (~200ms, reliable, no config). Falls back to SearXNG + trafilatura article fetch when Wikipedia misses and SearXNG is configured. Always available (no `requires` field).
|
|
|
|
2. **Wikipedia sources in research pipeline.** During sub-query execution, `wiki_search` runs alongside `_search_searxng`. Wikipedia articles merge into the source pool and get deduplicated by URL.
|
|
|
|
Shared Wikipedia logic lives in a new `wikipedia.py` service module.
|
|
|
|
## Components
|
|
|
|
### `src/fabledassistant/services/wikipedia.py` (new)
|
|
|
|
Two async functions:
|
|
|
|
**`wiki_summary(query: str) -> dict | None`**
|
|
- Direct title lookup via `https://en.wikipedia.org/api/rest_v1/page/summary/{title}`
|
|
- Returns `{"title": str, "extract": str, "url": str}` on hit
|
|
- Returns `None` on 404, disambiguation pages (`"type": "disambiguation"`), network errors, or empty extracts
|
|
- 5-second timeout
|
|
- User-Agent: `"FabledAssistant/1.0 (https://fabledsword.com)"`
|
|
|
|
**`wiki_search(query: str, limit: int = 3) -> list[dict]`**
|
|
- Search via `https://en.wikipedia.org/w/api.php?action=query&list=search&srsearch={query}&srlimit={limit}&format=json`
|
|
- For each search result, fetch its summary via the summary endpoint to get the extract
|
|
- Returns `[{"title": str, "extract": str, "url": str}, ...]`
|
|
- Returns `[]` on any failure
|
|
- Same timeout and User-Agent as above
|
|
|
|
### `src/fabledassistant/services/tools/web.py` (modified)
|
|
|
|
**Remove:** `search_web_tool`
|
|
|
|
**Add:** `lookup_tool`
|
|
|
|
```
|
|
@tool(
|
|
name="lookup",
|
|
description="Look up a topic, concept, or factual question. Returns a concise
|
|
answer from Wikipedia or web sources. Use for definitions,
|
|
explanations, 'what is X', 'how does Y work'. For comprehensive
|
|
written reports saved as notes, use research_topic instead.",
|
|
parameters={
|
|
"query": {"type": "string", "description": "The topic or question to look up"},
|
|
},
|
|
required=["query"],
|
|
)
|
|
```
|
|
|
|
No `requires` field — always available.
|
|
|
|
**Logic:**
|
|
1. Call `wiki_summary(query)`
|
|
2. If Wikipedia returns a result: return `{"success": True, "type": "lookup", "source": "wikipedia", "data": {"title": ..., "extract": ..., "url": ...}}`
|
|
3. If Wikipedia misses and `Config.searxng_enabled()`:
|
|
- Call `_search_searxng(query)` to get search results
|
|
- Fetch top 1-2 result URLs via `_fetch_full_article` (from `rss.py`, trafilatura-based)
|
|
- Return `{"success": True, "type": "lookup", "source": "web", "data": {"query": ..., "results": [...], "content": ...}}`
|
|
4. If Wikipedia misses and no SearXNG: return `{"success": True, "type": "lookup", "source": "none", "data": {"query": ..., "message": "No results found. You can answer from your own knowledge."}}`
|
|
|
|
### `src/fabledassistant/services/research.py` (modified)
|
|
|
|
**In Step 2 (parallel search):**
|
|
- For each sub-query, run `wiki_search(query, limit=1)` concurrently with `_search_searxng(query)`
|
|
- Merge Wikipedia results into the per-query result list
|
|
|
|
**In Step 3 (deduplication):**
|
|
- When deduplicating URLs, Wikipedia URLs (`wikipedia.org`) are checked against SearXNG results
|
|
- If a Wikipedia article URL already appears in SearXNG results, skip the duplicate
|
|
|
|
**Wikipedia article content for synthesis:**
|
|
- The `extract` from `wiki_search` is used as the source content (no additional fetch needed, unlike SearXNG URLs which require `fetch_url_content`)
|
|
- This means Wikipedia sources are available immediately without an HTTP fetch step
|
|
|
|
## Error Handling
|
|
|
|
- All Wikipedia API failures (network, timeout, malformed JSON) return `None`/`[]` silently
|
|
- `lookup` never raises — always returns a response the model can work with
|
|
- In the research pipeline, Wikipedia is purely additive; its failure never degrades existing SearXNG-based research
|
|
- Disambiguation pages are detected via `"type": "disambiguation"` in the summary response and treated as a miss
|
|
|
|
## Testing
|
|
|
|
### `tests/test_wikipedia.py` (new)
|
|
|
|
- `test_wiki_summary_returns_extract` — mock successful summary response, verify return shape
|
|
- `test_wiki_summary_returns_none_on_404` — mock 404, verify `None`
|
|
- `test_wiki_summary_returns_none_on_disambiguation` — mock disambiguation response, verify `None`
|
|
- `test_wiki_search_returns_results` — mock search API + summary fetches, verify list
|
|
- `test_wiki_search_returns_empty_on_failure` — mock network error, verify `[]`
|
|
|
|
### `tests/test_lookup_tool.py` (new)
|
|
|
|
- `test_lookup_wikipedia_hit` — mock `wiki_summary` returning data, verify tool returns wikipedia source
|
|
- `test_lookup_wikipedia_miss_searxng_fallback` — mock `wiki_summary` returning None, SearXNG returning results + article fetch, verify web source
|
|
- `test_lookup_wikipedia_miss_no_searxng` — mock both missing, verify graceful "no results" response
|
|
- `test_lookup_always_available` — verify the tool appears in `get_tools_for_user` regardless of SearXNG config
|
|
|
|
### `tests/test_research_pipeline.py` (add to existing)
|
|
|
|
- `test_research_includes_wikipedia_sources` — mock `wiki_search` alongside SearXNG, verify Wikipedia results appear in source pool
|
|
|
|
All tests mock HTTP calls — no live API hits.
|
|
|
|
## What Doesn't Change
|
|
|
|
- `read_article` tool — stays as-is (explicit URL fetch, different purpose)
|
|
- `research_topic` tool definition — stays as-is (same name, description, parameters)
|
|
- `generation_task.py` research interception — stays as-is
|
|
- `search_images` tool — stays as-is
|
|
- `_search_searxng` and `_search_searxng_images` — stay as-is
|
|
- `_fetch_full_article` in `rss.py` — stays as-is, reused by `lookup` for SearXNG fallback
|