Files
FabledScribe/docs/superpowers/plans/2026-04-05-article-reading.md
T

696 lines
23 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Article Reading Implementation Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** Add a `read_article` tool so the LLM can fetch any URL, fix the history builder so tool context survives follow-up turns, redesign the Discuss button to inject article content as a persisted tool exchange, and remove the RSS content character cap.
**Architecture:** Four independent changes executed in dependency order: (1) content cap removal, (2) `read_article` tool, (3) history builder fix (prerequisite for everything persisting across follow-ups), (4) Discuss endpoint + frontend. Each task is independently committable.
**Tech Stack:** Python/Quart, SQLAlchemy async, trafilatura (already installed), httpx (already installed), Vue 3 + TypeScript frontend.
---
## File map
| Action | Path | Responsibility |
|---|---|---|
| Modify | `src/fabledassistant/services/rss.py` | Remove `CONTENT_MAX_CHARS` truncation |
| Modify | `src/fabledassistant/services/tools.py` | Add `_URL_TOOLS` list, add `read_article` to `get_tools_for_user`, add handler in `execute_tool` |
| Modify | `src/fabledassistant/routes/chat.py` | Fix history builder to replay tool_calls |
| Modify | `src/fabledassistant/services/chat.py` | Add `tool_calls` parameter to `add_message` |
| Modify | `src/fabledassistant/routes/briefing.py` | Add `POST /api/briefing/articles/<item_id>/discuss` endpoint |
| Modify | `frontend/src/views/BriefingView.vue` | Replace `discussArticle()` to call new endpoint |
| Modify | `tests/test_rss_service.py` | Update truncation test, add no-truncation test |
| Create | `tests/test_article_reading.py` | Tests for `read_article` tool and history builder |
---
## Task 1: Remove RSS content cap
**Files:**
- Modify: `src/fabledassistant/services/rss.py:17-18,83,213`
- Modify: `tests/test_rss_service.py:19-26`
The `CONTENT_MAX_CHARS = 50_000` constant and all uses of `[:CONTENT_MAX_CHARS]` are removed.
Trafilatura extracts only article body text, so content is naturally bounded.
- [ ] **Step 1: Update the truncation test to assert no truncation**
In `tests/test_rss_service.py`, replace the existing `test_extract_item_truncates_content` test:
```python
def test_extract_item_does_not_truncate_content():
"""extract_item() should store content without truncation."""
from fabledassistant.services.rss import extract_item
long_text = "x" * 100_000
entry = MagicMock()
entry.get = lambda k, d="": {"summary": long_text, "title": "", "link": "", "id": "g"}.get(k, d)
entry.content = []
entry.published_parsed = None
item = extract_item(entry)
assert len(item["content"]) == 100_000
```
- [ ] **Step 2: Run the test to confirm it fails**
```bash
cd /home/bvandeusen/Nextcloud/Projects/fabledassistant
make test ARGS="tests/test_rss_service.py::test_extract_item_does_not_truncate_content -v"
```
Expected: FAIL (current code truncates to 50_000).
- [ ] **Step 3: Remove CONTENT_MAX_CHARS from rss.py**
In `src/fabledassistant/services/rss.py`:
Remove lines 1718:
```python
# Safety cap on stored content — effectively unlimited for typical articles
CONTENT_MAX_CHARS = 50_000
```
Change line 83 from:
```python
content = _html_to_text(content)[:CONTENT_MAX_CHARS]
```
to:
```python
content = _html_to_text(content)
```
Change line 213 from:
```python
item.content = full_text[:CONTENT_MAX_CHARS]
```
to:
```python
item.content = full_text
```
- [ ] **Step 4: Run all rss tests**
```bash
make test ARGS="tests/test_rss_service.py -v"
```
Expected: all pass. The `test_extract_item_truncates_content` test name no longer exists (replaced in Step 1).
- [ ] **Step 5: Commit**
```bash
git add src/fabledassistant/services/rss.py tests/test_rss_service.py
git commit -m "feat(rss): remove article content character cap"
```
---
## Task 2: Add `read_article` tool
**Files:**
- Modify: `src/fabledassistant/services/tools.py`
- Create: `tests/test_article_reading.py`
The tool uses `_fetch_full_article` from `rss.py` (lazy import inside `execute_tool` to avoid circular dependencies). Added unconditionally to all users via a new `_URL_TOOLS` list.
- [ ] **Step 1: Write failing tests**
Create `tests/test_article_reading.py`:
```python
import json
import pytest
from unittest.mock import AsyncMock, patch
@pytest.mark.asyncio
async def test_read_article_success():
"""read_article tool returns article content on success."""
from fabledassistant.services.tools import execute_tool
with patch(
"fabledassistant.services.rss._fetch_full_article",
new=AsyncMock(return_value="Article text here."),
):
result = await execute_tool(
user_id=1,
tool_name="read_article",
arguments={"url": "https://example.com/article"},
)
assert result["success"] is True
assert result["type"] == "article_content"
assert result["url"] == "https://example.com/article"
assert result["content"] == "Article text here."
assert result["truncated"] is False
@pytest.mark.asyncio
async def test_read_article_fetch_failure():
"""read_article tool returns success=False when fetch returns None."""
from fabledassistant.services.tools import execute_tool
with patch(
"fabledassistant.services.rss._fetch_full_article",
new=AsyncMock(return_value=None),
):
result = await execute_tool(
user_id=1,
tool_name="read_article",
arguments={"url": "https://example.com/bad"},
)
assert result["success"] is False
assert "Could not fetch" in result["error"]
@pytest.mark.asyncio
async def test_read_article_truncates_at_40k():
"""read_article tool truncates content at 40_000 chars and sets truncated=True."""
from fabledassistant.services.tools import execute_tool
long_content = "x" * 50_000
with patch(
"fabledassistant.services.rss._fetch_full_article",
new=AsyncMock(return_value=long_content),
):
result = await execute_tool(
user_id=1,
tool_name="read_article",
arguments={"url": "https://example.com/long"},
)
assert result["success"] is True
assert len(result["content"]) == 40_000
assert result["truncated"] is True
@pytest.mark.asyncio
async def test_read_article_empty_url():
"""read_article tool returns success=False when url is empty."""
from fabledassistant.services.tools import execute_tool
result = await execute_tool(
user_id=1,
tool_name="read_article",
arguments={"url": ""},
)
assert result["success"] is False
```
- [ ] **Step 2: Run tests to confirm they fail**
```bash
make test ARGS="tests/test_article_reading.py -v"
```
Expected: all 4 fail with "read_article not handled" or AttributeError.
- [ ] **Step 3: Add `_URL_TOOLS` list and register it in `get_tools_for_user`**
In `src/fabledassistant/services/tools.py`, add the `_URL_TOOLS` list immediately after the `_SEARCH_TOOLS` block (around line 836):
```python
_URL_TOOLS = [
{
"type": "function",
"function": {
"name": "read_article",
"description": (
"Fetch and read the full text of a web page or article from a URL. "
"Use when the user shares a URL and wants you to read it, or to get "
"the full content of a linked page. "
"Do NOT use search_web for URLs — use this tool instead."
),
"parameters": {
"type": "object",
"properties": {
"url": {"type": "string", "description": "The URL to fetch and read"}
},
"required": ["url"],
},
},
}
]
```
In `get_tools_for_user` (around line 1034), add `_URL_TOOLS` unconditionally after `_CORE_TOOLS`:
```python
async def get_tools_for_user(user_id: int) -> list[dict]:
"""Build the tool list for a user based on their configured integrations."""
tools = list(_CORE_TOOLS)
tools.extend(_URL_TOOLS)
tools.extend(_RAG_TOOLS)
tools.extend(_ENTITY_TOOLS)
if await is_caldav_configured(user_id):
tools.extend(_CALDAV_TOOLS)
if Config.searxng_enabled():
tools.extend(_SEARCH_TOOLS)
tools.extend(_RESEARCH_TOOLS)
tools.extend(_IMAGE_TOOLS)
logger.debug("User %d: %d tools available", user_id, len(tools))
return tools
```
- [ ] **Step 4: Add `read_article` handler in `execute_tool`**
In `src/fabledassistant/services/tools.py`, in the `execute_tool` function, find the `elif tool_name == "search_web":` block (around line 1771). Add the new handler immediately before it:
```python
elif tool_name == "read_article":
from fabledassistant.services.rss import _fetch_full_article
url = arguments.get("url", "").strip()
if not url:
return {"success": False, "error": "No URL provided"}
content = await _fetch_full_article(url)
if not content:
return {"success": False, "error": f"Could not fetch article content from {url}"}
_TOOL_CONTENT_CAP = 40_000
truncated = len(content) > _TOOL_CONTENT_CAP
return {
"success": True,
"type": "article_content",
"url": url,
"content": content[:_TOOL_CONTENT_CAP],
"truncated": truncated,
}
```
- [ ] **Step 5: Run the tests**
```bash
make test ARGS="tests/test_article_reading.py -v"
```
Expected: all 4 pass.
- [ ] **Step 6: Run full test suite**
```bash
make test
```
Expected: all pass.
- [ ] **Step 7: Commit**
```bash
git add src/fabledassistant/services/tools.py tests/test_article_reading.py
git commit -m "feat(tools): add read_article tool using trafilatura extraction"
```
---
## Task 3: Fix history builder
**Files:**
- Modify: `src/fabledassistant/routes/chat.py:162-166`
- Modify: `tests/test_article_reading.py` (add history builder tests)
The loop that builds `history` for `run_generation` currently drops `tool_calls`. This fix replays the full tool exchange so the LLM sees prior tool results on follow-up turns.
- [ ] **Step 1: Add history builder tests**
Append to `tests/test_article_reading.py`:
```python
def test_history_builder_plain_messages():
"""Messages without tool_calls are added as {role, content} unchanged."""
import json
messages = [
type("M", (), {"role": "system", "content": "sys", "tool_calls": None})(),
type("M", (), {"role": "user", "content": "hello", "tool_calls": None})(),
type("M", (), {"role": "assistant", "content": "hi", "tool_calls": None})(),
]
history = _build_history(messages)
assert history == [
{"role": "user", "content": "hello"},
{"role": "assistant", "content": "hi"},
]
def test_history_builder_with_tool_calls():
"""Messages with tool_calls emit an assistant entry + tool result entries."""
import json
tool_calls_data = [
{
"function": "read_article",
"arguments": {"url": "https://example.com"},
"result": {"success": True, "content": "Article text"},
}
]
messages = [
type("M", (), {"role": "user", "content": "read this", "tool_calls": None})(),
type("M", (), {
"role": "assistant",
"content": "",
"tool_calls": tool_calls_data,
})(),
type("M", (), {"role": "user", "content": "follow up", "tool_calls": None})(),
]
history = _build_history(messages)
assert history[0] == {"role": "user", "content": "read this"}
assert history[1]["role"] == "assistant"
assert history[1]["tool_calls"] == [
{"function": {"name": "read_article", "arguments": {"url": "https://example.com"}}}
]
assert history[2] == {"role": "tool", "content": json.dumps({"success": True, "content": "Article text"})}
assert history[3] == {"role": "user", "content": "follow up"}
def _build_history(messages):
"""Inline copy of the fixed history builder for testing."""
import json
history = []
for msg in messages:
if msg.role == "system":
continue
msg_dict = {"role": msg.role, "content": msg.content or ""}
if msg.tool_calls:
msg_dict["tool_calls"] = [
{"function": {"name": tc["function"], "arguments": tc["arguments"]}}
for tc in msg.tool_calls
]
history.append(msg_dict)
for tc in msg.tool_calls:
history.append({"role": "tool", "content": json.dumps(tc.get("result", {}))})
else:
history.append(msg_dict)
return history
```
- [ ] **Step 2: Run the tests to confirm they pass**
(These tests use `_build_history` defined inline — they test the logic directly, not the route. They should pass immediately.)
```bash
make test ARGS="tests/test_article_reading.py::test_history_builder_plain_messages tests/test_article_reading.py::test_history_builder_with_tool_calls -v"
```
Expected: both pass.
- [ ] **Step 3: Apply the fix to `chat.py`**
In `src/fabledassistant/routes/chat.py`, replace lines 162166:
```python
# Build history from existing messages (excluding system and the placeholder)
history = []
for msg in conv.messages:
if msg.role != "system":
history.append({"role": msg.role, "content": msg.content})
```
with:
```python
# Build history from existing messages (excluding system and the placeholder).
# Tool calls from prior turns are replayed as assistant tool_call + tool result
# messages so the LLM retains tool context on follow-up turns.
history = []
for msg in conv.messages:
if msg.role == "system":
continue
msg_dict = {"role": msg.role, "content": msg.content or ""}
if msg.tool_calls:
msg_dict["tool_calls"] = [
{"function": {"name": tc["function"], "arguments": tc["arguments"]}}
for tc in msg.tool_calls
]
history.append(msg_dict)
for tc in msg.tool_calls:
history.append({"role": "tool", "content": json.dumps(tc.get("result", {}))})
else:
history.append(msg_dict)
```
`json` is already imported at the top of `chat.py`.
- [ ] **Step 4: Run full test suite**
```bash
make test
```
Expected: all pass.
- [ ] **Step 5: Commit**
```bash
git add src/fabledassistant/routes/chat.py tests/test_article_reading.py
git commit -m "fix(chat): replay tool_calls in history so tool context survives follow-up turns"
```
---
## Task 4: Extend `add_message` to accept `tool_calls`
**Files:**
- Modify: `src/fabledassistant/services/chat.py:183-207`
The Discuss endpoint (Task 5) needs to store a synthetic assistant message with `tool_calls`. The existing `add_message` doesn't support this parameter.
- [ ] **Step 1: Update `add_message` signature and body**
In `src/fabledassistant/services/chat.py`, replace the `add_message` function (lines 183207):
```python
async def add_message(
conversation_id: int,
role: str,
content: str,
context_note_id: int | None = None,
status: str | None = None,
tool_calls: list | None = None,
) -> Message:
async with async_session() as session:
kwargs: dict = dict(
conversation_id=conversation_id,
role=role,
content=content,
context_note_id=context_note_id,
)
if status is not None:
kwargs["status"] = status
if tool_calls is not None:
kwargs["tool_calls"] = tool_calls
msg = Message(**kwargs)
session.add(msg)
# Touch conversation updated_at
conv = await session.get(Conversation, conversation_id)
if conv:
conv.updated_at = datetime.now(timezone.utc)
await session.commit()
await session.refresh(msg)
return msg
```
- [ ] **Step 2: Run full test suite**
```bash
make test
```
Expected: all pass (existing callers only use positional/keyword args that are unchanged).
- [ ] **Step 3: Commit**
```bash
git add src/fabledassistant/services/chat.py
git commit -m "feat(chat): add tool_calls parameter to add_message"
```
---
## Task 5: Add Discuss endpoint and update frontend
**Files:**
- Modify: `src/fabledassistant/routes/briefing.py`
- Modify: `frontend/src/views/BriefingView.vue`
New route: `POST /api/briefing/articles/<item_id>/discuss`. Fetches stored article from DB, stores a synthetic `read_article` tool exchange plus the user message, then triggers generation. Frontend replaces the inline-content approach with a call to this endpoint.
- [ ] **Step 1: Add the discuss endpoint to briefing.py**
At the top of `src/fabledassistant/routes/briefing.py`, add these imports (after the existing imports):
```python
from fabledassistant.models.rss_feed import RssItem, RssFeed
from fabledassistant.services.chat import add_message, get_conversation
from fabledassistant.services.generation_buffer import GenerationState, create_buffer, get_buffer
from fabledassistant.services.generation_task import run_generation
from fabledassistant.services.settings import get_setting
```
Note: `get_setting` and `asyncio` are already imported. Add only what is missing.
Then add the new route at the end of `briefing.py` (before any final lines), after the `list_news` route:
```python
@briefing_bp.route("/articles/<int:item_id>/discuss", methods=["POST"])
@_REQUIRE
async def discuss_article(item_id: int):
"""Pre-load a briefing article as a read_article tool exchange and trigger generation."""
uid = g.user.id
data = await request.get_json() or {}
conv_id = data.get("conv_id")
if not conv_id:
return jsonify({"error": "conv_id is required"}), 400
# Verify article belongs to this user (via feed ownership)
async with async_session() as session:
result = await session.execute(
select(RssItem).join(RssFeed, RssItem.feed_id == RssFeed.id)
.where(RssItem.id == item_id, RssFeed.user_id == uid)
)
item = result.scalar_one_or_none()
if item is None:
return jsonify({"error": "Article not found"}), 404
# Verify conversation belongs to this user
conv = await get_conversation(uid, conv_id)
if conv is None:
return jsonify({"error": "Conversation not found"}), 404
# Reject if generation already running
existing = get_buffer(conv_id)
if existing and existing.state == GenerationState.RUNNING:
return jsonify({"error": "Generation already in progress"}), 409
article_content = item.content or ""
# Store synthetic assistant message: read_article was already called with stored content
synthetic_tool_calls = [
{
"function": "read_article",
"arguments": {"url": item.url},
"result": {
"success": True,
"type": "article_content",
"url": item.url,
"content": article_content,
"truncated": False,
},
}
]
await add_message(conv_id, "assistant", "", status="complete", tool_calls=synthetic_tool_calls)
# Store user message
await add_message(conv_id, "user", "Please summarize and discuss this article.")
# Reload conversation so history includes the two new messages
conv = await get_conversation(uid, conv_id)
# Build history (using the fixed builder from chat.py logic — duplicated here)
history = []
for msg in conv.messages:
if msg.role == "system":
continue
msg_dict = {"role": msg.role, "content": msg.content or ""}
if msg.tool_calls:
msg_dict["tool_calls"] = [
{"function": {"name": tc["function"], "arguments": tc["arguments"]}}
for tc in msg.tool_calls
]
history.append(msg_dict)
for tc in msg.tool_calls:
history.append({"role": "tool", "content": json.dumps(tc.get("result", {}))})
else:
history.append(msg_dict)
model = await get_setting(uid, "default_model", "") or ""
from fabledassistant.config import Config as _Config
if not model:
model = _Config.OLLAMA_MODEL
# Create placeholder assistant message and generation buffer
assistant_msg = await add_message(conv_id, "assistant", "", status="generating")
try:
buf = create_buffer(conv_id, assistant_msg.id)
except RuntimeError:
return jsonify({"error": "Generation already in progress"}), 409
asyncio.create_task(run_generation(
buf, history, model,
uid, conv_id, conv.title,
"Please summarize and discuss this article.",
think=True,
))
return jsonify({"assistant_message_id": assistant_msg.id, "status": "generating"}), 202
```
- [ ] **Step 2: Run full test suite**
```bash
make test
```
Expected: all pass.
- [ ] **Step 3: Update `discussArticle` in BriefingView.vue**
In `frontend/src/views/BriefingView.vue`, replace the `discussArticle` function:
```typescript
async function discussArticle(item: NewsItem) {
if (!todayConvId.value || chatStore.streaming) return
if (!isToday.value) selectedConvId.value = todayConvId.value
await nextTick(() => {
document.querySelector('.briefing-center')?.scrollIntoView({ behavior: 'smooth', block: 'nearest' })
})
try {
await apiPost<{ assistant_message_id: number }>(
`/api/briefing/articles/${item.id}/discuss`,
{ conv_id: todayConvId.value },
)
} catch {
return
}
// Reload conversation so the new messages appear (including the generating placeholder),
// then reconnect to the SSE stream using the existing reconnectIfGenerating helper.
await chatStore.fetchConversation(todayConvId.value)
await chatStore.reconnectIfGenerating(todayConvId.value)
}
```
`reconnectIfGenerating` is already exported from `useChatStore`. It finds the assistant message in `status="generating"` state and connects to the SSE stream automatically. No changes to `chat.ts` are needed.
- [ ] **Step 4: TypeScript check**
```bash
cd /home/bvandeusen/Nextcloud/Projects/fabledassistant
npm --prefix frontend run type-check
```
Expected: no errors.
- [ ] **Step 5: Commit**
```bash
git add src/fabledassistant/routes/briefing.py frontend/src/views/BriefingView.vue frontend/src/stores/chat.ts
git commit -m "feat(briefing): add discuss endpoint and update frontend to use persisted article context"
```
---
## Task 6: Final verification
- [ ] **Step 1: Run full test suite**
```bash
cd /home/bvandeusen/Nextcloud/Projects/fabledassistant
make test
```
Expected: all tests pass.
- [ ] **Step 2: TypeScript check**
```bash
npm --prefix frontend run type-check
```
Expected: no errors.
- [ ] **Step 3: Push**
```bash
git push origin dev
```