- Add trafilatura + html2text to dependencies
- Replace custom HTMLStripper with html2text for RSS feed content
- Fetch full article text via httpx + trafilatura after each new item is stored;
falls back to RSS-provided content if fetch/extraction fails
- Raise CONTENT_MAX_CHARS from 2000 to 50000 (TEXT column, no migration needed)
- Re-embed items with full article content once enrichment completes
- Startup backfill enriches existing items with short content (<1000 chars)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
feedparser returns HTML in content/summary fields for many feeds.
Raw tags were being stored in the DB and passed to the LLM/embeddings.
Added a stdlib HTMLParser-based stripper in extract_item() — block elements
become newlines, script/style content is dropped, plain text passes through.
No new dependencies required.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Embed RSS items at fetch time (nomic-embed-text); backfill at startup
- Semantic news search injected into chat system prompt ("Recent News You've Seen")
when items match query above 0.55 cosine threshold (independent of note RAG)
- "Discuss in chat" button on news cards — creates a seeded conversation with
the article title + full content, navigates directly to the new chat
- Briefing compilation now passes 500-char article excerpts (not just headlines)
to the LLM and uses 8192 num_ctx to accommodate the larger prompt
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
bvandeusen