feat: full article fetching with trafilatura + html2text cleanup

- Add trafilatura + html2text to dependencies
- Replace custom HTMLStripper with html2text for RSS feed content
- Fetch full article text via httpx + trafilatura after each new item is stored;
  falls back to RSS-provided content if fetch/extraction fails
- Raise CONTENT_MAX_CHARS from 2000 to 50000 (TEXT column, no migration needed)
- Re-embed items with full article content once enrichment completes
- Startup backfill enriches existing items with short content (<1000 chars)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
2026-03-30 16:33:27 -04:00
parent 0b05b03987
commit e613485474
4 changed files with 157 additions and 54 deletions
+2
View File
@@ -19,6 +19,8 @@ dependencies = [
"caldav>=1.3",
"icalendar>=5.0",
"feedparser>=6.0",
"html2text>=2024.2",
"trafilatura>=1.12",
"APScheduler>=3.10,<4.0",
"pywebpush>=2.0",
]