feat: full article fetching with trafilatura + html2text cleanup
- Add trafilatura + html2text to dependencies - Replace custom HTMLStripper with html2text for RSS feed content - Fetch full article text via httpx + trafilatura after each new item is stored; falls back to RSS-provided content if fetch/extraction fails - Raise CONTENT_MAX_CHARS from 2000 to 50000 (TEXT column, no migration needed) - Re-embed items with full article content once enrichment completes - Startup backfill enriches existing items with short content (<1000 chars) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
@@ -19,6 +19,8 @@ dependencies = [
|
||||
"caldav>=1.3",
|
||||
"icalendar>=5.0",
|
||||
"feedparser>=6.0",
|
||||
"html2text>=2024.2",
|
||||
"trafilatura>=1.12",
|
||||
"APScheduler>=3.10,<4.0",
|
||||
"pywebpush>=2.0",
|
||||
]
|
||||
|
||||
Reference in New Issue
Block a user