feat: full article fetching with trafilatura + html2text cleanup
- Add trafilatura + html2text to dependencies - Replace custom HTMLStripper with html2text for RSS feed content - Fetch full article text via httpx + trafilatura after each new item is stored; falls back to RSS-provided content if fetch/extraction fails - Raise CONTENT_MAX_CHARS from 2000 to 50000 (TEXT column, no migration needed) - Re-embed items with full article content once enrichment completes - Startup backfill enriches existing items with short content (<1000 chars) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
@@ -266,6 +266,11 @@ def create_app() -> Quart:
|
||||
await backfill_rss_item_embeddings()
|
||||
except Exception:
|
||||
logger.warning("RSS embedding backfill failed", exc_info=True)
|
||||
try:
|
||||
from fabledassistant.services.embeddings import backfill_rss_article_content
|
||||
await backfill_rss_article_content()
|
||||
except Exception:
|
||||
logger.warning("RSS article content backfill failed", exc_info=True)
|
||||
|
||||
asyncio.create_task(_delayed_backfill())
|
||||
|
||||
|
||||
Reference in New Issue
Block a user