5 Commits

Include renames

Author	SHA1	Message	Date
bvandeusenandClaude Opus 4.8	96c29c370b	feat(ingest): localize inline post-body images to local copies (Phase 2) ... Render a post body faithfully by serving our stored copies of inline images instead of hotlinking the public CDN. The join key is the CDN filehash (32-hex MD5) shared between a body <img src> and the media URL we downloaded (the same identity extract_media dedups by): - utils.paths.filehash_from_url — one source of truth for the extractor; patreon_client._filehash now delegates so capture- and render-time hashing cannot drift. - ImageRecord gains source_url (provenance) + source_filehash (indexed match key); migration 0051. - the per-media sidecar carries the file's source_url; the importer persists it (NULL-only) on the ImageRecord via _apply_sidecar. - post_feed_service.get_post remaps body <img src> -> /images/<path> for every inline image whose filehash maps to a stored image of THIS artist; unmatched / pre-Phase-2 images keep hotlinking. Pre-existing on-disk images have no filehash yet, so they fall back to hotlinking until re-downloaded; localization is forward-looking. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-14 16:39:58 -04:00
bvandeusen	bd3f996582	fix(sidecar): correct external_post_id + post_url derivation for non-Patreon platforms ... Audit of one sample sidecar per platform on the operator's /mnt/Data/Patreon/ archive surfaced three parser bugs that have been silently corrupting non-Patreon Posts since FC-3 shipped: 1. SubscribeStar `id` vs `post_id` confusion. gallery-dl puts the per-attachment id in `id` (e.g. 711509) and the actual post id in `post_id` (e.g. 360360). FC's external_post_id chain had `id` winning, so every multi-image SubscribeStar post was fragmented into N Post rows in the database. Reorder the chain to `("post_id", "id", "index", "message_id")` — Patreon/Pixiv (no `post_id`), HF (uses `index`), Discord (uses `message_id`) all unaffected. 2. Discord `message` field not captured. Discord posts put the body in `message`, not `content`. Append it to the description fallback chain `("content", "description", "caption", "message")`. 3. post_url is the file URL on SubscribeStar/Pixiv/HF/Discord. New `_derive_post_url(platform, data)` helper synthesizes proper permalinks from per-platform fields: subscribestar → https://www.subscribestar.com/posts/<post_id> pixiv → https://www.pixiv.net/artworks/<id> hentaifoundry → https://www.hentai-foundry.com/pictures/user/<user>/<index> discord → https://discord.com/channels/<server>/<channel>/<message> Patreon's bare `url` IS a real permalink and is used as-is. For the four file-URL platforms, the bare `url` is NEVER trusted: derive or return None rather than persist a CDN URL. Tests: - `test_parse_core_fields_and_id_priority` flipped to assert post_id wins over id. - New `test_parse_id_used_when_no_post_id` covers the Patreon real shape. - New `test_parse_message_used_as_description_fallback` covers Discord bodies. - Five new tests cover per-platform post_url derivation (SubscribeStar/Pixiv/HF/Discord/Patreon-untouched + missing-fields → None). Cleanup migration alembic 0025_fix_subscribestar_post_ids: - For each SubscribeStar Post: find a related ImageRecord.path, walk to its sidecar JSON, read `post_id`, overwrite Post.external_post_id + post_url with the corrected values. - After all updates, every group of Posts under one source sharing the same NEW external_post_id is a fragment-set — merge to a canonical row using the same ImageProvenance pre-delete + repoint dance as alembic 0022 (banked pattern). - Pure-SQL backfill of Pixiv post_url: replace any `i.pximg.net`-shape url with the derived `/artworks/<id>` permalink. - HF and Discord post_url backfills skipped — HF would need the `user` field (not stored on Post), Discord needs server/channel triple. Both will be corrected by a deep-scan re-applying sidecars through the new parser. Idempotent: re-running on already-corrected data is a no-op.	2026-05-27 15:35:25 -04:00
bvandeusen	ae8c78ae09	fix(sidecar): synthesize post_title from content first-line when title is empty (subscribestar) ... SubscribeStar gallery-dl always writes `title: ""` and embeds the leading sentence inside `content` HTML. Confirmed against the operator's /mnt/Data/Patreon/Cheunart/subscribestar/ dump 2026-05-27: every post's JSON has `title: ""` and a content like `<div>Lets say hello to you guys with my Belle <br><br><br></div>`. FC's sidecar parser, treating empty strings as missing, had been leaving post_title NULL on every subscribestar post since FC-3 shipped. Fix at two layers: 1. `backend/app/utils/sidecar.py` — new `_first_line_text(body, limit)` helper strips HTML tags, collapses whitespace, returns the first non-empty line truncated to 120 chars with ellipsis. `parse_sidecar` now falls back to this when `title` resolves to None and a `content`/`description`/`caption` value is present. Patreon's non-empty titles short-circuit the fallback so existing behavior is unchanged. Four new tests in test_sidecar_util.py pin: derivation from content, truncation at 120 chars, explicit-title precedence, no-content no-fallback. 2. `alembic 0024_backfill_post_title_from_description` — backfills the same logic across existing Post rows where `post_title IS NULL OR post_title = ''` AND description is present. Idempotent (re-running is a no-op once titles are populated). Downgrade is a no-op since there's no safe way to tell derived rows from genuine ones. After deploy + migration: subscribestar posts will surface a meaningful title in PostCard, post feed search, etc.	2026-05-27 14:44:09 -04:00
bvandeusen	0978fbac66	fix(sidecar): strip gallery-dl 'NN_' numbering prefix when locating sidecars — fixes 'deep scan refresh count high but 0 Posts created' (operator-flagged 2026-05-26) — Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-26 00:27:43 -04:00
bvandeusenandClaude Opus 4.7	ef7e4d92a9	feat(sidecar): minimal gallery-dl sidecar find + parse util ... Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-18 14:44:37 -04:00