bvandeusen/FabledCurator
1
0
Fork 0
Code Issues 1 Pull Requests Actions Packages Projects Releases 33 Wiki Activity

fix(sidecar): correct external_post_id + post_url derivation for non-Patreon platforms

Browse Source 
Audit of one sample sidecar per platform on the operator's
/mnt/Data/Patreon/ archive surfaced three parser bugs that have been
silently corrupting non-Patreon Posts since FC-3 shipped:

1. SubscribeStar `id` vs `post_id` confusion. gallery-dl puts the
   per-attachment id in `id` (e.g. 711509) and the actual post id in
   `post_id` (e.g. 360360). FC's external_post_id chain had `id`
   winning, so every multi-image SubscribeStar post was fragmented into
   N Post rows in the database. Reorder the chain to
   `("post_id", "id", "index", "message_id")` — Patreon/Pixiv (no
   `post_id`), HF (uses `index`), Discord (uses `message_id`) all
   unaffected.

2. Discord `message` field not captured. Discord posts put the body in
   `message`, not `content`. Append it to the description fallback chain
   `("content", "description", "caption", "message")`.

3. post_url is the file URL on SubscribeStar/Pixiv/HF/Discord. New
   `_derive_post_url(platform, data)` helper synthesizes proper
   permalinks from per-platform fields:
     subscribestar → https://www.subscribestar.com/posts/<post_id>
     pixiv         → https://www.pixiv.net/artworks/<id>
     hentaifoundry → https://www.hentai-foundry.com/pictures/user/<user>/<index>
     discord       → https://discord.com/channels/<server>/<channel>/<message>
   Patreon's bare `url` IS a real permalink and is used as-is. For the
   four file-URL platforms, the bare `url` is NEVER trusted: derive or
   return None rather than persist a CDN URL.

Tests:
- `test_parse_core_fields_and_id_priority` flipped to assert post_id
  wins over id.
- New `test_parse_id_used_when_no_post_id` covers the Patreon real
  shape.
- New `test_parse_message_used_as_description_fallback` covers Discord
  bodies.
- Five new tests cover per-platform post_url derivation
  (SubscribeStar/Pixiv/HF/Discord/Patreon-untouched + missing-fields →
  None).

Cleanup migration alembic 0025_fix_subscribestar_post_ids:
- For each SubscribeStar Post: find a related ImageRecord.path, walk to
  its sidecar JSON, read `post_id`, overwrite Post.external_post_id +
  post_url with the corrected values.
- After all updates, every group of Posts under one source sharing the
  same NEW external_post_id is a fragment-set — merge to a canonical
  row using the same ImageProvenance pre-delete + repoint dance as
  alembic 0022 (banked pattern).
- Pure-SQL backfill of Pixiv post_url: replace any `i.pximg.net`-shape
  url with the derived `/artworks/<id>` permalink.
- HF and Discord post_url backfills skipped — HF would need the `user`
  field (not stored on Post), Discord needs server/channel triple.
  Both will be corrected by a deep-scan re-applying sidecars through
  the new parser.

Idempotent: re-running on already-corrected data is a no-op.
This commit is contained in:
Bryan Van Deusen
2026-05-27 15:35:25 -04:00
parent ae8c78ae09
commit bd3f996582
3 changed files with 412 additions and 5 deletions
Download Patch File Download Diff File Expand all files Collapse all files
tests/test_sidecar_util.py
+100 -2
View File
@@ -78,6 +78,10 @@ def test_parse_empty_dict_all_none():
def test_parse_core_fields_and_id_priority():
"""`post_id` MUST win over `id` (SubscribeStar duplicate-post fix).
Patreon sidecars in the wild don't expose post_id; this test uses
`category=patreon` but synthetically sets both fields to pin the
parser's precedence."""
sd = parse_sidecar({
"category": "patreon",
"id": 12345, "post_id": 999,
@@ -88,7 +92,7 @@ def test_parse_core_fields_and_id_priority():
"published_at": "2023-08-01T04:20:02Z",
})
assert sd.platform == "patreon"
assert sd.external_post_id == "12345" # 'id' wins over 'post_id'
assert sd.external_post_id == "999" # 'post_id' wins over 'id'
assert sd.post_url == "https://patreon.com/posts/12345"
assert sd.post_title == "Hello"
assert sd.description == "<p>body</p>"
@@ -96,13 +100,25 @@ def test_parse_core_fields_and_id_priority():
assert sd.post_date.year == 2023 and sd.post_date.tzinfo is not None
def test_parse_id_used_when_no_post_id():
"""Without post_id (Patreon/Pixiv/Discord real shape), `id` wins."""
sd = parse_sidecar({"id": 12345, "url": "https://example.test/p/1"})
assert sd.external_post_id == "12345"
def test_parse_description_precedence_and_images_count():
sd = parse_sidecar({"description": "d", "caption": "c",
"images": [1, 2, 3]})
assert sd.description == "d" # content>description>caption
assert sd.description == "d" # content>description>caption>message
assert sd.attachment_count == 3 # len(images) fallback
def test_parse_message_used_as_description_fallback():
"""Discord posts have `message` not `content`; FC must surface it."""
sd = parse_sidecar({"category": "discord", "message": "hello channel"})
assert sd.description == "hello channel"
def test_parse_date_epoch_and_unparseable_and_naive():
assert parse_sidecar({"timestamp": 1690857602}).post_date.tzinfo is not None
assert parse_sidecar({"date": "not-a-date"}).post_date is None
@@ -143,6 +159,88 @@ def test_parse_title_no_fallback_when_no_content():
assert sd.post_title is None
def test_parse_subscribestar_post_url_derived_and_post_id_wins():
"""SubscribeStar gallery-dl puts the per-attachment id in `id` and
the actual post id in `post_id`. The bare `url` is the file URL —
must be ignored and a derived permalink used instead."""
sd = parse_sidecar({
"category": "subscribestar",
"id": 711509, "post_id": 360360,
"url": "/post_uploads?payload=opaque",
"title": "",
"content": "<div>hello</div>",
})
assert sd.external_post_id == "360360"
assert sd.post_url == "https://www.subscribestar.com/posts/360360"
def test_parse_pixiv_post_url_derived():
"""Pixiv's `url` is the image URL (i.pximg.net); must be replaced
with the post permalink under /artworks/<id>."""
sd = parse_sidecar({
"category": "pixiv",
"id": 140466853,
"url": "https://i.pximg.net/img-original/img/2026/01/28/10/28/24/140466853_p0.jpg",
"title": "Nerissa x Jailbird",
})
assert sd.external_post_id == "140466853"
assert sd.post_url == "https://www.pixiv.net/artworks/140466853"
def test_parse_hentaifoundry_post_url_derived():
"""HF sidecars omit `url` entirely and use `index`+`user` for the
post's natural key. Synthesize the canonical /pictures/user/<u>/<i>
permalink."""
sd = parse_sidecar({
"category": "hentaifoundry",
"index": 1182595,
"user": "HolyMeh",
"title": "Annigosa",
})
assert sd.external_post_id == "1182595"
assert sd.post_url == "https://www.hentai-foundry.com/pictures/user/HolyMeh/1182595"
def test_parse_discord_post_url_derived_and_message_id_wins():
"""Discord posts use `message_id` for the post key and the
server/channel/message triple for the permalink."""
sd = parse_sidecar({
"category": "discord",
"message_id": "1195924119762505818",
"channel_id": "968315530597498880",
"server_id": "771088957849075793",
"message": "channel body text",
"url": "https://cdn.discordapp.com/attachments/file.png",
})
assert sd.external_post_id == "1195924119762505818"
assert sd.post_url == (
"https://discord.com/channels/771088957849075793/"
"968315530597498880/1195924119762505818"
)
assert sd.description == "channel body text"
def test_parse_patreon_post_url_kept_as_is():
"""Patreon's bare `url` IS a real permalink — must not be replaced."""
sd = parse_sidecar({
"category": "patreon",
"id": 47074733,
"url": "https://www.patreon.com/posts/barbara-genshin-47074733",
})
assert sd.post_url == "https://www.patreon.com/posts/barbara-genshin-47074733"
def test_parse_derived_url_returns_none_when_fields_missing():
"""If the per-platform fields needed to derive the URL are missing,
return None rather than fall back to the file `url`."""
sd = parse_sidecar({
"category": "subscribestar",
"url": "/post_uploads?payload=opaque",
# no post_id
})
assert sd.post_url is None
def test_parse_ignores_non_str_junk():
sd = parse_sidecar({"category": 5, "title": 7, "page_count": "x",
"id": True})