fix(sidecar): correct external_post_id + post_url derivation for non-Patreon platforms
Audit of one sample sidecar per platform on the operator's
/mnt/Data/Patreon/ archive surfaced three parser bugs that have been
silently corrupting non-Patreon Posts since FC-3 shipped:
1. SubscribeStar `id` vs `post_id` confusion. gallery-dl puts the
per-attachment id in `id` (e.g. 711509) and the actual post id in
`post_id` (e.g. 360360). FC's external_post_id chain had `id`
winning, so every multi-image SubscribeStar post was fragmented into
N Post rows in the database. Reorder the chain to
`("post_id", "id", "index", "message_id")` — Patreon/Pixiv (no
`post_id`), HF (uses `index`), Discord (uses `message_id`) all
unaffected.
2. Discord `message` field not captured. Discord posts put the body in
`message`, not `content`. Append it to the description fallback chain
`("content", "description", "caption", "message")`.
3. post_url is the file URL on SubscribeStar/Pixiv/HF/Discord. New
`_derive_post_url(platform, data)` helper synthesizes proper
permalinks from per-platform fields:
subscribestar → https://www.subscribestar.com/posts/<post_id>
pixiv → https://www.pixiv.net/artworks/<id>
hentaifoundry → https://www.hentai-foundry.com/pictures/user/<user>/<index>
discord → https://discord.com/channels/<server>/<channel>/<message>
Patreon's bare `url` IS a real permalink and is used as-is. For the
four file-URL platforms, the bare `url` is NEVER trusted: derive or
return None rather than persist a CDN URL.
Tests:
- `test_parse_core_fields_and_id_priority` flipped to assert post_id
wins over id.
- New `test_parse_id_used_when_no_post_id` covers the Patreon real
shape.
- New `test_parse_message_used_as_description_fallback` covers Discord
bodies.
- Five new tests cover per-platform post_url derivation
(SubscribeStar/Pixiv/HF/Discord/Patreon-untouched + missing-fields →
None).
Cleanup migration alembic 0025_fix_subscribestar_post_ids:
- For each SubscribeStar Post: find a related ImageRecord.path, walk to
its sidecar JSON, read `post_id`, overwrite Post.external_post_id +
post_url with the corrected values.
- After all updates, every group of Posts under one source sharing the
same NEW external_post_id is a fragment-set — merge to a canonical
row using the same ImageProvenance pre-delete + repoint dance as
alembic 0022 (banked pattern).
- Pure-SQL backfill of Pixiv post_url: replace any `i.pximg.net`-shape
url with the derived `/artworks/<id>` permalink.
- HF and Discord post_url backfills skipped — HF would need the `user`
field (not stored on Post), Discord needs server/channel triple.
Both will be corrected by a deep-scan re-applying sidecars through
the new parser.
Idempotent: re-running on already-corrected data is a no-op.
This commit is contained in:
@@ -111,8 +111,16 @@ def parse_sidecar(data: dict) -> SidecarData:
|
||||
cat = data.get("category")
|
||||
platform = cat if isinstance(cat, str) and cat.strip() else None
|
||||
|
||||
# external_post_id lookup order: post_id MUST come before id.
|
||||
# SubscribeStar gallery-dl writes the per-attachment id in `id`
|
||||
# (e.g. 711509) and the actual post id in `post_id` (e.g. 360360);
|
||||
# picking `id` first fragments every multi-image subscribestar post
|
||||
# into N distinct Post rows in FC. Patreon/Pixiv have no `post_id`
|
||||
# so `id` still wins for them; HF uses `index`, Discord uses
|
||||
# `message_id` — all reached via the remaining chain entries.
|
||||
# Operator-flagged 2026-05-27 during the sidecar audit.
|
||||
external_post_id = None
|
||||
for k in ("id", "post_id", "index", "message_id"):
|
||||
for k in ("post_id", "id", "index", "message_id"):
|
||||
v = data.get(k)
|
||||
if isinstance(v, bool):
|
||||
continue
|
||||
@@ -138,7 +146,11 @@ def parse_sidecar(data: dict) -> SidecarData:
|
||||
if post_date is not None:
|
||||
break
|
||||
|
||||
description = _first_str(data, ("content", "description", "caption"))
|
||||
# `message` is Discord gallery-dl's body field (no `content`); added
|
||||
# 2026-05-27 to the description fallback chain.
|
||||
description = _first_str(
|
||||
data, ("content", "description", "caption", "message"),
|
||||
)
|
||||
|
||||
# SubscribeStar posts always write `title: ""` and put the leading
|
||||
# sentence inside `content` (confirmed against the operator's
|
||||
@@ -150,13 +162,60 @@ def parse_sidecar(data: dict) -> SidecarData:
|
||||
if post_title is None and description:
|
||||
post_title = _first_line_text(description)
|
||||
|
||||
# post_url derivation: SubscribeStar/Pixiv/HF/Discord put the FILE
|
||||
# download URL in `url`, not a post permalink. Synthesize the
|
||||
# permalink from per-platform fields when possible. Patreon's `url`
|
||||
# IS a permalink and is used as-is. For the four file-URL platforms,
|
||||
# the bare `url` is NEVER trusted — derive or return None rather
|
||||
# than persist a CDN URL in post.post_url.
|
||||
if platform in _DERIVED_URL_PLATFORMS:
|
||||
post_url = _derive_post_url(platform, data)
|
||||
else:
|
||||
post_url = _first_str(data, ("url", "post_url"))
|
||||
|
||||
return SidecarData(
|
||||
platform=platform,
|
||||
external_post_id=external_post_id,
|
||||
post_url=_first_str(data, ("url", "post_url")),
|
||||
post_url=post_url,
|
||||
post_title=post_title,
|
||||
description=description,
|
||||
attachment_count=attachment_count,
|
||||
post_date=post_date,
|
||||
raw=data,
|
||||
)
|
||||
|
||||
|
||||
_DERIVED_URL_PLATFORMS = frozenset({
|
||||
"subscribestar", "pixiv", "hentaifoundry", "discord",
|
||||
})
|
||||
|
||||
|
||||
def _derive_post_url(platform: str, data: dict) -> str | None:
|
||||
"""Synthesize the post-permalink URL from per-platform metadata.
|
||||
|
||||
gallery-dl writes the file-download URL in `url` for these four
|
||||
platforms; we need a real permalink for the PostCard "open original"
|
||||
button. Returns None if the platform-specific fields are missing
|
||||
(rare in well-formed sidecars but defensive).
|
||||
"""
|
||||
if platform == "subscribestar":
|
||||
pid = data.get("post_id")
|
||||
if isinstance(pid, (str, int)) and str(pid).strip():
|
||||
return f"https://www.subscribestar.com/posts/{pid}"
|
||||
elif platform == "pixiv":
|
||||
pid = data.get("id")
|
||||
if isinstance(pid, (str, int)) and not isinstance(pid, bool) and str(pid).strip():
|
||||
return f"https://www.pixiv.net/artworks/{pid}"
|
||||
elif platform == "hentaifoundry":
|
||||
user = _first_str(data, ("user", "artist"))
|
||||
idx = data.get("index")
|
||||
if user and isinstance(idx, (str, int)) and not isinstance(idx, bool) and str(idx).strip():
|
||||
return f"https://www.hentai-foundry.com/pictures/user/{user}/{idx}"
|
||||
elif platform == "discord":
|
||||
sid = data.get("server_id")
|
||||
cid = data.get("channel_id")
|
||||
mid = data.get("message_id")
|
||||
if all(isinstance(v, (str, int)) and not isinstance(v, bool) and str(v).strip()
|
||||
for v in (sid, cid, mid)):
|
||||
return f"https://discord.com/channels/{sid}/{cid}/{mid}"
|
||||
return None
|
||||
|
||||
Reference in New Issue
Block a user