fix(sidecar): correct external_post_id + post_url derivation for non-Patreon platforms
Audit of one sample sidecar per platform on the operator's
/mnt/Data/Patreon/ archive surfaced three parser bugs that have been
silently corrupting non-Patreon Posts since FC-3 shipped:
1. SubscribeStar `id` vs `post_id` confusion. gallery-dl puts the
per-attachment id in `id` (e.g. 711509) and the actual post id in
`post_id` (e.g. 360360). FC's external_post_id chain had `id`
winning, so every multi-image SubscribeStar post was fragmented into
N Post rows in the database. Reorder the chain to
`("post_id", "id", "index", "message_id")` — Patreon/Pixiv (no
`post_id`), HF (uses `index`), Discord (uses `message_id`) all
unaffected.
2. Discord `message` field not captured. Discord posts put the body in
`message`, not `content`. Append it to the description fallback chain
`("content", "description", "caption", "message")`.
3. post_url is the file URL on SubscribeStar/Pixiv/HF/Discord. New
`_derive_post_url(platform, data)` helper synthesizes proper
permalinks from per-platform fields:
subscribestar → https://www.subscribestar.com/posts/<post_id>
pixiv → https://www.pixiv.net/artworks/<id>
hentaifoundry → https://www.hentai-foundry.com/pictures/user/<user>/<index>
discord → https://discord.com/channels/<server>/<channel>/<message>
Patreon's bare `url` IS a real permalink and is used as-is. For the
four file-URL platforms, the bare `url` is NEVER trusted: derive or
return None rather than persist a CDN URL.
Tests:
- `test_parse_core_fields_and_id_priority` flipped to assert post_id
wins over id.
- New `test_parse_id_used_when_no_post_id` covers the Patreon real
shape.
- New `test_parse_message_used_as_description_fallback` covers Discord
bodies.
- Five new tests cover per-platform post_url derivation
(SubscribeStar/Pixiv/HF/Discord/Patreon-untouched + missing-fields →
None).
Cleanup migration alembic 0025_fix_subscribestar_post_ids:
- For each SubscribeStar Post: find a related ImageRecord.path, walk to
its sidecar JSON, read `post_id`, overwrite Post.external_post_id +
post_url with the corrected values.
- After all updates, every group of Posts under one source sharing the
same NEW external_post_id is a fragment-set — merge to a canonical
row using the same ImageProvenance pre-delete + repoint dance as
alembic 0022 (banked pattern).
- Pure-SQL backfill of Pixiv post_url: replace any `i.pximg.net`-shape
url with the derived `/artworks/<id>` permalink.
- HF and Discord post_url backfills skipped — HF would need the `user`
field (not stored on Post), Discord needs server/channel triple.
Both will be corrected by a deep-scan re-applying sidecars through
the new parser.
Idempotent: re-running on already-corrected data is a no-op.
This commit is contained in:
@@ -0,0 +1,250 @@
|
||||
"""subscribestar: correct external_post_id + post_url, merge fragmented Posts
|
||||
|
||||
Revision ID: 0025
|
||||
Revises: 0024
|
||||
Create Date: 2026-05-27
|
||||
|
||||
Closes the operator-flagged 2026-05-27 SubscribeStar data-fragmentation
|
||||
issue: gallery-dl writes the per-attachment id in `id` and the actual
|
||||
post id in `post_id`. FC's sidecar parser had `id` winning the
|
||||
external_post_id fallback chain, so every multi-image SubscribeStar post
|
||||
was fragmented into N Post rows (one per attachment) in the database.
|
||||
|
||||
The parser fix (sidecar.py, same commit) reorders the chain so `post_id`
|
||||
wins. This migration cleans up the existing fragmented data.
|
||||
|
||||
Strategy per SubscribeStar Post in the database:
|
||||
1. Find a related ImageRecord via ImageProvenance to get an on-disk
|
||||
path. Walk to its sidecar JSON (gallery-dl's NN_ numbering prefix
|
||||
convention is honored).
|
||||
2. Read the sidecar; if it has `post_id`, that's the correct
|
||||
external_post_id.
|
||||
3. Update Post.external_post_id (= post_id) and Post.post_url (=
|
||||
derived https://www.subscribestar.com/posts/<post_id>).
|
||||
|
||||
After all rows are updated, every group of Posts under the same source
|
||||
that share the same NEW external_post_id is a duplicate set —
|
||||
fragments of the same actual post. Merge them: pick canonical (lowest
|
||||
id), repoint ImageProvenance + ImageRecord.primary_post_id +
|
||||
PostAttachment to canonical, dedupe ImageProvenance against alembic
|
||||
0021's UNIQUE, delete drop Posts.
|
||||
|
||||
Also: pure-SQL backfill of Pixiv post_url (Pixiv sidecars also write a
|
||||
file URL in `url`; the post permalink is /artworks/<id> and FC's parser
|
||||
now derives it). Pixiv external_post_id is already correct (gallery-dl's
|
||||
`id` matches the URL pattern) so no per-row sidecar IO needed.
|
||||
|
||||
HF + Discord post_url backfills are skipped here — HF needs the user
|
||||
field (not stored on Post directly), Discord needs server/channel
|
||||
triple. Both will eventually be corrected by a deep-scan that
|
||||
re-applies sidecars via the new parser.
|
||||
|
||||
Idempotent: re-running on already-corrected data is a no-op (the post_id
|
||||
in sidecar matches what's already in DB).
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import re
|
||||
from pathlib import Path
|
||||
from typing import Sequence, Union
|
||||
|
||||
from alembic import op
|
||||
from sqlalchemy import text
|
||||
|
||||
revision: str = "0025"
|
||||
down_revision: Union[str, None] = "0024"
|
||||
branch_labels: Union[str, Sequence[str], None] = None
|
||||
depends_on: Union[str, Sequence[str], None] = None
|
||||
|
||||
|
||||
# Mirror of sidecar._NUMBERING_PREFIX. Kept inline so the migration is
|
||||
# self-contained (the operator's banked rule:
|
||||
# reference_postgres_enum_swap_drop_checks.md says migrations shouldn't
|
||||
# import from runtime app code).
|
||||
_NUMBERING_PREFIX = re.compile(r"^\d+_(.+)$")
|
||||
|
||||
|
||||
def _find_sidecar(media_path: Path) -> Path | None:
|
||||
"""gallery-dl writes the sidecar under the unprefixed stem
|
||||
(`HOLLOW-ICHIGO.json`) while the media file gets a NN_ ordering
|
||||
prefix (`01_HOLLOW-ICHIGO.png`). Try in order:
|
||||
1. <stem>.json next to the media
|
||||
2. <media>.json next to the media (full-name variant)
|
||||
3. strip the NN_ prefix from the stem, then <stripped>.json
|
||||
"""
|
||||
if not media_path:
|
||||
return None
|
||||
cand = media_path.with_suffix(".json")
|
||||
if cand.is_file():
|
||||
return cand
|
||||
cand = media_path.parent / f"{media_path.name}.json"
|
||||
if cand.is_file():
|
||||
return cand
|
||||
m = _NUMBERING_PREFIX.match(media_path.stem)
|
||||
if m:
|
||||
cand = media_path.parent / f"{m.group(1)}.json"
|
||||
if cand.is_file():
|
||||
return cand
|
||||
return None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
conn = op.get_bind()
|
||||
|
||||
# ── PART 1: SubscribeStar post_id correction ─────────────────────
|
||||
ss_posts = conn.execute(text("""
|
||||
SELECT p.id, p.external_post_id, p.source_id, p.post_url
|
||||
FROM post p
|
||||
JOIN source s ON s.id = p.source_id
|
||||
WHERE s.platform = 'subscribestar'
|
||||
""")).fetchall()
|
||||
|
||||
sidecars_read = 0
|
||||
posts_updated = 0
|
||||
posts_missing_sidecar = 0
|
||||
for post_row in ss_posts:
|
||||
path = _first_attachment_path(conn, post_row.id)
|
||||
if not path:
|
||||
posts_missing_sidecar += 1
|
||||
continue
|
||||
sidecar = _find_sidecar(Path(path))
|
||||
if sidecar is None:
|
||||
posts_missing_sidecar += 1
|
||||
continue
|
||||
try:
|
||||
data = json.loads(sidecar.read_text(encoding="utf-8"))
|
||||
except (OSError, json.JSONDecodeError):
|
||||
posts_missing_sidecar += 1
|
||||
continue
|
||||
sidecars_read += 1
|
||||
post_id = data.get("post_id")
|
||||
if not isinstance(post_id, (str, int)) or isinstance(post_id, bool):
|
||||
continue
|
||||
post_id_str = str(post_id).strip()
|
||||
if not post_id_str:
|
||||
continue
|
||||
new_url = f"https://www.subscribestar.com/posts/{post_id_str}"
|
||||
# Idempotent: skip if already correct.
|
||||
if (post_row.external_post_id == post_id_str
|
||||
and post_row.post_url == new_url):
|
||||
continue
|
||||
conn.execute(
|
||||
text("""
|
||||
UPDATE post
|
||||
SET external_post_id = :epid, post_url = :url
|
||||
WHERE id = :id
|
||||
"""),
|
||||
{"epid": post_id_str, "url": new_url, "id": post_row.id},
|
||||
)
|
||||
posts_updated += 1
|
||||
|
||||
print(
|
||||
f"0025: subscribestar — read {sidecars_read} sidecars, "
|
||||
f"updated {posts_updated} Posts, "
|
||||
f"{posts_missing_sidecar} Posts had no resolvable sidecar"
|
||||
)
|
||||
|
||||
# ── PART 2: Merge SubscribeStar fragments now sharing epid ───────
|
||||
# After Part 1, each group of Posts under one source with the SAME
|
||||
# new external_post_id is a fragment-set of the same actual post.
|
||||
# Merge to one canonical row. Pre-handle the same ImageProvenance
|
||||
# collision pattern as alembic 0022 (uq_image_provenance_image_post).
|
||||
fragment_groups = conn.execute(text("""
|
||||
SELECT source_id, external_post_id, ARRAY_AGG(id ORDER BY id ASC) AS post_ids
|
||||
FROM post p
|
||||
JOIN source s ON s.id = p.source_id
|
||||
WHERE s.platform = 'subscribestar'
|
||||
AND p.external_post_id IS NOT NULL
|
||||
GROUP BY source_id, external_post_id
|
||||
HAVING COUNT(*) > 1
|
||||
""")).fetchall()
|
||||
|
||||
merged = 0
|
||||
for grp in fragment_groups:
|
||||
post_ids = list(grp.post_ids)
|
||||
keep_id, *drop_ids = post_ids
|
||||
for drop_id in drop_ids:
|
||||
# Pre-DELETE colliding ImageProvenance under drop_ that
|
||||
# already exist under keep (alembic 0022 banked the pattern).
|
||||
conn.execute(
|
||||
text("""
|
||||
DELETE FROM image_provenance
|
||||
WHERE post_id = :drop_
|
||||
AND image_record_id IN (
|
||||
SELECT image_record_id FROM image_provenance
|
||||
WHERE post_id = :keep
|
||||
)
|
||||
"""),
|
||||
{"keep": keep_id, "drop_": drop_id},
|
||||
)
|
||||
conn.execute(
|
||||
text("""
|
||||
UPDATE image_provenance SET post_id = :keep
|
||||
WHERE post_id = :drop_
|
||||
"""),
|
||||
{"keep": keep_id, "drop_": drop_id},
|
||||
)
|
||||
conn.execute(
|
||||
text("""
|
||||
UPDATE image_record SET primary_post_id = :keep
|
||||
WHERE primary_post_id = :drop_
|
||||
"""),
|
||||
{"keep": keep_id, "drop_": drop_id},
|
||||
)
|
||||
conn.execute(
|
||||
text("""
|
||||
UPDATE post_attachment SET post_id = :keep
|
||||
WHERE post_id = :drop_
|
||||
"""),
|
||||
{"keep": keep_id, "drop_": drop_id},
|
||||
)
|
||||
conn.execute(
|
||||
text("DELETE FROM post WHERE id = :drop_"),
|
||||
{"drop_": drop_id},
|
||||
)
|
||||
merged += 1
|
||||
print(f"0025: subscribestar — merged {merged} duplicate Post fragments")
|
||||
|
||||
# ── PART 3: Pixiv post_url backfill (pure SQL) ───────────────────
|
||||
# Pixiv's external_post_id is already correct (gallery-dl's `id` is
|
||||
# the post id). Only post_url needs derivation: replace anything
|
||||
# under i.pximg.net (the file URL) with the /artworks/<id> permalink.
|
||||
pixiv_updated = conn.execute(text("""
|
||||
UPDATE post p
|
||||
SET post_url = 'https://www.pixiv.net/artworks/' || p.external_post_id
|
||||
FROM source s
|
||||
WHERE p.source_id = s.id
|
||||
AND s.platform = 'pixiv'
|
||||
AND p.external_post_id IS NOT NULL
|
||||
AND (p.post_url IS NULL
|
||||
OR p.post_url LIKE 'https://i.pximg.net/%'
|
||||
OR p.post_url LIKE 'http://i.pximg.net/%')
|
||||
""")).rowcount
|
||||
print(f"0025: pixiv — backfilled post_url on {pixiv_updated} Posts")
|
||||
|
||||
|
||||
def _first_attachment_path(conn, post_id: int) -> str | None:
|
||||
"""Return any ImageRecord.path attached to this post (via
|
||||
ImageProvenance). Lowest-id row keeps the migration deterministic
|
||||
so re-running on the same DB picks the same sidecar."""
|
||||
row = conn.execute(
|
||||
text("""
|
||||
SELECT ir.path
|
||||
FROM image_provenance ip
|
||||
JOIN image_record ir ON ir.id = ip.image_record_id
|
||||
WHERE ip.post_id = :pid
|
||||
ORDER BY ip.id ASC
|
||||
LIMIT 1
|
||||
"""),
|
||||
{"pid": post_id},
|
||||
).first()
|
||||
return row[0] if row else None
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
# Lossy: external_post_id values were overwritten with the correct
|
||||
# post_id; original per-attachment ids weren't preserved. Post-merge
|
||||
# also deleted drop rows. No safe restore. To roll back the schema
|
||||
# invariant, fork from 0024 and re-run sidecar imports.
|
||||
pass
|
||||
Reference in New Issue
Block a user