fix(subscriptions): hide sidecar synthetic Sources + prefer real on lookup
Two coupled bugs surfaced 2026-05-31 by the Subscriptions UI showing "phantom" subscriptions like `sidecar:patreon:dpmaker`: 1. `SourceService.list()` returned every Source, no filter on URL. alembic 0022 (2026-05-26) consolidated old per-post-URL Sources into one canonical row per (artist, platform); when no real campaign URL was salvageable it rewrote the canonical to `sidecar:<plat>:<slug>` enabled=false as a disabled anchor. The UI then listed those anchors as if they were polls — disabled, but visible. Fix: `list()` excludes `url LIKE 'sidecar:%'` by default; `include_synthetic=True` opts back in for admin tooling. 2. `importer._source_for_sidecar` picked the lowest-id Source for (artist, platform). When alembic 0022 had rewritten a per-post row into a synthetic anchor (lower id) AND the operator later added the real subscription (higher id), every gallery-dl download silently attached its Post to the SYNTHETIC instead of the real Source. Fix: prefer a non-`sidecar:%` URL when one exists; fall back to the synthetic; only create a new synthetic when nothing exists for (artist, platform). alembic 0028 is the data half: for every (artist, platform) with both a synthetic AND a real Source, pre-merge Post+ImageProvenance collisions on the canonical, bulk-repoint Posts/ImageProvenance/ DownloadEvent.source_id onto the real Source, and delete the synthetic. Lone synthetics (no real twin) are left intact — they anchor real imported content the operator may still want; the list-filter hides them so they no longer surface as phantoms.
This commit is contained in:
@@ -0,0 +1,190 @@
|
||||
"""collapse-sidecar-synthetic: repoint Posts/ImageProvenance/DownloadEvents
|
||||
from `sidecar:<platform>:<slug>` synthetic Source anchors onto the real
|
||||
Source for the same (artist, platform) when one exists, then delete the
|
||||
synthetic.
|
||||
|
||||
Revision ID: 0028
|
||||
Revises: 0027
|
||||
Create Date: 2026-05-31
|
||||
|
||||
Background: alembic 0022 (2026-05-26) consolidated the old per-post-URL
|
||||
Source rows into one canonical Source per (artist, platform). When NO
|
||||
real campaign URL was salvageable among the candidates, it rewrote the
|
||||
canonical row to url='sidecar:<platform>:<slug>' enabled=false as a
|
||||
disabled anchor for any Posts already attached.
|
||||
|
||||
That was fine while it was the only Source for that artist+platform.
|
||||
But: the unique constraint on Source is (artist_id, platform, url), not
|
||||
(artist_id, platform). When the operator later added the real
|
||||
subscription via the UI / extension / etc., a SECOND row landed —
|
||||
the real one — with id > the synthetic. Both coexisted.
|
||||
|
||||
Two follow-on problems surfaced 2026-05-31:
|
||||
|
||||
1. The Subscriptions UI listed both rows. The synthetic was disabled
|
||||
so the scheduler never polled it, but it looked like a phantom
|
||||
subscription. (Fixed in same commit by SourceService.list filter.)
|
||||
2. importer._source_for_sidecar picked Source by `ORDER BY id ASC
|
||||
LIMIT 1`, so EVERY gallery-dl download since the real Source was
|
||||
added attached its Post to the SYNTHETIC anchor, not the real
|
||||
Source. (Fixed in same commit by preferring non-sidecar URLs.)
|
||||
|
||||
This migration is the data half of the cleanup: for every (artist,
|
||||
platform) with both a synthetic AND a real Source, repoint the
|
||||
synthetic's children (Posts, ImageProvenance, DownloadEvents) onto the
|
||||
real Source and delete the synthetic. Reuses the same epid/provenance
|
||||
collision dance from alembic 0022 because the same uniqueness
|
||||
constraints fire row-by-row during bulk UPDATEs.
|
||||
|
||||
Lone synthetic anchors — those where no real Source for the same
|
||||
(artist, platform) exists (e.g., filesystem-imported artist with no
|
||||
subscription added) — are LEFT INTACT. They anchor real imported
|
||||
content; deleting them would CASCADE-delete the Posts the operator
|
||||
imported. The SourceService.list filter hides them from the UI; the
|
||||
operator can delete them by hand if they want the underlying imports
|
||||
gone.
|
||||
"""
|
||||
from typing import Sequence, Union
|
||||
|
||||
from alembic import op
|
||||
from sqlalchemy import text
|
||||
|
||||
revision: str = "0028"
|
||||
down_revision: Union[str, None] = "0027"
|
||||
branch_labels: Union[str, Sequence[str], None] = None
|
||||
depends_on: Union[str, Sequence[str], None] = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
conn = op.get_bind()
|
||||
|
||||
# Find (artist_id, platform) groups where BOTH a sidecar synthetic
|
||||
# and at least one real Source exist.
|
||||
groups = conn.execute(text("""
|
||||
SELECT artist_id, platform
|
||||
FROM source
|
||||
GROUP BY artist_id, platform
|
||||
HAVING bool_or(url LIKE 'sidecar:%')
|
||||
AND bool_or(url NOT LIKE 'sidecar:%')
|
||||
""")).fetchall()
|
||||
|
||||
for artist_id, platform in groups:
|
||||
rows = conn.execute(
|
||||
text("""
|
||||
SELECT id, url FROM source
|
||||
WHERE artist_id = :a AND platform = :p
|
||||
ORDER BY id ASC
|
||||
"""),
|
||||
{"a": artist_id, "p": platform},
|
||||
).fetchall()
|
||||
|
||||
synthetic_ids = [sid for sid, url in rows if url.startswith("sidecar:")]
|
||||
real_rows = [(sid, url) for sid, url in rows if not url.startswith("sidecar:")]
|
||||
if not synthetic_ids or not real_rows:
|
||||
continue # belt+suspenders; the GROUP BY already filtered
|
||||
|
||||
# Canonical real: lowest-id non-sidecar Source.
|
||||
canonical_id = real_rows[0][0]
|
||||
|
||||
# STEP A: PRE-merge Post collisions on (canonical, external_post_id).
|
||||
# Mirror alembic 0022's pre-merge logic — when synth has Post X
|
||||
# epid=N and real has Post Y epid=N, the bulk UPDATE below would
|
||||
# trip uq_post_source_external_id row-by-row. Group all Posts
|
||||
# under (canonical + synthetics) by epid; for any group >1,
|
||||
# pick a keep (prefer one already under canonical, else lowest
|
||||
# id) and merge the rest into it.
|
||||
all_posts = conn.execute(
|
||||
text("""
|
||||
SELECT external_post_id, id, source_id
|
||||
FROM post
|
||||
WHERE source_id = :canonical OR source_id = ANY(:synths)
|
||||
ORDER BY external_post_id, id
|
||||
"""),
|
||||
{"canonical": canonical_id, "synths": synthetic_ids},
|
||||
).fetchall()
|
||||
by_epid: dict = {}
|
||||
for epid, post_id, src_id in all_posts:
|
||||
by_epid.setdefault(epid, []).append((post_id, src_id))
|
||||
for _epid, posts in by_epid.items():
|
||||
if len(posts) <= 1:
|
||||
continue
|
||||
canonical_side = [p for p in posts if p[1] == canonical_id]
|
||||
keep_id = canonical_side[0][0] if canonical_side else posts[0][0]
|
||||
drop_ids = [p[0] for p in posts if p[0] != keep_id]
|
||||
for drop_id in drop_ids:
|
||||
# Pre-delete image_provenance rows under drop_ whose
|
||||
# image_record_id already has provenance under keep —
|
||||
# avoids tripping uq_image_provenance_image_post (0021)
|
||||
# row-by-row during the repoint UPDATE.
|
||||
conn.execute(
|
||||
text("""
|
||||
DELETE FROM image_provenance
|
||||
WHERE post_id = :drop_
|
||||
AND image_record_id IN (
|
||||
SELECT image_record_id FROM image_provenance
|
||||
WHERE post_id = :keep
|
||||
)
|
||||
"""),
|
||||
{"keep": keep_id, "drop_": drop_id},
|
||||
)
|
||||
conn.execute(
|
||||
text("""
|
||||
UPDATE image_provenance SET post_id = :keep
|
||||
WHERE post_id = :drop_
|
||||
"""),
|
||||
{"keep": keep_id, "drop_": drop_id},
|
||||
)
|
||||
conn.execute(
|
||||
text("""
|
||||
UPDATE image_record SET primary_post_id = :keep
|
||||
WHERE primary_post_id = :drop_
|
||||
"""),
|
||||
{"keep": keep_id, "drop_": drop_id},
|
||||
)
|
||||
conn.execute(
|
||||
text("DELETE FROM post WHERE id = :drop_"),
|
||||
{"drop_": drop_id},
|
||||
)
|
||||
|
||||
# STEP B: Bulk reparent the remaining Posts off the synthetics.
|
||||
conn.execute(
|
||||
text("""
|
||||
UPDATE post SET source_id = :canonical
|
||||
WHERE source_id = ANY(:synths)
|
||||
"""),
|
||||
{"canonical": canonical_id, "synths": synthetic_ids},
|
||||
)
|
||||
|
||||
# STEP C: Reparent ImageProvenance.source_id (denormalized FK;
|
||||
# no UNIQUE on source_id, safe bulk).
|
||||
conn.execute(
|
||||
text("""
|
||||
UPDATE image_provenance SET source_id = :canonical
|
||||
WHERE source_id = ANY(:synths)
|
||||
"""),
|
||||
{"canonical": canonical_id, "synths": synthetic_ids},
|
||||
)
|
||||
|
||||
# STEP D: Reparent any DownloadEvent.source_id. Synthetics are
|
||||
# enabled=false so the scheduler never created events for them;
|
||||
# this is belt+suspenders for any rows planted by manual force
|
||||
# or older code paths.
|
||||
conn.execute(
|
||||
text("""
|
||||
UPDATE download_event SET source_id = :canonical
|
||||
WHERE source_id = ANY(:synths)
|
||||
"""),
|
||||
{"canonical": canonical_id, "synths": synthetic_ids},
|
||||
)
|
||||
|
||||
# STEP E: Drop the now-empty synthetics.
|
||||
conn.execute(
|
||||
text("DELETE FROM source WHERE id = ANY(:synths)"),
|
||||
{"synths": synthetic_ids},
|
||||
)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
# Lossy migration — synthetic Sources deleted, Posts repointed and
|
||||
# potentially merged. No safe downgrade.
|
||||
pass
|
||||
Reference in New Issue
Block a user