feat(model): nullable Post.source_id + denormalized Post.artist_id; retire sidecar synthetics
Operator-asked 2026-06-01 after the Dymkens orphan investigation (Scribe plan #540). The pre-2030 sidecar-synthetic Source pattern (`sidecar:<platform>:<slug>` enabled=false rows) existed solely to satisfy `Post.source_id NOT NULL`, and leaked into the Subscriptions UI as phantom subscriptions. Now the data model says what's true: filesystem-imported content with no live subscription has NULL source_id, full stop. ## Schema (alembic 0030) - `post.artist_id` — NEW NOT NULL FK to artist (CASCADE). Backfilled from source.artist_id in the migration. Indexed for the artist-filter queries. - `post.source_id` — NOT NULL → nullable; FK ondelete CASCADE → SET NULL. Deleting a Source detaches its Posts instead of destroying archived content (subscription ends, archive stays). - `image_provenance.source_id` — same nullable + SET NULL. - Partial unique index `uq_post_artist_external_id_null_source` on (artist_id, external_post_id) WHERE source_id IS NULL — guards filesystem-import dedup since the existing source-bound unique ignores NULLs (Postgres treats NULL != NULL). - Sidecar synthetic Sources deleted: NULL out FKs in post, image_provenance first, then DELETE FROM source WHERE url LIKE 'sidecar:%'. The Dymkens cleanup. ## Model + service changes - `Post.source_id` → `Mapped[int | None]`; new `Post.artist_id` denormalized. - `ImageProvenance.source_id` → `Mapped[int | None]`. - Importer: `_source_for_sidecar` (synthetic-creating) → `_lookup_source_for_sidecar` (returns None when no subscription). `_find_or_create_post` takes required `artist_id`; matches on (source_id, external_post_id) for source-bound posts or (artist_id, external_post_id) for NULL-source posts. - Service queries switched off the Source detour to use Post.artist_id directly: post_feed_service.scroll/around/get_post (LEFT JOIN to Source so NULL-source posts surface); artist_service date_row/ activity/post_count; provenance_service.for_image/for_post (LEFT JOIN); gallery_service._provenance_exists_where_artist via Post.artist_id instead of ImageProvenance.source_id → Source. - `_to_dict` and provenance dict-builders emit `"source": null` for NULL-source rows. ## Frontend - `ProvenancePanel.vue` + `PostCard.vue`: render `e.source?.platform ?? 'filesystem import'` so NULL-source posts get a clear "filesystem import" affordance instead of a NaN crash. ## Tests - `test_importer_upsert_helpers`: removed the four synthetic-anchor tests; added `_find_or_create_post_idempotent_with_null_source` (dedup via the partial unique index) and `_lookup_source_for_sidecar_returns_*` (existing-subscription + none cases). The existing `_find_or_create_post_idempotent` now also passes `artist_id` and asserts it. - 8 other test files updated: every direct `Post(...)` construction gains `artist_id=<artist>.id`. The `_seed_post` helper in `test_post_feed_service` looks up artist_id from the source row so callsites stay one-arg. ## Verification on deploy After alembic 0030 runs: - `SELECT COUNT(*) FROM source WHERE url LIKE 'sidecar:%'` → 0. - `SELECT COUNT(*) FROM post WHERE source_id IS NULL` → count of filesystem-imported posts (Dymkens + any other historical). - Every `post.artist_id` non-null; consistent with source.artist_id for source-bound rows. - Subscriptions tab: no Dymkens phantom row. - Artist detail → Posts/Gallery: Dymkens's content still reachable via Post.artist_id. - Provenance panel renders "filesystem import" chip for NULL-source posts; PostCard same. ## Out of scope - UI to manage/delete orphan NULL-source Posts. Data model is right; UI follows if operator wants it.
This commit is contained in:
@@ -212,10 +212,10 @@ class Importer:
|
||||
which would lose the surrounding scan's progress — and re-run `stmt`
|
||||
(scalar_one) to return the row the other worker created.
|
||||
|
||||
Centralizes the pattern shared by _find_or_create_source,
|
||||
_source_for_sidecar, and _find_or_create_post. The plain
|
||||
SELECT-then-INSERT version lost races under the 5-min recovery sweep
|
||||
(operator-flagged 2026-05-26)."""
|
||||
Centralizes the pattern shared by _find_or_create_source and
|
||||
_find_or_create_post. The plain SELECT-then-INSERT version lost
|
||||
races under the 5-min recovery sweep (operator-flagged
|
||||
2026-05-26)."""
|
||||
existing = self.session.execute(stmt).scalar_one_or_none()
|
||||
if existing is not None:
|
||||
return existing
|
||||
@@ -258,47 +258,25 @@ class Importer:
|
||||
lambda: Source(artist_id=artist_id, platform=platform, url=url),
|
||||
)
|
||||
|
||||
def _source_for_sidecar(
|
||||
self, *, artist_id: int, platform: str, artist_slug: str,
|
||||
) -> Source:
|
||||
"""Sidecar-import Source resolver. Used by both filesystem imports
|
||||
and gallery-dl downloads (both write sidecar JSON, both flow through
|
||||
_apply_sidecar / _capture_attachment).
|
||||
def _lookup_source_for_sidecar(
|
||||
self, *, artist_id: int, platform: str,
|
||||
) -> Source | None:
|
||||
"""Find the real subscription Source for (artist, platform), or
|
||||
None if no subscription exists.
|
||||
|
||||
Source represents a subscription feed (one per artist+platform — the
|
||||
URL polled by the FC-3 downloader). The filesystem importer used to
|
||||
call _find_or_create_source(url=sd.post_url), creating one Source
|
||||
row per post URL — 100s of junk Sources per artist, all with
|
||||
enabled=True, polluting the artist detail page and tricking the
|
||||
subscription checker into trying to poll patreon post URLs as feeds.
|
||||
Operator-flagged 2026-05-26; consolidated via alembic 0022.
|
||||
|
||||
Resolution order: prefer a real (non-sidecar) Source over a
|
||||
synthetic anchor. When alembic 0022 ran, it may have rewritten
|
||||
per-post Sources into `sidecar:<platform>:<slug>` synthetic
|
||||
anchors. If the operator later added the real subscription, both
|
||||
rows now coexist. A naive `ORDER BY id ASC LIMIT 1` lookup would
|
||||
pick the older synthetic and silently attach every gallery-dl
|
||||
download to the wrong Source — operator-flagged 2026-05-31 after
|
||||
the Subscriptions UI surfaced the phantom anchors. Pick the real
|
||||
one when one exists; fall back to the synthetic; only create a
|
||||
new synthetic when nothing exists for (artist, platform).
|
||||
Pre-alembic-0030 this method would CREATE a synthetic
|
||||
`sidecar:<platform>:<slug>` Source when no real one existed —
|
||||
because `Post.source_id` was NOT NULL and the importer needed
|
||||
something to attach Posts to. Alembic 0030 relaxed both
|
||||
`Post.source_id` and `ImageProvenance.source_id` to nullable, so
|
||||
synthetic anchors are obsolete; the importer now leaves
|
||||
source_id as None when no subscription exists for the (artist,
|
||||
platform). Operator-asked 2026-06-01: synthetic Sources had
|
||||
leaked into the Subscriptions UI as phantom subscriptions and
|
||||
the operator wanted the data model to truthfully say "this
|
||||
content has no live subscription."
|
||||
"""
|
||||
real_stmt = (
|
||||
select(Source)
|
||||
.where(
|
||||
Source.artist_id == artist_id,
|
||||
Source.platform == platform,
|
||||
~Source.url.like("sidecar:%"),
|
||||
)
|
||||
.order_by(Source.id.asc())
|
||||
.limit(1)
|
||||
)
|
||||
real = self.session.execute(real_stmt).scalar_one_or_none()
|
||||
if real is not None:
|
||||
return real
|
||||
|
||||
any_stmt = (
|
||||
stmt = (
|
||||
select(Source)
|
||||
.where(
|
||||
Source.artist_id == artist_id,
|
||||
@@ -307,29 +285,37 @@ class Importer:
|
||||
.order_by(Source.id.asc())
|
||||
.limit(1)
|
||||
)
|
||||
return self._get_or_create(
|
||||
any_stmt,
|
||||
lambda: Source(
|
||||
artist_id=artist_id,
|
||||
platform=platform,
|
||||
url=f"sidecar:{platform}:{artist_slug}",
|
||||
enabled=False,
|
||||
),
|
||||
)
|
||||
return self.session.execute(stmt).scalar_one_or_none()
|
||||
|
||||
def _find_or_create_post(
|
||||
self, *, source_id: int, external_post_id: str,
|
||||
self, *, source_id: int | None, external_post_id: str,
|
||||
artist_id: int,
|
||||
) -> Post:
|
||||
"""Race-safe find-or-create on `post` keyed by
|
||||
(source_id, external_post_id). Mirrors `_find_or_create_source`
|
||||
— same savepoint + IntegrityError-recovery pattern."""
|
||||
stmt = select(Post).where(
|
||||
Post.source_id == source_id,
|
||||
Post.external_post_id == external_post_id,
|
||||
)
|
||||
"""Race-safe find-or-create on `post`. Keyed by
|
||||
(source_id, external_post_id) when source_id is set — the
|
||||
`uq_post_source_external_id` constraint guards. For NULL-source
|
||||
posts the existence check matches on (artist_id, external_post_id),
|
||||
which the partial unique index `uq_post_artist_external_id_null_source`
|
||||
(alembic 0030) guards. Same savepoint + IntegrityError-recovery
|
||||
pattern as the rest of the helpers."""
|
||||
if source_id is not None:
|
||||
stmt = select(Post).where(
|
||||
Post.source_id == source_id,
|
||||
Post.external_post_id == external_post_id,
|
||||
)
|
||||
else:
|
||||
stmt = select(Post).where(
|
||||
Post.source_id.is_(None),
|
||||
Post.artist_id == artist_id,
|
||||
Post.external_post_id == external_post_id,
|
||||
)
|
||||
return self._get_or_create(
|
||||
stmt,
|
||||
lambda: Post(source_id=source_id, external_post_id=external_post_id),
|
||||
lambda: Post(
|
||||
source_id=source_id,
|
||||
artist_id=artist_id,
|
||||
external_post_id=external_post_id,
|
||||
),
|
||||
)
|
||||
|
||||
def import_one(self, source: Path) -> ImportResult:
|
||||
@@ -370,12 +356,14 @@ class Importer:
|
||||
return None
|
||||
sd = parse_sidecar(data)
|
||||
platform = sd.platform or "unknown"
|
||||
src = self._source_for_sidecar(
|
||||
artist_id=artist.id, platform=platform, artist_slug=artist.slug,
|
||||
src = self._lookup_source_for_sidecar(
|
||||
artist_id=artist.id, platform=platform,
|
||||
)
|
||||
epid = sd.external_post_id or sc.stem
|
||||
return self._find_or_create_post(
|
||||
source_id=src.id, external_post_id=epid,
|
||||
source_id=src.id if src else None,
|
||||
external_post_id=epid,
|
||||
artist_id=artist.id,
|
||||
)
|
||||
|
||||
def _capture_attachment(
|
||||
@@ -858,14 +846,15 @@ class Importer:
|
||||
src = explicit_source
|
||||
else:
|
||||
platform = sd.platform or "unknown"
|
||||
src = self._source_for_sidecar(
|
||||
src = self._lookup_source_for_sidecar(
|
||||
artist_id=artist.id, platform=platform,
|
||||
artist_slug=artist.slug,
|
||||
)
|
||||
|
||||
epid = sd.external_post_id or sc.stem
|
||||
post = self._find_or_create_post(
|
||||
source_id=src.id, external_post_id=epid,
|
||||
source_id=src.id if src else None,
|
||||
external_post_id=epid,
|
||||
artist_id=artist.id,
|
||||
)
|
||||
if sd.post_url is not None:
|
||||
post.post_url = sd.post_url
|
||||
@@ -901,7 +890,7 @@ class Importer:
|
||||
ImageProvenance(
|
||||
image_record_id=record.id,
|
||||
post_id=post.id,
|
||||
source_id=src.id,
|
||||
source_id=src.id if src else None,
|
||||
captured_metadata=sd.raw,
|
||||
)
|
||||
)
|
||||
|
||||
Reference in New Issue
Block a user