feat(model): nullable Post.source_id + denormalized Post.artist_id; retire sidecar synthetics
CI / lint (push) Failing after 3s
CI / backend-lint-and-test (push) Successful in 11s
CI / frontend-build (push) Successful in 17s
CI / intimp (push) Failing after 2m15s
CI / intapi (push) Failing after 2m18s
CI / intcore (push) Failing after 2m17s

Operator-asked 2026-06-01 after the Dymkens orphan investigation
(Scribe plan #540). The pre-2030 sidecar-synthetic Source pattern
(`sidecar:<platform>:<slug>` enabled=false rows) existed solely to
satisfy `Post.source_id NOT NULL`, and leaked into the Subscriptions
UI as phantom subscriptions. Now the data model says what's true:
filesystem-imported content with no live subscription has NULL
source_id, full stop.

## Schema (alembic 0030)

- `post.artist_id` — NEW NOT NULL FK to artist (CASCADE). Backfilled
  from source.artist_id in the migration. Indexed for the artist-filter
  queries.
- `post.source_id` — NOT NULL → nullable; FK ondelete CASCADE → SET
  NULL. Deleting a Source detaches its Posts instead of destroying
  archived content (subscription ends, archive stays).
- `image_provenance.source_id` — same nullable + SET NULL.
- Partial unique index `uq_post_artist_external_id_null_source` on
  (artist_id, external_post_id) WHERE source_id IS NULL — guards
  filesystem-import dedup since the existing source-bound unique
  ignores NULLs (Postgres treats NULL != NULL).
- Sidecar synthetic Sources deleted: NULL out FKs in post,
  image_provenance first, then DELETE FROM source WHERE url LIKE
  'sidecar:%'. The Dymkens cleanup.

## Model + service changes

- `Post.source_id` → `Mapped[int | None]`; new `Post.artist_id`
  denormalized.
- `ImageProvenance.source_id` → `Mapped[int | None]`.
- Importer: `_source_for_sidecar` (synthetic-creating) →
  `_lookup_source_for_sidecar` (returns None when no subscription).
  `_find_or_create_post` takes required `artist_id`; matches on
  (source_id, external_post_id) for source-bound posts or
  (artist_id, external_post_id) for NULL-source posts.
- Service queries switched off the Source detour to use Post.artist_id
  directly: post_feed_service.scroll/around/get_post (LEFT JOIN to
  Source so NULL-source posts surface); artist_service date_row/
  activity/post_count; provenance_service.for_image/for_post (LEFT
  JOIN); gallery_service._provenance_exists_where_artist via
  Post.artist_id instead of ImageProvenance.source_id → Source.
- `_to_dict` and provenance dict-builders emit `"source": null` for
  NULL-source rows.

## Frontend

- `ProvenancePanel.vue` + `PostCard.vue`: render `e.source?.platform
  ?? 'filesystem import'` so NULL-source posts get a clear
  "filesystem import" affordance instead of a NaN crash.

## Tests

- `test_importer_upsert_helpers`: removed the four synthetic-anchor
  tests; added `_find_or_create_post_idempotent_with_null_source`
  (dedup via the partial unique index) and
  `_lookup_source_for_sidecar_returns_*` (existing-subscription +
  none cases). The existing `_find_or_create_post_idempotent` now
  also passes `artist_id` and asserts it.
- 8 other test files updated: every direct `Post(...)` construction
  gains `artist_id=<artist>.id`. The `_seed_post` helper in
  `test_post_feed_service` looks up artist_id from the source row so
  callsites stay one-arg.

## Verification on deploy

After alembic 0030 runs:
- `SELECT COUNT(*) FROM source WHERE url LIKE 'sidecar:%'` → 0.
- `SELECT COUNT(*) FROM post WHERE source_id IS NULL` → count of
  filesystem-imported posts (Dymkens + any other historical).
- Every `post.artist_id` non-null; consistent with source.artist_id
  for source-bound rows.
- Subscriptions tab: no Dymkens phantom row.
- Artist detail → Posts/Gallery: Dymkens's content still reachable
  via Post.artist_id.
- Provenance panel renders "filesystem import" chip for NULL-source
  posts; PostCard same.

## Out of scope

- UI to manage/delete orphan NULL-source Posts. Data model is right;
  UI follows if operator wants it.
This commit is contained in:
2026-06-01 14:17:52 -04:00
parent 94e7d20792
commit 2f66de2928
23 changed files with 358 additions and 192 deletions
+57 -68
View File
@@ -212,10 +212,10 @@ class Importer:
which would lose the surrounding scan's progress — and re-run `stmt`
(scalar_one) to return the row the other worker created.
Centralizes the pattern shared by _find_or_create_source,
_source_for_sidecar, and _find_or_create_post. The plain
SELECT-then-INSERT version lost races under the 5-min recovery sweep
(operator-flagged 2026-05-26)."""
Centralizes the pattern shared by _find_or_create_source and
_find_or_create_post. The plain SELECT-then-INSERT version lost
races under the 5-min recovery sweep (operator-flagged
2026-05-26)."""
existing = self.session.execute(stmt).scalar_one_or_none()
if existing is not None:
return existing
@@ -258,47 +258,25 @@ class Importer:
lambda: Source(artist_id=artist_id, platform=platform, url=url),
)
def _source_for_sidecar(
self, *, artist_id: int, platform: str, artist_slug: str,
) -> Source:
"""Sidecar-import Source resolver. Used by both filesystem imports
and gallery-dl downloads (both write sidecar JSON, both flow through
_apply_sidecar / _capture_attachment).
def _lookup_source_for_sidecar(
self, *, artist_id: int, platform: str,
) -> Source | None:
"""Find the real subscription Source for (artist, platform), or
None if no subscription exists.
Source represents a subscription feed (one per artist+platform — the
URL polled by the FC-3 downloader). The filesystem importer used to
call _find_or_create_source(url=sd.post_url), creating one Source
row per post URL — 100s of junk Sources per artist, all with
enabled=True, polluting the artist detail page and tricking the
subscription checker into trying to poll patreon post URLs as feeds.
Operator-flagged 2026-05-26; consolidated via alembic 0022.
Resolution order: prefer a real (non-sidecar) Source over a
synthetic anchor. When alembic 0022 ran, it may have rewritten
per-post Sources into `sidecar:<platform>:<slug>` synthetic
anchors. If the operator later added the real subscription, both
rows now coexist. A naive `ORDER BY id ASC LIMIT 1` lookup would
pick the older synthetic and silently attach every gallery-dl
download to the wrong Source — operator-flagged 2026-05-31 after
the Subscriptions UI surfaced the phantom anchors. Pick the real
one when one exists; fall back to the synthetic; only create a
new synthetic when nothing exists for (artist, platform).
Pre-alembic-0030 this method would CREATE a synthetic
`sidecar:<platform>:<slug>` Source when no real one existed
because `Post.source_id` was NOT NULL and the importer needed
something to attach Posts to. Alembic 0030 relaxed both
`Post.source_id` and `ImageProvenance.source_id` to nullable, so
synthetic anchors are obsolete; the importer now leaves
source_id as None when no subscription exists for the (artist,
platform). Operator-asked 2026-06-01: synthetic Sources had
leaked into the Subscriptions UI as phantom subscriptions and
the operator wanted the data model to truthfully say "this
content has no live subscription."
"""
real_stmt = (
select(Source)
.where(
Source.artist_id == artist_id,
Source.platform == platform,
~Source.url.like("sidecar:%"),
)
.order_by(Source.id.asc())
.limit(1)
)
real = self.session.execute(real_stmt).scalar_one_or_none()
if real is not None:
return real
any_stmt = (
stmt = (
select(Source)
.where(
Source.artist_id == artist_id,
@@ -307,29 +285,37 @@ class Importer:
.order_by(Source.id.asc())
.limit(1)
)
return self._get_or_create(
any_stmt,
lambda: Source(
artist_id=artist_id,
platform=platform,
url=f"sidecar:{platform}:{artist_slug}",
enabled=False,
),
)
return self.session.execute(stmt).scalar_one_or_none()
def _find_or_create_post(
self, *, source_id: int, external_post_id: str,
self, *, source_id: int | None, external_post_id: str,
artist_id: int,
) -> Post:
"""Race-safe find-or-create on `post` keyed by
(source_id, external_post_id). Mirrors `_find_or_create_source`
— same savepoint + IntegrityError-recovery pattern."""
stmt = select(Post).where(
Post.source_id == source_id,
Post.external_post_id == external_post_id,
)
"""Race-safe find-or-create on `post`. Keyed by
(source_id, external_post_id) when source_id is set — the
`uq_post_source_external_id` constraint guards. For NULL-source
posts the existence check matches on (artist_id, external_post_id),
which the partial unique index `uq_post_artist_external_id_null_source`
(alembic 0030) guards. Same savepoint + IntegrityError-recovery
pattern as the rest of the helpers."""
if source_id is not None:
stmt = select(Post).where(
Post.source_id == source_id,
Post.external_post_id == external_post_id,
)
else:
stmt = select(Post).where(
Post.source_id.is_(None),
Post.artist_id == artist_id,
Post.external_post_id == external_post_id,
)
return self._get_or_create(
stmt,
lambda: Post(source_id=source_id, external_post_id=external_post_id),
lambda: Post(
source_id=source_id,
artist_id=artist_id,
external_post_id=external_post_id,
),
)
def import_one(self, source: Path) -> ImportResult:
@@ -370,12 +356,14 @@ class Importer:
return None
sd = parse_sidecar(data)
platform = sd.platform or "unknown"
src = self._source_for_sidecar(
artist_id=artist.id, platform=platform, artist_slug=artist.slug,
src = self._lookup_source_for_sidecar(
artist_id=artist.id, platform=platform,
)
epid = sd.external_post_id or sc.stem
return self._find_or_create_post(
source_id=src.id, external_post_id=epid,
source_id=src.id if src else None,
external_post_id=epid,
artist_id=artist.id,
)
def _capture_attachment(
@@ -858,14 +846,15 @@ class Importer:
src = explicit_source
else:
platform = sd.platform or "unknown"
src = self._source_for_sidecar(
src = self._lookup_source_for_sidecar(
artist_id=artist.id, platform=platform,
artist_slug=artist.slug,
)
epid = sd.external_post_id or sc.stem
post = self._find_or_create_post(
source_id=src.id, external_post_id=epid,
source_id=src.id if src else None,
external_post_id=epid,
artist_id=artist.id,
)
if sd.post_url is not None:
post.post_url = sd.post_url
@@ -901,7 +890,7 @@ class Importer:
ImageProvenance(
image_record_id=record.id,
post_id=post.id,
source_id=src.id,
source_id=src.id if src else None,
captured_metadata=sd.raw,
)
)