feat(model): nullable Post.source_id + denormalized Post.artist_id; retire sidecar synthetics
CI / lint (push) Failing after 3s
CI / backend-lint-and-test (push) Successful in 11s
CI / frontend-build (push) Successful in 17s
CI / intimp (push) Failing after 2m15s
CI / intapi (push) Failing after 2m18s
CI / intcore (push) Failing after 2m17s

Operator-asked 2026-06-01 after the Dymkens orphan investigation
(Scribe plan #540). The pre-2030 sidecar-synthetic Source pattern
(`sidecar:<platform>:<slug>` enabled=false rows) existed solely to
satisfy `Post.source_id NOT NULL`, and leaked into the Subscriptions
UI as phantom subscriptions. Now the data model says what's true:
filesystem-imported content with no live subscription has NULL
source_id, full stop.

## Schema (alembic 0030)

- `post.artist_id` — NEW NOT NULL FK to artist (CASCADE). Backfilled
  from source.artist_id in the migration. Indexed for the artist-filter
  queries.
- `post.source_id` — NOT NULL → nullable; FK ondelete CASCADE → SET
  NULL. Deleting a Source detaches its Posts instead of destroying
  archived content (subscription ends, archive stays).
- `image_provenance.source_id` — same nullable + SET NULL.
- Partial unique index `uq_post_artist_external_id_null_source` on
  (artist_id, external_post_id) WHERE source_id IS NULL — guards
  filesystem-import dedup since the existing source-bound unique
  ignores NULLs (Postgres treats NULL != NULL).
- Sidecar synthetic Sources deleted: NULL out FKs in post,
  image_provenance first, then DELETE FROM source WHERE url LIKE
  'sidecar:%'. The Dymkens cleanup.

## Model + service changes

- `Post.source_id` → `Mapped[int | None]`; new `Post.artist_id`
  denormalized.
- `ImageProvenance.source_id` → `Mapped[int | None]`.
- Importer: `_source_for_sidecar` (synthetic-creating) →
  `_lookup_source_for_sidecar` (returns None when no subscription).
  `_find_or_create_post` takes required `artist_id`; matches on
  (source_id, external_post_id) for source-bound posts or
  (artist_id, external_post_id) for NULL-source posts.
- Service queries switched off the Source detour to use Post.artist_id
  directly: post_feed_service.scroll/around/get_post (LEFT JOIN to
  Source so NULL-source posts surface); artist_service date_row/
  activity/post_count; provenance_service.for_image/for_post (LEFT
  JOIN); gallery_service._provenance_exists_where_artist via
  Post.artist_id instead of ImageProvenance.source_id → Source.
- `_to_dict` and provenance dict-builders emit `"source": null` for
  NULL-source rows.

## Frontend

- `ProvenancePanel.vue` + `PostCard.vue`: render `e.source?.platform
  ?? 'filesystem import'` so NULL-source posts get a clear
  "filesystem import" affordance instead of a NaN crash.

## Tests

- `test_importer_upsert_helpers`: removed the four synthetic-anchor
  tests; added `_find_or_create_post_idempotent_with_null_source`
  (dedup via the partial unique index) and
  `_lookup_source_for_sidecar_returns_*` (existing-subscription +
  none cases). The existing `_find_or_create_post_idempotent` now
  also passes `artist_id` and asserts it.
- 8 other test files updated: every direct `Post(...)` construction
  gains `artist_id=<artist>.id`. The `_seed_post` helper in
  `test_post_feed_service` looks up artist_id from the source row so
  callsites stay one-arg.

## Verification on deploy

After alembic 0030 runs:
- `SELECT COUNT(*) FROM source WHERE url LIKE 'sidecar:%'` → 0.
- `SELECT COUNT(*) FROM post WHERE source_id IS NULL` → count of
  filesystem-imported posts (Dymkens + any other historical).
- Every `post.artist_id` non-null; consistent with source.artist_id
  for source-bound rows.
- Subscriptions tab: no Dymkens phantom row.
- Artist detail → Posts/Gallery: Dymkens's content still reachable
  via Post.artist_id.
- Provenance panel renders "filesystem import" chip for NULL-source
  posts; PostCard same.

## Out of scope

- UI to manage/delete orphan NULL-source Posts. Data model is right;
  UI follows if operator wants it.
This commit is contained in:
2026-06-01 14:17:52 -04:00
parent 94e7d20792
commit 2f66de2928
23 changed files with 358 additions and 192 deletions
@@ -0,0 +1,141 @@
"""nullable post.source_id + denormalized post.artist_id; retire sidecar synthetics
Revision ID: 0030
Revises: 0029
Create Date: 2026-06-01
Operator-asked 2026-06-01 after the Dymkens orphan investigation: the
sidecar synthetic Source pattern (`sidecar:<platform>:<slug>` rows
with enabled=false) was technically correct but misled the operator
into thinking they had phantom subscriptions. The synthetics existed
solely to satisfy `Post.source_id NOT NULL` for filesystem-imported
content with no real subscription.
This migration makes the data model honest:
1. **Post gets a denormalized `artist_id` column** so artist filters
work without traversing `Post → Source.artist_id`. Backfilled from
the existing Source linkage, then NOT NULL'd.
2. **`Post.source_id` becomes nullable**, FK ondelete `CASCADE` → `SET
NULL`. Deleting a Source detaches its Posts instead of destroying
imported content (semantically: subscription ends, archive stays).
3. **`ImageProvenance.source_id` becomes nullable** with the same FK
semantic change.
4. **Sidecar synthetic Sources are deleted** — first NULL out the
FKs from Post + ImageProvenance pointing at them (so the implicit
CASCADE doesn't fire), then delete. DownloadEvent FK is unchanged
(still CASCADE'd, NOT NULL'd) — synthetics have `enabled=false`
so no events exist for them.
Uniqueness handling: the existing `uq_post_source_external_id`
(source_id, external_post_id) keeps working for source-bound Posts
(Postgres treats NULL != NULL so NULL-source rows aren't deduped by
it). A second partial unique index covers the NULL-source case on
(artist_id, external_post_id) so filesystem-imported posts still
dedupe within an artist.
"""
from typing import Sequence, Union
import sqlalchemy as sa
from alembic import op
from sqlalchemy import text
revision: str = "0030"
down_revision: Union[str, None] = "0029"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
conn = op.get_bind()
# Step 1: add Post.artist_id, initially nullable for backfill.
op.add_column(
"post",
sa.Column("artist_id", sa.Integer, nullable=True),
)
op.create_foreign_key(
"post_artist_id_fkey", "post", "artist",
["artist_id"], ["id"], ondelete="CASCADE",
)
# Step 2: backfill from Source.artist_id (every existing Post has a
# Source today, so every row gets populated).
conn.execute(text("""
UPDATE post p
SET artist_id = s.artist_id
FROM source s
WHERE p.source_id = s.id AND p.artist_id IS NULL
"""))
# Sanity: count any remaining NULLs. Should be zero pre-this-migration.
remaining = conn.execute(text(
"SELECT COUNT(*) FROM post WHERE artist_id IS NULL"
)).scalar_one()
if remaining:
raise RuntimeError(
f"alembic 0030: {remaining} post rows have no resolvable "
f"artist_id after backfill. Investigate before continuing."
)
# Step 3: enforce NOT NULL + add index for artist-filter queries.
op.alter_column("post", "artist_id", nullable=False)
op.create_index("ix_post_artist_id", "post", ["artist_id"])
# Step 4: relax post.source_id + flip FK to SET NULL.
op.alter_column("post", "source_id", nullable=True)
op.drop_constraint("post_source_id_fkey", "post", type_="foreignkey")
op.create_foreign_key(
"post_source_id_fkey", "post", "source",
["source_id"], ["id"], ondelete="SET NULL",
)
# Step 5: relax image_provenance.source_id + flip FK to SET NULL.
op.alter_column("image_provenance", "source_id", nullable=True)
op.drop_constraint(
"image_provenance_source_id_fkey", "image_provenance",
type_="foreignkey",
)
op.create_foreign_key(
"image_provenance_source_id_fkey", "image_provenance", "source",
["source_id"], ["id"], ondelete="SET NULL",
)
# Step 6: partial unique index on (artist_id, external_post_id) for
# NULL-source Posts. The existing uq_post_source_external_id keeps
# guarding source-bound rows; NULL-source rows now dedupe within
# an artist.
op.execute(
"CREATE UNIQUE INDEX uq_post_artist_external_id_null_source "
"ON post (artist_id, external_post_id) "
"WHERE source_id IS NULL"
)
# Step 7: retire sidecar synthetic Sources. NULL out the references
# FIRST (the new FK is SET NULL so CASCADE wouldn't fire anyway, but
# being explicit makes the intent clear). Then delete the synthetic
# source rows. Any DownloadEvent rows under synthetics CASCADE-die
# with the source — synthetics have enabled=false so there shouldn't
# be any in practice.
conn.execute(text("""
UPDATE post
SET source_id = NULL
WHERE source_id IN (SELECT id FROM source WHERE url LIKE 'sidecar:%')
"""))
conn.execute(text("""
UPDATE image_provenance
SET source_id = NULL
WHERE source_id IN (SELECT id FROM source WHERE url LIKE 'sidecar:%')
"""))
deleted = conn.execute(text(
"DELETE FROM source WHERE url LIKE 'sidecar:%' RETURNING id"
)).rowcount
print(f"alembic 0030: deleted {deleted} sidecar synthetic source rows")
def downgrade() -> None:
# Lossy migration — the deleted sidecar synthetics can't be
# restored from the orphan post.source_id / image_provenance.source_id
# values, and the partial unique index encodes a constraint that
# NULL-source Posts may now exist. No safe downgrade.
pass
+6 -2
View File
@@ -34,8 +34,12 @@ class ImageProvenance(Base):
post_id: Mapped[int] = mapped_column(
ForeignKey("post.id", ondelete="CASCADE"), nullable=False, index=True
)
source_id: Mapped[int] = mapped_column(
ForeignKey("source.id", ondelete="CASCADE"), nullable=False, index=True
# Nullable since alembic 0030 — provenance rows for filesystem-imported
# content with no subscription have NULL source_id. FK ondelete SET
# NULL so deleting a Source detaches its provenance rows instead of
# destroying the linkage between image and post.
source_id: Mapped[int | None] = mapped_column(
ForeignKey("source.id", ondelete="SET NULL"), nullable=True, index=True
)
captured_metadata: Mapped[dict | None] = mapped_column(JSON, nullable=True)
captured_at: Mapped[datetime] = mapped_column(
+20 -4
View File
@@ -1,6 +1,9 @@
"""Post — provenance anchor for content downloaded from a Source.
"""Post — provenance anchor for one creator post (may contain many images).
A Post is one creator post; it may contain many images/videos.
`source_id` is nullable since alembic 0030 — filesystem-imported posts
with no live subscription have NULL source_id. `artist_id` is the
denormalized always-present link to the creator (added in 0030 so
artist-filter queries don't depend on the Source detour).
"""
from datetime import datetime
@@ -14,12 +17,25 @@ from .base import Base
class Post(Base):
__tablename__ = "post"
__table_args__ = (
# Source-bound dedup. Postgres treats NULL != NULL so rows
# with source_id IS NULL aren't deduped by this constraint;
# the partial unique index `uq_post_artist_external_id_null_source`
# (created in alembic 0030) covers that case via
# (artist_id, external_post_id).
UniqueConstraint("source_id", "external_post_id", name="uq_post_source_external_id"),
)
id: Mapped[int] = mapped_column(Integer, primary_key=True)
source_id: Mapped[int] = mapped_column(
ForeignKey("source.id", ondelete="CASCADE"), nullable=False, index=True
source_id: Mapped[int | None] = mapped_column(
ForeignKey("source.id", ondelete="SET NULL"), nullable=True, index=True
)
# Denormalized; always equals source.artist_id when source_id is set
# (the importer is responsible for keeping them consistent on insert).
# Filter queries (artist detail, artist-scoped posts feed) use this
# directly instead of joining through Source.
artist_id: Mapped[int] = mapped_column(
ForeignKey("artist.id", ondelete="CASCADE"),
nullable=False, index=True,
)
external_post_id: Mapped[str] = mapped_column(String(128), nullable=False)
post_url: Mapped[str | None] = mapped_column(Text, nullable=True)
+9 -7
View File
@@ -58,13 +58,17 @@ class ArtistService:
)
).scalar_one()
# Posts under this artist that have at least one image attached.
# Use Post.artist_id (alembic 0030) for the artist filter; keep
# the ImageProvenance JOIN so date bounds reflect only image-
# bearing posts (matches the original semantic). NULL-source
# posts now surface too.
date_row = (
await self.session.execute(
select(func.min(Post.post_date), func.max(Post.post_date))
.select_from(Post)
.join(ImageProvenance, ImageProvenance.post_id == Post.id)
.join(Source, Source.id == ImageProvenance.source_id)
.where(Source.artist_id == aid)
.where(Post.artist_id == aid)
)
).first()
dmin, dmax = date_row if date_row else (None, None)
@@ -98,14 +102,14 @@ class ArtistService:
)
).all()
# Same Post.artist_id direct filter — counts NULL-source posts too.
month = func.date_trunc("month", Post.post_date).label("m")
activity = (
await self.session.execute(
select(month, func.count(func.distinct(ImageProvenance.image_record_id)))
.select_from(Post)
.join(ImageProvenance, ImageProvenance.post_id == Post.id)
.join(Source, Source.id == ImageProvenance.source_id)
.where(and_(Source.artist_id == aid, Post.post_date.isnot(None)))
.where(and_(Post.artist_id == aid, Post.post_date.isnot(None)))
.group_by(month)
.order_by(month)
)
@@ -114,9 +118,7 @@ class ArtistService:
post_count = (
await self.session.execute(
select(func.count(func.distinct(Post.id)))
.select_from(Post)
.join(Source, Source.id == Post.source_id)
.where(Source.artist_id == aid)
.where(Post.artist_id == aid)
)
).scalar_one()
+8 -2
View File
@@ -133,10 +133,16 @@ def _provenance_clause(post_id, artist_id):
ImageProvenance.post_id == post_id,
)
if artist_id is not None:
# Use Post.artist_id (alembic 0030 denormalized column) instead
# of joining through ImageProvenance.source_id → Source.artist_id.
# The denormalization is the always-present linkage; the source
# path now drops NULL-source provenance rows (filesystem-imported
# content) which would otherwise vanish from artist-filtered
# gallery views.
return exists().where(
ImageProvenance.image_record_id == ImageRecord.id,
ImageProvenance.source_id == Source.id,
Source.artist_id == artist_id,
ImageProvenance.post_id == Post.id,
Post.artist_id == artist_id,
)
return None
+53 -64
View File
@@ -212,10 +212,10 @@ class Importer:
which would lose the surrounding scan's progress — and re-run `stmt`
(scalar_one) to return the row the other worker created.
Centralizes the pattern shared by _find_or_create_source,
_source_for_sidecar, and _find_or_create_post. The plain
SELECT-then-INSERT version lost races under the 5-min recovery sweep
(operator-flagged 2026-05-26)."""
Centralizes the pattern shared by _find_or_create_source and
_find_or_create_post. The plain SELECT-then-INSERT version lost
races under the 5-min recovery sweep (operator-flagged
2026-05-26)."""
existing = self.session.execute(stmt).scalar_one_or_none()
if existing is not None:
return existing
@@ -258,47 +258,25 @@ class Importer:
lambda: Source(artist_id=artist_id, platform=platform, url=url),
)
def _source_for_sidecar(
self, *, artist_id: int, platform: str, artist_slug: str,
) -> Source:
"""Sidecar-import Source resolver. Used by both filesystem imports
and gallery-dl downloads (both write sidecar JSON, both flow through
_apply_sidecar / _capture_attachment).
def _lookup_source_for_sidecar(
self, *, artist_id: int, platform: str,
) -> Source | None:
"""Find the real subscription Source for (artist, platform), or
None if no subscription exists.
Source represents a subscription feed (one per artist+platform — the
URL polled by the FC-3 downloader). The filesystem importer used to
call _find_or_create_source(url=sd.post_url), creating one Source
row per post URL — 100s of junk Sources per artist, all with
enabled=True, polluting the artist detail page and tricking the
subscription checker into trying to poll patreon post URLs as feeds.
Operator-flagged 2026-05-26; consolidated via alembic 0022.
Resolution order: prefer a real (non-sidecar) Source over a
synthetic anchor. When alembic 0022 ran, it may have rewritten
per-post Sources into `sidecar:<platform>:<slug>` synthetic
anchors. If the operator later added the real subscription, both
rows now coexist. A naive `ORDER BY id ASC LIMIT 1` lookup would
pick the older synthetic and silently attach every gallery-dl
download to the wrong Source — operator-flagged 2026-05-31 after
the Subscriptions UI surfaced the phantom anchors. Pick the real
one when one exists; fall back to the synthetic; only create a
new synthetic when nothing exists for (artist, platform).
Pre-alembic-0030 this method would CREATE a synthetic
`sidecar:<platform>:<slug>` Source when no real one existed
because `Post.source_id` was NOT NULL and the importer needed
something to attach Posts to. Alembic 0030 relaxed both
`Post.source_id` and `ImageProvenance.source_id` to nullable, so
synthetic anchors are obsolete; the importer now leaves
source_id as None when no subscription exists for the (artist,
platform). Operator-asked 2026-06-01: synthetic Sources had
leaked into the Subscriptions UI as phantom subscriptions and
the operator wanted the data model to truthfully say "this
content has no live subscription."
"""
real_stmt = (
select(Source)
.where(
Source.artist_id == artist_id,
Source.platform == platform,
~Source.url.like("sidecar:%"),
)
.order_by(Source.id.asc())
.limit(1)
)
real = self.session.execute(real_stmt).scalar_one_or_none()
if real is not None:
return real
any_stmt = (
stmt = (
select(Source)
.where(
Source.artist_id == artist_id,
@@ -307,29 +285,37 @@ class Importer:
.order_by(Source.id.asc())
.limit(1)
)
return self._get_or_create(
any_stmt,
lambda: Source(
artist_id=artist_id,
platform=platform,
url=f"sidecar:{platform}:{artist_slug}",
enabled=False,
),
)
return self.session.execute(stmt).scalar_one_or_none()
def _find_or_create_post(
self, *, source_id: int, external_post_id: str,
self, *, source_id: int | None, external_post_id: str,
artist_id: int,
) -> Post:
"""Race-safe find-or-create on `post` keyed by
(source_id, external_post_id). Mirrors `_find_or_create_source`
— same savepoint + IntegrityError-recovery pattern."""
"""Race-safe find-or-create on `post`. Keyed by
(source_id, external_post_id) when source_id is set — the
`uq_post_source_external_id` constraint guards. For NULL-source
posts the existence check matches on (artist_id, external_post_id),
which the partial unique index `uq_post_artist_external_id_null_source`
(alembic 0030) guards. Same savepoint + IntegrityError-recovery
pattern as the rest of the helpers."""
if source_id is not None:
stmt = select(Post).where(
Post.source_id == source_id,
Post.external_post_id == external_post_id,
)
else:
stmt = select(Post).where(
Post.source_id.is_(None),
Post.artist_id == artist_id,
Post.external_post_id == external_post_id,
)
return self._get_or_create(
stmt,
lambda: Post(source_id=source_id, external_post_id=external_post_id),
lambda: Post(
source_id=source_id,
artist_id=artist_id,
external_post_id=external_post_id,
),
)
def import_one(self, source: Path) -> ImportResult:
@@ -370,12 +356,14 @@ class Importer:
return None
sd = parse_sidecar(data)
platform = sd.platform or "unknown"
src = self._source_for_sidecar(
artist_id=artist.id, platform=platform, artist_slug=artist.slug,
src = self._lookup_source_for_sidecar(
artist_id=artist.id, platform=platform,
)
epid = sd.external_post_id or sc.stem
return self._find_or_create_post(
source_id=src.id, external_post_id=epid,
source_id=src.id if src else None,
external_post_id=epid,
artist_id=artist.id,
)
def _capture_attachment(
@@ -858,14 +846,15 @@ class Importer:
src = explicit_source
else:
platform = sd.platform or "unknown"
src = self._source_for_sidecar(
src = self._lookup_source_for_sidecar(
artist_id=artist.id, platform=platform,
artist_slug=artist.slug,
)
epid = sd.external_post_id or sc.stem
post = self._find_or_create_post(
source_id=src.id, external_post_id=epid,
source_id=src.id if src else None,
external_post_id=epid,
artist_id=artist.id,
)
if sd.post_url is not None:
post.post_url = sd.post_url
@@ -901,7 +890,7 @@ class Importer:
ImageProvenance(
image_record_id=record.id,
post_id=post.id,
source_id=src.id,
source_id=src.id if src else None,
captured_metadata=sd.raw,
)
)
+21 -9
View File
@@ -69,13 +69,19 @@ class PostFeedService:
raise ValueError("direction must be 'older' or 'newer'")
sort_key = _sort_key()
# Artist via the denormalized Post.artist_id (alembic 0030);
# Source via LEFT JOIN since post.source_id can now be NULL for
# filesystem-imported posts with no live subscription. A
# platform= filter implicitly excludes NULL-source posts (they
# have no platform); an artist_id= filter still surfaces them
# because Post.artist_id is always set.
stmt = (
select(Post, Artist, Source)
.join(Source, Post.source_id == Source.id)
.join(Artist, Source.artist_id == Artist.id)
.join(Artist, Post.artist_id == Artist.id)
.outerjoin(Source, Post.source_id == Source.id)
)
if artist_id is not None:
stmt = stmt.where(Source.artist_id == artist_id)
stmt = stmt.where(Post.artist_id == artist_id)
if platform is not None:
stmt = stmt.where(Source.platform == platform)
if cursor:
@@ -135,8 +141,8 @@ class PostFeedService:
cursor for each end. Returns None if the post doesn't exist."""
anchor = (await self.session.execute(
select(Post, Artist, Source)
.join(Source, Post.source_id == Source.id)
.join(Artist, Source.artist_id == Artist.id)
.join(Artist, Post.artist_id == Artist.id)
.outerjoin(Source, Post.source_id == Source.id)
.where(Post.id == post_id)
)).one_or_none()
if anchor is None:
@@ -168,8 +174,8 @@ class PostFeedService:
async def get_post(self, post_id: int) -> dict | None:
row = (await self.session.execute(
select(Post, Artist, Source)
.join(Source, Post.source_id == Source.id)
.join(Artist, Source.artist_id == Artist.id)
.join(Artist, Post.artist_id == Artist.id)
.outerjoin(Source, Post.source_id == Source.id)
.where(Post.id == post_id)
)).one_or_none()
if row is None:
@@ -260,7 +266,7 @@ class PostFeedService:
return out
def _to_dict(
self, post: Post, artist: Artist, source: Source,
self, post: Post, artist: Artist, source: Source | None,
thumbs_map: dict, atts_map: dict,
) -> dict:
plain_full = html_to_plain(post.description) if post.description else None
@@ -269,6 +275,9 @@ class PostFeedService:
else:
description_plain, truncated = truncate_at_word(plain_full, DESCRIPTION_LIMIT)
thumbs_entry = thumbs_map.get(post.id, {"thumbs": [], "more": 0})
# `source` is null for filesystem-imported posts with no live
# subscription (alembic 0030). Frontend renders that as a
# "filesystem import" affordance instead of a platform chip.
return {
"id": post.id,
"external_post_id": post.external_post_id,
@@ -279,7 +288,10 @@ class PostFeedService:
"description_plain": description_plain,
"description_truncated": truncated,
"artist": {"id": artist.id, "name": artist.name, "slug": artist.slug},
"source": {"id": source.id, "platform": source.platform},
"source": (
{"id": source.id, "platform": source.platform}
if source is not None else None
),
"thumbnails": thumbs_entry["thumbs"],
"thumbnails_more": thumbs_entry["more"],
"attachments": atts_map.get(post.id, []),
+12 -6
View File
@@ -70,11 +70,15 @@ class ProvenanceService:
rec = await self.session.get(ImageRecord, image_id)
if rec is None:
return None
# Artist via Post.artist_id (alembic 0030); Source via LEFT JOIN
# since both Post.source_id and ImageProvenance.source_id can be
# NULL for filesystem-imported content. Frontend renders source=
# null as "filesystem import."
stmt = (
select(ImageProvenance, Post, Source, Artist)
.join(Post, Post.id == ImageProvenance.post_id)
.join(Source, Source.id == ImageProvenance.source_id)
.join(Artist, Artist.id == Source.artist_id)
.join(Artist, Artist.id == Post.artist_id)
.outerjoin(Source, Source.id == ImageProvenance.source_id)
.where(ImageProvenance.image_record_id == image_id)
.order_by(ImageProvenance.captured_at.asc(),
ImageProvenance.id.asc())
@@ -90,7 +94,7 @@ class ProvenanceService:
"captured_at": ip.captured_at.isoformat()
if ip.captured_at else None,
"post": _post_dict(post),
"source": _source_dict(src),
"source": _source_dict(src) if src is not None else None,
"artist": _artist_dict(art),
}
for ip, post, src, art in rows
@@ -99,10 +103,12 @@ class ProvenanceService:
}
async def for_post(self, post_id: int) -> dict | None:
# Same LEFT JOIN to Source — get_post must succeed for a
# NULL-source post.
stmt = (
select(Post, Source, Artist)
.join(Source, Source.id == Post.source_id)
.join(Artist, Artist.id == Source.artist_id)
.join(Artist, Artist.id == Post.artist_id)
.outerjoin(Source, Source.id == Post.source_id)
.where(Post.id == post_id)
)
row = (await self.session.execute(stmt)).first()
@@ -111,7 +117,7 @@ class ProvenanceService:
post, src, art = row
return {
"post": _post_dict(post),
"source": _source_dict(src),
"source": _source_dict(src) if src is not None else None,
"artist": _artist_dict(art),
"attachments": await self._attachments_for_posts([post.id]),
}
+1 -1
View File
@@ -157,7 +157,7 @@ class SourceService:
if artist_id is not None:
stmt = stmt.where(Source.artist_id == artist_id)
if not include_synthetic:
# Filesystem-import sidecar anchors (importer._source_for_sidecar)
# Pre-alembic-0030 sidecar synthetic anchors
# have url='sidecar:<platform>:<slug>' and exist only to give
# imported Posts a NOT-NULL Source FK. They aren't pollable
# feeds; the Subscriptions UI used to render them as phantom
@@ -19,7 +19,12 @@
v-for="e in state.entries" :key="e.provenance_id" class="fc-prov__card"
>
<div class="fc-prov__head">
<span class="fc-prov__platform">{{ e.source.platform }}</span>
<!-- Posts with no live subscription have source=null (alembic
0030); render an explicit "filesystem import" affordance
instead of a platform chip. -->
<span class="fc-prov__platform">
{{ e.source?.platform ?? 'filesystem import' }}
</span>
<span v-if="postDate(e)" class="fc-prov__date">{{ postDate(e) }}</span>
</div>
<button
+7 -2
View File
@@ -7,7 +7,12 @@
@keydown.enter="onCardClick"
>
<div class="fc-post-card__head">
<v-chip size="x-small" variant="tonal">{{ post.source.platform }}</v-chip>
<!-- Posts with no live subscription have source=null (alembic
0030); show a "filesystem import" affordance instead of a
platform chip. -->
<v-chip size="x-small" variant="tonal">
{{ post.source?.platform ?? 'filesystem import' }}
</v-chip>
<RouterLink
:to="{ name: 'artist', params: { slug: post.artist.slug } }"
class="fc-post-card__artist"
@@ -25,7 +30,7 @@
v-if="post.post_url"
:href="post.post_url" target="_blank" rel="noopener"
icon="mdi-open-in-new" size="x-small" variant="text"
:aria-label="`open original post on ${post.source.platform}`"
:aria-label="`open original post on ${post.source?.platform ?? 'web'}`"
@click.stop
/>
<v-btn
+2 -2
View File
@@ -37,8 +37,8 @@ async def test_artist_overview_post_count(client, db):
)
db.add(s)
await db.flush()
db.add(Post(source_id=s.id, external_post_id="p1"))
db.add(Post(source_id=s.id, external_post_id="p2"))
db.add(Post(source_id=s.id, artist_id=a.id, external_post_id="p1"))
db.add(Post(source_id=s.id, artist_id=a.id, external_post_id="p2"))
await db.flush()
await db.commit()
resp = await client.get("/api/artist/lyra")
+4 -3
View File
@@ -25,7 +25,7 @@ async def seeded_post(db):
db.add(source)
await db.flush()
post = Post(
source_id=source.id, external_post_id="API1",
source_id=source.id, artist_id=artist.id, external_post_id="API1",
post_title="Hello", post_url="https://p/alice-api/1",
post_date=datetime.now(UTC),
description="<p>hi</p>",
@@ -123,7 +123,8 @@ async def post_timeline(db):
posts = []
for i in range(5):
p = Post(
source_id=source.id, external_post_id=f"TL{i}",
source_id=source.id, artist_id=artist.id,
external_post_id=f"TL{i}",
post_title=f"post {i}", post_date=base + timedelta(days=i),
)
db.add(p)
@@ -189,7 +190,7 @@ async def test_detail_returns_uncapped_thumbnails(client, db):
db.add(s)
await db.flush()
p = Post(
source_id=s.id, external_post_id="DETAIL10",
source_id=s.id, artist_id=a.id, external_post_id="DETAIL10",
post_title="big post", description="<p>body</p>",
)
db.add(p)
+2 -1
View File
@@ -26,7 +26,8 @@ async def _seed_full(db):
url="https://patreon.test/alice")
db.add(source)
await db.flush()
post = Post(source_id=source.id, external_post_id="555",
post = Post(source_id=source.id, artist_id=artist.id,
external_post_id="555",
post_url="https://patreon.test/p/555", post_title="Set 1",
post_date=datetime(2023, 8, 1, tzinfo=UTC),
description="<p>hi</p>", attachment_count=2)
+1 -1
View File
@@ -25,7 +25,7 @@ async def _fixture(db):
db.add(src)
await db.flush()
post = Post(
source_id=src.id, external_post_id="p1",
source_id=src.id, artist_id=artist.id, external_post_id="p1",
post_date=datetime(2026, 3, 1, tzinfo=UTC),
)
db.add(post)
+1 -1
View File
@@ -35,7 +35,7 @@ async def _post(db, artist_name, slug, ext):
url=f"https://patreon.test/{slug}")
db.add(s)
await db.flush()
p = Post(source_id=s.id, external_post_id=ext)
p = Post(source_id=s.id, artist_id=a.id, external_post_id=ext)
db.add(p)
await db.flush()
return a, s, p
+2 -1
View File
@@ -152,7 +152,8 @@ async def _seed_image_with_post(
db.add(source)
await db.flush()
post = Post(
source_id=source.id, external_post_id=external_post_id,
source_id=source.id, artist_id=artist.id,
external_post_id=external_post_id,
post_title="A Post", post_date=post_date,
)
db.add(post)
+45 -74
View File
@@ -88,11 +88,36 @@ def test_find_or_create_post_idempotent(importer, artist_row, db_sync):
)
p1 = importer._find_or_create_post(
source_id=src.id, external_post_id="ext-001",
artist_id=artist_row.id,
)
p2 = importer._find_or_create_post(
source_id=src.id, external_post_id="ext-001",
artist_id=artist_row.id,
)
assert p1.id == p2.id
assert p1.artist_id == artist_row.id
def test_find_or_create_post_idempotent_with_null_source(
importer, artist_row, db_sync,
):
"""Post.source_id is nullable since alembic 0030 — filesystem-imported
posts with no live subscription have NULL source_id. The partial
unique index `uq_post_artist_external_id_null_source` on
(artist_id, external_post_id) WHERE source_id IS NULL guards the
dedup; the helper matches on the same (artist_id, external_post_id)
keys when source_id is None."""
p1 = importer._find_or_create_post(
source_id=None, external_post_id="fs-001",
artist_id=artist_row.id,
)
p2 = importer._find_or_create_post(
source_id=None, external_post_id="fs-001",
artist_id=artist_row.id,
)
assert p1.id == p2.id
assert p1.source_id is None
assert p1.artist_id == artist_row.id
def test_find_or_create_source_recovers_from_integrity_error(
@@ -147,13 +172,13 @@ def test_find_or_create_source_recovers_from_integrity_error(
assert recovered.id == pre_existing.id
def test_source_for_sidecar_reuses_existing_subscription(
def test_lookup_source_for_sidecar_returns_existing_subscription(
importer, artist_row, db_sync,
):
"""The filesystem-import sidecar resolver should attach to whatever
Source already exists for (artist, platform) — the canonical subscription
Source — regardless of its URL. Without this, every imported post
spawned its own Source row.
"""The sidecar-import Source lookup returns the existing Source for
(artist, platform) when one exists — letting filesystem-imported
content attach to the artist's real subscription instead of being
orphaned.
"""
canonical = Source(
artist_id=artist_row.id, platform="patreon",
@@ -162,84 +187,30 @@ def test_source_for_sidecar_reuses_existing_subscription(
db_sync.add(canonical)
db_sync.flush()
resolved = importer._source_for_sidecar(
resolved = importer._lookup_source_for_sidecar(
artist_id=artist_row.id, platform="patreon",
artist_slug=artist_row.slug,
)
assert resolved is not None
assert resolved.id == canonical.id
def test_source_for_sidecar_creates_synthetic_anchor_when_none_exists(
def test_lookup_source_for_sidecar_returns_none_when_no_subscription(
importer, artist_row, db_sync,
):
"""No subscription Source for this (artist, platform) yet. The helper
creates one synthetic anchor (enabled=False, url='sidecar:<plat>:<slug>')
so subsequent imports reuse it instead of spawning per-post Sources.
"""Alembic 0030 made Post.source_id nullable; the importer no
longer creates synthetic `sidecar:<platform>:<slug>` anchor rows
when no real subscription exists. The lookup returns None and the
caller carries that through as a NULL source_id on the new Post.
"""
resolved = importer._source_for_sidecar(
resolved = importer._lookup_source_for_sidecar(
artist_id=artist_row.id, platform="pixiv",
artist_slug=artist_row.slug,
)
assert resolved.url == f"sidecar:pixiv:{artist_row.slug}"
assert resolved.enabled is False
assert resolved.artist_id == artist_row.id
assert resolved.platform == "pixiv"
assert resolved is None
# Second call returns the same row (no new Source spawned).
again = importer._source_for_sidecar(
artist_id=artist_row.id, platform="pixiv",
artist_slug=artist_row.slug,
# Verify no Source row was created as a side effect.
count = db_sync.execute(
select(Source).where(
Source.artist_id == artist_row.id, Source.platform == "pixiv",
)
assert again.id == resolved.id
def test_source_for_sidecar_distinct_platforms_distinct_anchors(
importer, artist_row, db_sync,
):
"""One synthetic anchor per (artist, platform). Different platforms get
different anchors even when no campaign Source exists for either.
"""
p = importer._source_for_sidecar(
artist_id=artist_row.id, platform="patreon",
artist_slug=artist_row.slug,
)
x = importer._source_for_sidecar(
artist_id=artist_row.id, platform="pixiv",
artist_slug=artist_row.slug,
)
assert p.id != x.id
assert p.platform == "patreon"
assert x.platform == "pixiv"
def test_source_for_sidecar_prefers_real_over_synthetic_when_both_exist(
importer, artist_row, db_sync,
):
"""When BOTH a synthetic anchor AND a real Source exist for the same
(artist, platform), the resolver must return the REAL one. This is the
fix for the 2026-05-31 phantom-subscription bug: alembic 0022 had
rewritten an older per-post Source row into a sidecar synthetic, and
the operator later added the real subscription. The old `ORDER BY
id ASC LIMIT 1` lookup picked the older synthetic (lower id),
silently attaching every gallery-dl download to the wrong Source.
"""
synthetic = Source(
artist_id=artist_row.id, platform="patreon",
url=f"sidecar:patreon:{artist_row.slug}", enabled=False,
)
db_sync.add(synthetic)
db_sync.flush()
real = Source(
artist_id=artist_row.id, platform="patreon",
url="https://www.patreon.com/testartist", enabled=True,
)
db_sync.add(real)
db_sync.flush()
assert synthetic.id < real.id # ordering precondition
resolved = importer._source_for_sidecar(
artist_id=artist_row.id, platform="patreon",
artist_slug=artist_row.slug,
)
assert resolved.id == real.id
assert resolved.url == "https://www.patreon.com/testartist"
).all()
assert count == []
+1 -1
View File
@@ -17,7 +17,7 @@ async def _post(db, **post_kwargs):
db.add(src)
await db.flush()
post = Post(
source_id=src.id, external_post_id="p1",
source_id=src.id, artist_id=artist.id, external_post_id="p1",
post_date=datetime(2026, 3, 1, tzinfo=UTC),
**post_kwargs,
)
+2 -2
View File
@@ -62,7 +62,7 @@ async def _run_backfill(db):
async def test_backfill_primary_post(db):
rec = await _img(db, 1)
a, s = await _artist_source(db, "Alice", "alice")
post = Post(source_id=s.id, external_post_id="1")
post = Post(source_id=s.id, artist_id=a.id, external_post_id="1")
db.add(post)
await db.flush()
rec.primary_post_id = post.id
@@ -78,7 +78,7 @@ async def test_backfill_primary_post(db):
async def test_backfill_provenance_fallback(db):
rec = await _img(db, 1)
a, s = await _artist_source(db, "Bob", "bob")
post = Post(source_id=s.id, external_post_id="2")
post = Post(source_id=s.id, artist_id=a.id, external_post_id="2")
db.add(post)
await db.flush()
db.add(ImageProvenance(image_record_id=rec.id, post_id=post.id,
+7 -1
View File
@@ -7,6 +7,7 @@ boundary, and the thumbnails/attachments composition.
from datetime import UTC, datetime, timedelta
import pytest
from sqlalchemy import select
from backend.app.models import (
Artist,
@@ -59,8 +60,13 @@ async def _seed_post(
db, source_id: int, *, external_id: str,
post_date=None, downloaded_at=None, title=None, description=None,
):
# Post.artist_id is NOT NULL since alembic 0030; look it up from
# the source so callsites don't have to thread the artist through.
artist_id = (await db.execute(
select(Source.artist_id).where(Source.id == source_id)
)).scalar_one()
p = Post(
source_id=source_id, external_post_id=external_id,
source_id=source_id, artist_id=artist_id, external_post_id=external_id,
post_title=title, post_date=post_date,
description=description,
)
+1 -1
View File
@@ -37,7 +37,7 @@ async def _seed_post(db, *, artist_name, slug, platform, ext_id,
db.add(source)
await db.flush()
post = Post(
source_id=source.id, external_post_id=ext_id,
source_id=source.id, artist_id=artist.id, external_post_id=ext_id,
post_url=f"https://{platform}.test/p/{ext_id}",
post_title=title, post_date=datetime(2023, 8, 1, tzinfo=UTC),
description=desc, attachment_count=count,
+1 -1
View File
@@ -138,7 +138,7 @@ async def test_update_changes_fields(db):
@pytest.mark.asyncio
async def test_list_hides_sidecar_synthetic_anchors(db):
"""Filesystem-import synthetic Sources (url='sidecar:<platform>:<slug>',
enabled=False — see importer._source_for_sidecar) used to leak into the
enabled=False — historical pre-alembic-0030 artifact) used to leak into the
Subscriptions UI as phantom subscriptions because list() didn't filter
them. They aren't pollable feeds; hide by default."""
artist = await _artist(db, "Alice")