feat(model): nullable Post.source_id + denormalized Post.artist_id; retire sidecar synthetics
Operator-asked 2026-06-01 after the Dymkens orphan investigation (Scribe plan #540). The pre-2030 sidecar-synthetic Source pattern (`sidecar:<platform>:<slug>` enabled=false rows) existed solely to satisfy `Post.source_id NOT NULL`, and leaked into the Subscriptions UI as phantom subscriptions. Now the data model says what's true: filesystem-imported content with no live subscription has NULL source_id, full stop. ## Schema (alembic 0030) - `post.artist_id` — NEW NOT NULL FK to artist (CASCADE). Backfilled from source.artist_id in the migration. Indexed for the artist-filter queries. - `post.source_id` — NOT NULL → nullable; FK ondelete CASCADE → SET NULL. Deleting a Source detaches its Posts instead of destroying archived content (subscription ends, archive stays). - `image_provenance.source_id` — same nullable + SET NULL. - Partial unique index `uq_post_artist_external_id_null_source` on (artist_id, external_post_id) WHERE source_id IS NULL — guards filesystem-import dedup since the existing source-bound unique ignores NULLs (Postgres treats NULL != NULL). - Sidecar synthetic Sources deleted: NULL out FKs in post, image_provenance first, then DELETE FROM source WHERE url LIKE 'sidecar:%'. The Dymkens cleanup. ## Model + service changes - `Post.source_id` → `Mapped[int | None]`; new `Post.artist_id` denormalized. - `ImageProvenance.source_id` → `Mapped[int | None]`. - Importer: `_source_for_sidecar` (synthetic-creating) → `_lookup_source_for_sidecar` (returns None when no subscription). `_find_or_create_post` takes required `artist_id`; matches on (source_id, external_post_id) for source-bound posts or (artist_id, external_post_id) for NULL-source posts. - Service queries switched off the Source detour to use Post.artist_id directly: post_feed_service.scroll/around/get_post (LEFT JOIN to Source so NULL-source posts surface); artist_service date_row/ activity/post_count; provenance_service.for_image/for_post (LEFT JOIN); gallery_service._provenance_exists_where_artist via Post.artist_id instead of ImageProvenance.source_id → Source. - `_to_dict` and provenance dict-builders emit `"source": null` for NULL-source rows. ## Frontend - `ProvenancePanel.vue` + `PostCard.vue`: render `e.source?.platform ?? 'filesystem import'` so NULL-source posts get a clear "filesystem import" affordance instead of a NaN crash. ## Tests - `test_importer_upsert_helpers`: removed the four synthetic-anchor tests; added `_find_or_create_post_idempotent_with_null_source` (dedup via the partial unique index) and `_lookup_source_for_sidecar_returns_*` (existing-subscription + none cases). The existing `_find_or_create_post_idempotent` now also passes `artist_id` and asserts it. - 8 other test files updated: every direct `Post(...)` construction gains `artist_id=<artist>.id`. The `_seed_post` helper in `test_post_feed_service` looks up artist_id from the source row so callsites stay one-arg. ## Verification on deploy After alembic 0030 runs: - `SELECT COUNT(*) FROM source WHERE url LIKE 'sidecar:%'` → 0. - `SELECT COUNT(*) FROM post WHERE source_id IS NULL` → count of filesystem-imported posts (Dymkens + any other historical). - Every `post.artist_id` non-null; consistent with source.artist_id for source-bound rows. - Subscriptions tab: no Dymkens phantom row. - Artist detail → Posts/Gallery: Dymkens's content still reachable via Post.artist_id. - Provenance panel renders "filesystem import" chip for NULL-source posts; PostCard same. ## Out of scope - UI to manage/delete orphan NULL-source Posts. Data model is right; UI follows if operator wants it.
This commit is contained in:
@@ -0,0 +1,141 @@
|
|||||||
|
"""nullable post.source_id + denormalized post.artist_id; retire sidecar synthetics
|
||||||
|
|
||||||
|
Revision ID: 0030
|
||||||
|
Revises: 0029
|
||||||
|
Create Date: 2026-06-01
|
||||||
|
|
||||||
|
Operator-asked 2026-06-01 after the Dymkens orphan investigation: the
|
||||||
|
sidecar synthetic Source pattern (`sidecar:<platform>:<slug>` rows
|
||||||
|
with enabled=false) was technically correct but misled the operator
|
||||||
|
into thinking they had phantom subscriptions. The synthetics existed
|
||||||
|
solely to satisfy `Post.source_id NOT NULL` for filesystem-imported
|
||||||
|
content with no real subscription.
|
||||||
|
|
||||||
|
This migration makes the data model honest:
|
||||||
|
|
||||||
|
1. **Post gets a denormalized `artist_id` column** so artist filters
|
||||||
|
work without traversing `Post → Source.artist_id`. Backfilled from
|
||||||
|
the existing Source linkage, then NOT NULL'd.
|
||||||
|
2. **`Post.source_id` becomes nullable**, FK ondelete `CASCADE` → `SET
|
||||||
|
NULL`. Deleting a Source detaches its Posts instead of destroying
|
||||||
|
imported content (semantically: subscription ends, archive stays).
|
||||||
|
3. **`ImageProvenance.source_id` becomes nullable** with the same FK
|
||||||
|
semantic change.
|
||||||
|
4. **Sidecar synthetic Sources are deleted** — first NULL out the
|
||||||
|
FKs from Post + ImageProvenance pointing at them (so the implicit
|
||||||
|
CASCADE doesn't fire), then delete. DownloadEvent FK is unchanged
|
||||||
|
(still CASCADE'd, NOT NULL'd) — synthetics have `enabled=false`
|
||||||
|
so no events exist for them.
|
||||||
|
|
||||||
|
Uniqueness handling: the existing `uq_post_source_external_id`
|
||||||
|
(source_id, external_post_id) keeps working for source-bound Posts
|
||||||
|
(Postgres treats NULL != NULL so NULL-source rows aren't deduped by
|
||||||
|
it). A second partial unique index covers the NULL-source case on
|
||||||
|
(artist_id, external_post_id) so filesystem-imported posts still
|
||||||
|
dedupe within an artist.
|
||||||
|
"""
|
||||||
|
from typing import Sequence, Union
|
||||||
|
|
||||||
|
import sqlalchemy as sa
|
||||||
|
from alembic import op
|
||||||
|
from sqlalchemy import text
|
||||||
|
|
||||||
|
revision: str = "0030"
|
||||||
|
down_revision: Union[str, None] = "0029"
|
||||||
|
branch_labels: Union[str, Sequence[str], None] = None
|
||||||
|
depends_on: Union[str, Sequence[str], None] = None
|
||||||
|
|
||||||
|
|
||||||
|
def upgrade() -> None:
|
||||||
|
conn = op.get_bind()
|
||||||
|
|
||||||
|
# Step 1: add Post.artist_id, initially nullable for backfill.
|
||||||
|
op.add_column(
|
||||||
|
"post",
|
||||||
|
sa.Column("artist_id", sa.Integer, nullable=True),
|
||||||
|
)
|
||||||
|
op.create_foreign_key(
|
||||||
|
"post_artist_id_fkey", "post", "artist",
|
||||||
|
["artist_id"], ["id"], ondelete="CASCADE",
|
||||||
|
)
|
||||||
|
|
||||||
|
# Step 2: backfill from Source.artist_id (every existing Post has a
|
||||||
|
# Source today, so every row gets populated).
|
||||||
|
conn.execute(text("""
|
||||||
|
UPDATE post p
|
||||||
|
SET artist_id = s.artist_id
|
||||||
|
FROM source s
|
||||||
|
WHERE p.source_id = s.id AND p.artist_id IS NULL
|
||||||
|
"""))
|
||||||
|
|
||||||
|
# Sanity: count any remaining NULLs. Should be zero pre-this-migration.
|
||||||
|
remaining = conn.execute(text(
|
||||||
|
"SELECT COUNT(*) FROM post WHERE artist_id IS NULL"
|
||||||
|
)).scalar_one()
|
||||||
|
if remaining:
|
||||||
|
raise RuntimeError(
|
||||||
|
f"alembic 0030: {remaining} post rows have no resolvable "
|
||||||
|
f"artist_id after backfill. Investigate before continuing."
|
||||||
|
)
|
||||||
|
|
||||||
|
# Step 3: enforce NOT NULL + add index for artist-filter queries.
|
||||||
|
op.alter_column("post", "artist_id", nullable=False)
|
||||||
|
op.create_index("ix_post_artist_id", "post", ["artist_id"])
|
||||||
|
|
||||||
|
# Step 4: relax post.source_id + flip FK to SET NULL.
|
||||||
|
op.alter_column("post", "source_id", nullable=True)
|
||||||
|
op.drop_constraint("post_source_id_fkey", "post", type_="foreignkey")
|
||||||
|
op.create_foreign_key(
|
||||||
|
"post_source_id_fkey", "post", "source",
|
||||||
|
["source_id"], ["id"], ondelete="SET NULL",
|
||||||
|
)
|
||||||
|
|
||||||
|
# Step 5: relax image_provenance.source_id + flip FK to SET NULL.
|
||||||
|
op.alter_column("image_provenance", "source_id", nullable=True)
|
||||||
|
op.drop_constraint(
|
||||||
|
"image_provenance_source_id_fkey", "image_provenance",
|
||||||
|
type_="foreignkey",
|
||||||
|
)
|
||||||
|
op.create_foreign_key(
|
||||||
|
"image_provenance_source_id_fkey", "image_provenance", "source",
|
||||||
|
["source_id"], ["id"], ondelete="SET NULL",
|
||||||
|
)
|
||||||
|
|
||||||
|
# Step 6: partial unique index on (artist_id, external_post_id) for
|
||||||
|
# NULL-source Posts. The existing uq_post_source_external_id keeps
|
||||||
|
# guarding source-bound rows; NULL-source rows now dedupe within
|
||||||
|
# an artist.
|
||||||
|
op.execute(
|
||||||
|
"CREATE UNIQUE INDEX uq_post_artist_external_id_null_source "
|
||||||
|
"ON post (artist_id, external_post_id) "
|
||||||
|
"WHERE source_id IS NULL"
|
||||||
|
)
|
||||||
|
|
||||||
|
# Step 7: retire sidecar synthetic Sources. NULL out the references
|
||||||
|
# FIRST (the new FK is SET NULL so CASCADE wouldn't fire anyway, but
|
||||||
|
# being explicit makes the intent clear). Then delete the synthetic
|
||||||
|
# source rows. Any DownloadEvent rows under synthetics CASCADE-die
|
||||||
|
# with the source — synthetics have enabled=false so there shouldn't
|
||||||
|
# be any in practice.
|
||||||
|
conn.execute(text("""
|
||||||
|
UPDATE post
|
||||||
|
SET source_id = NULL
|
||||||
|
WHERE source_id IN (SELECT id FROM source WHERE url LIKE 'sidecar:%')
|
||||||
|
"""))
|
||||||
|
conn.execute(text("""
|
||||||
|
UPDATE image_provenance
|
||||||
|
SET source_id = NULL
|
||||||
|
WHERE source_id IN (SELECT id FROM source WHERE url LIKE 'sidecar:%')
|
||||||
|
"""))
|
||||||
|
deleted = conn.execute(text(
|
||||||
|
"DELETE FROM source WHERE url LIKE 'sidecar:%' RETURNING id"
|
||||||
|
)).rowcount
|
||||||
|
print(f"alembic 0030: deleted {deleted} sidecar synthetic source rows")
|
||||||
|
|
||||||
|
|
||||||
|
def downgrade() -> None:
|
||||||
|
# Lossy migration — the deleted sidecar synthetics can't be
|
||||||
|
# restored from the orphan post.source_id / image_provenance.source_id
|
||||||
|
# values, and the partial unique index encodes a constraint that
|
||||||
|
# NULL-source Posts may now exist. No safe downgrade.
|
||||||
|
pass
|
||||||
@@ -34,8 +34,12 @@ class ImageProvenance(Base):
|
|||||||
post_id: Mapped[int] = mapped_column(
|
post_id: Mapped[int] = mapped_column(
|
||||||
ForeignKey("post.id", ondelete="CASCADE"), nullable=False, index=True
|
ForeignKey("post.id", ondelete="CASCADE"), nullable=False, index=True
|
||||||
)
|
)
|
||||||
source_id: Mapped[int] = mapped_column(
|
# Nullable since alembic 0030 — provenance rows for filesystem-imported
|
||||||
ForeignKey("source.id", ondelete="CASCADE"), nullable=False, index=True
|
# content with no subscription have NULL source_id. FK ondelete SET
|
||||||
|
# NULL so deleting a Source detaches its provenance rows instead of
|
||||||
|
# destroying the linkage between image and post.
|
||||||
|
source_id: Mapped[int | None] = mapped_column(
|
||||||
|
ForeignKey("source.id", ondelete="SET NULL"), nullable=True, index=True
|
||||||
)
|
)
|
||||||
captured_metadata: Mapped[dict | None] = mapped_column(JSON, nullable=True)
|
captured_metadata: Mapped[dict | None] = mapped_column(JSON, nullable=True)
|
||||||
captured_at: Mapped[datetime] = mapped_column(
|
captured_at: Mapped[datetime] = mapped_column(
|
||||||
|
|||||||
@@ -1,6 +1,9 @@
|
|||||||
"""Post — provenance anchor for content downloaded from a Source.
|
"""Post — provenance anchor for one creator post (may contain many images).
|
||||||
|
|
||||||
A Post is one creator post; it may contain many images/videos.
|
`source_id` is nullable since alembic 0030 — filesystem-imported posts
|
||||||
|
with no live subscription have NULL source_id. `artist_id` is the
|
||||||
|
denormalized always-present link to the creator (added in 0030 so
|
||||||
|
artist-filter queries don't depend on the Source detour).
|
||||||
"""
|
"""
|
||||||
|
|
||||||
from datetime import datetime
|
from datetime import datetime
|
||||||
@@ -14,12 +17,25 @@ from .base import Base
|
|||||||
class Post(Base):
|
class Post(Base):
|
||||||
__tablename__ = "post"
|
__tablename__ = "post"
|
||||||
__table_args__ = (
|
__table_args__ = (
|
||||||
|
# Source-bound dedup. Postgres treats NULL != NULL so rows
|
||||||
|
# with source_id IS NULL aren't deduped by this constraint;
|
||||||
|
# the partial unique index `uq_post_artist_external_id_null_source`
|
||||||
|
# (created in alembic 0030) covers that case via
|
||||||
|
# (artist_id, external_post_id).
|
||||||
UniqueConstraint("source_id", "external_post_id", name="uq_post_source_external_id"),
|
UniqueConstraint("source_id", "external_post_id", name="uq_post_source_external_id"),
|
||||||
)
|
)
|
||||||
|
|
||||||
id: Mapped[int] = mapped_column(Integer, primary_key=True)
|
id: Mapped[int] = mapped_column(Integer, primary_key=True)
|
||||||
source_id: Mapped[int] = mapped_column(
|
source_id: Mapped[int | None] = mapped_column(
|
||||||
ForeignKey("source.id", ondelete="CASCADE"), nullable=False, index=True
|
ForeignKey("source.id", ondelete="SET NULL"), nullable=True, index=True
|
||||||
|
)
|
||||||
|
# Denormalized; always equals source.artist_id when source_id is set
|
||||||
|
# (the importer is responsible for keeping them consistent on insert).
|
||||||
|
# Filter queries (artist detail, artist-scoped posts feed) use this
|
||||||
|
# directly instead of joining through Source.
|
||||||
|
artist_id: Mapped[int] = mapped_column(
|
||||||
|
ForeignKey("artist.id", ondelete="CASCADE"),
|
||||||
|
nullable=False, index=True,
|
||||||
)
|
)
|
||||||
external_post_id: Mapped[str] = mapped_column(String(128), nullable=False)
|
external_post_id: Mapped[str] = mapped_column(String(128), nullable=False)
|
||||||
post_url: Mapped[str | None] = mapped_column(Text, nullable=True)
|
post_url: Mapped[str | None] = mapped_column(Text, nullable=True)
|
||||||
|
|||||||
@@ -58,13 +58,17 @@ class ArtistService:
|
|||||||
)
|
)
|
||||||
).scalar_one()
|
).scalar_one()
|
||||||
|
|
||||||
|
# Posts under this artist that have at least one image attached.
|
||||||
|
# Use Post.artist_id (alembic 0030) for the artist filter; keep
|
||||||
|
# the ImageProvenance JOIN so date bounds reflect only image-
|
||||||
|
# bearing posts (matches the original semantic). NULL-source
|
||||||
|
# posts now surface too.
|
||||||
date_row = (
|
date_row = (
|
||||||
await self.session.execute(
|
await self.session.execute(
|
||||||
select(func.min(Post.post_date), func.max(Post.post_date))
|
select(func.min(Post.post_date), func.max(Post.post_date))
|
||||||
.select_from(Post)
|
.select_from(Post)
|
||||||
.join(ImageProvenance, ImageProvenance.post_id == Post.id)
|
.join(ImageProvenance, ImageProvenance.post_id == Post.id)
|
||||||
.join(Source, Source.id == ImageProvenance.source_id)
|
.where(Post.artist_id == aid)
|
||||||
.where(Source.artist_id == aid)
|
|
||||||
)
|
)
|
||||||
).first()
|
).first()
|
||||||
dmin, dmax = date_row if date_row else (None, None)
|
dmin, dmax = date_row if date_row else (None, None)
|
||||||
@@ -98,14 +102,14 @@ class ArtistService:
|
|||||||
)
|
)
|
||||||
).all()
|
).all()
|
||||||
|
|
||||||
|
# Same Post.artist_id direct filter — counts NULL-source posts too.
|
||||||
month = func.date_trunc("month", Post.post_date).label("m")
|
month = func.date_trunc("month", Post.post_date).label("m")
|
||||||
activity = (
|
activity = (
|
||||||
await self.session.execute(
|
await self.session.execute(
|
||||||
select(month, func.count(func.distinct(ImageProvenance.image_record_id)))
|
select(month, func.count(func.distinct(ImageProvenance.image_record_id)))
|
||||||
.select_from(Post)
|
.select_from(Post)
|
||||||
.join(ImageProvenance, ImageProvenance.post_id == Post.id)
|
.join(ImageProvenance, ImageProvenance.post_id == Post.id)
|
||||||
.join(Source, Source.id == ImageProvenance.source_id)
|
.where(and_(Post.artist_id == aid, Post.post_date.isnot(None)))
|
||||||
.where(and_(Source.artist_id == aid, Post.post_date.isnot(None)))
|
|
||||||
.group_by(month)
|
.group_by(month)
|
||||||
.order_by(month)
|
.order_by(month)
|
||||||
)
|
)
|
||||||
@@ -114,9 +118,7 @@ class ArtistService:
|
|||||||
post_count = (
|
post_count = (
|
||||||
await self.session.execute(
|
await self.session.execute(
|
||||||
select(func.count(func.distinct(Post.id)))
|
select(func.count(func.distinct(Post.id)))
|
||||||
.select_from(Post)
|
.where(Post.artist_id == aid)
|
||||||
.join(Source, Source.id == Post.source_id)
|
|
||||||
.where(Source.artist_id == aid)
|
|
||||||
)
|
)
|
||||||
).scalar_one()
|
).scalar_one()
|
||||||
|
|
||||||
|
|||||||
@@ -133,10 +133,16 @@ def _provenance_clause(post_id, artist_id):
|
|||||||
ImageProvenance.post_id == post_id,
|
ImageProvenance.post_id == post_id,
|
||||||
)
|
)
|
||||||
if artist_id is not None:
|
if artist_id is not None:
|
||||||
|
# Use Post.artist_id (alembic 0030 denormalized column) instead
|
||||||
|
# of joining through ImageProvenance.source_id → Source.artist_id.
|
||||||
|
# The denormalization is the always-present linkage; the source
|
||||||
|
# path now drops NULL-source provenance rows (filesystem-imported
|
||||||
|
# content) which would otherwise vanish from artist-filtered
|
||||||
|
# gallery views.
|
||||||
return exists().where(
|
return exists().where(
|
||||||
ImageProvenance.image_record_id == ImageRecord.id,
|
ImageProvenance.image_record_id == ImageRecord.id,
|
||||||
ImageProvenance.source_id == Source.id,
|
ImageProvenance.post_id == Post.id,
|
||||||
Source.artist_id == artist_id,
|
Post.artist_id == artist_id,
|
||||||
)
|
)
|
||||||
return None
|
return None
|
||||||
|
|
||||||
|
|||||||
@@ -212,10 +212,10 @@ class Importer:
|
|||||||
which would lose the surrounding scan's progress — and re-run `stmt`
|
which would lose the surrounding scan's progress — and re-run `stmt`
|
||||||
(scalar_one) to return the row the other worker created.
|
(scalar_one) to return the row the other worker created.
|
||||||
|
|
||||||
Centralizes the pattern shared by _find_or_create_source,
|
Centralizes the pattern shared by _find_or_create_source and
|
||||||
_source_for_sidecar, and _find_or_create_post. The plain
|
_find_or_create_post. The plain SELECT-then-INSERT version lost
|
||||||
SELECT-then-INSERT version lost races under the 5-min recovery sweep
|
races under the 5-min recovery sweep (operator-flagged
|
||||||
(operator-flagged 2026-05-26)."""
|
2026-05-26)."""
|
||||||
existing = self.session.execute(stmt).scalar_one_or_none()
|
existing = self.session.execute(stmt).scalar_one_or_none()
|
||||||
if existing is not None:
|
if existing is not None:
|
||||||
return existing
|
return existing
|
||||||
@@ -258,47 +258,25 @@ class Importer:
|
|||||||
lambda: Source(artist_id=artist_id, platform=platform, url=url),
|
lambda: Source(artist_id=artist_id, platform=platform, url=url),
|
||||||
)
|
)
|
||||||
|
|
||||||
def _source_for_sidecar(
|
def _lookup_source_for_sidecar(
|
||||||
self, *, artist_id: int, platform: str, artist_slug: str,
|
self, *, artist_id: int, platform: str,
|
||||||
) -> Source:
|
) -> Source | None:
|
||||||
"""Sidecar-import Source resolver. Used by both filesystem imports
|
"""Find the real subscription Source for (artist, platform), or
|
||||||
and gallery-dl downloads (both write sidecar JSON, both flow through
|
None if no subscription exists.
|
||||||
_apply_sidecar / _capture_attachment).
|
|
||||||
|
|
||||||
Source represents a subscription feed (one per artist+platform — the
|
Pre-alembic-0030 this method would CREATE a synthetic
|
||||||
URL polled by the FC-3 downloader). The filesystem importer used to
|
`sidecar:<platform>:<slug>` Source when no real one existed —
|
||||||
call _find_or_create_source(url=sd.post_url), creating one Source
|
because `Post.source_id` was NOT NULL and the importer needed
|
||||||
row per post URL — 100s of junk Sources per artist, all with
|
something to attach Posts to. Alembic 0030 relaxed both
|
||||||
enabled=True, polluting the artist detail page and tricking the
|
`Post.source_id` and `ImageProvenance.source_id` to nullable, so
|
||||||
subscription checker into trying to poll patreon post URLs as feeds.
|
synthetic anchors are obsolete; the importer now leaves
|
||||||
Operator-flagged 2026-05-26; consolidated via alembic 0022.
|
source_id as None when no subscription exists for the (artist,
|
||||||
|
platform). Operator-asked 2026-06-01: synthetic Sources had
|
||||||
Resolution order: prefer a real (non-sidecar) Source over a
|
leaked into the Subscriptions UI as phantom subscriptions and
|
||||||
synthetic anchor. When alembic 0022 ran, it may have rewritten
|
the operator wanted the data model to truthfully say "this
|
||||||
per-post Sources into `sidecar:<platform>:<slug>` synthetic
|
content has no live subscription."
|
||||||
anchors. If the operator later added the real subscription, both
|
|
||||||
rows now coexist. A naive `ORDER BY id ASC LIMIT 1` lookup would
|
|
||||||
pick the older synthetic and silently attach every gallery-dl
|
|
||||||
download to the wrong Source — operator-flagged 2026-05-31 after
|
|
||||||
the Subscriptions UI surfaced the phantom anchors. Pick the real
|
|
||||||
one when one exists; fall back to the synthetic; only create a
|
|
||||||
new synthetic when nothing exists for (artist, platform).
|
|
||||||
"""
|
"""
|
||||||
real_stmt = (
|
stmt = (
|
||||||
select(Source)
|
|
||||||
.where(
|
|
||||||
Source.artist_id == artist_id,
|
|
||||||
Source.platform == platform,
|
|
||||||
~Source.url.like("sidecar:%"),
|
|
||||||
)
|
|
||||||
.order_by(Source.id.asc())
|
|
||||||
.limit(1)
|
|
||||||
)
|
|
||||||
real = self.session.execute(real_stmt).scalar_one_or_none()
|
|
||||||
if real is not None:
|
|
||||||
return real
|
|
||||||
|
|
||||||
any_stmt = (
|
|
||||||
select(Source)
|
select(Source)
|
||||||
.where(
|
.where(
|
||||||
Source.artist_id == artist_id,
|
Source.artist_id == artist_id,
|
||||||
@@ -307,29 +285,37 @@ class Importer:
|
|||||||
.order_by(Source.id.asc())
|
.order_by(Source.id.asc())
|
||||||
.limit(1)
|
.limit(1)
|
||||||
)
|
)
|
||||||
return self._get_or_create(
|
return self.session.execute(stmt).scalar_one_or_none()
|
||||||
any_stmt,
|
|
||||||
lambda: Source(
|
|
||||||
artist_id=artist_id,
|
|
||||||
platform=platform,
|
|
||||||
url=f"sidecar:{platform}:{artist_slug}",
|
|
||||||
enabled=False,
|
|
||||||
),
|
|
||||||
)
|
|
||||||
|
|
||||||
def _find_or_create_post(
|
def _find_or_create_post(
|
||||||
self, *, source_id: int, external_post_id: str,
|
self, *, source_id: int | None, external_post_id: str,
|
||||||
|
artist_id: int,
|
||||||
) -> Post:
|
) -> Post:
|
||||||
"""Race-safe find-or-create on `post` keyed by
|
"""Race-safe find-or-create on `post`. Keyed by
|
||||||
(source_id, external_post_id). Mirrors `_find_or_create_source`
|
(source_id, external_post_id) when source_id is set — the
|
||||||
— same savepoint + IntegrityError-recovery pattern."""
|
`uq_post_source_external_id` constraint guards. For NULL-source
|
||||||
stmt = select(Post).where(
|
posts the existence check matches on (artist_id, external_post_id),
|
||||||
Post.source_id == source_id,
|
which the partial unique index `uq_post_artist_external_id_null_source`
|
||||||
Post.external_post_id == external_post_id,
|
(alembic 0030) guards. Same savepoint + IntegrityError-recovery
|
||||||
)
|
pattern as the rest of the helpers."""
|
||||||
|
if source_id is not None:
|
||||||
|
stmt = select(Post).where(
|
||||||
|
Post.source_id == source_id,
|
||||||
|
Post.external_post_id == external_post_id,
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
stmt = select(Post).where(
|
||||||
|
Post.source_id.is_(None),
|
||||||
|
Post.artist_id == artist_id,
|
||||||
|
Post.external_post_id == external_post_id,
|
||||||
|
)
|
||||||
return self._get_or_create(
|
return self._get_or_create(
|
||||||
stmt,
|
stmt,
|
||||||
lambda: Post(source_id=source_id, external_post_id=external_post_id),
|
lambda: Post(
|
||||||
|
source_id=source_id,
|
||||||
|
artist_id=artist_id,
|
||||||
|
external_post_id=external_post_id,
|
||||||
|
),
|
||||||
)
|
)
|
||||||
|
|
||||||
def import_one(self, source: Path) -> ImportResult:
|
def import_one(self, source: Path) -> ImportResult:
|
||||||
@@ -370,12 +356,14 @@ class Importer:
|
|||||||
return None
|
return None
|
||||||
sd = parse_sidecar(data)
|
sd = parse_sidecar(data)
|
||||||
platform = sd.platform or "unknown"
|
platform = sd.platform or "unknown"
|
||||||
src = self._source_for_sidecar(
|
src = self._lookup_source_for_sidecar(
|
||||||
artist_id=artist.id, platform=platform, artist_slug=artist.slug,
|
artist_id=artist.id, platform=platform,
|
||||||
)
|
)
|
||||||
epid = sd.external_post_id or sc.stem
|
epid = sd.external_post_id or sc.stem
|
||||||
return self._find_or_create_post(
|
return self._find_or_create_post(
|
||||||
source_id=src.id, external_post_id=epid,
|
source_id=src.id if src else None,
|
||||||
|
external_post_id=epid,
|
||||||
|
artist_id=artist.id,
|
||||||
)
|
)
|
||||||
|
|
||||||
def _capture_attachment(
|
def _capture_attachment(
|
||||||
@@ -858,14 +846,15 @@ class Importer:
|
|||||||
src = explicit_source
|
src = explicit_source
|
||||||
else:
|
else:
|
||||||
platform = sd.platform or "unknown"
|
platform = sd.platform or "unknown"
|
||||||
src = self._source_for_sidecar(
|
src = self._lookup_source_for_sidecar(
|
||||||
artist_id=artist.id, platform=platform,
|
artist_id=artist.id, platform=platform,
|
||||||
artist_slug=artist.slug,
|
|
||||||
)
|
)
|
||||||
|
|
||||||
epid = sd.external_post_id or sc.stem
|
epid = sd.external_post_id or sc.stem
|
||||||
post = self._find_or_create_post(
|
post = self._find_or_create_post(
|
||||||
source_id=src.id, external_post_id=epid,
|
source_id=src.id if src else None,
|
||||||
|
external_post_id=epid,
|
||||||
|
artist_id=artist.id,
|
||||||
)
|
)
|
||||||
if sd.post_url is not None:
|
if sd.post_url is not None:
|
||||||
post.post_url = sd.post_url
|
post.post_url = sd.post_url
|
||||||
@@ -901,7 +890,7 @@ class Importer:
|
|||||||
ImageProvenance(
|
ImageProvenance(
|
||||||
image_record_id=record.id,
|
image_record_id=record.id,
|
||||||
post_id=post.id,
|
post_id=post.id,
|
||||||
source_id=src.id,
|
source_id=src.id if src else None,
|
||||||
captured_metadata=sd.raw,
|
captured_metadata=sd.raw,
|
||||||
)
|
)
|
||||||
)
|
)
|
||||||
|
|||||||
@@ -69,13 +69,19 @@ class PostFeedService:
|
|||||||
raise ValueError("direction must be 'older' or 'newer'")
|
raise ValueError("direction must be 'older' or 'newer'")
|
||||||
|
|
||||||
sort_key = _sort_key()
|
sort_key = _sort_key()
|
||||||
|
# Artist via the denormalized Post.artist_id (alembic 0030);
|
||||||
|
# Source via LEFT JOIN since post.source_id can now be NULL for
|
||||||
|
# filesystem-imported posts with no live subscription. A
|
||||||
|
# platform= filter implicitly excludes NULL-source posts (they
|
||||||
|
# have no platform); an artist_id= filter still surfaces them
|
||||||
|
# because Post.artist_id is always set.
|
||||||
stmt = (
|
stmt = (
|
||||||
select(Post, Artist, Source)
|
select(Post, Artist, Source)
|
||||||
.join(Source, Post.source_id == Source.id)
|
.join(Artist, Post.artist_id == Artist.id)
|
||||||
.join(Artist, Source.artist_id == Artist.id)
|
.outerjoin(Source, Post.source_id == Source.id)
|
||||||
)
|
)
|
||||||
if artist_id is not None:
|
if artist_id is not None:
|
||||||
stmt = stmt.where(Source.artist_id == artist_id)
|
stmt = stmt.where(Post.artist_id == artist_id)
|
||||||
if platform is not None:
|
if platform is not None:
|
||||||
stmt = stmt.where(Source.platform == platform)
|
stmt = stmt.where(Source.platform == platform)
|
||||||
if cursor:
|
if cursor:
|
||||||
@@ -135,8 +141,8 @@ class PostFeedService:
|
|||||||
cursor for each end. Returns None if the post doesn't exist."""
|
cursor for each end. Returns None if the post doesn't exist."""
|
||||||
anchor = (await self.session.execute(
|
anchor = (await self.session.execute(
|
||||||
select(Post, Artist, Source)
|
select(Post, Artist, Source)
|
||||||
.join(Source, Post.source_id == Source.id)
|
.join(Artist, Post.artist_id == Artist.id)
|
||||||
.join(Artist, Source.artist_id == Artist.id)
|
.outerjoin(Source, Post.source_id == Source.id)
|
||||||
.where(Post.id == post_id)
|
.where(Post.id == post_id)
|
||||||
)).one_or_none()
|
)).one_or_none()
|
||||||
if anchor is None:
|
if anchor is None:
|
||||||
@@ -168,8 +174,8 @@ class PostFeedService:
|
|||||||
async def get_post(self, post_id: int) -> dict | None:
|
async def get_post(self, post_id: int) -> dict | None:
|
||||||
row = (await self.session.execute(
|
row = (await self.session.execute(
|
||||||
select(Post, Artist, Source)
|
select(Post, Artist, Source)
|
||||||
.join(Source, Post.source_id == Source.id)
|
.join(Artist, Post.artist_id == Artist.id)
|
||||||
.join(Artist, Source.artist_id == Artist.id)
|
.outerjoin(Source, Post.source_id == Source.id)
|
||||||
.where(Post.id == post_id)
|
.where(Post.id == post_id)
|
||||||
)).one_or_none()
|
)).one_or_none()
|
||||||
if row is None:
|
if row is None:
|
||||||
@@ -260,7 +266,7 @@ class PostFeedService:
|
|||||||
return out
|
return out
|
||||||
|
|
||||||
def _to_dict(
|
def _to_dict(
|
||||||
self, post: Post, artist: Artist, source: Source,
|
self, post: Post, artist: Artist, source: Source | None,
|
||||||
thumbs_map: dict, atts_map: dict,
|
thumbs_map: dict, atts_map: dict,
|
||||||
) -> dict:
|
) -> dict:
|
||||||
plain_full = html_to_plain(post.description) if post.description else None
|
plain_full = html_to_plain(post.description) if post.description else None
|
||||||
@@ -269,6 +275,9 @@ class PostFeedService:
|
|||||||
else:
|
else:
|
||||||
description_plain, truncated = truncate_at_word(plain_full, DESCRIPTION_LIMIT)
|
description_plain, truncated = truncate_at_word(plain_full, DESCRIPTION_LIMIT)
|
||||||
thumbs_entry = thumbs_map.get(post.id, {"thumbs": [], "more": 0})
|
thumbs_entry = thumbs_map.get(post.id, {"thumbs": [], "more": 0})
|
||||||
|
# `source` is null for filesystem-imported posts with no live
|
||||||
|
# subscription (alembic 0030). Frontend renders that as a
|
||||||
|
# "filesystem import" affordance instead of a platform chip.
|
||||||
return {
|
return {
|
||||||
"id": post.id,
|
"id": post.id,
|
||||||
"external_post_id": post.external_post_id,
|
"external_post_id": post.external_post_id,
|
||||||
@@ -279,7 +288,10 @@ class PostFeedService:
|
|||||||
"description_plain": description_plain,
|
"description_plain": description_plain,
|
||||||
"description_truncated": truncated,
|
"description_truncated": truncated,
|
||||||
"artist": {"id": artist.id, "name": artist.name, "slug": artist.slug},
|
"artist": {"id": artist.id, "name": artist.name, "slug": artist.slug},
|
||||||
"source": {"id": source.id, "platform": source.platform},
|
"source": (
|
||||||
|
{"id": source.id, "platform": source.platform}
|
||||||
|
if source is not None else None
|
||||||
|
),
|
||||||
"thumbnails": thumbs_entry["thumbs"],
|
"thumbnails": thumbs_entry["thumbs"],
|
||||||
"thumbnails_more": thumbs_entry["more"],
|
"thumbnails_more": thumbs_entry["more"],
|
||||||
"attachments": atts_map.get(post.id, []),
|
"attachments": atts_map.get(post.id, []),
|
||||||
|
|||||||
@@ -70,11 +70,15 @@ class ProvenanceService:
|
|||||||
rec = await self.session.get(ImageRecord, image_id)
|
rec = await self.session.get(ImageRecord, image_id)
|
||||||
if rec is None:
|
if rec is None:
|
||||||
return None
|
return None
|
||||||
|
# Artist via Post.artist_id (alembic 0030); Source via LEFT JOIN
|
||||||
|
# since both Post.source_id and ImageProvenance.source_id can be
|
||||||
|
# NULL for filesystem-imported content. Frontend renders source=
|
||||||
|
# null as "filesystem import."
|
||||||
stmt = (
|
stmt = (
|
||||||
select(ImageProvenance, Post, Source, Artist)
|
select(ImageProvenance, Post, Source, Artist)
|
||||||
.join(Post, Post.id == ImageProvenance.post_id)
|
.join(Post, Post.id == ImageProvenance.post_id)
|
||||||
.join(Source, Source.id == ImageProvenance.source_id)
|
.join(Artist, Artist.id == Post.artist_id)
|
||||||
.join(Artist, Artist.id == Source.artist_id)
|
.outerjoin(Source, Source.id == ImageProvenance.source_id)
|
||||||
.where(ImageProvenance.image_record_id == image_id)
|
.where(ImageProvenance.image_record_id == image_id)
|
||||||
.order_by(ImageProvenance.captured_at.asc(),
|
.order_by(ImageProvenance.captured_at.asc(),
|
||||||
ImageProvenance.id.asc())
|
ImageProvenance.id.asc())
|
||||||
@@ -90,7 +94,7 @@ class ProvenanceService:
|
|||||||
"captured_at": ip.captured_at.isoformat()
|
"captured_at": ip.captured_at.isoformat()
|
||||||
if ip.captured_at else None,
|
if ip.captured_at else None,
|
||||||
"post": _post_dict(post),
|
"post": _post_dict(post),
|
||||||
"source": _source_dict(src),
|
"source": _source_dict(src) if src is not None else None,
|
||||||
"artist": _artist_dict(art),
|
"artist": _artist_dict(art),
|
||||||
}
|
}
|
||||||
for ip, post, src, art in rows
|
for ip, post, src, art in rows
|
||||||
@@ -99,10 +103,12 @@ class ProvenanceService:
|
|||||||
}
|
}
|
||||||
|
|
||||||
async def for_post(self, post_id: int) -> dict | None:
|
async def for_post(self, post_id: int) -> dict | None:
|
||||||
|
# Same LEFT JOIN to Source — get_post must succeed for a
|
||||||
|
# NULL-source post.
|
||||||
stmt = (
|
stmt = (
|
||||||
select(Post, Source, Artist)
|
select(Post, Source, Artist)
|
||||||
.join(Source, Source.id == Post.source_id)
|
.join(Artist, Artist.id == Post.artist_id)
|
||||||
.join(Artist, Artist.id == Source.artist_id)
|
.outerjoin(Source, Source.id == Post.source_id)
|
||||||
.where(Post.id == post_id)
|
.where(Post.id == post_id)
|
||||||
)
|
)
|
||||||
row = (await self.session.execute(stmt)).first()
|
row = (await self.session.execute(stmt)).first()
|
||||||
@@ -111,7 +117,7 @@ class ProvenanceService:
|
|||||||
post, src, art = row
|
post, src, art = row
|
||||||
return {
|
return {
|
||||||
"post": _post_dict(post),
|
"post": _post_dict(post),
|
||||||
"source": _source_dict(src),
|
"source": _source_dict(src) if src is not None else None,
|
||||||
"artist": _artist_dict(art),
|
"artist": _artist_dict(art),
|
||||||
"attachments": await self._attachments_for_posts([post.id]),
|
"attachments": await self._attachments_for_posts([post.id]),
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -157,7 +157,7 @@ class SourceService:
|
|||||||
if artist_id is not None:
|
if artist_id is not None:
|
||||||
stmt = stmt.where(Source.artist_id == artist_id)
|
stmt = stmt.where(Source.artist_id == artist_id)
|
||||||
if not include_synthetic:
|
if not include_synthetic:
|
||||||
# Filesystem-import sidecar anchors (importer._source_for_sidecar)
|
# Pre-alembic-0030 sidecar synthetic anchors
|
||||||
# have url='sidecar:<platform>:<slug>' and exist only to give
|
# have url='sidecar:<platform>:<slug>' and exist only to give
|
||||||
# imported Posts a NOT-NULL Source FK. They aren't pollable
|
# imported Posts a NOT-NULL Source FK. They aren't pollable
|
||||||
# feeds; the Subscriptions UI used to render them as phantom
|
# feeds; the Subscriptions UI used to render them as phantom
|
||||||
|
|||||||
@@ -19,7 +19,12 @@
|
|||||||
v-for="e in state.entries" :key="e.provenance_id" class="fc-prov__card"
|
v-for="e in state.entries" :key="e.provenance_id" class="fc-prov__card"
|
||||||
>
|
>
|
||||||
<div class="fc-prov__head">
|
<div class="fc-prov__head">
|
||||||
<span class="fc-prov__platform">{{ e.source.platform }}</span>
|
<!-- Posts with no live subscription have source=null (alembic
|
||||||
|
0030); render an explicit "filesystem import" affordance
|
||||||
|
instead of a platform chip. -->
|
||||||
|
<span class="fc-prov__platform">
|
||||||
|
{{ e.source?.platform ?? 'filesystem import' }}
|
||||||
|
</span>
|
||||||
<span v-if="postDate(e)" class="fc-prov__date">{{ postDate(e) }}</span>
|
<span v-if="postDate(e)" class="fc-prov__date">{{ postDate(e) }}</span>
|
||||||
</div>
|
</div>
|
||||||
<button
|
<button
|
||||||
|
|||||||
@@ -7,7 +7,12 @@
|
|||||||
@keydown.enter="onCardClick"
|
@keydown.enter="onCardClick"
|
||||||
>
|
>
|
||||||
<div class="fc-post-card__head">
|
<div class="fc-post-card__head">
|
||||||
<v-chip size="x-small" variant="tonal">{{ post.source.platform }}</v-chip>
|
<!-- Posts with no live subscription have source=null (alembic
|
||||||
|
0030); show a "filesystem import" affordance instead of a
|
||||||
|
platform chip. -->
|
||||||
|
<v-chip size="x-small" variant="tonal">
|
||||||
|
{{ post.source?.platform ?? 'filesystem import' }}
|
||||||
|
</v-chip>
|
||||||
<RouterLink
|
<RouterLink
|
||||||
:to="{ name: 'artist', params: { slug: post.artist.slug } }"
|
:to="{ name: 'artist', params: { slug: post.artist.slug } }"
|
||||||
class="fc-post-card__artist"
|
class="fc-post-card__artist"
|
||||||
@@ -25,7 +30,7 @@
|
|||||||
v-if="post.post_url"
|
v-if="post.post_url"
|
||||||
:href="post.post_url" target="_blank" rel="noopener"
|
:href="post.post_url" target="_blank" rel="noopener"
|
||||||
icon="mdi-open-in-new" size="x-small" variant="text"
|
icon="mdi-open-in-new" size="x-small" variant="text"
|
||||||
:aria-label="`open original post on ${post.source.platform}`"
|
:aria-label="`open original post on ${post.source?.platform ?? 'web'}`"
|
||||||
@click.stop
|
@click.stop
|
||||||
/>
|
/>
|
||||||
<v-btn
|
<v-btn
|
||||||
|
|||||||
@@ -37,8 +37,8 @@ async def test_artist_overview_post_count(client, db):
|
|||||||
)
|
)
|
||||||
db.add(s)
|
db.add(s)
|
||||||
await db.flush()
|
await db.flush()
|
||||||
db.add(Post(source_id=s.id, external_post_id="p1"))
|
db.add(Post(source_id=s.id, artist_id=a.id, external_post_id="p1"))
|
||||||
db.add(Post(source_id=s.id, external_post_id="p2"))
|
db.add(Post(source_id=s.id, artist_id=a.id, external_post_id="p2"))
|
||||||
await db.flush()
|
await db.flush()
|
||||||
await db.commit()
|
await db.commit()
|
||||||
resp = await client.get("/api/artist/lyra")
|
resp = await client.get("/api/artist/lyra")
|
||||||
|
|||||||
@@ -25,7 +25,7 @@ async def seeded_post(db):
|
|||||||
db.add(source)
|
db.add(source)
|
||||||
await db.flush()
|
await db.flush()
|
||||||
post = Post(
|
post = Post(
|
||||||
source_id=source.id, external_post_id="API1",
|
source_id=source.id, artist_id=artist.id, external_post_id="API1",
|
||||||
post_title="Hello", post_url="https://p/alice-api/1",
|
post_title="Hello", post_url="https://p/alice-api/1",
|
||||||
post_date=datetime.now(UTC),
|
post_date=datetime.now(UTC),
|
||||||
description="<p>hi</p>",
|
description="<p>hi</p>",
|
||||||
@@ -123,7 +123,8 @@ async def post_timeline(db):
|
|||||||
posts = []
|
posts = []
|
||||||
for i in range(5):
|
for i in range(5):
|
||||||
p = Post(
|
p = Post(
|
||||||
source_id=source.id, external_post_id=f"TL{i}",
|
source_id=source.id, artist_id=artist.id,
|
||||||
|
external_post_id=f"TL{i}",
|
||||||
post_title=f"post {i}", post_date=base + timedelta(days=i),
|
post_title=f"post {i}", post_date=base + timedelta(days=i),
|
||||||
)
|
)
|
||||||
db.add(p)
|
db.add(p)
|
||||||
@@ -189,7 +190,7 @@ async def test_detail_returns_uncapped_thumbnails(client, db):
|
|||||||
db.add(s)
|
db.add(s)
|
||||||
await db.flush()
|
await db.flush()
|
||||||
p = Post(
|
p = Post(
|
||||||
source_id=s.id, external_post_id="DETAIL10",
|
source_id=s.id, artist_id=a.id, external_post_id="DETAIL10",
|
||||||
post_title="big post", description="<p>body</p>",
|
post_title="big post", description="<p>body</p>",
|
||||||
)
|
)
|
||||||
db.add(p)
|
db.add(p)
|
||||||
|
|||||||
@@ -26,7 +26,8 @@ async def _seed_full(db):
|
|||||||
url="https://patreon.test/alice")
|
url="https://patreon.test/alice")
|
||||||
db.add(source)
|
db.add(source)
|
||||||
await db.flush()
|
await db.flush()
|
||||||
post = Post(source_id=source.id, external_post_id="555",
|
post = Post(source_id=source.id, artist_id=artist.id,
|
||||||
|
external_post_id="555",
|
||||||
post_url="https://patreon.test/p/555", post_title="Set 1",
|
post_url="https://patreon.test/p/555", post_title="Set 1",
|
||||||
post_date=datetime(2023, 8, 1, tzinfo=UTC),
|
post_date=datetime(2023, 8, 1, tzinfo=UTC),
|
||||||
description="<p>hi</p>", attachment_count=2)
|
description="<p>hi</p>", attachment_count=2)
|
||||||
|
|||||||
@@ -25,7 +25,7 @@ async def _fixture(db):
|
|||||||
db.add(src)
|
db.add(src)
|
||||||
await db.flush()
|
await db.flush()
|
||||||
post = Post(
|
post = Post(
|
||||||
source_id=src.id, external_post_id="p1",
|
source_id=src.id, artist_id=artist.id, external_post_id="p1",
|
||||||
post_date=datetime(2026, 3, 1, tzinfo=UTC),
|
post_date=datetime(2026, 3, 1, tzinfo=UTC),
|
||||||
)
|
)
|
||||||
db.add(post)
|
db.add(post)
|
||||||
|
|||||||
@@ -35,7 +35,7 @@ async def _post(db, artist_name, slug, ext):
|
|||||||
url=f"https://patreon.test/{slug}")
|
url=f"https://patreon.test/{slug}")
|
||||||
db.add(s)
|
db.add(s)
|
||||||
await db.flush()
|
await db.flush()
|
||||||
p = Post(source_id=s.id, external_post_id=ext)
|
p = Post(source_id=s.id, artist_id=a.id, external_post_id=ext)
|
||||||
db.add(p)
|
db.add(p)
|
||||||
await db.flush()
|
await db.flush()
|
||||||
return a, s, p
|
return a, s, p
|
||||||
|
|||||||
@@ -152,7 +152,8 @@ async def _seed_image_with_post(
|
|||||||
db.add(source)
|
db.add(source)
|
||||||
await db.flush()
|
await db.flush()
|
||||||
post = Post(
|
post = Post(
|
||||||
source_id=source.id, external_post_id=external_post_id,
|
source_id=source.id, artist_id=artist.id,
|
||||||
|
external_post_id=external_post_id,
|
||||||
post_title="A Post", post_date=post_date,
|
post_title="A Post", post_date=post_date,
|
||||||
)
|
)
|
||||||
db.add(post)
|
db.add(post)
|
||||||
|
|||||||
@@ -88,11 +88,36 @@ def test_find_or_create_post_idempotent(importer, artist_row, db_sync):
|
|||||||
)
|
)
|
||||||
p1 = importer._find_or_create_post(
|
p1 = importer._find_or_create_post(
|
||||||
source_id=src.id, external_post_id="ext-001",
|
source_id=src.id, external_post_id="ext-001",
|
||||||
|
artist_id=artist_row.id,
|
||||||
)
|
)
|
||||||
p2 = importer._find_or_create_post(
|
p2 = importer._find_or_create_post(
|
||||||
source_id=src.id, external_post_id="ext-001",
|
source_id=src.id, external_post_id="ext-001",
|
||||||
|
artist_id=artist_row.id,
|
||||||
)
|
)
|
||||||
assert p1.id == p2.id
|
assert p1.id == p2.id
|
||||||
|
assert p1.artist_id == artist_row.id
|
||||||
|
|
||||||
|
|
||||||
|
def test_find_or_create_post_idempotent_with_null_source(
|
||||||
|
importer, artist_row, db_sync,
|
||||||
|
):
|
||||||
|
"""Post.source_id is nullable since alembic 0030 — filesystem-imported
|
||||||
|
posts with no live subscription have NULL source_id. The partial
|
||||||
|
unique index `uq_post_artist_external_id_null_source` on
|
||||||
|
(artist_id, external_post_id) WHERE source_id IS NULL guards the
|
||||||
|
dedup; the helper matches on the same (artist_id, external_post_id)
|
||||||
|
keys when source_id is None."""
|
||||||
|
p1 = importer._find_or_create_post(
|
||||||
|
source_id=None, external_post_id="fs-001",
|
||||||
|
artist_id=artist_row.id,
|
||||||
|
)
|
||||||
|
p2 = importer._find_or_create_post(
|
||||||
|
source_id=None, external_post_id="fs-001",
|
||||||
|
artist_id=artist_row.id,
|
||||||
|
)
|
||||||
|
assert p1.id == p2.id
|
||||||
|
assert p1.source_id is None
|
||||||
|
assert p1.artist_id == artist_row.id
|
||||||
|
|
||||||
|
|
||||||
def test_find_or_create_source_recovers_from_integrity_error(
|
def test_find_or_create_source_recovers_from_integrity_error(
|
||||||
@@ -147,13 +172,13 @@ def test_find_or_create_source_recovers_from_integrity_error(
|
|||||||
assert recovered.id == pre_existing.id
|
assert recovered.id == pre_existing.id
|
||||||
|
|
||||||
|
|
||||||
def test_source_for_sidecar_reuses_existing_subscription(
|
def test_lookup_source_for_sidecar_returns_existing_subscription(
|
||||||
importer, artist_row, db_sync,
|
importer, artist_row, db_sync,
|
||||||
):
|
):
|
||||||
"""The filesystem-import sidecar resolver should attach to whatever
|
"""The sidecar-import Source lookup returns the existing Source for
|
||||||
Source already exists for (artist, platform) — the canonical subscription
|
(artist, platform) when one exists — letting filesystem-imported
|
||||||
Source — regardless of its URL. Without this, every imported post
|
content attach to the artist's real subscription instead of being
|
||||||
spawned its own Source row.
|
orphaned.
|
||||||
"""
|
"""
|
||||||
canonical = Source(
|
canonical = Source(
|
||||||
artist_id=artist_row.id, platform="patreon",
|
artist_id=artist_row.id, platform="patreon",
|
||||||
@@ -162,84 +187,30 @@ def test_source_for_sidecar_reuses_existing_subscription(
|
|||||||
db_sync.add(canonical)
|
db_sync.add(canonical)
|
||||||
db_sync.flush()
|
db_sync.flush()
|
||||||
|
|
||||||
resolved = importer._source_for_sidecar(
|
resolved = importer._lookup_source_for_sidecar(
|
||||||
artist_id=artist_row.id, platform="patreon",
|
artist_id=artist_row.id, platform="patreon",
|
||||||
artist_slug=artist_row.slug,
|
|
||||||
)
|
)
|
||||||
|
assert resolved is not None
|
||||||
assert resolved.id == canonical.id
|
assert resolved.id == canonical.id
|
||||||
|
|
||||||
|
|
||||||
def test_source_for_sidecar_creates_synthetic_anchor_when_none_exists(
|
def test_lookup_source_for_sidecar_returns_none_when_no_subscription(
|
||||||
importer, artist_row, db_sync,
|
importer, artist_row, db_sync,
|
||||||
):
|
):
|
||||||
"""No subscription Source for this (artist, platform) yet. The helper
|
"""Alembic 0030 made Post.source_id nullable; the importer no
|
||||||
creates one synthetic anchor (enabled=False, url='sidecar:<plat>:<slug>')
|
longer creates synthetic `sidecar:<platform>:<slug>` anchor rows
|
||||||
so subsequent imports reuse it instead of spawning per-post Sources.
|
when no real subscription exists. The lookup returns None and the
|
||||||
|
caller carries that through as a NULL source_id on the new Post.
|
||||||
"""
|
"""
|
||||||
resolved = importer._source_for_sidecar(
|
resolved = importer._lookup_source_for_sidecar(
|
||||||
artist_id=artist_row.id, platform="pixiv",
|
artist_id=artist_row.id, platform="pixiv",
|
||||||
artist_slug=artist_row.slug,
|
|
||||||
)
|
)
|
||||||
assert resolved.url == f"sidecar:pixiv:{artist_row.slug}"
|
assert resolved is None
|
||||||
assert resolved.enabled is False
|
|
||||||
assert resolved.artist_id == artist_row.id
|
|
||||||
assert resolved.platform == "pixiv"
|
|
||||||
|
|
||||||
# Second call returns the same row (no new Source spawned).
|
# Verify no Source row was created as a side effect.
|
||||||
again = importer._source_for_sidecar(
|
count = db_sync.execute(
|
||||||
artist_id=artist_row.id, platform="pixiv",
|
select(Source).where(
|
||||||
artist_slug=artist_row.slug,
|
Source.artist_id == artist_row.id, Source.platform == "pixiv",
|
||||||
)
|
)
|
||||||
assert again.id == resolved.id
|
).all()
|
||||||
|
assert count == []
|
||||||
|
|
||||||
def test_source_for_sidecar_distinct_platforms_distinct_anchors(
|
|
||||||
importer, artist_row, db_sync,
|
|
||||||
):
|
|
||||||
"""One synthetic anchor per (artist, platform). Different platforms get
|
|
||||||
different anchors even when no campaign Source exists for either.
|
|
||||||
"""
|
|
||||||
p = importer._source_for_sidecar(
|
|
||||||
artist_id=artist_row.id, platform="patreon",
|
|
||||||
artist_slug=artist_row.slug,
|
|
||||||
)
|
|
||||||
x = importer._source_for_sidecar(
|
|
||||||
artist_id=artist_row.id, platform="pixiv",
|
|
||||||
artist_slug=artist_row.slug,
|
|
||||||
)
|
|
||||||
assert p.id != x.id
|
|
||||||
assert p.platform == "patreon"
|
|
||||||
assert x.platform == "pixiv"
|
|
||||||
|
|
||||||
|
|
||||||
def test_source_for_sidecar_prefers_real_over_synthetic_when_both_exist(
|
|
||||||
importer, artist_row, db_sync,
|
|
||||||
):
|
|
||||||
"""When BOTH a synthetic anchor AND a real Source exist for the same
|
|
||||||
(artist, platform), the resolver must return the REAL one. This is the
|
|
||||||
fix for the 2026-05-31 phantom-subscription bug: alembic 0022 had
|
|
||||||
rewritten an older per-post Source row into a sidecar synthetic, and
|
|
||||||
the operator later added the real subscription. The old `ORDER BY
|
|
||||||
id ASC LIMIT 1` lookup picked the older synthetic (lower id),
|
|
||||||
silently attaching every gallery-dl download to the wrong Source.
|
|
||||||
"""
|
|
||||||
synthetic = Source(
|
|
||||||
artist_id=artist_row.id, platform="patreon",
|
|
||||||
url=f"sidecar:patreon:{artist_row.slug}", enabled=False,
|
|
||||||
)
|
|
||||||
db_sync.add(synthetic)
|
|
||||||
db_sync.flush()
|
|
||||||
real = Source(
|
|
||||||
artist_id=artist_row.id, platform="patreon",
|
|
||||||
url="https://www.patreon.com/testartist", enabled=True,
|
|
||||||
)
|
|
||||||
db_sync.add(real)
|
|
||||||
db_sync.flush()
|
|
||||||
assert synthetic.id < real.id # ordering precondition
|
|
||||||
|
|
||||||
resolved = importer._source_for_sidecar(
|
|
||||||
artist_id=artist_row.id, platform="patreon",
|
|
||||||
artist_slug=artist_row.slug,
|
|
||||||
)
|
|
||||||
assert resolved.id == real.id
|
|
||||||
assert resolved.url == "https://www.patreon.com/testartist"
|
|
||||||
|
|||||||
@@ -17,7 +17,7 @@ async def _post(db, **post_kwargs):
|
|||||||
db.add(src)
|
db.add(src)
|
||||||
await db.flush()
|
await db.flush()
|
||||||
post = Post(
|
post = Post(
|
||||||
source_id=src.id, external_post_id="p1",
|
source_id=src.id, artist_id=artist.id, external_post_id="p1",
|
||||||
post_date=datetime(2026, 3, 1, tzinfo=UTC),
|
post_date=datetime(2026, 3, 1, tzinfo=UTC),
|
||||||
**post_kwargs,
|
**post_kwargs,
|
||||||
)
|
)
|
||||||
|
|||||||
@@ -62,7 +62,7 @@ async def _run_backfill(db):
|
|||||||
async def test_backfill_primary_post(db):
|
async def test_backfill_primary_post(db):
|
||||||
rec = await _img(db, 1)
|
rec = await _img(db, 1)
|
||||||
a, s = await _artist_source(db, "Alice", "alice")
|
a, s = await _artist_source(db, "Alice", "alice")
|
||||||
post = Post(source_id=s.id, external_post_id="1")
|
post = Post(source_id=s.id, artist_id=a.id, external_post_id="1")
|
||||||
db.add(post)
|
db.add(post)
|
||||||
await db.flush()
|
await db.flush()
|
||||||
rec.primary_post_id = post.id
|
rec.primary_post_id = post.id
|
||||||
@@ -78,7 +78,7 @@ async def test_backfill_primary_post(db):
|
|||||||
async def test_backfill_provenance_fallback(db):
|
async def test_backfill_provenance_fallback(db):
|
||||||
rec = await _img(db, 1)
|
rec = await _img(db, 1)
|
||||||
a, s = await _artist_source(db, "Bob", "bob")
|
a, s = await _artist_source(db, "Bob", "bob")
|
||||||
post = Post(source_id=s.id, external_post_id="2")
|
post = Post(source_id=s.id, artist_id=a.id, external_post_id="2")
|
||||||
db.add(post)
|
db.add(post)
|
||||||
await db.flush()
|
await db.flush()
|
||||||
db.add(ImageProvenance(image_record_id=rec.id, post_id=post.id,
|
db.add(ImageProvenance(image_record_id=rec.id, post_id=post.id,
|
||||||
|
|||||||
@@ -7,6 +7,7 @@ boundary, and the thumbnails/attachments composition.
|
|||||||
from datetime import UTC, datetime, timedelta
|
from datetime import UTC, datetime, timedelta
|
||||||
|
|
||||||
import pytest
|
import pytest
|
||||||
|
from sqlalchemy import select
|
||||||
|
|
||||||
from backend.app.models import (
|
from backend.app.models import (
|
||||||
Artist,
|
Artist,
|
||||||
@@ -59,8 +60,13 @@ async def _seed_post(
|
|||||||
db, source_id: int, *, external_id: str,
|
db, source_id: int, *, external_id: str,
|
||||||
post_date=None, downloaded_at=None, title=None, description=None,
|
post_date=None, downloaded_at=None, title=None, description=None,
|
||||||
):
|
):
|
||||||
|
# Post.artist_id is NOT NULL since alembic 0030; look it up from
|
||||||
|
# the source so callsites don't have to thread the artist through.
|
||||||
|
artist_id = (await db.execute(
|
||||||
|
select(Source.artist_id).where(Source.id == source_id)
|
||||||
|
)).scalar_one()
|
||||||
p = Post(
|
p = Post(
|
||||||
source_id=source_id, external_post_id=external_id,
|
source_id=source_id, artist_id=artist_id, external_post_id=external_id,
|
||||||
post_title=title, post_date=post_date,
|
post_title=title, post_date=post_date,
|
||||||
description=description,
|
description=description,
|
||||||
)
|
)
|
||||||
|
|||||||
@@ -37,7 +37,7 @@ async def _seed_post(db, *, artist_name, slug, platform, ext_id,
|
|||||||
db.add(source)
|
db.add(source)
|
||||||
await db.flush()
|
await db.flush()
|
||||||
post = Post(
|
post = Post(
|
||||||
source_id=source.id, external_post_id=ext_id,
|
source_id=source.id, artist_id=artist.id, external_post_id=ext_id,
|
||||||
post_url=f"https://{platform}.test/p/{ext_id}",
|
post_url=f"https://{platform}.test/p/{ext_id}",
|
||||||
post_title=title, post_date=datetime(2023, 8, 1, tzinfo=UTC),
|
post_title=title, post_date=datetime(2023, 8, 1, tzinfo=UTC),
|
||||||
description=desc, attachment_count=count,
|
description=desc, attachment_count=count,
|
||||||
|
|||||||
@@ -138,7 +138,7 @@ async def test_update_changes_fields(db):
|
|||||||
@pytest.mark.asyncio
|
@pytest.mark.asyncio
|
||||||
async def test_list_hides_sidecar_synthetic_anchors(db):
|
async def test_list_hides_sidecar_synthetic_anchors(db):
|
||||||
"""Filesystem-import synthetic Sources (url='sidecar:<platform>:<slug>',
|
"""Filesystem-import synthetic Sources (url='sidecar:<platform>:<slug>',
|
||||||
enabled=False — see importer._source_for_sidecar) used to leak into the
|
enabled=False — historical pre-alembic-0030 artifact) used to leak into the
|
||||||
Subscriptions UI as phantom subscriptions because list() didn't filter
|
Subscriptions UI as phantom subscriptions because list() didn't filter
|
||||||
them. They aren't pollable feeds; hide by default."""
|
them. They aren't pollable feeds; hide by default."""
|
||||||
artist = await _artist(db, "Alice")
|
artist = await _artist(db, "Alice")
|
||||||
|
|||||||
Reference in New Issue
Block a user