feat(model): nullable Post.source_id + denormalized Post.artist_id; retire sidecar synthetics
Operator-asked 2026-06-01 after the Dymkens orphan investigation (Scribe plan #540). The pre-2030 sidecar-synthetic Source pattern (`sidecar:<platform>:<slug>` enabled=false rows) existed solely to satisfy `Post.source_id NOT NULL`, and leaked into the Subscriptions UI as phantom subscriptions. Now the data model says what's true: filesystem-imported content with no live subscription has NULL source_id, full stop. ## Schema (alembic 0030) - `post.artist_id` — NEW NOT NULL FK to artist (CASCADE). Backfilled from source.artist_id in the migration. Indexed for the artist-filter queries. - `post.source_id` — NOT NULL → nullable; FK ondelete CASCADE → SET NULL. Deleting a Source detaches its Posts instead of destroying archived content (subscription ends, archive stays). - `image_provenance.source_id` — same nullable + SET NULL. - Partial unique index `uq_post_artist_external_id_null_source` on (artist_id, external_post_id) WHERE source_id IS NULL — guards filesystem-import dedup since the existing source-bound unique ignores NULLs (Postgres treats NULL != NULL). - Sidecar synthetic Sources deleted: NULL out FKs in post, image_provenance first, then DELETE FROM source WHERE url LIKE 'sidecar:%'. The Dymkens cleanup. ## Model + service changes - `Post.source_id` → `Mapped[int | None]`; new `Post.artist_id` denormalized. - `ImageProvenance.source_id` → `Mapped[int | None]`. - Importer: `_source_for_sidecar` (synthetic-creating) → `_lookup_source_for_sidecar` (returns None when no subscription). `_find_or_create_post` takes required `artist_id`; matches on (source_id, external_post_id) for source-bound posts or (artist_id, external_post_id) for NULL-source posts. - Service queries switched off the Source detour to use Post.artist_id directly: post_feed_service.scroll/around/get_post (LEFT JOIN to Source so NULL-source posts surface); artist_service date_row/ activity/post_count; provenance_service.for_image/for_post (LEFT JOIN); gallery_service._provenance_exists_where_artist via Post.artist_id instead of ImageProvenance.source_id → Source. - `_to_dict` and provenance dict-builders emit `"source": null` for NULL-source rows. ## Frontend - `ProvenancePanel.vue` + `PostCard.vue`: render `e.source?.platform ?? 'filesystem import'` so NULL-source posts get a clear "filesystem import" affordance instead of a NaN crash. ## Tests - `test_importer_upsert_helpers`: removed the four synthetic-anchor tests; added `_find_or_create_post_idempotent_with_null_source` (dedup via the partial unique index) and `_lookup_source_for_sidecar_returns_*` (existing-subscription + none cases). The existing `_find_or_create_post_idempotent` now also passes `artist_id` and asserts it. - 8 other test files updated: every direct `Post(...)` construction gains `artist_id=<artist>.id`. The `_seed_post` helper in `test_post_feed_service` looks up artist_id from the source row so callsites stay one-arg. ## Verification on deploy After alembic 0030 runs: - `SELECT COUNT(*) FROM source WHERE url LIKE 'sidecar:%'` → 0. - `SELECT COUNT(*) FROM post WHERE source_id IS NULL` → count of filesystem-imported posts (Dymkens + any other historical). - Every `post.artist_id` non-null; consistent with source.artist_id for source-bound rows. - Subscriptions tab: no Dymkens phantom row. - Artist detail → Posts/Gallery: Dymkens's content still reachable via Post.artist_id. - Provenance panel renders "filesystem import" chip for NULL-source posts; PostCard same. ## Out of scope - UI to manage/delete orphan NULL-source Posts. Data model is right; UI follows if operator wants it.
This commit is contained in:
@@ -0,0 +1,141 @@
|
||||
"""nullable post.source_id + denormalized post.artist_id; retire sidecar synthetics
|
||||
|
||||
Revision ID: 0030
|
||||
Revises: 0029
|
||||
Create Date: 2026-06-01
|
||||
|
||||
Operator-asked 2026-06-01 after the Dymkens orphan investigation: the
|
||||
sidecar synthetic Source pattern (`sidecar:<platform>:<slug>` rows
|
||||
with enabled=false) was technically correct but misled the operator
|
||||
into thinking they had phantom subscriptions. The synthetics existed
|
||||
solely to satisfy `Post.source_id NOT NULL` for filesystem-imported
|
||||
content with no real subscription.
|
||||
|
||||
This migration makes the data model honest:
|
||||
|
||||
1. **Post gets a denormalized `artist_id` column** so artist filters
|
||||
work without traversing `Post → Source.artist_id`. Backfilled from
|
||||
the existing Source linkage, then NOT NULL'd.
|
||||
2. **`Post.source_id` becomes nullable**, FK ondelete `CASCADE` → `SET
|
||||
NULL`. Deleting a Source detaches its Posts instead of destroying
|
||||
imported content (semantically: subscription ends, archive stays).
|
||||
3. **`ImageProvenance.source_id` becomes nullable** with the same FK
|
||||
semantic change.
|
||||
4. **Sidecar synthetic Sources are deleted** — first NULL out the
|
||||
FKs from Post + ImageProvenance pointing at them (so the implicit
|
||||
CASCADE doesn't fire), then delete. DownloadEvent FK is unchanged
|
||||
(still CASCADE'd, NOT NULL'd) — synthetics have `enabled=false`
|
||||
so no events exist for them.
|
||||
|
||||
Uniqueness handling: the existing `uq_post_source_external_id`
|
||||
(source_id, external_post_id) keeps working for source-bound Posts
|
||||
(Postgres treats NULL != NULL so NULL-source rows aren't deduped by
|
||||
it). A second partial unique index covers the NULL-source case on
|
||||
(artist_id, external_post_id) so filesystem-imported posts still
|
||||
dedupe within an artist.
|
||||
"""
|
||||
from typing import Sequence, Union
|
||||
|
||||
import sqlalchemy as sa
|
||||
from alembic import op
|
||||
from sqlalchemy import text
|
||||
|
||||
revision: str = "0030"
|
||||
down_revision: Union[str, None] = "0029"
|
||||
branch_labels: Union[str, Sequence[str], None] = None
|
||||
depends_on: Union[str, Sequence[str], None] = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
conn = op.get_bind()
|
||||
|
||||
# Step 1: add Post.artist_id, initially nullable for backfill.
|
||||
op.add_column(
|
||||
"post",
|
||||
sa.Column("artist_id", sa.Integer, nullable=True),
|
||||
)
|
||||
op.create_foreign_key(
|
||||
"post_artist_id_fkey", "post", "artist",
|
||||
["artist_id"], ["id"], ondelete="CASCADE",
|
||||
)
|
||||
|
||||
# Step 2: backfill from Source.artist_id (every existing Post has a
|
||||
# Source today, so every row gets populated).
|
||||
conn.execute(text("""
|
||||
UPDATE post p
|
||||
SET artist_id = s.artist_id
|
||||
FROM source s
|
||||
WHERE p.source_id = s.id AND p.artist_id IS NULL
|
||||
"""))
|
||||
|
||||
# Sanity: count any remaining NULLs. Should be zero pre-this-migration.
|
||||
remaining = conn.execute(text(
|
||||
"SELECT COUNT(*) FROM post WHERE artist_id IS NULL"
|
||||
)).scalar_one()
|
||||
if remaining:
|
||||
raise RuntimeError(
|
||||
f"alembic 0030: {remaining} post rows have no resolvable "
|
||||
f"artist_id after backfill. Investigate before continuing."
|
||||
)
|
||||
|
||||
# Step 3: enforce NOT NULL + add index for artist-filter queries.
|
||||
op.alter_column("post", "artist_id", nullable=False)
|
||||
op.create_index("ix_post_artist_id", "post", ["artist_id"])
|
||||
|
||||
# Step 4: relax post.source_id + flip FK to SET NULL.
|
||||
op.alter_column("post", "source_id", nullable=True)
|
||||
op.drop_constraint("post_source_id_fkey", "post", type_="foreignkey")
|
||||
op.create_foreign_key(
|
||||
"post_source_id_fkey", "post", "source",
|
||||
["source_id"], ["id"], ondelete="SET NULL",
|
||||
)
|
||||
|
||||
# Step 5: relax image_provenance.source_id + flip FK to SET NULL.
|
||||
op.alter_column("image_provenance", "source_id", nullable=True)
|
||||
op.drop_constraint(
|
||||
"image_provenance_source_id_fkey", "image_provenance",
|
||||
type_="foreignkey",
|
||||
)
|
||||
op.create_foreign_key(
|
||||
"image_provenance_source_id_fkey", "image_provenance", "source",
|
||||
["source_id"], ["id"], ondelete="SET NULL",
|
||||
)
|
||||
|
||||
# Step 6: partial unique index on (artist_id, external_post_id) for
|
||||
# NULL-source Posts. The existing uq_post_source_external_id keeps
|
||||
# guarding source-bound rows; NULL-source rows now dedupe within
|
||||
# an artist.
|
||||
op.execute(
|
||||
"CREATE UNIQUE INDEX uq_post_artist_external_id_null_source "
|
||||
"ON post (artist_id, external_post_id) "
|
||||
"WHERE source_id IS NULL"
|
||||
)
|
||||
|
||||
# Step 7: retire sidecar synthetic Sources. NULL out the references
|
||||
# FIRST (the new FK is SET NULL so CASCADE wouldn't fire anyway, but
|
||||
# being explicit makes the intent clear). Then delete the synthetic
|
||||
# source rows. Any DownloadEvent rows under synthetics CASCADE-die
|
||||
# with the source — synthetics have enabled=false so there shouldn't
|
||||
# be any in practice.
|
||||
conn.execute(text("""
|
||||
UPDATE post
|
||||
SET source_id = NULL
|
||||
WHERE source_id IN (SELECT id FROM source WHERE url LIKE 'sidecar:%')
|
||||
"""))
|
||||
conn.execute(text("""
|
||||
UPDATE image_provenance
|
||||
SET source_id = NULL
|
||||
WHERE source_id IN (SELECT id FROM source WHERE url LIKE 'sidecar:%')
|
||||
"""))
|
||||
deleted = conn.execute(text(
|
||||
"DELETE FROM source WHERE url LIKE 'sidecar:%' RETURNING id"
|
||||
)).rowcount
|
||||
print(f"alembic 0030: deleted {deleted} sidecar synthetic source rows")
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
# Lossy migration — the deleted sidecar synthetics can't be
|
||||
# restored from the orphan post.source_id / image_provenance.source_id
|
||||
# values, and the partial unique index encodes a constraint that
|
||||
# NULL-source Posts may now exist. No safe downgrade.
|
||||
pass
|
||||
Reference in New Issue
Block a user