Merge pull request 'v26.05.26.3 — hotfix: migration 0022 pre-merge across ENTIRE (canonical+others) group' (#25) from dev into main

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

.forgejo/workflows/ci.yml

@@ -24,13 +24,17 @@ jobs:
     steps:
       - uses: actions/checkout@v4
 
-      - name: Cache pip wheels
+      # Cache step removed 2026-05-26: act_runner's cache backend has been
-        uses: actions/cache@v4
+      # broken on this homelab runner since 2026-05-15 (first as request-
-        with:
+      # timeout warnings, then as hard "Cannot find module .../dist/restore/
-          path: ~/.cache/pip
+      # index.js" failures that tank the whole job). The cache step targeted
-          key: pip-${{ runner.os }}-py314-${{ hashFiles('requirements.txt') }}
+      # ~/.cache/pip but the install below uses `uv pip install` primarily,
-          restore-keys: |
+      # whose own cache lives at ~/.cache/uv — so the cache step's real
-            pip-${{ runner.os }}-py314-
+      # benefit was marginal even when working. Cost of removal: ~30s of
+      # wheel downloads per job. Future re-enable: mount ~/.cache/uv as a
+      # docker volume at the runner level (skips actions/cache entirely),
+      # or fix the runner-side cache backend (clear /var/run/act/actions/*,
+      # pin act_runner version, etc.).
 
       - name: Install Python deps
         # ruff is pre-installed in the ci-python image (see CI-Runner/CI-python/
@@ -124,13 +128,6 @@ jobs:
           --health-retries 10
     steps:
       - uses: actions/checkout@v4
-      - name: Cache pip wheels
-        uses: actions/cache@v4
-        with:
-          path: ~/.cache/pip
-          key: pip-${{ runner.os }}-py314-${{ hashFiles('requirements.txt') }}
-          restore-keys: |
-            pip-${{ runner.os }}-py314-
       - name: API integration shard (resolve service IPs, migrate, test)
         run: |
           set -eux
@@ -189,13 +186,6 @@ jobs:
           --health-retries 10
     steps:
       - uses: actions/checkout@v4
-      - name: Cache pip wheels
-        uses: actions/cache@v4
-        with:
-          path: ~/.cache/pip
-          key: pip-${{ runner.os }}-py314-${{ hashFiles('requirements.txt') }}
-          restore-keys: |
-            pip-${{ runner.os }}-py314-
       - name: Importer integration shard (resolve service IPs, migrate, test)
         run: |
           set -eux
@@ -254,13 +244,6 @@ jobs:
           --health-retries 10
     steps:
       - uses: actions/checkout@v4
-      - name: Cache pip wheels
-        uses: actions/cache@v4
-        with:
-          path: ~/.cache/pip
-          key: pip-${{ runner.os }}-py314-${{ hashFiles('requirements.txt') }}
-          restore-keys: |
-            pip-${{ runner.os }}-py314-
       - name: Core integration shard (everything not api / importer / migration / phash / sidecar / scan / archive / backfill)
         run: |
           set -eux

alembic/versions/0022_source_per_artist_platform.py

@@ -16,12 +16,21 @@ filesystem importer was misusing Source as a per-post key.
 Migration steps per (artist_id, platform) group with >1 Source:
   1. Pick canonical — prefer a URL NOT matching '/posts/<id>$' (real
      campaign URL like /cw/Atole); else min(id).
-  2. Reparent Posts and ImageProvenance off the other Sources onto canonical.
+  2. PRE-merge any Posts under non-canonical sources whose
-  3. Merge any Posts that collide on (canonical_source_id, external_post_id)
+     external_post_id ALREADY exists under the canonical source. (Same
-     by repointing ImageProvenance + ImageRecord.primary_post_id to the
+     gallery-dl post imported via two different sidecar paths can plant
-     earliest Post, dedupe ImageProvenance, delete the loser Posts.
+     two Post rows with identical external_post_id under different
-  4. Delete the orphan Source rows.
+     Sources for the same artist.) Repoint ImageProvenance +
-  5. If the canonical Source's URL still looks like a per-post URL (no
+     ImageRecord.primary_post_id to the canonical-side Post, dedupe
+     ImageProvenance against alembic 0021's uq, then delete the
+     non-canonical-side Post. This MUST happen before step 3 — Postgres
+     fires uq_post_source_external_id row-by-row during the bulk UPDATE
+     and the merge-after-reparent ordering 500s on first collision
+     (operator-hit during v26.05.26.1 deploy, 2026-05-26).
+  3. Reparent remaining Posts onto canonical (no collisions possible now).
+  4. Reparent ImageProvenance.source_id off the non-canonical sources.
+  5. Delete the orphan Source rows.
+  6. If the canonical Source's URL still looks like a per-post URL (no
      campaign URL existed among candidates), rewrite it to
      'sidecar:<platform>:<artist_slug>' so the artist detail page shows
      something readable.
@@ -74,7 +83,74 @@ def upgrade() -> None:
         if not other_ids:
             continue
 
-        # Reparent Posts off the other Sources.
+        # STEP 2: PRE-merge ALL Posts with duplicate external_post_id
+        # across the entire (canonical + others) group, BEFORE the bulk
+        # reparent. Two cases must both be handled:
+        #   (A) canonical has Post X with epid=N; an "other" source has
+        #       Post Y with epid=N → after bulk UPDATE, (canonical, N)
+        #       collides with itself.
+        #   (B) two different "other" sources each have a Post with
+        #       epid=N; canonical has none → after bulk UPDATE, both
+        #       are repointed to (canonical, N) and the second collides.
+        # The earlier version of this migration only handled (A); the
+        # operator's deploy 2026-05-26 tripped (B) at line 139.
+        # Fix: group ALL Posts in the (artist, platform) by epid; for
+        # any group with count>1, pick the keep (prefer one already
+        # under canonical; else lowest id) and merge the rest into it.
+        all_posts = conn.execute(
+            text("""
+                SELECT external_post_id, id, source_id
+                FROM post
+                WHERE source_id = :canonical OR source_id = ANY(:others)
+                ORDER BY external_post_id, id
+            """),
+            {"canonical": canonical_id, "others": other_ids},
+        ).fetchall()
+        by_epid: dict = {}
+        for epid, post_id, src_id in all_posts:
+            by_epid.setdefault(epid, []).append((post_id, src_id))
+        for epid, posts in by_epid.items():
+            if len(posts) <= 1:
+                continue
+            # Prefer a Post already under canonical as the keep.
+            canonical_posts = [p for p in posts if p[1] == canonical_id]
+            if canonical_posts:
+                keep_id = canonical_posts[0][0]
+            else:
+                keep_id = posts[0][0]  # already sorted by id ASC
+            drop_ids = [p[0] for p in posts if p[0] != keep_id]
+            for drop_id in drop_ids:
+                conn.execute(
+                    text("""
+                        UPDATE image_provenance SET post_id = :keep
+                        WHERE post_id = :drop_
+                    """),
+                    {"keep": keep_id, "drop_": drop_id},
+                )
+                conn.execute(
+                    text("""
+                        UPDATE image_record SET primary_post_id = :keep
+                        WHERE primary_post_id = :drop_
+                    """),
+                    {"keep": keep_id, "drop_": drop_id},
+                )
+                # Repointed provenance may now collide on
+                # uq_image_provenance_image_post (alembic 0021). Dedupe.
+                conn.execute(text("""
+                    DELETE FROM image_provenance ip1
+                    USING image_provenance ip2
+                    WHERE ip1.image_record_id = ip2.image_record_id
+                      AND ip1.post_id = ip2.post_id
+                      AND ip1.id > ip2.id
+                """))
+                conn.execute(
+                    text("DELETE FROM post WHERE id = :drop_"),
+                    {"drop_": drop_id},
+                )
+
+        # STEP 3: Bulk reparent the remaining Posts off the other
+        # Sources. After step 2, no collisions on
+        # (canonical, external_post_id) are possible.
         conn.execute(
             text("""
                 UPDATE post SET source_id = :canonical
@@ -82,7 +158,9 @@ def upgrade() -> None:
             """),
             {"canonical": canonical_id, "others": other_ids},
         )
-        # Reparent ImageProvenance.source_id similarly (denormalized FK).
+
+        # STEP 4: Reparent ImageProvenance.source_id (denormalized FK).
+        # No UNIQUE on source_id; safe bulk update.
         conn.execute(
             text("""
                 UPDATE image_provenance SET source_id = :canonical
@@ -91,53 +169,7 @@ def upgrade() -> None:
             {"canonical": canonical_id, "others": other_ids},
         )
 
-        # Merge any Posts colliding on (canonical, external_post_id) after
+        # STEP 5: Drop the orphan Sources.
-        # reparent. In practice within a single (artist, platform) group
-        # these should be rare — each gallery-dl post has a unique id — but
-        # safe to handle.
-        post_collisions = conn.execute(
-            text("""
-                SELECT external_post_id, ARRAY_AGG(id ORDER BY id) AS ids
-                FROM post
-                WHERE source_id = :canonical
-                GROUP BY external_post_id
-                HAVING COUNT(*) > 1
-            """),
-            {"canonical": canonical_id},
-        ).fetchall()
-        for _epid, post_ids in post_collisions:
-            keep = post_ids[0]
-            drops = post_ids[1:]
-            conn.execute(
-                text("""
-                    UPDATE image_provenance SET post_id = :keep
-                    WHERE post_id = ANY(:drops)
-                """),
-                {"keep": keep, "drops": drops},
-            )
-            conn.execute(
-                text("""
-                    UPDATE image_record SET primary_post_id = :keep
-                    WHERE primary_post_id = ANY(:drops)
-                """),
-                {"keep": keep, "drops": drops},
-            )
-            # Repointed provenance rows may now collide on
-            # uq_image_provenance_image_post (alembic 0021). Same dedupe
-            # pattern: keep min(id) per (image_record_id, post_id).
-            conn.execute(text("""
-                DELETE FROM image_provenance ip1
-                USING image_provenance ip2
-                WHERE ip1.image_record_id = ip2.image_record_id
-                  AND ip1.post_id = ip2.post_id
-                  AND ip1.id > ip2.id
-            """))
-            conn.execute(
-                text("DELETE FROM post WHERE id = ANY(:drops)"),
-                {"drops": drops},
-            )
-
-        # Drop the orphan Sources.
         conn.execute(
             text("DELETE FROM source WHERE id = ANY(:others)"),
             {"others": other_ids},