Migration lock safety + remove the merge's full-table scan (the real 0040-hang fix) #85

Merged

bvandeusen merged 2 commits from dev into main 2026-06-08 01:03:07 -04:00

Conversation 0 Commits 2 Files Changed 3

+52 -67

bvandeusen commented 2026-06-08 01:03:00 -04:00

Owner

Copy Link

Copy Source

Root-cause fix for the migration hang + tag-standardization timeout. CI green (run #271, e90e6b2).

The real cause (operator-diagnosed)

Web has always been a single replica — there was no migration race. 0040's ALTER series_page hung because a concurrent tag merge had locked series_page and was then stuck for minutes in _create_protective_aliases, an unindexed full scan of image_record (tagger_predictions JSON, ~59k rows). The same scan starved the tag-standardization task into its 40-min timeout. One slow query, holding a lock inside a merge, blocked both.

Fixes

• perf(tags) — drop the scan (e90e6b2). The tagger's tag_to_category map is one-to-one, and a tag's kind is set from that category when created — so kind already is the tagger's category for the name. The scan only ever rediscovered the kind. The merge now builds the protective alias from src_kind directly: no scan, no lock-holding slow step. This removes the actual blocker (fixes both symptoms at the source).
• fix(alembic) — lock_timeout, drop advisory lock (8e98e79). Sets lock_timeout (default 30s, MIGRATION_LOCK_TIMEOUT override) on the migration connection — a blocked DDL now fails fast with "canceling statement due to lock timeout" and the entrypoint exits non-zero so the deploy surfaces it / retries, instead of hanging forever. Reverts the pg_advisory_xact_lock (it treated a race that wasn't the cause and added a new indefinite-hang mode).

No schema change (alembic head stays 0042).

🤖 Generated with Claude Code

Root-cause fix for the migration hang + tag-standardization timeout. CI green (run #271, `e90e6b2`). ## The real cause (operator-diagnosed) Web has always been a single replica — there was no migration race. 0040's `ALTER series_page` hung because a concurrent **tag merge** had locked `series_page` and was then stuck for minutes in `_create_protective_aliases`, an **unindexed full scan of `image_record`** (`tagger_predictions` JSON, ~59k rows). The same scan starved the tag-standardization task into its 40-min timeout. One slow query, holding a lock inside a merge, blocked both. ## Fixes - **`perf(tags)` — drop the scan (`e90e6b2`).** The tagger's `tag_to_category` map is one-to-one, and a tag's `kind` is set from that category when created — so `kind` already *is* the tagger's category for the name. The scan only ever rediscovered the kind. The merge now builds the protective alias from `src_kind` directly: no scan, no lock-holding slow step. This removes the actual blocker (fixes both symptoms at the source). - **`fix(alembic)` — lock_timeout, drop advisory lock (`8e98e79`).** Sets `lock_timeout` (default 30s, `MIGRATION_LOCK_TIMEOUT` override) on the migration connection — a blocked DDL now fails fast with "canceling statement due to lock timeout" and the entrypoint exits non-zero so the deploy surfaces it / retries, instead of hanging forever. Reverts the `pg_advisory_xact_lock` (it treated a race that wasn't the cause and added a new indefinite-hang mode). No schema change (alembic head stays 0042). 🤖 Generated with [Claude Code](https://claude.com/claude-code)

bvandeusen added 2 commits 2026-06-08 01:03:00 -04:00

fix(alembic): lock_timeout on migrations, drop the advisory lock ... 8e98e79968

Reverses the advisory-lock approach (7309d1d) — it treated a replica race that
wasn't the cause and added a new indefinite-hang mode (a sibling/stale migrator
holding the xact lock).

Real cause of the 0040 hang (operator-diagnosed 2026-06-07): web has always been
a single replica. The migration's ALTER series_page queued behind a concurrent
tag-merge that held a series_page lock for minutes — _do_merge repoints
series_page then runs _create_protective_aliases, an unindexed full scan of
image_record (JSON column, ~59k rows). Migrations ran with no lock_timeout, so
the DDL hung indefinitely and silently.

Fix: SET lock_timeout (default 30s, env-overridable) on the migration connection
before alembic's transaction. A blocked DDL now fails fast with 'canceling
statement due to lock timeout'; the entrypoint exits non-zero so the deploy
retries / surfaces loudly instead of wedging. General protection for every
future migration. (The slow _create_protective_aliases scan — the actual lock
holder — is the separate perf fix still under discussion.)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

perf(tags): protective-alias uses tag kind, drops the image_record full scan ... e90e6b2c34

_create_protective_aliases scanned every image_record's tagger_predictions JSON
(unindexed full scan, ~59k rows) to find the categories a merged-away tag's name
was predicted under. That scan ran inside the merge transaction AFTER it had
locked series_page — on a large library it held that lock for minutes and is what
blocked migration 0040 (and starved the standardization task into its 40-min
timeout).

The scan was redundant: the tagger's tag_to_category map is one-to-one (a name has
exactly one category) and a tag's kind is set from that category when created, so
kind already IS the tagger's category for the name. The scan only ever rediscovered
the kind. Build the single protective alias from src_kind directly — no scan, no
lock-holding slow step in the merge.

Rewrote test_alias_per_observed_prediction_category (which encoded the
can't-actually-happen one-name-two-categories case) → test_protective_alias_uses_tag_kind.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

bvandeusen merged commit e46893fefd into main 2026-06-08 01:03:07 -04:00

Sign in to join this conversation.

Reviewers

No Reviewers

Labels

No items

No Label

Milestone

No items

No Milestone

Projects

Clear projects

No project

Assignees

Clear assignees

bvandeusen

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: bvandeusen/FabledCurator#85