fix(tags): normalize task fails fast on lock + logs progress
CI / lint (push) Successful in 2s
CI / frontend-build (push) Successful in 22s
CI / backend-lint-and-test (push) Successful in 26s
CI / integration (push) Successful in 3m5s

normalize_tags_task ran to the 40-min hard limit with zero logs (operator-
flagged 2026-06-07). Cause: a per-group merge repoints series_page (via
_repoint_series_pages); during the wedged 0040 migration that held ACCESS
EXCLUSIVE on series_page, the merge's UPDATE blocked on that lock. The time-box
check is at the top of the group loop, so a statement blocked mid-group never
yields back to it — the task sat until the Celery hard kill. No logs because the
only log fired per *finished* group.

- Set lock_timeout=30s on the normalize session (opt-in server_settings on the
  async factory). A blocked merge now raises, the per-group handler rolls back +
  counts an error, and the loop continues — one stuck group can't strand the
  chunk, and the budget checkpoint stays effective.
- Log group count at start + a heartbeat every 25 groups, so a long/slow run is
  diagnosable instead of silent.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-06-07 21:02:39 -04:00
parent 9770dd3474
commit a00a2786e3
3 changed files with 42 additions and 4 deletions
+10 -1
View File
@@ -126,7 +126,16 @@ def normalize_tags_task(self) -> dict:
from ._async_session import async_session_factory
async def _run() -> dict:
async_factory, async_engine = async_session_factory()
# lock_timeout=30s: a per-group merge repoints FKs across image_tag and
# series_page; if a statement blocks on a lock (e.g. behind a schema
# migration holding ACCESS EXCLUSIVE on series_page — the exact wedge that
# made this task run to the 40-min hard limit with no progress,
# operator-flagged 2026-06-07), it now fails fast. The per-group handler
# catches it (rollback + error++) and the loop continues, so one blocked
# group can't strand the whole chunk.
async_factory, async_engine = async_session_factory(
server_settings={"lock_timeout": "30s"}
)
try:
async with async_factory() as session:
# normalize_existing_tags commits per group internally.