fix(tags): normalize task fails fast on lock + logs progress
CI / lint (push) Successful in 2s
CI / frontend-build (push) Successful in 22s
CI / backend-lint-and-test (push) Successful in 26s
CI / integration (push) Successful in 3m5s

normalize_tags_task ran to the 40-min hard limit with zero logs (operator-
flagged 2026-06-07). Cause: a per-group merge repoints series_page (via
_repoint_series_pages); during the wedged 0040 migration that held ACCESS
EXCLUSIVE on series_page, the merge's UPDATE blocked on that lock. The time-box
check is at the top of the group loop, so a statement blocked mid-group never
yields back to it — the task sat until the Celery hard kill. No logs because the
only log fired per *finished* group.

- Set lock_timeout=30s on the normalize session (opt-in server_settings on the
  async factory). A blocked merge now raises, the per-group handler rolls back +
  counts an error, and the loop continues — one stuck group can't strand the
  chunk, and the budget checkpoint stays effective.
- Log group count at start + a heartbeat every 25 groups, so a long/slow run is
  diagnosable instead of silent.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-06-07 21:02:39 -04:00
parent 9770dd3474
commit a00a2786e3
3 changed files with 42 additions and 4 deletions
+18 -3
View File
@@ -15,8 +15,15 @@ from sqlalchemy.pool import NullPool
from ..config import get_config
def async_session_factory():
"""Return ``(sessionmaker, engine)`` bound to a fresh async engine."""
def async_session_factory(*, server_settings: dict | None = None):
"""Return ``(sessionmaker, engine)`` bound to a fresh async engine.
``server_settings`` (optional) are applied by asyncpg as per-connection GUCs
on connect. Because NullPool opens a fresh real connection per checkout, every
transaction in the task inherits them — used to set ``lock_timeout`` so a
statement blocked on a lock fails fast instead of hanging to the Celery hard
limit (operator-flagged normalize_tags timeout 2026-06-07).
"""
cfg = get_config()
# NullPool: this engine lives for ONE task (created + disposed per
# asyncio.run loop), so intra-task connection pooling buys nothing and
@@ -26,5 +33,13 @@ def async_session_factory():
# phase 3 (asyncpg ConnectionDoesNotExistError, Anduo #40014). NullPool
# opens a fresh real connection on each checkout, so phase 3 always
# reconnects clean; pre_ping is then redundant.
engine = create_async_engine(cfg.database_url, future=True, poolclass=NullPool)
connect_args = {}
if server_settings:
connect_args["server_settings"] = {
k: str(v) for k, v in server_settings.items()
}
engine = create_async_engine(
cfg.database_url, future=True, poolclass=NullPool,
connect_args=connect_args,
)
return async_sessionmaker(engine, class_=AsyncSession, expire_on_commit=False), engine