fix(tags): normalize task fails fast on lock + logs progress
normalize_tags_task ran to the 40-min hard limit with zero logs (operator- flagged 2026-06-07). Cause: a per-group merge repoints series_page (via _repoint_series_pages); during the wedged 0040 migration that held ACCESS EXCLUSIVE on series_page, the merge's UPDATE blocked on that lock. The time-box check is at the top of the group loop, so a statement blocked mid-group never yields back to it — the task sat until the Celery hard kill. No logs because the only log fired per *finished* group. - Set lock_timeout=30s on the normalize session (opt-in server_settings on the async factory). A blocked merge now raises, the per-group handler rolls back + counts an error, and the loop continues — one stuck group can't strand the chunk, and the budget checkpoint stays effective. - Log group count at start + a heartbeat every 25 groups, so a long/slow run is diagnosable instead of silent. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -15,8 +15,15 @@ from sqlalchemy.pool import NullPool
|
||||
from ..config import get_config
|
||||
|
||||
|
||||
def async_session_factory():
|
||||
"""Return ``(sessionmaker, engine)`` bound to a fresh async engine."""
|
||||
def async_session_factory(*, server_settings: dict | None = None):
|
||||
"""Return ``(sessionmaker, engine)`` bound to a fresh async engine.
|
||||
|
||||
``server_settings`` (optional) are applied by asyncpg as per-connection GUCs
|
||||
on connect. Because NullPool opens a fresh real connection per checkout, every
|
||||
transaction in the task inherits them — used to set ``lock_timeout`` so a
|
||||
statement blocked on a lock fails fast instead of hanging to the Celery hard
|
||||
limit (operator-flagged normalize_tags timeout 2026-06-07).
|
||||
"""
|
||||
cfg = get_config()
|
||||
# NullPool: this engine lives for ONE task (created + disposed per
|
||||
# asyncio.run loop), so intra-task connection pooling buys nothing and
|
||||
@@ -26,5 +33,13 @@ def async_session_factory():
|
||||
# phase 3 (asyncpg ConnectionDoesNotExistError, Anduo #40014). NullPool
|
||||
# opens a fresh real connection on each checkout, so phase 3 always
|
||||
# reconnects clean; pre_ping is then redundant.
|
||||
engine = create_async_engine(cfg.database_url, future=True, poolclass=NullPool)
|
||||
connect_args = {}
|
||||
if server_settings:
|
||||
connect_args["server_settings"] = {
|
||||
k: str(v) for k, v in server_settings.items()
|
||||
}
|
||||
engine = create_async_engine(
|
||||
cfg.database_url, future=True, poolclass=NullPool,
|
||||
connect_args=connect_args,
|
||||
)
|
||||
return async_sessionmaker(engine, class_=AsyncSession, expire_on_commit=False), engine
|
||||
|
||||
Reference in New Issue
Block a user