fix(tags): normalize task fails fast on lock + logs progress
normalize_tags_task ran to the 40-min hard limit with zero logs (operator- flagged 2026-06-07). Cause: a per-group merge repoints series_page (via _repoint_series_pages); during the wedged 0040 migration that held ACCESS EXCLUSIVE on series_page, the merge's UPDATE blocked on that lock. The time-box check is at the top of the group loop, so a statement blocked mid-group never yields back to it — the task sat until the Celery hard kill. No logs because the only log fired per *finished* group. - Set lock_timeout=30s on the normalize session (opt-in server_settings on the async factory). A blocked merge now raises, the per-group handler rolls back + counts an error, and the loop continues — one stuck group can't strand the chunk, and the budget checkpoint stays effective. - Log group count at start + a heartbeat every 25 groups, so a long/slow run is diagnosable instead of silent. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -126,7 +126,16 @@ def normalize_tags_task(self) -> dict:
|
||||
from ._async_session import async_session_factory
|
||||
|
||||
async def _run() -> dict:
|
||||
async_factory, async_engine = async_session_factory()
|
||||
# lock_timeout=30s: a per-group merge repoints FKs across image_tag and
|
||||
# series_page; if a statement blocks on a lock (e.g. behind a schema
|
||||
# migration holding ACCESS EXCLUSIVE on series_page — the exact wedge that
|
||||
# made this task run to the 40-min hard limit with no progress,
|
||||
# operator-flagged 2026-06-07), it now fails fast. The per-group handler
|
||||
# catches it (rollback + error++) and the loop continues, so one blocked
|
||||
# group can't strand the whole chunk.
|
||||
async_factory, async_engine = async_session_factory(
|
||||
server_settings={"lock_timeout": "30s"}
|
||||
)
|
||||
try:
|
||||
async with async_factory() as session:
|
||||
# normalize_existing_tags commits per group internally.
|
||||
|
||||
Reference in New Issue
Block a user