fix(tags): normalize task fails fast on lock + logs progress
CI / lint (push) Successful in 2s
CI / frontend-build (push) Successful in 22s
CI / backend-lint-and-test (push) Successful in 26s
CI / integration (push) Successful in 3m5s

normalize_tags_task ran to the 40-min hard limit with zero logs (operator-
flagged 2026-06-07). Cause: a per-group merge repoints series_page (via
_repoint_series_pages); during the wedged 0040 migration that held ACCESS
EXCLUSIVE on series_page, the merge's UPDATE blocked on that lock. The time-box
check is at the top of the group loop, so a statement blocked mid-group never
yields back to it — the task sat until the Celery hard kill. No logs because the
only log fired per *finished* group.

- Set lock_timeout=30s on the normalize session (opt-in server_settings on the
  async factory). A blocked merge now raises, the per-group handler rolls back +
  counts an error, and the loop continues — one stuck group can't strand the
  chunk, and the budget checkpoint stays effective.
- Log group count at start + a heartbeat every 25 groups, so a long/slow run is
  diagnosable instead of silent.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-06-07 21:02:39 -04:00
parent 9770dd3474
commit a00a2786e3
3 changed files with 42 additions and 4 deletions
+14
View File
@@ -743,6 +743,10 @@ async def normalize_existing_tags(
"sample": sample,
}
start = time.monotonic()
log.info(
"normalize_existing_tags: %d group(s) need changes (budget=%ss)",
len(touched), time_budget_seconds,
)
for done, (key, members) in enumerate(touched):
# Time-box: stop cleanly before the Celery limit kills us mid-group and
# strands the run as a timeout. The caller re-enqueues to finish the
@@ -754,6 +758,16 @@ async def normalize_existing_tags(
summary["partial"] = True
summary["remaining"] = len(touched) - done
break
# Heartbeat so a long run is diagnosable instead of silent — the timeout
# operator-flagged 2026-06-07 produced zero logs because the only log was
# per finished group and it was stuck mid-group on a lock.
if done and done % 25 == 0:
log.info(
"normalize_existing_tags: %d/%d groups (%d merged, %d errors, "
"%.0fs elapsed)",
done, len(touched), summary["merged"], summary["errors"],
time.monotonic() - start,
)
canonical = key[2]
names_by_id = dict(members)
# Survivor: prefer a member already named canonically (no rename, no