fix(tags): time-box + self-resume the tag standardization (stop the 40-min timeout)
normalize_tags_task timed out at the 40-min hard limit on a large back-catalog
(the first run recases the whole booru vocabulary) — operator-flagged, and it
monopolized the concurrency-1 maintenance queue while doing so.
normalize_existing_tags now takes time_budget_seconds: the live run stops
cleanly at the budget and reports {partial, remaining}. The task runs 600s
chunks and re-enqueues itself until nothing remains (idempotent — commits per
group, so the next chunk skips already-canonical groups). Short chunks let the
recovery sweep and other maintenance tasks interleave instead of being blocked
for 40 minutes.
Frontend: the Standardize button is now fire-and-forget ("Queued — runs in the
background; re-run Preview to confirm") instead of poll-until-done, which would
have falsely reported "complete" after the first chunk.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -1,6 +1,7 @@
|
||||
"""Tag CRUD + autocomplete + image-tag association."""
|
||||
|
||||
import logging
|
||||
import time
|
||||
from collections.abc import Sequence
|
||||
from dataclasses import dataclass
|
||||
|
||||
@@ -637,7 +638,10 @@ def _best_connected(tag_ids: list[int], counts: dict[int, int]) -> int:
|
||||
|
||||
|
||||
async def normalize_existing_tags(
|
||||
session: AsyncSession, *, dry_run: bool = False
|
||||
session: AsyncSession,
|
||||
*,
|
||||
dry_run: bool = False,
|
||||
time_budget_seconds: float | None = None,
|
||||
) -> dict:
|
||||
"""Convert the back-catalog to the #701 canonical tag form.
|
||||
|
||||
@@ -647,6 +651,14 @@ async def normalize_existing_tags(
|
||||
survivor to the canonical form. Idempotent: a group that is already a lone
|
||||
canonical tag is a no-op, so re-running is safe.
|
||||
|
||||
A first run over a fresh back-catalog can touch tens of thousands of tags
|
||||
(the whole booru-derived vocabulary needs recasing) and won't finish inside
|
||||
one Celery time limit — it timed out at 40 min (operator-flagged 2026-06-07).
|
||||
`time_budget_seconds` time-boxes the live run: it stops cleanly at the budget
|
||||
and reports `partial`/`remaining` so the caller can re-enqueue and continue.
|
||||
Because it commits per group and is idempotent, the next run just picks up
|
||||
the groups still needing change.
|
||||
|
||||
dry_run=True returns a projection (counts + a sample of the changes) with no
|
||||
mutations. Live runs commit per group and isolate failures per group so one
|
||||
bad group can't strand the rest.
|
||||
@@ -656,7 +668,7 @@ async def normalize_existing_tags(
|
||||
"total_changes": T, "sample": [{"to", "from": [...], "kind", "merge"}]}
|
||||
Returns (live):
|
||||
{"groups_processed", "merged", "renamed", "aliases_created", "errors",
|
||||
"sample": [...]}
|
||||
"total_changes", "remaining", "partial", "sample": [...]}
|
||||
"""
|
||||
rows = (
|
||||
await session.execute(
|
||||
@@ -715,9 +727,23 @@ async def normalize_existing_tags(
|
||||
"renamed": 0,
|
||||
"aliases_created": 0,
|
||||
"errors": 0,
|
||||
"total_changes": len(touched),
|
||||
"remaining": len(touched),
|
||||
"partial": False,
|
||||
"sample": sample,
|
||||
}
|
||||
for key, members in touched:
|
||||
start = time.monotonic()
|
||||
for done, (key, members) in enumerate(touched):
|
||||
# Time-box: stop cleanly before the Celery limit kills us mid-group and
|
||||
# strands the run as a timeout. The caller re-enqueues to finish the
|
||||
# rest (idempotent — already-canonical groups are skipped next pass).
|
||||
if (
|
||||
time_budget_seconds is not None
|
||||
and time.monotonic() - start >= time_budget_seconds
|
||||
):
|
||||
summary["partial"] = True
|
||||
summary["remaining"] = len(touched) - done
|
||||
break
|
||||
canonical = key[2]
|
||||
names_by_id = dict(members)
|
||||
# Survivor: prefer a member already named canonically (no rename, no
|
||||
@@ -752,4 +778,7 @@ async def normalize_existing_tags(
|
||||
log.warning(
|
||||
"tag normalize failed for group %r: %s", canonical, exc
|
||||
)
|
||||
else:
|
||||
# Loop finished without hitting the time budget — nothing left to do.
|
||||
summary["remaining"] = 0
|
||||
return summary
|
||||
|
||||
Reference in New Issue
Block a user