feat(translation): 8h sweep + drain button + detection probe (#1376)
CI / lint (push) Successful in 3s
CI / frontend-build (push) Successful in 19s
CI / backend-lint-and-test (push) Successful in 46s
CI / integration (push) Successful in 3m55s

Throughput: translate_posts now runs every 8h (was daily) as the
steady-state cadence for newly-imported posts, and the Settings
"Translate now" button runs it in drain mode (run-until-done, no reset)
so one press clears the whole untranslated backlog instead of a single
300-post chunk. The interrupt/backoff re-enqueue now preserves the drain
flag so a bulk drain resumes cleanly after an Interpreter restart.

Misdetection groundwork: surface the detector's confidence from the
Interpreter client (it was in the detectedLanguage payload but discarded)
and add a read-only "Test translation" box — POST /settings/translation/
probe + TranslationCard UI — that shows detected language + confidence +
engine + result for pasted text, without saving. Lets the operator see
why a short/abbreviation-heavy English title gets mis-detected so the
detection guard (min-length + confidence floor) can be tuned from real
numbers. The guard itself follows once the mis-detected cases are probed.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CgZP9v2otxVJymiYsnVuMy
This commit is contained in:
2026-07-08 20:13:49 -04:00
parent a017771621
commit ffdbdbaf07
8 changed files with 274 additions and 20 deletions
+33 -11
View File
@@ -7,8 +7,10 @@ same pattern as the other backfill sweeps. No-op unless translation is enabled,
a base URL is set, and the service is healthy.
Two entry points:
- ``translate_posts`` — the daily untranslated sweep (translated_source_lang IS
NULL only); also the Settings "Translate now" button.
- ``translate_posts`` — the periodic untranslated sweep (every 8h;
translated_source_lang IS NULL only), one bounded chunk per fire. The Settings
"Translate now" button calls it with ``drain=True`` to chase the tail
run-until-done and clear the whole backlog in a single press.
- ``retranslate_posts`` — after a model change, resets the stored translation
columns for a scoped set of posts (all, or a given set of artists) so the
untranslated selection re-runs them, then chases the tail until drained
@@ -56,13 +58,19 @@ def _interrupt_backoff(retry_after) -> int:
name="backend.app.tasks.translation.translate_posts",
soft_time_limit=1800, time_limit=2100,
)
def translate_posts() -> str:
def translate_posts(drain: bool = False) -> str:
"""Translate untranslated non-English post title/description (default target
en) via Interpreter. Per-post [title, description] batch → per-post detected
language. Passthrough / already-target posts are marked handled (source ==
target) with no stored translation. 503 or a connection error interrupts the
run (retry next cycle); 400 stops it (fix config); posts done so far stay
committed. No-op unless configured + healthy. Returns a summary string."""
committed. No-op unless configured + healthy. Returns a summary string.
``drain`` (the Settings "Translate now" button) chases the tail
run-until-done: on a clean chunk with work still remaining it re-enqueues
itself until the untranslated backlog is zero, like ``retranslate_posts``. The
periodic beat leaves it False → one bounded chunk per fire (the 8h cadence
drives the rest, enough for the trickle of newly-imported posts)."""
SessionLocal = _sync_session_factory()
with SessionLocal() as session:
ready = _translation_config(session)
@@ -73,14 +81,28 @@ def translate_posts() -> str:
status, translated, retry_after = _translate_batch(
session, posts, base_url, target,
)
# Steady-state daily sweep: one chunk per run on success (the beat drives
# the rest — no tail-chase). But if Interpreter drained mid-chunk, don't
# make a manual "Translate now" wait a whole day — re-enqueue after its
# Retry-After hint (or the default backoff). Self-terminating: once the
# service is down the health gate returns "interpreter unavailable" early.
if status == "interrupted" and _count_untranslated(session, None):
# Drain mode (manual "Translate now"): chase the tail until the backlog is
# zero, so one press clears the whole pile instead of one 300-chunk.
# Termination is guaranteed — every handled post leaves the untranslated
# set, so the remaining count strictly decreases each chunk.
if status == "ok" and drain:
remaining = _count_untranslated(session, None)
if remaining:
translate_posts.apply_async(
(), {"drain": True}, countdown=_RETRANSLATE_COUNTDOWN,
)
log.info(
"translate_posts: draining — %d remaining → re-enqueued",
remaining,
)
# Interpreter drained mid-chunk (graceful restart): don't make a manual
# "Translate now" wait a whole beat cycle — re-enqueue after its
# Retry-After hint (or the default backoff), preserving `drain` so a bulk
# drain resumes cleanly. Self-terminating: once the service is down the
# health gate returns "interpreter unavailable" early.
elif status == "interrupted" and _count_untranslated(session, None):
delay = _interrupt_backoff(retry_after)
translate_posts.apply_async((), {}, countdown=delay)
translate_posts.apply_async((), {"drain": drain}, countdown=delay)
log.info(
"translate_posts: interrupted (service draining) → retry in %ss",
delay,